Treat the LLM Like an Unreliable API: Designing AI Pipelines That Hold Up

Imagine a third-party API with these properties.

It takes two to twenty seconds to answer. It charges you for every call. It sometimes returns a different answer to the exact same request. Now and then it returns something that isn’t valid JSON, even though you asked for JSON. It never returns an error when it’s wrong. It just answers, confidently, in perfect grammar.

If a vendor sold you that API, you’d wrap it in so many guards that it couldn’t hurt you. You’d add timeouts, retries, validation, logging and a fallback. You wouldn’t let it touch anything important without a check.

That API is a large language model. And yet, on a lot of codebases I see, the model call is the least defended line in the whole system: one await, one JSON.parse, and the result goes straight into the database.

Most of the fix is the same discipline from Integrating Messy APIs. A few parts are specific to models, because no normal API is wrong while sounding right.

The examples are TypeScript and provider-neutral. Assume a small function like this, implemented with whatever SDK you use:

type ModelRequest = {
  model: string;      // e.g. the provider's model id
  system: string;
  input: string;
  signal?: AbortSignal;
};

type ModelResponse = { text: string; inputTokens: number; outputTokens: number };

declare function callModel(req: ModelRequest): Promise<ModelResponse>;

Rule 1: Keep facts out of the model

The cheapest, fastest and most reliable model call is the one you don’t make.

Before you send anything to a model, ask: do I already know this? If the answer lives in your database, a config file or a simple rule, use deterministic code. Deterministic means the same input always gives the same output. Code is deterministic. A model is not.

A common mistake looks like this:

// Before: asking the model for things the database already knows
const prompt = `Here are tomorrow's bookings: ${JSON.stringify(bookings)}.
Here are the guest messages: ${JSON.stringify(messages)}.
Write a summary of who arrives tomorrow and what they need.`;

The model now has to count arrivals, read dates and copy names. It will usually get that right. “Usually” is the problem.

// After: code handles the facts, the model only reads the messy text
const arrivals = bookings.filter((b) => b.checkIn === tomorrow); // exact
const notes = await Promise.all(
  messages.map((m) => extractNote(m.text)) // fuzzy: the model's job
);

The model only gets the part that is genuinely fuzzy: free text written by a human. Everything else stays in code you can test.

This is the first rule in Why Santara Sends One Briefing, Not a Chatbot, and it shapes every rule below. The less you ask the model, the less it can get wrong.

Rule 2: Define the output shape, and check it every time

If your code is going to use the model’s answer, the answer needs a shape. Free text is for humans. Code needs structure.

Most providers now offer some form of “structured output” or JSON mode. Use it, but don’t treat it as a guarantee. Validate the result in your own code anyway, like a webhook body from a partner. A schema library like Zod makes this short.

Here’s an example schema for extracting a note from a guest message. Notice the second variant:

import { z } from "zod";

const Note = z.discriminatedUnion("status", [
  z.object({
    status: z.literal("ok"),
    arrivalTime: z.string().regex(/^\d{2}:\d{2}$/).nullable(),
    mentionedDate: z.iso.date().nullable(),   // "YYYY-MM-DD" if the guest names a date
    needsReply: z.boolean(),
    requests: z.array(z.enum(["early_checkin", "late_checkout", "luggage", "parking", "other"])),
  }),
  z.object({
    status: z.literal("unsure"),
    reason: z.string().max(200),
  }),
]);

type Note = z.infer<typeof Note>;

A discriminated union is a type that is one of several shapes, told apart by one field (here, status). The model must return either a confident ok result or an explicit unsure with a reason.

Give the model a way to say “I don’t know”

That unsure variant is the most important line in the schema.

If your schema only allows a confident answer, the model will always give one. It has no other legal move. So an ambiguous message produces a guess dressed up as a fact, and nothing downstream can tell the difference.

With an unsure path, ambiguity becomes data. Your code can route those cases to a human or count them. A system that says “I’m not sure” now and then is far more useful than one that is quietly wrong now and then.

Say so in the prompt, too: “If the message is ambiguous or you can’t tell, return status unsure with a short reason. Do not guess.”

Rule 3: Wrap every call: timeout, retry, validate, fall back

Now the wrapper. This is the one function every model call in your system goes through. It handles the boring failures so the rest of your code doesn’t have to.

What it needs to do:

Time out. A call that hangs for two minutes is worse than one that fails in ten seconds.
Retry transient errors with backoff. Wait a bit longer each time, with some randomness (called jitter) so many clients don’t retry at the same moment.
Validate the output against the schema.
If validation fails, retry once with the error. Telling the model exactly what was wrong often fixes it.
If it still fails, fall back to a safe default, usually “unsure,” and flag it for a human.

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function callWithRetry(req: ModelRequest, attempts = 3, timeoutMs = 20_000) {
  for (let i = 0; ; i++) {
    try {
      return await callModel({ ...req, signal: AbortSignal.timeout(timeoutMs) });
    } catch (err) {
      if (i >= attempts - 1 || !isTransient(err)) throw err;
      const base = 500 * 2 ** i;                        // 0.5s, 1s, 2s...
      await sleep(base + Math.random() * base);         // plus jitter
    }
  }
}

AbortSignal.timeout() is built into Node and browsers. Pass the signal through to your HTTP client or SDK so the request is actually cancelled, not just ignored. isTransient is yours to write: usually timeouts, HTTP 429 and 5xx count, while 400-level errors like a bad request don’t, because retrying them just fails again.

Then validation on top:

async function extract<T>(schema: z.ZodType<T>, req: ModelRequest): Promise<
  { ok: true; value: T } | { ok: false; reason: string }
> {
  let res = await callWithRetry(req);
  let parsed = safeParseJson(schema, res.text);
  if (parsed.success) return { ok: true, value: parsed.data };

  // One repair attempt: show the model its own mistake
  res = await callWithRetry({
    ...req,
    input: `${req.input}\n\nYour previous answer was invalid:\n${parsed.error}\nReturn only valid JSON matching the schema.`,
  });
  parsed = safeParseJson(schema, res.text);
  if (parsed.success) return { ok: true, value: parsed.data };

  return { ok: false, reason: parsed.error };   // caller falls back and flags it
}

function safeParseJson<T>(schema: z.ZodType<T>, text: string) {
  try {
    const r = schema.safeParse(JSON.parse(text));
    return r.success ? { success: true as const, data: r.data }
                     : { success: false as const, error: z.prettifyError(r.error) };
  } catch {
    return { success: false as const, error: "Output was not valid JSON." };
  }
}

Two things about this design.

The repair loop is capped at one. If the model fails twice on the same input, a third try rarely helps and costs money. Stop and flag it.

Failure is a normal return value, not an exception. In a pipeline, ok: false usually means: store the item as “needs human review” and move on. One bad message shouldn’t stop the other ninety-nine.

Here’s the whole shape of it:

Known facts never reach the model, bad output gets one repair attempt, and anything uncertain goes to a person instead of into your data.

Rule 4: Make repeated calls safe and cheap

Background jobs retry. Webhooks arrive twice. A user double-clicks. If each of those triggers a model call, you pay twice, and you might get two different answers for the same input, which is worse.

Two old tools solve this.

Idempotency keys. An idempotency key is an ID that says “this is the same piece of work as before.” For a pipeline, it’s usually something like extract-note:<messageId>:<promptVersion>. Before calling the model, check whether a result for that key already exists. If yes, return it.

Caching identical requests. Go one step further and key on the content itself: a hash of the model id, the prompt version and the exact input. Same request, same stored answer.

import { createHash } from "node:crypto";

function cacheKey(req: ModelRequest, promptVersion: string) {
  return createHash("sha256")
    .update(JSON.stringify([req.model, promptVersion, req.system, req.input]))
    .digest("hex");
}

Beyond saving money, it makes the system stable. Without it, a retry can quietly change what your users see. Just make sure everything that affects the answer, including things like “today is…”, is part of the key.

Rule 5: Log everything you’d need to replay a call

When a model gives a weird answer in production, the first question is “what exactly did it see?” If you can’t answer that, you can’t debug it, and you can’t prove a fix works.

So store a record for every call:

type ModelCallLog = {
  id: string;
  createdAt: Date;
  feature: string;          // "extract-note"
  promptVersion: string;    // "extract-note@v7"
  model: string;            // exact model id, not "the default"
  input: string;            // the raw input you sent
  rawOutput: string;        // before parsing, even if it was invalid
  parsedOk: boolean;
  inputTokens: number;
  outputTokens: number;
  latencyMs: number;
  attempt: number;
};

A few notes:

Store the raw output, not just the parsed result. When validation fails, it’s the only clue to why.
Store the exact model id. An alias like “latest” can point to a different model next month.
Watch the personal data. Treat this table like any sensitive table: limited access, a retention period, deletion on request.

With this table, you can re-run a bad production case against a new prompt and compare. It’s the “keep the raw event, replay later” idea from the integrations post, applied to models.

Rule 6: Version your prompts like code

A prompt is code. It changes behavior, it has bugs, and a one-word edit can break things in ways nobody notices for a week.

So treat it that way:

Keep prompts in the repository, not in a dashboard someone edits on a Friday afternoon.
Version each one (extract-note@v7) and log the version with every call.
Change prompts through code review, with eval results in the pull request.
Change one thing at a time. Don’t switch models and rewrite the prompt in one release.

export const extractNotePrompt = {
  version: "extract-note@v7",
  system: [
    "You extract structured notes from short-term rental guest messages.",
    "Return only JSON matching the schema.",
    "If the message is ambiguous, return status \"unsure\" with a short reason. Do not guess.",
  ].join("\n"),
};

Not fancy, and it doesn’t need to be. Every output in your logs can now be traced to the exact prompt and model that produced it.

Rule 7: Cross-check the model against what you know

Validation tells you the output has the right shape. It doesn’t tell you it’s true.

A note can be perfectly valid JSON and still say a guest arrives on Friday when the booking says Saturday. Wherever you have real data, compare the model’s output to it.

function crossCheck(note: Extract<Note, { status: "ok" }>, booking: Booking) {
  const problems: string[] = [];
  if (note.mentionedDate && note.mentionedDate !== booking.checkIn) {
    problems.push(`Message mentions ${note.mentionedDate}, booking says ${booking.checkIn}`);
  }
  return problems;   // non-empty: show to a human, don't silently pick one
}

When the model and your data disagree, don’t let either one quietly win. Your data might be stale, or the model might have misread. Either way, a person should see it. That’s how the Santara briefing treats it at a high level, too: a mismatch is worth flagging, not hiding.

Rule 8: Evals, a small test suite for a non-deterministic function

You wouldn’t ship a change to your pricing code without tests. But teams ship prompt changes and model upgrades all the time with nothing more than “I tried three messages and it looked fine.”

An eval (short for evaluation) is just a test suite for model behavior. It doesn’t need a platform or a framework to start. It needs:

a fixed set of cases, 20 to 50 is plenty to begin with, taken from real inputs (with personal data removed or replaced) and including the weird ones;
simple pass/fail checks for each case, written in code;
a habit of running it on every prompt change and every model change.

type EvalCase = { name: string; input: string; check: (n: Note) => boolean };

const cases: EvalCase[] = [
  { name: "late arrival", input: "flight delayed, we land around 9pm",
    check: (n) => n.status === "ok" && n.arrivalTime === "21:00" },
  { name: "luggage", input: "can we leave bags after checkout?",
    check: (n) => n.status === "ok" && n.requests.includes("luggage") },
  { name: "ambiguous", input: "same as last time pls",
    check: (n) => n.status === "unsure" },
];

async function runEvals() {
  let passed = 0;
  for (const c of cases) {
    const r = await extract(Note, { model: MODEL, system: extractNotePrompt.system, input: c.input });
    const ok = r.ok && c.check(r.value);
    if (ok) passed++; else console.log(`FAIL ${c.name}`);
  }
  console.log(`${passed}/${cases.length} passed`);
}

Look at the third case. Test that the model says “unsure” when it should. A prompt change that makes the model more confident can look like an improvement and actually be a regression.

Some practical habits:

Every production bug becomes a case, exactly like regular tests.
Expect some noise. A case might flip between runs. Track the pass rate over time rather than demanding 100% every run.
Prefer checks in code over a model grading a model. A model judge has its place for fuzzy qualities like tone, but start with checks you can read.

Twenty cases and a script you run before merging will catch most embarrassing regressions. A very good return on an afternoon.

Rule 9: Set cost and latency budgets

A model call can cost far more and take far longer than a normal API call. Multiply by every message, user and retry, and costs grow in ways nobody planned. I wrote about the business side of this in The Hidden Costs of AI Tools.

On the engineering side, make the budget explicit:

Per call: a maximum output length (most APIs let you set one), a timeout, and a maximum number of attempts including repairs.
Per job: for example, “processing one user’s morning batch may make at most N model calls.” If a bug causes a loop, it hits the ceiling instead of your bill.
Per day: track tokens and cost per feature from the call log in Rule 5, and alert when it moves sharply.

Latency deserves the same care. If a user is waiting, a ten-second call is a broken page. Move model work into background jobs wherever you can, and let the user see the result when it’s ready.

Also, try the smallest model that passes your evals. Your eval suite is what makes that a safe decision instead of a hopeful one.

Rule 10: The model suggests, your code decides

This is the rule I’d never bend.

Never let a model’s output directly trigger an irreversible action. Sending a message to a customer, refunding money, cancelling a booking, deleting data. The model can propose the action. Something else has to approve it: either deterministic code with strict rules, or a person.

// The model returns a proposal, never a side effect
type Proposal = { action: "send_reply"; to: string; draft: string };

await db.proposals.insert({ ...proposal, status: "pending_approval" });
// A person (or a strict rule) approves it later. Only then does it run.

Why so strict? Because a model doesn’t fail by crashing. It fails with a confident wrong answer. In a “pending” list, that costs someone a few seconds. In a sent email, it can’t be taken back.

This gets even more important when you let a model choose its own steps and call tools. I wrote more about that in What an “AI Agent” Actually Is. The short version: the more freedom the model has, the more these guards matter, not less.

The bigger picture

A common pattern is that people treat the model as the clever part of the system, and so trust it more than anything else. It should be the opposite. The model is the least predictable part, so it deserves the most defensive code.

Most of these rules aren’t new. We already use timeouts, retries, idempotency, validation, logging and tests for payment providers and booking platforms. The genuinely new parts are an “unsure” path, cross-checking against facts you already have, and evals instead of trusting a demo.

Do that, and the model becomes what it should be: a useful reader of messy human text, inside a system that doesn’t need it to be perfect.

Checklist

Sources: Zod documentation (discriminated unions, safeParse, z.prettifyError), MDN: AbortSignal.timeout().