How to validate LLM output before it reaches users

A language model will happily return something that looks right and is wrong. The JSON parses, the headings are there, the tone is perfect, and a required field is missing or a number came from nowhere. If you want to validate LLM output before it reaches users, you need checks that run in code, every time, with a clear decision about what happens when one fails.

I learned to treat model output the same way I treat a request body from an unknown client: untrusted until proven otherwise. In REQ Agent, the CLI I built to turn client discovery material into client, design and dev handoff documents, nothing gets written to disk until the generated documents pass validation for structure, feature coverage and design-to-dev token consistency. The same layered approach works for almost any LLM feature.

Layer one: can you even parse it?

The first gate is mechanical. If you asked for JSON, parse it and validate it against a schema. If you asked for Markdown with specific sections, parse the headings. Anything that fails here is a hard failure, and it is also the cheapest failure to recover from, because a retry with the parse error fed back to the model usually fixes it.

Schema validation catches more than syntax. Enums, string lengths, required arrays, number ranges: all of these belong in the schema. A priority field that should be one of three values and comes back as "urgent-ish" is a schema failure, and the schema should say so. I keep this layer in a shared helper so every LLM call in a codebase goes through the same parse, validate and retry path.

Layer two: structural rules the schema can't express

Schemas describe shape. Plenty of rules live one level up:

• Every section in the template exists and has real content, beyond a heading followed by nothing.
• Lists that should be non-empty are non-empty.
• IDs referenced in one part of the output exist in another part.
• Dates fall in a sensible range and money values are positive.

These checks are plain functions. They return a list of problems, and an empty list means the output passed. Keeping the return type as a list of readable messages pays off twice: you can log them, and you can feed them straight back to the model on a retry.

Layer three: consistency against the source of truth

This is the layer most teams skip, and it catches the failures that hurt most. The output has to agree with the input it came from, and with any sibling outputs generated alongside it.

In a document pipeline, that means feature coverage: every feature in the internal spec has to appear in the developer handoff. It also means token consistency: if the design document calls a screen "Order history", the dev handoff cannot call it "Past purchases". Each document reads fine alone. The inconsistency only shows up when you compare them, which is exactly why a program should do the comparing.

A coverage check can be very simple:

type Feature = { id: string; name: string };
type ValidationIssue = { check: string; message: string };

export function checkFeatureCoverage(
  features: Feature[],
  documents: Record<string, string>
): ValidationIssue[] {
  const issues: ValidationIssue[] = [];
  for (const [docName, body] of Object.entries(documents)) {
    const text = body.toLowerCase();
    for (const feature of features) {
      if (!text.includes(feature.name.toLowerCase())) {
        issues.push({
          check: "feature-coverage",
          message: `"${feature.name}" (${feature.id}) is missing from ${docName}`,
        });
      }
    }
  }
  return issues;
}

Substring matching is crude, and that is fine for a first version. It fails loudly on the common problem (a feature silently dropped) and you can tighten it later, for example by asking the model to tag each section with feature IDs and checking the IDs instead of the names.

Layer four: rules about what must never ship

Some outputs are well formed, consistent and still unacceptable. A support reply that promises a refund policy you don't have. A summary that includes an email address from the input. Generated HTML containing a script tag. These checks are product rules, and they are worth writing down explicitly:

• Strip or escape anything that will be rendered as HTML.
• Scan for personal data that should never leave the system.
• Block claims the product cannot back up, using an allowlist of things the model is permitted to promise.

For anything where the model's output triggers an action, such as sending an email or updating a record, treat validation as a security boundary. The model's output is input to your system, and it gets the same suspicion as anything else coming over the wire.

Decide what failure means before it happens

Validation is only useful if a failed check leads somewhere sensible. My default ladder:

1. Retry with feedback. Send the issues back to the model and ask for a corrected version. Cap it at two or three attempts.
2. Fall back. Use a simpler template, a cached previous answer or a smaller task the model handles reliably.
3. Route to a person. Put the output in a review queue with the failed checks attached, so the reviewer knows exactly what to look at.
4. Fail closed. If none of that works, show the user an honest error and write nothing.

What you never want is the fifth option, the one that happens by default when nobody decides: ship the output anyway and hope. In REQ Agent, a failed check means no files are written, because a half-correct document on disk gets forwarded to a client faster than anyone expects.

Retry counts and fallback paths are worth tracking over time. If one check fails constantly, the prompt or the intermediate format needs work, and a recurring failure is a far better signal than a vague sense that "the AI is flaky". Building a proper test set around these checks is the next step, which I cover in how to evaluate an LLM feature before you ship it.

Where I'd start

Put every model call behind one function that parses, validates against a schema and retries with the errors. Add structural and consistency checks as plain functions that return readable issues. Decide your failure ladder up front, and make "write nothing" a legitimate outcome. If you want to see these checks in the context of a full pipeline, I walk through REQ Agent's design in the two-stage LLM pipeline post.