How to evaluate an LLM feature before you ship it

Most LLM features get tested the same way at first: someone tries a few prompts, the answers look good, and it ships. Then a prompt tweak or a model update quietly breaks a case nobody re-checked. If you want to evaluate an LLM feature before you ship it, you need the same thing you'd want for any other code: a fixed set of inputs, checks that pass or fail, and a way to run them on every change.

Here's the evaluation setup I'd put in place for any LLM feature, from the first dataset to evals running in CI.

Start with a golden dataset

A golden dataset is a list of realistic inputs with the outcome you expect for each. It's the single most useful thing you can build for an LLM feature, and it doesn't need to be big. Thirty good cases beat three hundred random ones.

What to put in it:

• The common cases. The inputs you expect most often, in the shapes users actually send them.
• The hard cases. Ambiguous inputs, missing fields, very long inputs, other languages, typos.
• The cases that must fail safely. Inputs where the right answer is "I can't do this" or a validation error.
• Every bug you've fixed. Each production failure becomes a new case, so it can never come back unnoticed.

Store it as files in the repo, next to the code. When the expected behaviour changes, the dataset changes in the same pull request, and the reviewer sees both.

Assertions first, judges second

The cheapest and most reliable checks are plain code. Before reaching for anything clever, write down what you can verify deterministically:

• Output parses and matches the schema.
• Required fields are present and have the right types.
• Extracted values match the expected values, exactly or within a tolerance.
• Forbidden content is absent: placeholder text, leaked system prompt, a different language than requested.
• Length and format stay within bounds.

REQ Agent, a CLI I built that generates client, designer and dev-handoff documents from discovery notes, uses this kind of check at runtime: it validates generated documents for structure, feature coverage and design-to-dev token consistency before writing them. The same checks make good evals, since a check that guards production can also grade a test run. I go deeper on runtime checks in how to validate LLM output before it reaches users, and on schema checks in getting reliable JSON from LLMs.

With Node's built-in test runner, an eval over a golden dataset is a normal test file:

import test from "node:test";
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { extractInvoice } from "../src/extract-invoice.js";

type Case = {
  id: string;
  input: string;
  expected: { currency: string; total: number; minLineItems: number };
};

const cases: Case[] = JSON.parse(
  await readFile(new URL("./golden/invoices.json", import.meta.url), "utf8"),
);

for (const c of cases) {
  test(`invoice extraction: ${c.id}`, async () => {
    const out = await extractInvoice(c.input);
    assert.equal(out.currency, c.expected.currency);
    assert.ok(Math.abs(out.total - c.expected.total) < 0.01, `total was ${out.total}`);
    assert.ok(out.lineItems.length >= c.expected.minLineItems);
    assert.doesNotMatch(out.notes ?? "", /lorem ipsum|as an ai/i);
  });
}

Each case gets its own named test, so a failure tells you exactly which input broke.

LLM-as-judge, with care

Some qualities can't be checked with an assertion: is the summary faithful to the source, is the tone right, does the answer address the question? For those, a second model call that grades the output, an LLM-as-judge, is useful. It also has failure modes of its own:

• Ask narrow questions. "Does the summary state anything that isn't in the source? Answer yes or no and quote the claim" works far better than "rate this summary from 1 to 10".
• Give it a rubric. Spell out what pass and fail look like, with an example of each.
• Check the judge. Grade a sample of outputs by hand and compare with the judge's verdicts. If they disagree often, fix the rubric before trusting the scores.
• Prefer pass/fail over scores. A numeric score drifts between runs. A binary verdict on a specific question is easier to track.
• Keep the judge stable. Changing the judge's prompt or model changes your measurement, so treat it like a versioned part of the test suite.

My rule: assertions for everything they can cover, a judge only for the qualities they can't.

Run evals in CI as regression tests

Evals only help if they run when things change. Wire them into CI and run them on any change to:

• the prompt or system instructions,
• the model or its settings,
• the tools or retrieval logic that feed the model,
• the parsing and validation code around it.

LLM outputs vary, so expect some noise. Two habits help. Set a pass threshold per suite instead of requiring a perfect run on every flaky case, and keep the hard, must-not-break cases in a separate strict suite that has to pass completely. Model calls cost money, so run the full suite on pull requests that touch the feature, and a small smoke set on everything else.

Evaluate the failure paths too

Users will hit the edges, so test them on purpose:

• Timeouts and provider errors. Does the feature degrade gracefully, or show a broken screen?
• Empty or useless outputs. Does your code catch them before the user sees them?
• Hostile inputs. Inputs that try to override instructions or pull out data they shouldn't.
• Cost per request. Record token usage per case, so a prompt change that doubles cost shows up in review.

After launch, production logs become your next source of eval cases. Logging inputs, outputs and failures safely is its own topic, which I cover in logging and observability for LLM features.

Where I'd start

Write twenty to thirty golden cases this week, including a few that must fail safely. Turn every check you can into a plain assertion, add a judge only for what's left, and run the suite in CI whenever the prompt or model changes. After that, every production bug becomes one more case.