Verify a refactor, even an AI-written one, by diffing output

A refactor promises that behaviour stays the same while the code changes. Reading the diff tells you whether the new code looks right. It does not tell you whether it returns the same thing. The fastest way I know to verify a refactor is to run the old code and the new code on the same inputs and diff what comes out. This post shows how I do that, the inputs I pick, and the other checks I use when there is no old output to compare against. It matters even more when an assistant or agent wrote the new version.

How I verify refactor output with a diff harness

On the Buffdudes admin dashboard, five revenue cards each loaded every transaction in the selected date range (43k rows, 47 columns) into Node and added them up in JavaScript. The rewrite moved that work into Postgres so only the totals come back. The full story is in moving aggregation into Postgres.

A change like that is easy to get subtly wrong. A sum that treats nulls differently, a date boundary that is inclusive in one version and exclusive in the other, a number that comes back as a string from the database driver. Each of those passes a code review and changes what the admin sees.

So I diffed old and new responses on two local databases, over 7 date ranges, for all five cards: 35 of 35 identical, key order included.

A harness for this is small. The shape I'd use, with placeholder paths, running the old build and the new build side by side:

const OLD = "http://localhost:3001";
const NEW = "http://localhost:3002";

const cards = ["card-a", "card-b"]; // the endpoints under test
const ranges = [
  { from: "2025-01-01", to: "2025-01-31" },
  { from: "2025-02-15", to: "2025-03-14" },
];

let same = 0;
let total = 0;
for (const card of cards) {
  for (const { from, to } of ranges) {
    const qs = `?from=${from}&to=${to}`;
    const [a, b] = await Promise.all(
      [OLD, NEW].map((base) =>
        fetch(`${base}/admin/stats/${card}${qs}`).then((r) => r.text())
      )
    );
    total++;
    if (a === b) same++;
    else console.log(`DIFF ${card} ${from}..${to}\n${a}\n${b}`);
  }
}
console.log(`${same}/${total} identical`);

Comparing raw response text is deliberate. It catches key order, number formatting and whitespace, which a deep-equality check would forgive. A deep comparison treats {a: 1, b: 2} and {b: 2, a: 1} as equal, while JSON.stringify output and response bodies keep insertion order. If a client, a test or a cached response depends on that order, you want to know.

Pick inputs that hit the edges

A diff harness is only as good as its inputs. The 7 date ranges for the dashboard were chosen to hit the places where aggregation code breaks:

• the full default range
• all time
• the last 30 days
• a single day
• a calendar month
• a range starting mid-month
• an empty range

The empty range checks what "no data" looks like: zero, null, or a missing key. The single day and mid-month start check boundaries. All time checks the largest input. If the old and new code agree on all of those, I trust them on the ranges in between.

This is what characterization tests are: tests that record what the code does today, whether or not anyone wrote that behaviour down, so you can tell when a change alters it. A diff harness is a characterization test that uses the old code itself as the expected value, so you never have to write the expected output by hand. GitHub took the same idea into production with Scientist, which runs the old and new code paths on live calls and compares their results.

When there is no old output to diff

Some changes are fixes, so the old output was wrong by definition. Diffing against it tells you only that something changed.

For those, I test the logic directly. On Fotobank, a stock photo marketplace, a fix to how carts merge shipped with unit tests for the merge logic. Logic like cart merging usually comes down to a handful of cases you can name, such as the same item on both sides or one side empty, and each one becomes a small test with an obvious expected result. That is a better fit than a harness when you can state the right answer in a sentence.

Check the build that ships

Sometimes the thing to verify is a build artifact rather than a function.

On a Flutter app, a fix kept a test-store API key out of production builds. Reading the code suggested the key could no longer be compiled in. To prove it, I built with a canary key, a unique string with no other purpose, and searched the outputs: 0 occurrences in the production APK, 1 in the staging APK. The 1 matters as much as the 0, since it shows the search would have found the key if it were there. The details are in keeping test API keys out of production builds.

Check against real data and real deployments

Local data is tidy. Production is where the shapes you did not think of live.

For database work, I re-check changes against a production snapshot before shipping. Index changes on Buffdudes were checked that way: run the real queries on a copy of real data, with and without the change, and read the plans. A query that is fast on a small local database can still scan on production-sized tables.

For a website, the end-to-end tests run against a real deployment, including its redirects. Redirect rules often live in hosting config rather than in the app, so a test against a local dev server never exercises them. Running the suite against the deployed site checks the thing users hit.

Why this matters more for AI-generated code

When an assistant or agent rewrites a module, the diff is often large and looks confident. Reviewing it line by line is slow, and the mistakes in generated code tend to be the plausible kind: a condition that is almost right, an edge case quietly dropped, a renamed field. AI generated code review is where a diff harness pays off most. Instead of reading 300 lines and hoping, you run the old and new versions on the same inputs and look at the handful of cases that differ.

It also gives the review a clear standard. "35 of 35 identical" is a claim anyone on the team can rerun. I still read the code, because the harness only covers the inputs I chose, but I read it knowing where it agrees. The wider review habits are in using AI in code review without lowering the bar, and if the agent also wrote the tests, reviewing AI-generated tests covers what to check.

What is a characterization test?

A characterization test records what existing code does today and fails if a change alters it. It describes current behaviour rather than intended behaviour, which makes it a safety net for refactoring code that has no spec or few tests. Diffing old and new output over a fixed set of inputs is the quickest way to build one.

The short version

Run the old and new code on the same inputs and compare the raw output, key order included. Choose inputs that hit edges: empty, single item, boundaries, everything. Use unit tests when the old output was wrong, canary values when the thing that ships is a build, and production snapshots or real deployments when local data is too tidy. If you want this kind of check built into how your team reviews changes, that is part of what I offer through my services.