Reviewing AI-generated tests: what to check before merging

Letting an assistant write your tests is one of the most tempting uses of AI in a codebase. Coverage goes up, the CI badge goes green, and the pull request looks thorough. Reviewing AI-generated tests is also where I spend more care than almost anywhere else, because a test suite that passes for the wrong reasons is worse than a thin one. It tells you the code is safe when nobody has checked.

Here is what I look for before approving generated tests.

Tests that assert the implementation

The most common problem: the model reads the function, then writes a test that restates it. If the function returns price * 0.9, the test asserts price * 0.9. That test will pass forever, including after someone breaks the business rule, because it was derived from the code instead of from the requirement.

A good test comes from the spec. "Members get 10% off orders over 50" gives you cases at 49.99, 50, and 50.01, plus a non-member at 100. A generated test usually gives you one happy path with whatever numbers appeared in the code.

What I check:

• Can I trace each assertion back to a requirement, acceptance criterion, or bug report?
• If I deliberately break the logic, does at least one test fail?
• Are the expected values written as literals, or computed with the same formula as the code under test?

When expected values are calculated with the same expression as the implementation, the test proves nothing.

Logic mocked out of existence

Generated tests love mocks. Mock the repository, mock the HTTP client, mock the queue, mock the clock. Sometimes the model mocks the very function the test claims to cover, and the test ends up asserting that a mock returned what the mock was told to return.

Mocks are fine at real boundaries: third-party APIs, the clock, randomness. Inside your own code they hide exactly the bugs you care about, such as a wrong SQL filter, a missing transaction, or a broken mapping between layers.

My default for backend code is to run tests against a real Postgres (in Docker or a CI service container) for anything touching queries. An N+1 or a wrong index shape will never show up behind a mocked repository.

Missing edge cases

Models are good at the case you described and weak at the cases you forgot. Before approving, I look for:

• Empty inputs: empty arrays, empty strings, zero, null, missing optional fields.
• Boundaries: exactly at a limit, one under, one over.
• Error paths: the downstream service times out, the DB write fails, the input fails validation.
• Permissions: a user requesting another tenant's record. On multi-tenant systems this is the test I care about most.
• Concurrency and idempotency: the same webhook delivered twice, two requests updating the same row.
• Time: time zones, end of month, daylight saving changes, anything with "today" in it.

If none of these appear, the suite is incomplete regardless of the coverage number.

Assertions that cannot fail

Some generated tests have no meaningful assertion at all. A few patterns to watch for:

// Passes even if createOrder throws inside a forgotten promise
it("creates an order", async () => {
  const result = createOrder(input);
  expect(result).toBeDefined();
});

// Better: await it, assert the actual contract
it("creates an order with the member discount", async () => {
  const result = await createOrder({ ...input, total: 100, member: true });
  expect(result.total).toBe(90);
  expect(result.status).toBe("pending");
});

Also watch for toBeTruthy() on objects (always true), snapshot tests nobody will read, assertions inside callbacks that never run, and try/catch blocks that swallow the failure the test was meant to detect.

Tests that weaken to pass

When a generated test fails, an assistant asked to "fix the tests" will sometimes change the expectation to match the current output. The test goes green and the bug is now encoded as correct behaviour.

The rule I'd set on any team: a changed assertion in a PR needs a reason in the description. If the expected value moved, someone has to say why the old value was wrong. That one rule catches a lot.

A review checklist

When I review a PR with generated tests, I run through this:

1. Does each test map to a stated requirement or bug?
2. Are expected values literals derived from the spec?
3. Are mocks only at external boundaries?
4. Are the edge cases above covered where they apply?
5. Does every async call get awaited, and does every test assert something that could fail?
6. If an existing assertion changed, is the reason written down?
7. If I break the code on purpose, do the tests catch it?

The last one is the fastest honest check. Flip a comparison, delete a line, return early. If the suite stays green, it was decoration.

Where generated tests do help

None of this means avoiding AI for tests. I use it to scaffold files, generate fixtures, enumerate cases I then prune, and write the repetitive parts of table-driven tests. The workflow that works for me is to write the list of cases myself, usually straight from the spec, then let the assistant turn that list into code. With Spec-Driven Development the edge cases are already written down before anyone opens a test file, which makes this easy.

The same principle applies to production code in general, which I wrote about in using AI in code review without lowering the bar. For LLM features specifically, unit tests are only part of the picture; you also need an evaluation set for the feature itself.

The short version

AI-generated tests are fast to write and easy to over-trust. Check that assertions come from requirements, that mocks stay at the boundaries, that edge cases exist, and that the suite fails when you break the code. You define the cases; the assistant writes the boilerplate.