End-to-end testing found 16 bugs in done tickets

End-to-end testing a compliance platform against the running app turned up sixteen defects. As my commit put it: "Every one of them sat behind a ticket that had been marked done." The unit tests passed, and the suite was large and green. None of that told us whether a person could use the features.

This post is about what those sixteen bugs looked like, why unit tests could not see them, and the habits I now follow so "done" means someone has used the thing.

What end-to-end testing caught that unit tests missed

The platform has a Python backend and a React frontend, and a large unit test suite. The bugs fell into a few families, and each family is a kind of failure that lives between components.

Data silently dropped at the boundary. The worst one: the request body for raising a risk lost four rating justifications on the way in, and the model was set to extra="ignore", so nothing complained. The result was that no risk raised through the interface could be submitted for review at all.

Half a feature. Writes that worked with reads that did not, for closure data and a routed role. Changes that published no event, so nothing downstream reacted. Six backend features with no screen. An import that was only reachable from the empty state, "so a tenant could import exactly once, ever."

Things that never run in tests. The frontend start script pointed at an output path the build never wrote, which is why the production server would not boot. No unit test starts the production server.

One wrong key. A single comment was stored with author_user_id instead of user_id, and the entire action items list returned 500 for everybody. The existing test asserted the comment body and never the keys. It now asserts the key set. I found it by opening the page.

Pydantic ignores extra fields by default

The dropped justifications deserve a closer look, because the default behaviour is the trap. The Pydantic config docs list extra as "ignore" by default: unknown fields are discarded quietly. That is friendly for loose inputs and dangerous for a form that must arrive complete.

from pydantic import BaseModel, ConfigDict

class RaiseRiskRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    title: str
    likelihood: int
    impact: int

With extra="forbid", a field the client sends in the wrong place becomes a validation error you see on the first try. My default for request bodies that represent a user's form is to forbid extras, and to allow them only where the client legitimately sends more than the server reads.

A browser bug jsdom could not reproduce

One bug only exists in a real browser. On a multi-step form, clicking Next submitted the whole form. React reused the same DOM node for the Next button and the Submit button, and flipped its type from button to submit in the middle of the click. The browser then ran the default action of a submit button.

The fix was to key the two buttons apart so React creates separate elements. The React docs on preserving and resetting state explain why: same component at the same position means React keeps the node.

{isLastStep ? (
  <button key="submit" type="submit">Submit</button>
) : (
  <button key="next" type="button" onClick={goNext}>Next</button>
)}

It is covered by a browser test, "because jsdom does not reproduce the browser's timing for a click's default action and would pass either way." A component test would have stayed green with or without the fix.

What is the difference between end-to-end tests and unit tests?

A unit test checks one piece of code in isolation, with its collaborators faked or controlled. An end-to-end test drives the running system the way a user does, through the real interface, real server and real database. Unit tests tell you each piece does what its author meant. End-to-end tests tell you the pieces agree with each other and that the feature works for a person.

Both matter, and they fail differently. Every one of the sixteen bugs above passed its unit tests.

Tests that pass while checking nothing

Two more findings from the same period changed how I read a green suite.

mypy --strict reported 95 errors in the test modules, mostly missing return annotations. Under strict mode that matters, because mypy does not check the body of an unannotated function, so those test functions had never been type checked. After the fix, 5408 tests passed and mypy reported no issues in 3497 files.

An end-to-end suite also hunted for records nobody had touched yet, "which works once and never again." Each test now creates the state it needs. A test that depends on leftover data will pass on a fresh database and fail on the second run, or worse, pass against the wrong record.

Verify the work breakdown against the code

After the sixteen bugs, I checked the whole work breakdown against the code instead of against ticket status. It counted 494 items: 449 built, 26 partial and 19 not built. The largest gap was a trigger service that was unit-tested but that nothing called. Its tests passed because they called it directly.

That check is cheap compared with finding the gaps in front of a client. It works best when the work is written down as a spec first, which is one reason I use spec-driven development: there is a list to verify against. A typical verification line on a change now reads like this one from the same project: 22 Playwright tests over the real interface, 4,977 unit tests, ruff and mypy clean.

Where I'd start

• Before closing a ticket, open the feature in the running app and use it once as a user would.
• Write at least one end-to-end test per user-visible flow, through the real interface.
• Forbid extra fields on request bodies that represent a form.
• Assert keys and shapes, as well as values, when a test checks stored data.
• Make each test create its own state.
• Periodically check the backlog against the code, looking for features with no screen and services nothing calls.

If an AI agent wrote the code or the tests, these checks matter more, since generated tests tend to mirror the implementation. I covered what to look for in reviewing AI-generated tests, how to verify a refactor by diffing old and new output, and how I let an AI agent test a Flutter app.