The API Tests Passed. Existing Clients Broke.

The original API created a user from a request like this:

POST /users?name=ann&age=30

The rebuild expected name and age in a JSON body instead. That request to the rebuild returned {"error":"name is required"}. A client sending it could no longer create a user, even though the generated visible tests and held-out test all passed.

This happened in a cold run of Rebuild Dossier against a third-party Express todo API. The coding agent rebuilt the app from extracted contracts and tests, without the source. It implemented all ten routes, passed the four visible checks and the one held-out check, then failed a side-by-side request comparison with the original.

The route was present. The contract was missing.

For POST /users, the extracted contract contained only the route declaration. It did not say that the handler read req.query.name. The generated check sent an empty JSON object and accepted any response with a status below 500. That check could reject a crash. It could not distinguish the original query-string API from a new JSON-body API.

The status predicate was weaker than it looked: a missing route can return 404 and still satisfy “below 500.” A green result here meant the server answered without a 5xx error. It said nothing about whether this request created a user, where the inputs belonged, or what came back.

These are compatibility breaks, not cosmetic differences. A client depends on where fields go, what shape comes back, and whether the service starts and retains data. A route list alone records none of that.

Check the request a real client sends

For an API rebuild, I would save a small set of representative exchanges from the existing app before writing the replacement. Each example should pin the method, path, query string, headers, body, status, and response shape. Then I would replay the exact requests against the rebuild.

1. Use real parameter locations. A test that sends {} cannot tell whether a working client uses query parameters, JSON, or form data.
2. Assert the response contract. “Status below 500” is a useful smoke check. Compatibility needs the expected status, fields, and envelope.
3. Exercise state across requests. Create a record, fetch it, restart the process, and fetch it again if persistence is part of the original behavior.
4. Run the documented entry point. An importable app object is not enough if existing users launch it with node index.js.

A side-by-side check needs some care. Generated IDs and timestamps can differ between runs, so compare stable fields and state changes rather than raw response bytes. But do not normalize away a changed envelope, missing field, or different parameter location: those are the contract you are trying to protect.

What changed after this run

The extractor was later changed to follow Express handlers into their source, including some imported functions and local callees. It could then put the behavior, not just the route declaration, into the contract and aim mutation checks at the handler. On this same todo app, the regenerated contract includes the code that reads req.query.name.

That is a better starting point for a rebuild, but it is not a complete compatibility guarantee. In this run, persistence and server startup also differed. Those behaviors live outside a single route handler and need their own checks. The extractor fix does not retroactively make the earlier all-green rebuild compatible, and the log does not report a new full rebuild of this app after the fix.

This is one third-party app and one rebuild, not a failure rate. The useful lesson is narrower and concrete: if a generated test never sends a request that an existing client sends, its green result cannot certify that client will still work.