I wrote a program that does nothing. Whatever you ask it, it answers HTTP 200 and stops. It has no login, no data and no features.

Then I ran it against the test suite that Rebuild Dossier generated for a real application. It passed 20 of 20 visible tests and 12 of 12 held-out test files. A held-out test is one the coding agent never saw while building.

Then I ran it against a separate set of checks I wrote by hand, each one describing something the original app actually does. The stub matched 0 of 77.

Why a stub can pass

This was a deliberately constructed control, so the stub was built to find the weakest point in the suite. It found one. Many generated checks ask whether the server answers without crashing, not whether it answers correctly. A response that is not a server error satisfies that question, and so does a response that does nothing.

The tests also came from the same extraction as the spec. If the extraction missed a behavior, the spec does not mention it and no test checks it. Both artifacts share the gap, so the suite agrees with the spec and still disagrees with the original app.

In the study under review at ICSE 2027, the clearest shared gap was authentication. Rebuilds from the weaker model tier answered every authentication-rejection check incorrectly, and the generated tests did not notice. I wrote about a smaller version of this in The API Tests Passed. Existing Clients Broke.

What this does not show

The limits matter as much as the number:

  • The 77 checks cover database-independent requirements from 132 written checks.
  • I wrote them, so they reflect what I thought to ask. An outside check could find more gaps or fewer.
  • They compare responses by key, which is coarse.
  • It is one application. The result is not a failure rate for coding agents, and it does not say the agents are bad at rebuilding apps.

The claim is narrow. A green generated suite can be satisfied by software that does nothing, so green alone does not show a rebuild is correct.

Four checks to run before trusting a green run

  1. Run the dumbest program that could pass. Write a stub that returns the same success response to everything and point your tests at it. If it passes, the suite measures less than you thought.
  2. Take at least some checks from the original, not the extraction. Write requirements from how the existing app behaves and score the rebuild against them separately.
  3. Test the rejections. Wrong password, missing token, bad input. A suite that only exercises the happy path cannot tell a working login from no login.
  4. Report the claim at the size of the test. "The server answered without a 5xx error" is a different sentence from "the feature works."

The baseline is in the repository, so you can run the stub yourself and read the scoring protocol, including its limits.