The original todo app had a working list. You could add an item, change it, and remove it. The rebuilt page could do none of those things. Its visible page test still passed.
I found this while testing Rebuild Dossier, a tool that extracts contracts and tests from an existing application before a coding agent rebuilds it. For this run, the agent received the generated material without the original source. The generated page test checked that three fixed strings appeared on screen. The agent returned a page displaying those strings.
What the test actually proved
The page capture had recorded text, not an interaction. There was no assertion for creating a todo, seeing it join the list, editing its title, or deleting it. The test did exactly what it was written to do. It could not tell a working interface from a static one with the right words.
| Part | What happened |
|---|---|
| Original app | A person could add, edit, and delete todos through the interface. |
| Generated check | Three fixed strings had to appear in the rendered page. |
| Passing rebuild | The strings appeared in a static page; the list did not work. |
The agent also built GET, POST, PUT, and DELETE endpoints. That API work did not connect the page to a usable list. It was a separate issue with the build order; the static page itself satisfied the narrow check. Both details matter when diagnosing the result.
The missing unit was the user action
A text assertion is useful when the requirement is text. “Manage todos” is a sequence of state changes. To check that feature, I would drive the same sequence through the original and rebuilt interfaces:
- Create an item and confirm it appears in the list.
- Edit the item and confirm the old title is replaced.
- Delete the item and confirm it disappears.
That browser path matters even if the API endpoints each pass their own tests. It checks the connection between the controls, state, requests, and rendered result. A page can have working endpoints and still leave a user staring at a static list.
The same distinction applies beyond CRUD. A captured screenshot may freeze an animated counter halfway through its transition. In another Rebuild Dossier run, a coding agent hardcoded that captured intermediate number and passed the available checks without reproducing the animation. The capture timing bug was later fixed. The useful question remains: did the test record the intended behavior, or only one moment of it?
A review checklist for generated tests
Before treating a green run as a faithful rebuild, I would look at the tests as carefully as the code:
- Write down the feature in verbs. “Add, edit, delete a todo” is more testable than “todo page renders.”
- Mark what the capture missed. Authentication, delayed data, interaction, and animation can all be absent from a page snapshot. An uncovered route should be reported as uncovered.
- Exercise the path a user takes. Test the interface through its controls, then compare the same actions against the original app.
- State the claim at the test’s level. If the check only proves three strings are present, report that. Save “the feature works” for a check that actually exercises the feature.
This was one blind rebuild of one third-party app, so it does not tell us how often the failure happens. It does show where the check stopped: three strings rendered. A passing test is useful only when you can say what behavior it actually exercised.