Research / index 001
Two lines of work: verifying what coding agents actually produce, and auditing where standard machine-learning evaluation hides unstable decisions. Profiles: ORCID · Google Scholar.
Rebuild Dossier is an open-source MCP server that extracts interface contracts and tests from an existing Express or Next.js application. It packages those materials for a fresh coding agent and adds hooks that enforce parts of the rebuild workflow. The tool produces a spec and evaluation materials; the agent performs the rebuild.
A separate paper on the same tool, submitted to ICSE 2027 (Software Engineering in Practice track) and currently under review. When one pipeline derives both a rebuild's specification and its acceptance tests from the same extraction, the two can share a blind spot. In a pre-registered study of 30 frozen rebuilds, a do-nothing stub passed every generated visible and held-out test, while a requirements oracle frozen before scoring, and 59 checks written independently by an external researcher, showed the weaker-model rebuilds answering every authentication-rejection check incorrectly. The paper proposes a three-step audit to run before trusting a green rebuild.
An audit of a tree classifier ranking stocks by probability of beating a market benchmark, across 188 monthly cross-sectional decisions over a 200-stock U.S. equity universe. Correcting validation-label leakage and return-aggregation errors left test AUC nearly unchanged (0.553 → 0.551), but replaced 61.1% of the selected portfolio (mean Jaccard overlap 0.389) — aggregate accuracy survived while the actual decisions did not. A frozen 12-minus-1-month momentum rule outperformed the corrected model and a fixed ensemble once the ledger was fixed.
What the evaluation found
In a sealed follow-up on one application, all 30 rebuild snapshots passed the 20 visible tests. A separate requirements check found substantial differences in authentication and response behavior. The check covered 77 database-independent requirements out of 132 written checks; it was author-written and used a coarse response-key comparison. The result identifies a concrete limit of the generated suite in that setting, not a general failure rate for coding agents.
The chronological findings log records the experiments, bugs, fixes, and later corrections. The arXiv paper is a versioned publication; the log is the living record. For the latest requirements check, see the archived oracle and results.
From the workbench
These articles unpack individual failures as engineering problems: what the original application did, what the generated check measured, and what a better verification path would exercise.