Published by Sean Carr
Fewer Findings Did Not Mean the AI Fix Loop Was Finished
Saved review rounds show declining finding counts, repair regressions, and decisions to stop patching without a clean reviewer verdict.
At a Glance
| Telemetry Metric | Measured Value | Context & Observability |
|---|
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
A repair is another code change. Sending an AI agent a review finding does not establish that the next version closes it, or that the repair leaves neighboring behavior intact. We examined saved rounds from AI engineering work rather than counting an agent's completion message as the outcome. Across five audit-repair sequences, the reported blocking findings fell from 25 to 11 to 6. That downward trend was useful, but it concealed different endings. One sequence reached a reviewer PASS at its third review. The other four still had blocking findings and received explicit circuit-breaker rulings that ended further review rounds and prescribed a final repair. Those rulings were decisions about how to proceed, not clean reviewer verdicts. Data from October 10 and 13, 2026, analyzed October 2026. This extends our earlier gateway and inbox case studies by comparing the shape and stopping conditions of additional repair sequences. It does not estimate model rankings or a general defect rate.
Method
The work used AI builder lanes and a separate Codex adversarial reviewer. We read the saved machine-readable reports alongside their human-readable versions and the final ruling briefs. The reports identify Codex as the adversarial reviewer. We do not infer the builder's exact model from a commit author or attribute a comparative advantage to the reviewer family.
The primary set consists of the five audit sequences with saved first, second, and third reviews. We count each report's blocking finding field. This excludes nonblocking findings and does not equate a failed formatting check with a blocking code finding. A finding can recur across reports, so adding rounds counts review observations, not distinct defects.
We also inspected later task reports with repeated reviews. The table below includes two contrasting examples to test whether a smaller count or an early PASS reliably described the next saved report. These examples are descriptive, not a randomly sampled replication.
What We Measured
The circuit breaker appears in four of these five sequences after the third saved review, following two repair opportunities. That is its observed use in this set, not its frequency across all engineering work. The records do not give a comparable time to PASS for the other sequences because their loops ended by ruling.
Experimental Notes & Observability Details
- Repair-caused findings need a different count from findings discovered later. In sequence E, the second report explicitly describes one introduced problem. In sequence A, the third report explicitly describes two regressions. These are three report-attributed repair regressions, a lower bound from explicit descriptions rather than a complete causal classification. We have not established how many of all later-round findings were introduced by the immediately preceding repair. Some reports flag missing context, and a later discovery alone is insufficient evidence of introduction.
- These later reports covered changing code and sometimes broader surrounding changes. They show that the saved verdict sequence was not uniformly improving. They do not show that an earlier PASS was incorrect for the exact code and scope it reviewed.
What We Think It Suggests
- Review counts describe workload remaining under a particular review scope. They are weaker evidence for progress when the fix changes the scope or moves the problem into a neighboring behavior. A declining count can accompany successful repairs and newly introduced failures at once.
- The AI element is the interaction between generations: an agent receives a concrete criticism, produces a plausible repair, and another model examines what that repair actually does. The reports contain examples where the repair changed behavior in a way the next reviewer challenged. This is a reason to preserve repair history and examine the latest change, rather than treating the first review as a checklist that can only shrink.
- Stopping rules also need honest outcome labels. A final ruling may be a practical way to break an unproductive loop. Its existence cannot be counted as reviewer PASS or evidence that the resulting repair was independently cleared. Keeping those outcomes separate makes the workflow easier to assess without overstating what an AI review achieved.
Limits
- This is a small, selected set of related engineering tasks. It does not measure how often review loops churn across the organization or across model families.
- Reports are reviewer judgments, sometimes conditional on omitted context. We did not independently reproduce every finding or test a live system.
- Review scope and code changed between rounds. Finding totals are not a controlled measure of code quality, and repeated findings are not independent observations.
- The causal count is incomplete. Explicit regression descriptions support a lower bound, not an introduced-by-fix percentage. Final rulings also leave outcomes censored for any rounds-to-PASS calculation.
Detailed Analysis & Field Observations
| Audit-repair sequence | First review | Second review | Third review | Recorded ending |
|---|---|---|---|---|
| A | 3 | 2 | 2 | Circuit-breaker ruling |
| B | 4 | 3 | 2 | Circuit-breaker ruling |
| C | 9 | 2 | 1 | Circuit-breaker ruling |
| D | 7 | 3 | 1 | Circuit-breaker ruling |
| E | 2 | 1 | 0 | Reviewer PASS |
| Later task sequence | Initial review | Next review | Following review |
|---|---|---|---|
| Scrubbing-related change | 1 blocking | 1 blocking | 3 blocking |
| Image-processing change | PASS, 0 blocking | BLOCK, 1 blocking | BLOCK, 1 blocking |