Published by Sean Carr
Find the Bug Class, Not the Bug
AI agents fixed a database cutover one symptom at a time, and one flow kept breaking. A syntax-aware census sized the whole class in one pass, and the guarded rewrite tool accepted four of the 777 calls it counted under its guards.
At a Glance
One flow, repeated breaks
5 in 2 rounds
Related entries on new worker account creation and first sign-in, each with a different immediate cause
Records with no database home
3 singly, 11 in one sweep
Three surfaced one at a time as bugs; one inventory sweep then listed 11 more record types
Legacy data calls counted
708 in 155 files
Syntax-aware census, plus 69 calls through older wrappers in 27 files
Rewritten automatically
4 of 777 calls
385 left for reviewed manual change; 388 held by other running lanes
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Related worker flow entries | 5 in 2 rounds | An access-policy synchronization defect, a role repair that wrote to a different store than it read, user profiles with no table, the same policy defect still present in stored data a round later, and an access-policy gap on first sign-in. |
| Storage-home bugs | 3 singly, 11 in one sweep | The bug log names this the biggest repeat and gives the root cause as no single inventory of the paths the app reads and writes. |
| Failing check rows to root causes | 227 rows, 22 causes | 15 application or configuration causes, 4 problems in the check script itself and 3 fixture or product-decision items. One cause, a database configuration gap, accounted for 60 of the 227 rows. |
| Direct legacy data calls | 708 in 155 files | Plus 69 calls through older wrapper functions in 27 files, for 777 calls in all. |
| Path-related candidate sites | 995 sites in 235 files | Syntactic matches for path construction, path helpers and validators, and path literals outside the modules meant to own storage locations. Candidates, not confirmed hand-built paths. |
| Fake-API test candidates | 901 files | Test files that use or implement a fake of the platform API. Candidates, not confirmed fakes: some are real-path adapters sharing the same interface. |
| Places permissions are defined or decided | At least 6 | Two in the platform core, three in the application server and one in the web app, from one inventory; the count does not establish six independent authorities. The application server alone had 615 direct permission-check sites in 109 files; the web app had 218 direct permission calls in 94 files. |
| Automatic rewrites | 4 of 777 calls | 385 manual and 388 skipped. A final whole-repository dry run proposed no further automatic rewrites under the tool's guards; 4 is the subset the tool accepted, not a measured maximum. The committed baseline for direct calls fell from 708 to 704. |
When an AI agent reports a bug fixed, ask for the count of other sites with the same flaw, produced by a tool, before accepting the fix.
Accept "fixed" for a flaw that could exist elsewhere without a number for how many other places were checked.
Keep a count baseline for each known bad pattern and fail any change that raises it.
Expect an automatic rewrite to clear the class on its own. Here 207 of the 385 manual cases, 27% of the 773 calls not rewritten, needed a destination that did not exist yet.
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
During a database cutover, AI agent lanes fixed bugs as they surfaced, one log entry per bug. The flow for creating a new worker account and signing that worker in appears in five related entries across two fix rounds, each with a different immediate cause. One access-policy synchronization defect recorded as repaired in the first round was still blocking new worker sign-ins in the second: the first entry had already noted that stored data was not migrated, and the repair did not reach it. Storage gaps turned up the same way: three record types broke one at a time, then a single sweep listed 11 more with no home in the new database. We then had agents count whole classes with a syntax-aware code-search and census tool we built. It found 708 direct legacy data calls in 155 files, 995 path-related candidate sites, 901 test files that are candidates for testing against a fake of the platform API, and permission definitions in at least six separately maintained places. Counting was the easy part. A guarded rewrite tool changed 4 of the 777 data calls automatically and left 385 for reviewed manual changes, 207 of them because the module those calls should move into did not exist yet. Data from October 21 to 22, 2026, analyzed October 23, 2026. This is a tentative case study of one codebase during one migration. We did not measure whether census-first triage reduced repeat fixes afterwards.
Method
The engineering work was done by AI agent lanes under a coordinating agent and human oversight. We read five kinds of saved records: the cutover bug log (one row per bug with its root cause and class, written by the lanes and the coordinating agent), a root-cause diagnosis of an end-to-end check run of 424 rows (227 failing), the census report and its machine-readable counts, a plan that mapped every place permissions were defined or checked, and the final dry run and report of the guarded rewrite tool. Where a machine-readable file existed, we recounted from it rather than from the prose.
Counting rules. A "worker flow entry" is a bug-log or diagnosis entry whose symptom is new worker account creation or a worker's first sign-in. A "storage-home" bug is one the log itself classes as a record type with no home in the new database, or reads and writes going to different stores. Census counts are syntactic: they match how the code is written, not what runs at runtime. A "place permissions are defined or decided" is a separately maintained permission list, catalog or decision point named in the permission map, not each individual check; the count does not establish that each place is an independent authority.
Conditions. The bug log's first 22 entries come from the first two logged fix rounds (October 21 into the early hours of October 22) and predate the census. On October 22 the team adopted an 11-question triage protocol that requires listing every site with the same flaw before a fix starts; the census tool was built and tested early that day, and the rewrite lane finished that morning. All measurements are from local checkouts and test runs, not production.
What We Measured
One flow appeared in five related bug entries across two rounds of one-at-a-time fixes. A census then sized whole classes syntactically in one pass, and the guarded rewrite changed 4 of the 777 data calls it counted.
Related worker flow entries
5 in 2 rounds
An access-policy synchronization defect, a role repair that wrote to a different store than it read, user profiles with no table, the same policy defect still present in stored data a round later, and an access-policy gap on first sign-in.
Storage-home bugs
3 singly, 11 in one sweep
The bug log names this the biggest repeat and gives the root cause as no single inventory of the paths the app reads and writes.
Failing check rows to root causes
227 rows, 22 causes
15 application or configuration causes, 4 problems in the check script itself and 3 fixture or product-decision items. One cause, a database configuration gap, accounted for 60 of the 227 rows.
Direct legacy data calls
708 in 155 files
Plus 69 calls through older wrapper functions in 27 files, for 777 calls in all.
Path-related candidate sites
995 sites in 235 files
Syntactic matches for path construction, path helpers and validators, and path literals outside the modules meant to own storage locations. Candidates, not confirmed hand-built paths.
Fake-API test candidates
901 files
Test files that use or implement a fake of the platform API. Candidates, not confirmed fakes: some are real-path adapters sharing the same interface.
Places permissions are defined or decided
At least 6
Two in the platform core, three in the application server and one in the web app, from one inventory; the count does not establish six independent authorities. The application server alone had 615 direct permission-check sites in 109 files; the web app had 218 direct permission calls in 94 files.
Automatic rewrites
4 of 777 calls
385 manual and 388 skipped. A final whole-repository dry run proposed no further automatic rewrites under the tool's guards; 4 is the subset the tool accepted, not a measured maximum. The committed baseline for direct calls fell from 708 to 704.
Experimental Notes & Observability Details
- Why 385 calls were left for manual change: 207 had no certified owner module or database home to move into, 85 used a dynamic or unregistered operation, 51 involved filters, guarded transactions, batches or cursors, and the remaining 42 were smaller cases such as unusual argument forms. The 207 are 54% of the 385 manual cases and 27% of the 773 calls not rewritten. All 388 skipped calls sat in files that other running lanes held at the time, so they were not assessed for a destination.
- An earlier informal count, made before this census, gave lower call and test-file figures. The census report states it did not reuse them, and we report only the census.
- The census tool itself had ten defects in its first version (nine in code, one wrong test expectation), each found during development and covered by a regression test before use. Its final owned suite was 39 passing tests.
What We Think It Suggests
- **One-at-a-time fixing closed the symptom in front of each agent.** Each worker-flow entry recorded a plausible local fix. The first entry already noted that its fix did not migrate stored data, yet the round moved on, and a round later the same access-policy synchronization defect was still blocking new worker sign-ins. The class, code changes that never reach stored data, had been diagnosed in the log but was repaired only where it was first seen.
- **A census turns "are there more?" from a judgment into a number.** The storage-home bugs show the contrast: three were found by hitting them, then one inventory sweep listed 11 more. The triage protocol now makes the count a required answer ("fixed 1 of N is not done"), and the diagnosis recorded sibling sites for each application root cause rather than only the failing row. In these entries, agents reported the bug they were shown as fixed. Whether other sites exist is a question a tool can answer with a count, within the limits of syntactic matching, rather than a question to leave to the agent's own judgment.
- **Counting a class is not the same as fixing it.** The rewrite tool was guarded: golden examples, refusals for anything non-trivial, idempotence checks and covering tests before any commit. Under those guards it changed 4 of 777 calls; the guards limit risk but do not establish that every rewrite is semantically safe. Of the 385 calls left for manual change, 207 had nowhere to go, because the per-record module or database home they should call did not exist yet. The census was most useful as a work list, a baseline that blocks growth, and a map of missing design, which the tool could report but not supply.
- **Policy definitions are the same problem in another form.** With permission definitions in at least six separately maintained places, a fix made in one of them can leave another disagreeing. The diagnosis recorded policy synchronization defects of that kind, and the response planned was one source that the other places read, rather than another patch to one of the six.
Limits
- **One codebase, one migration, two days.** These are observations from a single cutover with small counts. They do not estimate how often one-at-a-time fixing repeats across projects, teams or model families.
- **No before-and-after measurement.** The triage protocol and census arrived mid-stream. We did not measure whether repeat fixes fell afterwards, or time an agent finding all sites with the tool against plain text search on the same bugs.
- **The counts depend on our grouping.** "Five worker flow entries" and "at least six places" follow the counting rules above; a different grouping could give a different count, or more places if overrides and cached copies are counted separately.
- **Census counts are syntactic.** They follow how code is written, so dynamic calls and aliases can be missed or miscounted, a census list cannot certify that it is complete, the 995 path-related sites are candidates rather than confirmed hand-built paths, and the 901 fake-API test files are candidates rather than confirmed problems.
- **Model identities are not recorded per entry.** The log does not establish which model wrote each fix, so no model comparison follows.
Detailed Analysis & Field Observations
The cutover moved an application's data from a document store to Postgres behind the platform core. Agent lanes worked in rounds. Each round ran an end-to-end check, fixed what failed and wrote a row per bug: what broke, why, the class, the fix, and what should catch it next time. The log is honest about causes, and in hindsight it also shows a pattern the agents did not stop for. One flow, and a few kinds of cause, kept coming back.
| Worker flow entry | Round | Immediate cause | Recorded fix |
|---|---|---|---|
| 1 | First | Access-policy synchronization defect in the default worker role; stored data not migrated | Repair migration |
| 2 | First | Role repair reported 3 updates; nothing changed | Role writes moved to the same store as reads |
| 3 | First | User profiles had no table in the new database | New table and backfill |
| 4 | Second | The first-round policy defect was still present in stored data, so new worker sign-ins failed | Versioned migration for stored data |
| 5 | Second | Access-policy gap on first sign-in, so invite acceptance was never recorded | Profile read moved to the governed path |
The second-round diagnosis is where the method changed. Instead of fixing 227 failing rows one at a time, an agent traced them to 22 root causes, each with a list of sibling sites that shared the flaw. It also corrected an earlier log entry: the logs showed a different cause for one permission failure than the round's own note had recorded. One packaging list that left five rule files out of the platform core's image broke the platform's command-framework operations. That is one fix, not dozens.
The census came next. A tool we built parses Python with a concrete syntax tree and TypeScript with the compiler's own parser, records every site matching a pattern, and stores the count as a baseline that fails when it grows. The first run counted the legacy data calls, path-related candidates and fake-API test candidates above in a single pass. A separate plan mapped every place permissions were defined or checked, which is how the six separate places became visible.
The rewrite step tested how far a count goes. The rewrite tool refused any call that had filters, transactions, unusual arguments or no certified destination, and committed nothing without covering tests passing. It rewrote 4 calls. The other results were still useful: they list every syntactic match for the old pattern (aliases and dynamic calls can escape it), and they show that 207 of the 385 manual cases are waiting on a design decision, the owner module for each record type, rather than on mechanical editing.
Related studies: Patching on Top followed a repair series that was merged and then reverted; Fewer Findings Did Not Mean the AI Fix Loop Was Finished looked at review rounds that kept producing repairs; From Recurring AI Checks to Scripts and Preventing Code Slop and Enforcing Task Completion with Mechanical Gates cover moving checks from agent judgment into deterministic code. Two companion studies cover the same application in the same week: Are Your Tests Honest? runs its test suite against a fake of the platform API and against the real platform, and Working Is Not Organized measures how spread out its data access was a day after this census. This study adds the step before the fix: count the class first.