Published by Sean Carr
Are Your Tests Honest? One Suite Against a Fake Platform and the Real One
An AI-maintained test suite had 44 failing entries against a fake of our platform API and 1,230 against the real platform. A first-pass sort put most of the gap on the tests themselves, and the reported share of real bugs differed between the first pass and the fix agents' reports.
At a Glance
Against the fake
17,317 passed, 44 failing entries
43 failures and 1 error, 745 s on 8 workers
Against the real platform
16,368 passed, 1,230 failing entries
955 failures and 275 setup or teardown errors, 1,101 s on 8 workers
First-pass sort of 1,163 real-only failures
1,067 / 72 / 24
Test relied on the fake / test setup gap / possible real bug
Reported by fix agents (7 slices)
81 of 815
Labeled real bugs, 9.9%, versus 2.1% in the first pass, on different cohorts; one of the seven reports gave its supplied first-pass count
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Fake run | 17,317 passed, 44 failing entries | 43 failures and 1 error, with 39 skipped and 653 expected failures. 745 s wall time on 8 workers. |
| Real-platform run | 16,368 passed, 1,230 failing entries | 955 failures and 275 setup or teardown errors, with 39 skipped and 596 expected failures. A setup or teardown error is a separate entry and can accompany another outcome for the same test. 1,101 s wall time on 8 workers. |
| Pass share | 99.7% vs 93.0% | Passed entries divided by passed, failed and error entries, against the fake and against the real platform. |
| Real-only failures | About 1,190 | Tests that failed against the real platform and were not in the fake run's failure list. The sorting script counted 1,186; raw recounts from the run logs gave 1,181 and 1,195, so the exact unique-test count is unresolved. The script set aside 23 in files owned by other lanes and sorted the other 1,163, in 185 files. |
| First-pass sort | 1,067 / 72 / 24 | Of 1,163 failures: the test relied on the fake / the new test setup had a gap / possible real bug. 17 of the 24 possible bugs were in one test file. |
| Most common first error | 127 of 185 files | Files whose first listed real-platform error was a permission denial. They held 787 of the 1,163 sorted failures. This is the first error per file, not 787 separately diagnosed permission defects. |
| Fix agents' slice reports | 81 of 815 | Real-bug labels in the seven slices whose reports gave a count for the original slice: 11, 13, 5, 16, 1, 14 and 21, or 9.9%. The report with 1 gave the initial supplied classification rather than its own; the other six give 80 of 699. The first pass labeled 24 of all 1,163 sorted failures, 2.1%, as real bugs, so the two rates cover different cohorts. |
| Wall time between runs | About 48% more wall time | 1,101 s versus 745 s, an observed 47.8% difference between two runs on nearby code, not an isolated estimate of the real platform's overhead. In one measurement, a database cluster start of 7.3 s is shared, and the core takes about 3.7 s per worker, paid only by tests that call the platform. |
Run the suite at least once against the real dependency and compare the two failure lists.
Read a mostly green run against a fake as evidence that the system works.
Create test data through the same create operations the application uses.
Let tests insert partial records into a fake that would never accept them in production.
Treat an automatic failure sort as a starting estimate, and recount after someone reads each failure.
Report the first-pass count of real bugs as the number of real bugs.
Track known failures by name and compare each run against a base.
Let hundreds of known failures become background noise that a new bug can hide in.
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
AI agents wrote and maintain the test suite for one application on our platform. Its tests that called the platform used an in-memory fake of the platform API, built from a pinned copy of the platform's operation catalog and schema. On October 21, 2026, against that fake, the suite had 17,317 passing tests and 44 failure or error entries. We then pointed the suite at the real platform core and a real local database. Against the real platform it had 16,368 passing tests and 1,230 failure or error entries. The first-pass assessment put most of the gap on the tests. An automatic first-pass sort labeled 1,067 of the 1,163 real-only failures it sorted as tests that relied on the fake, mostly because their test data could not exist on the real platform. It labeled 72 as gaps in the new test setup and 24, or 2.1%, as possible real bugs. Seven fix-agent reports on their own slices labeled 81 of 815 failures, or 9.9%, as real bugs. The two cohorts differ, one of the seven reports gave its supplied first-pass count rather than its own, and neither count has been independently audited. The day before, the CEO had hit bugs in normal use, including a credential document upload that did not work and a shift-create request rejected as invalid. According to the orchestrating agent's review, no failing test had pointed at either one. Data from Oct 21 2026, analyzed Oct 23 2026. This is a single-suite case study.
Method
The suite belongs to one application on our platform and runs with pytest on 8 parallel workers. By default, its tests that call the platform used an in-memory fake of the platform API. The fake was built from a pinned copy of the platform's operation catalog and database schema. It models some platform behavior, including simplified permission and scoping checks, batch rollback and repeated-request handling, but it is not the platform's implementation: it does not run the platform's own validation code, real SQL or database constraints.
Three AI build lanes made the real-platform harness. One started a portable local database with one database per test worker and built the schema from the deploy SQL. One ran the platform core's own request handling, permission checks and database handler inside the test process. One pointed the existing tests at that core. Identity records were the one exception: they were loaded through the platform's existing test identity mechanism, not created end to end.
We compare two full runs from October 21, 2026: one with the fake selected and one against the real platform. The fake run finished first. The real run used the final integration tree a few hours later, so the two runs are on nearby code rather than identical code.
An agent-written script took the failures that occurred only against the real platform and gave each a first-pass class: the test relied on fake-only behavior, the new harness had a gap, or there was a possible real bug. The failures in other lanes' files were set aside. The rest were split into 10 slices of about 116 failures, one per fix agent. Each fix agent read its failures and reported a classification of its original slice; one of the seven reports we use gave the initial supplied classification instead of its own. We read those reports; we did not reclassify the failures ourselves.
For context, we also used a base comparison made that morning on the fake-mode suite. It matched each failing test against a clean export of an earlier commit and labeled it pre-existing, new, absent at base, or flaky on an isolated rerun.
What We Measured
The suite had 44 failure or error entries against the fake and 1,230 against the real platform, where about 1,190 tests failed only there. The first-pass sort put most of the real-only failures on the tests. Seven fix-agent slice reports labeled real bugs at about 4.8 times the first-pass rate, on a different cohort.
Fake run
17,317 passed, 44 failing entries
43 failures and 1 error, with 39 skipped and 653 expected failures. 745 s wall time on 8 workers.
Real-platform run
16,368 passed, 1,230 failing entries
955 failures and 275 setup or teardown errors, with 39 skipped and 596 expected failures. A setup or teardown error is a separate entry and can accompany another outcome for the same test. 1,101 s wall time on 8 workers.
Pass share
99.7% vs 93.0%
Passed entries divided by passed, failed and error entries, against the fake and against the real platform.
Real-only failures
About 1,190
Tests that failed against the real platform and were not in the fake run's failure list. The sorting script counted 1,186; raw recounts from the run logs gave 1,181 and 1,195, so the exact unique-test count is unresolved. The script set aside 23 in files owned by other lanes and sorted the other 1,163, in 185 files.
First-pass sort
1,067 / 72 / 24
Of 1,163 failures: the test relied on the fake / the new test setup had a gap / possible real bug. 17 of the 24 possible bugs were in one test file.
Most common first error
127 of 185 files
Files whose first listed real-platform error was a permission denial. They held 787 of the 1,163 sorted failures. This is the first error per file, not 787 separately diagnosed permission defects.
Fix agents' slice reports
81 of 815
Real-bug labels in the seven slices whose reports gave a count for the original slice: 11, 13, 5, 16, 1, 14 and 21, or 9.9%. The report with 1 gave the initial supplied classification rather than its own; the other six give 80 of 699. The first pass labeled 24 of all 1,163 sorted failures, 2.1%, as real bugs, so the two rates cover different cohorts.
Wall time between runs
About 48% more wall time
1,101 s versus 745 s, an observed 47.8% difference between two runs on nearby code, not an isolated estimate of the real platform's overhead. In one measurement, a database cluster start of 7.3 s is shared, and the core takes about 3.7 s per worker, paid only by tests that call the platform.
Experimental Notes & Observability Details
- The first-pass assessment attributed most of the gap to test data. The tests had inserted partial records straight into the fake: rows without required fields, or with status values the real schema does not allow. The real platform rejected them. In those cases the failure was in the test, not the application.
- The first-pass sort assigned one class per failure from its first error. A per-file roll-up of the same failures gave 1,101 / 62 / 0, because no file had a majority of possible real bugs. The fix agents' later counts were reported per failing test, not per distinct product bug. One agent noted that a test needing a data repair could still be counted as a real bug.
- The real bugs the fix agents described, by their own diagnosis, were ordinary integration faults that this suite surfaced once the real code ran; other kinds of test could have caught them too. A reader treated a returned map of rows and a truncation flag as a plain list. Valid database timestamps with varying fractional precision were rejected. A logging helper changed named arguments into a tuple and broke a library's warnings. A credential document reference pointed at the wrong table. We cannot tie any of these to the bugs the CEO hit.
- The real harness also slowed the fix agents. In the saved completion snapshot from the end of that day, 8 of the 10 fix lanes reported themselves blocked, and 5 of those 8 named a startup lock on the shared local database. Later reports show the lanes completing.
What We Think It Suggests
- **A fake built from the platform's catalog models some behavior, but not the platform's.** It caught wrong operation names and missing fields, and it ran simplified permission and scoping checks of its own. It was unlikely to catch a call the real platform would deny, a record the real schema would reject or a wrong query, because it did not run the platform's implementation or its SQL constraints. A pass against the fake meant the application agreed with the fake.
- **Fixtures fit what the tests ran against.** Test data that agents wrote and the fake accepted failed the real platform's validation. We did not establish why; one plausible reading is that the fake was the only target those fixtures had been checked against. That morning, as other agents fixed old failures, the fake-mode failures fell from 456 to 43 within a few hours, by recorded file times. That was useful work, but a run that gets greener against a fake is still measured against the fake. A green run is only as good as what stands behind it.
- **The reported real-bug rates differed.** An automatic first pass put 2.1% of the real-only failures down as real bugs. Seven fix-agent slice reports, reading failures in context, put 9.9% of their slices there, on a different cohort. Both are AI judgments and neither has been audited, so we cannot say what caused the difference or which is closer to the true rate. The first-pass number was the one that first reached our planning notes.
- **Hundreds of known failures can hide a new one.** On the morning of the study, 399 of the 465 failing entries in the fake-mode suite were already failing at an earlier base commit; 53 were new. When failure is normal, a new failure can sit in that list unnoticed unless each run is compared with a base. Comparing by test name is what separated the 53 new failures from the old ones.
Limits
- **One suite, one application, one day.** This is a case study of a single AI-maintained suite. It does not estimate how honest fake-backed suites are in general.
- **The two runs were not on identical code.** The fake run finished a few hours before the real run, which used the final integration tree. The real run collected 18,198 tests and the fake run 18,053, 145 more, and their outcome entries differ by 180, so some differences may come from code changes rather than the dependency.
- **The classifications are AI judgments that have not been audited.** The first-pass sort was a script that read the first error line. The re-sort came from the fix agents' own reports, and our charter does not accept self-reports as verification. Three slices gave no comparable count for their original failures, one of the seven comparable reports gave its supplied first-pass classification, the 9.9% and 2.1% rates cover different cohorts, and in the end-of-day snapshot 8 of the 10 lanes reported themselves blocked.
- **The link between the CEO's bugs and the tests is the orchestrating agent's account.** We did not rerun the suite at the commits where those bugs shipped, and we did not confirm that any real-only failure was the cause of either one.
- **The real-platform harness is not the whole system.** Identity records were loaded through a test mechanism, and the runs did not touch a deployed environment, a browser or production data. A pass against this harness still says less than a passing check on a deployed test environment.
Detailed Analysis & Field Observations
| Run (October 21, 2026) | Passed | Failed | Errors | Skipped | Expected failures | Wall time |
|---|---|---|---|---|---|---|
| Fake platform API | 17,317 | 43 | 1 | 39 | 653 | 745 s |
| Real platform core and database | 16,368 | 955 | 275 | 39 | 596 | 1,101 s |
| Who sorted the real-only failures | Relied on the fake | Test setup gap | Possible real bug | Sorted |
|---|---|---|---|---|
| First-pass script, per failure | 1,067 | 72 | 24 | 1,163 |
| First-pass script, per file majority | 1,101 | 62 | 0 | 1,163 |
| Fix-agent reports, seven original slices | not comparable | not comparable | 81 | 815 |
The question behind this study came from the CEO, after two bugs had reached him in normal use: are our tests really honest? The suite was large and mostly passing against the fake. The fake did what it was built to do. It checked that the application called operations that existed, with fields the catalog knew. It did not run the real platform's own checks on whether to allow the call, store the record or return the expected row.
Running the suite against the real platform was not about more tests. It changed what a pass means. Before, a pass meant the application agreed with the fake. After, a pass means the call went through the platform's permission checks, validation and real SQL against a real database. The real-platform run took about 48% more wall time than the fake run, an observed difference between two runs on different code trees rather than a measured cost of the change, and about 1,190 tests that had looked fine now failed.
The more useful finding is how the gap was read. The first summary that reached our planning notes said about 24 real bugs. That number came from a first-pass script. Seven reports from the agents who then fixed the failures, reading them one at a time, labeled 81 of 815 as real bugs. The test-honesty problem applies to the analysis too: a sorted number is only as reliable as whatever did the sorting.
This extends two of our earlier findings. In Eight Review Passes on an AI That Reads Strangers' Email, a suite once passed over a real transaction bug because the in-memory fake did not enforce the real database's rule. In Why We Never Trust a Builder Model's Own Status Report, a green status report did not match an independent check. Here, the gap is in the test suite itself, at the scale of a whole application. Two companion studies look at the same application in the same week: Find the Bug Class, Not the Bug follows its database cutover through the bug log and a code census, and Working Is Not Organized measures how spread out its data access had become.