Are Your Tests Honest? One Suite Against a Fake Platform and the Real One

An AI-maintained test suite had 44 failing entries against a fake of our platform API and 1,230 against the real platform. A first-pass sort put most of the gap on the tests themselves, and the reported share of real bugs differed between the first pass and the fix agents' reports.

Core Finding:AI agents wrote and maintain the test suite for one application on our platform. Its tests that called the platform used an in-memory fake of the platform API, built from a pinned copy of the platform's operation catalog and schema. On October 21, 2026, against that fake, the suite had 17,317 passing tests and 44 failure or error entries. We then pointed the suite at the real platform core and a real local database. Against the real platform it had 16,368 passing tests and 1,230 failure or error entries. The first-pass assessment put most of the gap on the tests. An automatic first-pass sort labeled 1,067 of the 1,163 real-only failures it sorted as tests that relied on the fake, mostly because their test data could not exist on the real platform. It labeled 72 as gaps in the new test setup and 24, or 2.1%, as possible real bugs. Seven fix-agent reports on their own slices labeled 81 of 815 failures, or 9.9%, as real bugs. The two cohorts differ, one of the seven reports gave its supplied first-pass count rather than its own, and neither count has been independently audited. The day before, the CEO had hit bugs in normal use, including a credential document upload that did not work and a shift-create request rejected as invalid. According to the orchestrating agent's review, no failing test had pointed at either one. Data from Oct 21 2026, analyzed Oct 23 2026. This is a single-suite case study.

Against the fake

17,317 passed, 44 failing entries

43 failures and 1 error, 745 s on 8 workers

Against the real platform

16,368 passed, 1,230 failing entries

955 failures and 275 setup or teardown errors, 1,101 s on 8 workers

First-pass sort of 1,163 real-only failures

1,067 / 72 / 24

Test relied on the fake / test setup gap / possible real bug

Reported by fix agents (7 slices)

81 of 815

Labeled real bugs, 9.9%, versus 2.1% in the first pass, on different cohorts; one of the seven reports gave its supplied first-pass count

Empirical Findings Table

The suite had 44 failure or error entries against the fake and 1,230 against the real platform, where about 1,190 tests failed only there. The first-pass sort put most of the real-only failures on the tests. Seven fix-agent slice reports labeled real bugs at about 4.8 times the first-pass rate, on a different cohort.

Telemetry Metric	Measured Value	Context & Observability
Fake run	17,317 passed, 44 failing entries	43 failures and 1 error, with 39 skipped and 653 expected failures. 745 s wall time on 8 workers.
Real-platform run	16,368 passed, 1,230 failing entries	955 failures and 275 setup or teardown errors, with 39 skipped and 596 expected failures. A setup or teardown error is a separate entry and can accompany another outcome for the same test. 1,101 s wall time on 8 workers.
Pass share	99.7% vs 93.0%	Passed entries divided by passed, failed and error entries, against the fake and against the real platform.
Real-only failures	About 1,190	Tests that failed against the real platform and were not in the fake run's failure list. The sorting script counted 1,186; raw recounts from the run logs gave 1,181 and 1,195, so the exact unique-test count is unresolved. The script set aside 23 in files owned by other lanes and sorted the other 1,163, in 185 files.
First-pass sort	1,067 / 72 / 24	Of 1,163 failures: the test relied on the fake / the new test setup had a gap / possible real bug. 17 of the 24 possible bugs were in one test file.
Most common first error	127 of 185 files	Files whose first listed real-platform error was a permission denial. They held 787 of the 1,163 sorted failures. This is the first error per file, not 787 separately diagnosed permission defects.
Fix agents' slice reports	81 of 815	Real-bug labels in the seven slices whose reports gave a count for the original slice: 11, 13, 5, 16, 1, 14 and 21, or 9.9%. The report with 1 gave the initial supplied classification rather than its own; the other six give 80 of 699. The first pass labeled 24 of all 1,163 sorted failures, 2.1%, as real bugs, so the two rates cover different cohorts.
Wall time between runs	About 48% more wall time	1,101 s versus 745 s, an observed 47.8% difference between two runs on nearby code, not an isolated estimate of the real platform's overhead. In one measurement, a database cluster start of 7.3 s is shared, and the core takes about 3.7 s per worker, paid only by tests that call the platform.

Engineering Action Directives

Run the suite at least once against the real dependency and compare the two failure lists.

DON'T

Read a mostly green run against a fake as evidence that the system works.

Create test data through the same create operations the application uses.

DON'T

Let tests insert partial records into a fake that would never accept them in production.

Treat an automatic failure sort as a starting estimate, and recount after someone reads each failure.

DON'T

Report the first-pass count of real bugs as the number of real bugs.

Track known failures by name and compare each run against a base.

DON'T

Let hundreds of known failures become background noise that a new bug can hide in.

Methodology Boundaries & Limits

**One suite, one application, one day.** This is a case study of a single AI-maintained suite. It does not estimate how honest fake-backed suites are in general.
**The two runs were not on identical code.** The fake run finished a few hours before the real run, which used the final integration tree. The real run collected 18,198 tests and the fake run 18,053, 145 more, and their outcome entries differ by 180, so some differences may come from code changes rather than the dependency.
**The classifications are AI judgments that have not been audited.** The first-pass sort was a script that read the first error line. The re-sort came from the fix agents' own reports, and our charter does not accept self-reports as verification. Three slices gave no comparable count for their original failures, one of the seven comparable reports gave its supplied first-pass classification, the 9.9% and 2.1% rates cover different cohorts, and in the end-of-day snapshot 8 of the 10 lanes reported themselves blocked.
**The link between the CEO's bugs and the tests is the orchestrating agent's account.** We did not rerun the suite at the commits where those bugs shipped, and we did not confirm that any real-only failure was the cause of either one.
**The real-platform harness is not the whole system.** Identity records were loaded through a test mechanism, and the runs did not touch a deployed environment, a browser or production data. A pass against this harness still says less than a passing check on a deployed test environment.