MeshTen Lab

Our working thesis

v0.1, working draft, October 2026

Estimated tokens behind this research program

~14 billion tokens

~414 billion including cached context re-reads

Measured: ~1.7 billion new tokens (input plus output) and ~49.7 billion cached re-reads across Claude, Codex and Gemini in the 30 days to October 15, 2026. Estimate: that 30-day rate multiplied across the 242 days since February 15, 2026. Earlier months were likely lighter, so this is probably an upper-end estimate. Local model inference is not included.

Published by Sean Carr

Autonomous agents become reliable through the system around the model, not the model alone.

Models propose; plain code decides; a different model checks the work; and a claim counts only when the evidence came from outside the builder. This is our working design principle, not a guarantee that any model, reviewer, or gate is correct.

This is a living thesis about agentic workflows and autonomous software agents. It changes as studies land. Claims can be weakened or dropped when new evidence cuts against them. The articles below report observations; the claims connect those observations into ideas we can test.

We reviewed all 49 published Lab articles for this version. Most findings come from one organization's workloads, and many are tentative case studies. Several articles revisit the same incident or benchmark; links are not independent replications. Supporting evidence means the observation fits the claim, not that the full claim has been established.

Why the studies range so widely

The studies span memory, retrieval, security review, local inference, 3D assets and a game engine on purpose. We nested tests of the same claims inside fields that were unfamiliar to us, to see whether those claims held where we could not lean on prior expertise. If plain-code gates, cross-model checks and evidence from outside the builder only worked in familiar territory, that would weaken the thesis. These different settings let us ask whether a checker can miss the thing it claims to check, and whether an outside reference makes an output easier to judge. In the game engine study, a separate full walk of state exposed stale hashes that early tests missed. In the 3D asset study, agents built a coastal scene against a locked visual target, with independent scoring and no permission to change acceptance thresholds. Breadth is not independence; most studies come from one team's workloads.

H1 · Status: tentative

Self-report is not evidence

Our working claim is that independent, cross-model verification catches mistakes that a builder misses. A completion message is a lead to check against tests, artifacts, and observed actions. A different model can challenge the builder's assumptions, but its findings also need checking. The published cases support separating building from checking; they do not isolate the benefit of changing model family from the benefit of a fresh review.

Supporting articles

Why We Never Trust a Builder Model's Own Status Report
Tentative incident: claimed success diverged from test output and independent review.
Article measurement
Independent review caught 2 P1 security defects in batch 1 and 2 P1s in batch 2.
Cross-Model Review: What a Second Model Catches
Fresh review found four additional high-severity findings; this overlaps the builder case and is not a model ranking.
Article measurement
an independent deep-read audit revised the platform security score down by 6 points after discovering 4 unindexed P1 findings.
Eight Review Passes on an AI That Reads Strangers' Email
A second model raised 17 findings after the first read; one service and scoped reviews only.
Article measurement
5 → 3 → 4 → 1 → 3 → 1 → 0 across seven passes. The count did not fall steadily.
A Speed Fix Introduced a Potential Tenant-Isolation Defect, and Independent Review Blocked It
A passing builder suite missed a potential defect blocked by review; no live exploit or same-family comparison was run.
Article measurement
The builder's tests passed but missed the potential tenant-isolation defect.
Model Bake-Off for Code-Built 3D Assets
A locked visual target and independent scoring made outputs checkable; one scene cannot isolate the effect of independent checking.
Article measurement
Zero self-judging allowed; implementing agents prohibited from altering visual acceptance thresholds.
A Fine-Tuned Code Author Failed Its Own Pre-Registered Test
Independent corpus inspection found 25 defective teaching records; this is one audit and tuning project.
Article measurement
25 records taught a broken idempotency-key derivation as the correct pattern
An Agent Can Understand the Request and Still Fail to Act
Observable follow-through passed 0/5 graded baseline cases, despite the ability to produce helpful replies; delivery was simulated.
Article measurement
Autonomous follow-through passed 0/5 graded cases, with the execution error in that category.

Evidence against / limits

Review severity is a judgment, not a measured production outcome. The cases are selected and sometimes overlap. A clean review of a supplied diff covers that scope only. We lack a controlled comparison of self-review, fresh review by the same family, and review by a different family with matched effort and independently established defects.

Eight Review Passes on an AI That Reads Strangers' Email
Six findings followed earlier fixes; three of those six came from the orchestrator's instructions. Reviewers and orchestrators were fallible too.
Article measurement
3 of those 6
A Second AI Audit Found Critical Findings, Then Verification Lowered One
A later audit found more urgent concerns, but verification lowered one P0 claim to P2. Verification used the later auditor's model family.
Article measurement
Its downgraded claim overlooked protections relevant to the asserted consequence.
Fewer Findings Did Not Mean the AI Fix Loop Was Finished
Only one of five sequences reached reviewer PASS; four stopped by ruling. Falling counts did not establish completion.
Article measurement
The circuit breaker appears in four of these five sequences after the third saved review, following two repair opportunities.
Patching on Top: An AI Fix Series That Was Merged, Then Reverted
A repair series was merged then reverted. Integration and local repairs did not establish a durable outcome.
Article measurement
A repair series labeled rounds 1 through 5 was merged and then explicitly reverted on October 9, 2026.

What would change our mind

On fresh tasks with known defects and equal review effort, self-review repeatedly catches as many real defects as independent review, with no increase in missed defects or false alarms. If a fresh reviewer helps but changing model family does not, we would drop the cross-model advantage from this claim.

How we will test this next

EXP-3: planned, pre-registered

The cases do not isolate a second model family from any fresh review pass. EXP-3 will use known-defect tasks with three equal-effort arms: self-review, a fresh same-family pass, and a different-family pass. A different-family gain of at least 10 percentage points over both other arms, with the planned statistical checks and few extra false findings, would upgrade H1; missing those conditions would weaken it. If the upper bound on its gain over a fresh same-family pass is below 10 points, we would drop the claimed family advantage.

Pre-registration recorded in commit 01fcdfdd. Study not yet run.

H2 · Status: contested

Plain code beats models at control decisions

We expect plain code to handle bounded routing, query planning, gating, recurring checks, and event recording better when the rule and input can be stated explicitly. A model can propose a plan or interpret an exception; code can apply the agreed rule and preserve what happened. The routing results support a narrow efficiency claim. They do not establish that general planning, unfamiliar retrieval, or policy judgment should become fixed rules.

Supporting articles

Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
On reused questions, routing removed 152,239 planner tokens with no detected accuracy difference; this concerns query planning only.
Article measurement
deterministic routing eliminated 152,239 planner tokens to zero
Record What the Agent Did Before Asking a Model to Summarize It
A raw event lookup took 0.2 seconds versus 16.6 for a model summary, but queries and outputs differed; no equivalent-answer speedup follows.
Article measurement
In the demonstration, a raw status query took 0.2 seconds. A summarized project query took 16.6 seconds and used one model call.
Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
Hook-only skip-told runs completed 56/58 and 29/29 items versus ungated 39/65 and 21/29; one repository and specific checks.
Article measurement
Hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%). Ungated skip-told runs completed 39/65 and 21/29 items respectively.
The Hidden Infrastructure Tax in Multi-Agent Coding: An Estimated Replay of 39,526 Calls and an RTX 5090 Lab
Hardened regex and a local two-stage cascade matched 100/100 on a 100-case synthetic PHI set (rules tuned in-sample) at 0.03 to 0.04 ms negative p50, and hybrid repo-filtered extraction reached 99.46% in-sample Micro F1 on 50 briefs. The CEO approved the local cascade on 2026-10-15 and it is now the default PHI admission check, with Claude Haiku as the fallback; an out-of-sample test is still planned.
Article measurement
Arm 4 Hybrid (Arm 0+ + repo-filtered qwen3:14b) matched 367/370 items (99.73% precision, 99.19% recall, Macro F1 99.40%, McNemar p = 2.65e-22 vs baseline Arm 0 at 59.98% F1 and p = 3.73e-09 vs Arm 0+ at 95.35% F1).

Evidence against / limits

A rule can encode the wrong policy, accept the wrong edit, read stale input, or never run. Faster recording does not verify authored handoff text or remote delivery. Script deployment has not been compared with a measured model baseline. The broad claim is contested because fixed retrieval rules failed badly on unfamiliar questions.

From Recurring AI Checks to Scripts: What Actually Ran?
Tentative rollout: two jobs never ran and active alarms awaited delivery. No measured model baseline or input-freshness comparison establishes superiority.
Article measurement
Two of four selected jobs were reported never to have run; active checks produced clean results and pending notifications.
The AI Proposes, Plain Code Decides: Who Decides What the Rule Means?
Four of 17 findings fit a retrospective policy grouping; one was intended behavior. Code still needs an agreed policy.
Article measurement
It does not describe four confirmed instances of a model escaping its intended authority.
A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
Compiled retrieval rules scored 0.45 out of sample versus 0.975 for agentic search. Fixed rules lost on unfamiliar retrieval.
Article measurement
Out of sample: E recall@5 0.45 and recall@10 0.45; hybrid F recall@5 0.45 on both runs vs the agent's 0.975 mean.
Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
Generic change checks were too weak: six incomplete runs passed. A gate does not establish that the right work happened.
Article measurement
Slop reduction and required items completed on skip-told briefs under mechanical gating.
Article qualification: Six incomplete runs passed because generic file-change checks accepted the wrong edit.
Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations
Rules written after seeing the questions caught six ambiguous cases but falsely flagged 19/158 correct answers (12%). This is in-sample control evidence, not a validated rule.
Article measurement
Rules written after inspecting the benchmark flagged all 6 ambiguous questions and falsely flagged 19/158 H-correct questions (12%). This was an in-sample result.
Our Determinism Hash Was Wrong on 74.3% of Steps
The deterministic checker itself was wrong on 74.3% of steps. Code needs a genuinely separate reference check too.
Article measurement
Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.

What would change our mind

On independently written, unseen control tasks with identical inputs and permissions, a model planner repeatedly improves correct decisions enough to outweigh its measured delay and token use, while fixed rules miss required actions or need frequent manual exceptions. That would narrow which decisions we assign to code.

How we will test this next

EXP-4: planned, pre-registered

Fixed retrieval rules fell to 0.45 out of sample in the published code-retrieval study, so familiar-task results leave H2 contested. EXP-4 will compare a plain-code router with a learned fallback against a model router, scored only on unseen control tasks and counting manual exceptions needed. A code-router accuracy gain of at least 5 percentage points with an uncertainty interval above zero and at most 5 exceptions per 100 tasks would upgrade the bounded claim; missing those conditions would weaken it. A clear model-router gain of at least 5 points, or more than 10 code-router exceptions per 100 tasks, would lead us to drop the claimed advantage in this setting.

A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample

Pre-registration recorded in commit 01fcdfdd. Study not yet run.

H3 · Status: tentative

What the agent sees beats how it is asked

Our hypothesis is that context, memory, retrieval architecture, and mechanical gates move results more than prompt wording alone. Useful source spans and explicit checks can give an agent facts and feedback that instructions cannot supply. Yet prompts can have large effects, and extra context can mislead. Existing studies compare different interventions on different tasks, so the proposed ranking remains tentative until a direct comparison holds source access and effort constant.

Supporting articles

Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
Catalog injection reduced observed helper duplication; zero events in these small samples is not a general zero rate.
Article measurement
Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs.
83% Fewer Tokens, Matched Accuracy on 100 Questions: Semantic Memory vs Pasting the Whole History
Structured memory and full context each scored 88/100, using 19.5k versus 113.6k tokens; the sampled tie is not equivalence.
Article measurement
Structured event memory matched full-context accuracy on this 100-question set (88/100) consuming 19,500 tokens (17.2% of Arm A, an 82.8% reduction).
Half the Tokens: Code-Search Hints Cut Agent Token Use 51% on a 30-Question Test
Hints cut tokens by 51% on 30 questions; holdout recall was 0.94 to 0.98. Pure indexing alone had poor recall.
Article measurement
Arm H consumed 57,438 tokens vs 118,105 for agentic grep on CODERET-NIGHT; 40,737 vs 91,844 on CODERET-E3.
Decoupled Context Pipeline Invariants: Verifying 205 Prompts Across 11 Switch States
Frozen outputs preserved 2,255 prompt states through a refactor; this measures pipeline parity, not better task answers.
Article measurement
Verification passed across all 205 prompts and 11 switch states with zero differences and 40 live hook captures matching identically.
Deconstructing Agent Retrieval Latency: Process Pools, Loopback Daemons, and Pipelining
Systems changes cut warm retrieval from 23.3 to 9.9 seconds with 30/30 matching outputs; this is latency evidence, not a prompt comparison.
Article measurement
All 30 tested questions produced byte-identical stdout and prompt matching with zero prompt misses.
Eliminating Quadratic Bottlenecks in Multi-Session Thread Reconstruction
Linear parsing and caching cut thread reconstruction from 19 to 1.5 seconds with matching output; the model still limits answer latency.
Article measurement
Table reconstruction across 1,293 threads dropped from 19.0s to 1.5s (12.6x speedup) with byte-identical output.
Sub-Second Agent Memory Recall via Asynchronous Speculative Prefetch
A cache hit returned in 0.29 seconds after background work; this measures perceived waiting, not full computation removed.
Article measurement
Cache-served responses returned in 0.29s wall time (annotated [prefetched 11s ago]).

Evidence against / limits

There is no common experiment establishing that architecture generally matters more than prompting. Memory comparisons sometimes grant different source access; extra tokens and ingestion work belong in the result. A sampled score tie is not equivalence. Stale comments and confident hints can preserve the wrong answer, and a prompt checklist produced a large gain in its own study.

A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
A compact rules-based context pipeline lost recall on new questions; lower token use was not enough.
Article measurement
F used 11,169 tokens per question out of sample, 12.7% of the agent's 88,075.
83% Fewer Tokens, Matched Accuracy on 100 Questions: Semantic Memory vs Pasting the Whole History
The planned token target was missed; structured context was less efficient than predicted and requires ingestion work.
Article measurement
Clause 2 (Arm S tokens <= 15% of full context) was REFUTED at 17.2%.
Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
The 46/50 versus 35/50 result includes unequal source access; another token measurement used 12% more median tokens.
Article measurement
The arms had different source access, so observed gains include corpus availability.
Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations
Memory scored 43/50 versus grep's 47/50 on one slice (McNemar p = 0.22), and model triage caught 0/6 ambiguous cases. This synthesis reuses earlier studies. The article's later analysis also attributed 28.6% of 42 misses to hint anchoring; its proposed recovery was estimated, not tested.
Article measurement
A lightweight model caught 0 of 6 ambiguous questions.
Article qualification: A confident incorrect memory hint is worse than no hint.
Auto-Context Prompt Injection: Measuring Code Slop and Helper Duplication Across 36 Controlled Sessions
Tentative trend: mechanical slop fell 6 to 1 (exact McNemar p = 0.062), the blind judge also found no significant difference (6 ON versus 9 OFF, paired McNemar p = 0.51). Both arms completed 17/18 tasks; context added 6% to 29% tokens.
Article measurement
Task completion rate: 17/18 ON sessions succeeded vs 17/18 OFF sessions.
Prompting Is Necessary, Not Sufficient: What Moved the Numbers in Six of Our Studies
Prompt checklists raised completion from 60% to 97%. This synthesis has no prompt-only versus tooling head-to-head and reuses studies above.
Article measurement
Showing checklist up front raised skip-told item completion on Claude from 60% (39/65) to 97% (35/36) with 0 send-backs
The Local Coder Ceiling: 100% on Single Functions, 0% on Multi-File Bug Fixes Under Default Config
A surgical prompt made one task pass 3/3 but broke another 3/3 to 0/3; prompts can move individual results substantially.
Article measurement
3/3 under the surgical system prompt (which broke task 002 from 3/3 to 0/3).
A Stale Safety Comment Parked a 10 ms Win for Three Days
Four of five records of a default were wrong; a stale instruction delayed a measured gain. Context must be checked for freshness.
Article measurement
Records of the default for this one switch: five; four were wrong.
A Fine-Tuned Code Author Failed Its Own Pre-Registered Test
Adding specifics did not repair the tuned model's abstention behavior: it flagged only 2/13 unknowns. More context is not a universal remedy.
Article measurement
Arm C (tuned, specifics in context): halt-and-flag 15% (2/13), average rubric 0.74, 9 hallucinations.

What would change our mind

In a pre-registered comparison on fresh tasks, with equal source access and total effort, prompt-only changes repeatedly match or exceed the accuracy and efficiency gains from retrieval, memory, and gates. Repeated losses from structured context would also weaken this claim.

How we will test this next

EXP-5: planned, pre-registered

Past studies compared different interventions on different tasks, leaving the proposed ranking tentative. EXP-5 will use identical fresh tasks, the same source access and total token budget, and vary only a prompt checklist versus retrieval architecture against a shared baseline. Retrieval beating the checklist by at least 10 percentage points with the planned statistical checks and no more than 10% extra tokens would upgrade H3; missing those conditions would weaken it. A clear checklist win of at least 10 points, or an upper bound below a 10-point retrieval gain, would lead us to drop the proposed large architecture advantage here.

Pre-registration recorded in commit 01fcdfdd. Study not yet run.

H4 · Status: tentative

The measuring setup is a leading source of error

We suspect harnesses, confinement, averages, and tuning on familiar questions are leading sources of error in agent evaluation. The setup can manufacture an apparent success, hide a failure, or blame a model for a broken tool. We therefore inspect intermediate artifacts, keep important failure categories separate, and test whether checkers can reject deliberately wrong behavior. Published examples establish that these errors occur; they do not yet rank them against all other causes.

Supporting articles

An Agent Can Understand the Request and Still Fail to Act
Category results moved in opposite directions; timeouts were excluded from conditional pass rates, so a single average would hide important distinctions.
Article measurement
Resource handling moved in the opposite direction: its reported repeat-mean pass rate was 26.7% for baseline and 0.0% with classification.
A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
Rules tuned on 50 questions scored 0.94 there and 0.45 on 20 independently authored questions.
Article measurement
Phase 1: compiled arm E in-sample recall@5 0.94
Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations
Review of 200 questions found ten bad answers and six ambiguous questions; benchmark error was part of the observed ceiling.
Article measurement
Pareto analysis of 200 holdout questions identified 10 bad gold answers and 6 ambiguous questions
When the Edit Harness Makes a Valid Model Patch Fail
A splice bug affected four of eight patch tasks. A clean post-fix reference still scored only 3/30, so fixing measurement did not fix model logic.
Article measurement
the whole-function configuration recorded 3/30, with the affected tasks now reaching logic failures rather than harness-induced syntax errors.
The Confinement Tax: Measuring Scope Leakage in Agentic Benchmark Harnesses
The sealed harness audited 1,934 calls and denied seven reads. The earlier unsealed run cannot establish a numeric confinement penalty.
Article measurement
All 1,934 tool calls audited; all 7 outside-root read attempts denied.
A Destructive Tool Choice Should Not Disappear Inside an Average
Two models each made destructive requests in 2/2 simulated trials; a third recorded 0/2. The tiny sample is not a deployment rate or ranking.
Article measurement
destructive command choices in 2/2 bait trials for one local coding model and 2/2 for another. A third local model recorded 0/2.
Our Determinism Hash Was Wrong on 74.3% of Steps
A cached checker gave wrong hashes on 22,290/30,000 steps; early golden tests missed the late failure.
Article measurement
Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.
39.7 Million Random Pairs Found Zero Bugs. 144 Targeted Pairs Found Two.
Random fuzz missed a case that a targeted table found twice; large counts did not establish coverage.
Article measurement
Random-float fuzz: 39.7 million pairs, 0 mismatches. Targeted class table: 144 pairs, exactly 2 mismatches, both signed-zero ties.
Our Culling Number Was Right and Still Misleading
Correct kernel timing omitted required maintenance costing 47.9 times as much on a live scene; system timings used a transcription.
Article measurement
Maintenance-to-cull ratio 47.9x; total-to-cull 49.0x.
Fixed Cost vs Pixel Cost: A Resolution Ladder With R^2 = 0.99993
A controlled ladder found a 48% reduction rather than the earlier roughly 20% spot estimate. The proposed cause of the discrepancy was not established.
Article measurement
4K to 1080p: a 48% reduction.
Frame Budget at 120 FPS: What We Cut and What It Bought
Capture readback dominated a headless total and was absent in windowed execution; measurement paths answered different questions.
Article measurement
Headless capture telemetry initially reported 16.6 ms total due to a 14.1 ms GPU to host readback artifact; windowed gameplay bypasses readback entirely.
Automated 30-Minute Engine Soak Testing: Measuring Frame Latency and Memory Drift
Longer automated runs exposed a 14 MB/minute leak in an early baseline. A later flat 30-minute run cannot rule out all leaks.
Article measurement
In early baseline tests, an unclipped texture streamer caused private bytes to expand at 14 MB/minute; automated soak tests caught the leak before integration.

Evidence against / limits

The cases were chosen because something went wrong, so they cannot estimate the prevalence of measurement error. A confined run establishes its own boundary, not a numeric penalty relative to an unconfined run. Targeted checks still cover only their chosen cases. Correct measurements can support a narrow real improvement, and model reasoning failures remain after tool defects are fixed.

Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
Among 40 classified errors, reasoning was the largest category (15), ahead of bad ground truth (9). This cuts against treating measurement as the dominant error source.
Article measurement
15 reasoning errors (37.5%), 10 retrieval misses (25.0%), 9 flawed ground-truth benchmarks (22.5%), 5 wrong-thread attributions (12.5%), and 1 judging variance (2.5%).
Five Optimizations That Measured Flat
Five correct optimizations measured flat or near-flat. Checked code and plausible attribution did not establish a useful end-to-end gain; small samples limit tiny-effect claims.
Article measurement
Five correct optimizations that measured flat on whole-frame time
Reaching 120 FPS at Native 4K: The 3.5 ms That Wasn't Where We Looked
Several plausible fixes measured neutral before the actual cost owner was isolated. Instrumentation narrowed the search but did not name the cause.
Article measurement
Candidate fixes in sequence: grid pre-filter neutral; static/dynamic partition neutral; flat cache 8%; supplied bounds zero; a static-prop premise falsified.
Asset Decode Was 83% of Boot. Parallelizing Textures Bought 5.6%.
The first fix moved its phase only 5.6%; the after-fix boot comparison was inadmissible. A correct phase attribution did not establish the expected overall win.
Article measurement
First version (texture decode only, byte-identical): -5.6%. Model loaders: about 66% of the decode phase, not yet parallelized.
The Hidden Infrastructure Tax in Multi-Agent Coding: An Estimated Replay of 39,526 Calls and an RTX 5090 Lab
Roughly 99% of the 121.82-hour summed concurrent call-time reduction came from software fixes and CPU worker sharding rather than the GPU, and the 100.0% PHI and 99.46% checklist rules were refined in-sample on the evaluation corpora.
Article measurement
roughly 99% of the time reduction comes from software and algorithmic fixes
Article qualification: prospective out-of-sample holdout testing is required to establish unseen generalization.

What would change our mind

A prospective audit of a broad, randomly sampled set of agent studies finds that independently rebuilding harnesses, sealing access, using fresh questions, and reporting failure categories rarely changes conclusions, while model errors explain most discrepancies. We would then drop 'leading' and retain a narrower measurement warning.

How we will test this next

EXP-6: planned, pre-registered

The cases were selected because something went wrong, so they cannot tell us how often measurement errors matter. EXP-6 will take a prospective random sample of agent studies, independently rebuild each harness and seal access, then count how often conclusions change because of measurement defects versus model reasoning failures. With 40 complete audits, measurement changes in at least 30% and a clear excess over reasoning failures would upgrade H4; missing those conditions would weaken it. If the upper bound on measurement changes is below 20%, or reasoning failures clearly exceed them by at least 10 percentage points, we would drop 'leading' from the claim.

Pre-registration recorded in commit 01fcdfdd. Study not yet run.

Open questions

When does parallel orchestration improve completed work after accounting for memory, cleanup, and coordination? We keep this outside H1 to H4. One concurrency incident series and separate inference or pipeline profiles do not establish a general resource-allocation rule, so we do not add H5 in this version.
How much of independent review's value comes from a fresh context, a different model family, additional effort, or a better specification?
Can we compare prompt wording, source access, retrieval structure, and gates in one study with matched effort and fresh tasks?

Article mapping

Every article has a disposition. Open an article to inspect its method and limits. Not used means it does not directly test these claims.

View the complete evidence map

H1 · SUPPORTS
Why We Never Trust a Builder Model's Own Status Report
Tentative incident: claimed success diverged from test output and independent review.
Independent review caught 2 P1 security defects in batch 1 and 2 P1s in batch 2.
H1 · SUPPORTS
Cross-Model Review: What a Second Model Catches
Fresh review found four additional high-severity findings; this overlaps the builder case and is not a model ranking.
an independent deep-read audit revised the platform security score down by 6 points after discovering 4 unindexed P1 findings.
H1 · SUPPORTS
Eight Review Passes on an AI That Reads Strangers' Email
A second model raised 17 findings after the first read; one service and scoped reviews only.
5 → 3 → 4 → 1 → 3 → 1 → 0 across seven passes. The count did not fall steadily.
H1 · COMPLICATES
Eight Review Passes on an AI That Reads Strangers' Email
Six findings followed earlier fixes; three of those six came from the orchestrator's instructions. Reviewers and orchestrators were fallible too.
3 of those 6
H1 · SUPPORTS
A Speed Fix Introduced a Potential Tenant-Isolation Defect, and Independent Review Blocked It
A passing builder suite missed a potential defect blocked by review; no live exploit or same-family comparison was run.
The builder's tests passed but missed the potential tenant-isolation defect.
H1 · COMPLICATES
A Second AI Audit Found Critical Findings, Then Verification Lowered One
A later audit found more urgent concerns, but verification lowered one P0 claim to P2. Verification used the later auditor's model family.
Its downgraded claim overlooked protections relevant to the asserted consequence.
H1 · COMPLICATES
Fewer Findings Did Not Mean the AI Fix Loop Was Finished
Only one of five sequences reached reviewer PASS; four stopped by ruling. Falling counts did not establish completion.
The circuit breaker appears in four of these five sequences after the third saved review, following two repair opportunities.
H1 · COMPLICATES
Patching on Top: An AI Fix Series That Was Merged, Then Reverted
A repair series was merged then reverted. Integration and local repairs did not establish a durable outcome.
A repair series labeled rounds 1 through 5 was merged and then explicitly reverted on October 9, 2026.
H1 · SUPPORTS
Model Bake-Off for Code-Built 3D Assets
A locked visual target and independent scoring made outputs checkable; one scene cannot isolate the effect of independent checking.
Zero self-judging allowed; implementing agents prohibited from altering visual acceptance thresholds.
H1 · SUPPORTS
A Fine-Tuned Code Author Failed Its Own Pre-Registered Test
Independent corpus inspection found 25 defective teaching records; this is one audit and tuning project.
25 records taught a broken idempotency-key derivation as the correct pattern
H1 · SUPPORTS
An Agent Can Understand the Request and Still Fail to Act
Observable follow-through passed 0/5 graded baseline cases, despite the ability to produce helpful replies; delivery was simulated.
Autonomous follow-through passed 0/5 graded cases, with the execution error in that category.
H4 · SUPPORTS
An Agent Can Understand the Request and Still Fail to Act
Category results moved in opposite directions; timeouts were excluded from conditional pass rates, so a single average would hide important distinctions.
Resource handling moved in the opposite direction: its reported repeat-mean pass rate was 26.7% for baseline and 0.0% with classification.
H2 · SUPPORTS
Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
On reused questions, routing removed 152,239 planner tokens with no detected accuracy difference; this concerns query planning only.
deterministic routing eliminated 152,239 planner tokens to zero
H2 · SUPPORTS
Record What the Agent Did Before Asking a Model to Summarize It
A raw event lookup took 0.2 seconds versus 16.6 for a model summary, but queries and outputs differed; no equivalent-answer speedup follows.
In the demonstration, a raw status query took 0.2 seconds. A summarized project query took 16.6 seconds and used one model call.
H2 · COMPLICATES
From Recurring AI Checks to Scripts: What Actually Ran?
Tentative rollout: two jobs never ran and active alarms awaited delivery. No measured model baseline or input-freshness comparison establishes superiority.
Two of four selected jobs were reported never to have run; active checks produced clean results and pending notifications.
H2 · COMPLICATES
The AI Proposes, Plain Code Decides: Who Decides What the Rule Means?
Four of 17 findings fit a retrospective policy grouping; one was intended behavior. Code still needs an agreed policy.
It does not describe four confirmed instances of a model escaping its intended authority.
H2 · COMPLICATES
A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
Compiled retrieval rules scored 0.45 out of sample versus 0.975 for agentic search. Fixed rules lost on unfamiliar retrieval.
Out of sample: E recall@5 0.45 and recall@10 0.45; hybrid F recall@5 0.45 on both runs vs the agent's 0.975 mean.
H3 · COMPLICATES
A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
A compact rules-based context pipeline lost recall on new questions; lower token use was not enough.
F used 11,169 tokens per question out of sample, 12.7% of the agent's 88,075.
H4 · SUPPORTS
A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
Rules tuned on 50 questions scored 0.94 there and 0.45 on 20 independently authored questions.
Phase 1: compiled arm E in-sample recall@5 0.94
H2 · SUPPORTS
Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
Hook-only skip-told runs completed 56/58 and 29/29 items versus ungated 39/65 and 21/29; one repository and specific checks.
Hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%). Ungated skip-told runs completed 39/65 and 21/29 items respectively.
H2 · COMPLICATES
Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
Generic change checks were too weak: six incomplete runs passed. A gate does not establish that the right work happened.
Slop reduction and required items completed on skip-told briefs under mechanical gating.
Article qualification: Six incomplete runs passed because generic file-change checks accepted the wrong edit.
H3 · SUPPORTS
Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
Catalog injection reduced observed helper duplication; zero events in these small samples is not a general zero rate.
Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs.
H3 · SUPPORTS
83% Fewer Tokens, Matched Accuracy on 100 Questions: Semantic Memory vs Pasting the Whole History
Structured memory and full context each scored 88/100, using 19.5k versus 113.6k tokens; the sampled tie is not equivalence.
Structured event memory matched full-context accuracy on this 100-question set (88/100) consuming 19,500 tokens (17.2% of Arm A, an 82.8% reduction).
H3 · COMPLICATES
83% Fewer Tokens, Matched Accuracy on 100 Questions: Semantic Memory vs Pasting the Whole History
The planned token target was missed; structured context was less efficient than predicted and requires ingestion work.
Clause 2 (Arm S tokens <= 15% of full context) was REFUTED at 17.2%.
H3 · COMPLICATES
Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
The 46/50 versus 35/50 result includes unequal source access; another token measurement used 12% more median tokens.
The arms had different source access, so observed gains include corpus availability.
H4 · SUPPORTS
Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations
Review of 200 questions found ten bad answers and six ambiguous questions; benchmark error was part of the observed ceiling.
Pareto analysis of 200 holdout questions identified 10 bad gold answers and 6 ambiguous questions
H2 · COMPLICATES
Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations
Rules written after seeing the questions caught six ambiguous cases but falsely flagged 19/158 correct answers (12%). This is in-sample control evidence, not a validated rule.
Rules written after inspecting the benchmark flagged all 6 ambiguous questions and falsely flagged 19/158 H-correct questions (12%). This was an in-sample result.
H3 · COMPLICATES
Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations
Memory scored 43/50 versus grep's 47/50 on one slice (McNemar p = 0.22), and model triage caught 0/6 ambiguous cases. This synthesis reuses earlier studies. The article's later analysis also attributed 28.6% of 42 misses to hint anchoring; its proposed recovery was estimated, not tested.
A lightweight model caught 0 of 6 ambiguous questions.
Article qualification: A confident incorrect memory hint is worse than no hint.
H4 · COMPLICATES
Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
Among 40 classified errors, reasoning was the largest category (15), ahead of bad ground truth (9). This cuts against treating measurement as the dominant error source.
15 reasoning errors (37.5%), 10 retrieval misses (25.0%), 9 flawed ground-truth benchmarks (22.5%), 5 wrong-thread attributions (12.5%), and 1 judging variance (2.5%).
H3 · SUPPORTS
Half the Tokens: Code-Search Hints Cut Agent Token Use 51% on a 30-Question Test
Hints cut tokens by 51% on 30 questions; holdout recall was 0.94 to 0.98. Pure indexing alone had poor recall.
Arm H consumed 57,438 tokens vs 118,105 for agentic grep on CODERET-NIGHT; 40,737 vs 91,844 on CODERET-E3.
H3 · COMPLICATES
Auto-Context Prompt Injection: Measuring Code Slop and Helper Duplication Across 36 Controlled Sessions
Tentative trend: mechanical slop fell 6 to 1 (exact McNemar p = 0.062), the blind judge also found no significant difference (6 ON versus 9 OFF, paired McNemar p = 0.51). Both arms completed 17/18 tasks; context added 6% to 29% tokens.
Task completion rate: 17/18 ON sessions succeeded vs 17/18 OFF sessions.
H3 · COMPLICATES
Prompting Is Necessary, Not Sufficient: What Moved the Numbers in Six of Our Studies
Prompt checklists raised completion from 60% to 97%. This synthesis has no prompt-only versus tooling head-to-head and reuses studies above.
Showing checklist up front raised skip-told item completion on Claude from 60% (39/65) to 97% (35/36) with 0 send-backs
H3 · COMPLICATES
The Local Coder Ceiling: 100% on Single Functions, 0% on Multi-File Bug Fixes Under Default Config
A surgical prompt made one task pass 3/3 but broke another 3/3 to 0/3; prompts can move individual results substantially.
3/3 under the surgical system prompt (which broke task 002 from 3/3 to 0/3).
H3 · COMPLICATES
A Stale Safety Comment Parked a 10 ms Win for Three Days
Four of five records of a default were wrong; a stale instruction delayed a measured gain. Context must be checked for freshness.
Records of the default for this one switch: five; four were wrong.
H3 · SUPPORTS
Decoupled Context Pipeline Invariants: Verifying 205 Prompts Across 11 Switch States
Frozen outputs preserved 2,255 prompt states through a refactor; this measures pipeline parity, not better task answers.
Verification passed across all 205 prompts and 11 switch states with zero differences and 40 live hook captures matching identically.
H3 · SUPPORTS
Deconstructing Agent Retrieval Latency: Process Pools, Loopback Daemons, and Pipelining
Systems changes cut warm retrieval from 23.3 to 9.9 seconds with 30/30 matching outputs; this is latency evidence, not a prompt comparison.
All 30 tested questions produced byte-identical stdout and prompt matching with zero prompt misses.
H3 · SUPPORTS
Eliminating Quadratic Bottlenecks in Multi-Session Thread Reconstruction
Linear parsing and caching cut thread reconstruction from 19 to 1.5 seconds with matching output; the model still limits answer latency.
Table reconstruction across 1,293 threads dropped from 19.0s to 1.5s (12.6x speedup) with byte-identical output.
H3 · SUPPORTS
Sub-Second Agent Memory Recall via Asynchronous Speculative Prefetch
A cache hit returned in 0.29 seconds after background work; this measures perceived waiting, not full computation removed.
Cache-served responses returned in 0.29s wall time (annotated [prefetched 11s ago]).
H3 · COMPLICATES
A Fine-Tuned Code Author Failed Its Own Pre-Registered Test
Adding specifics did not repair the tuned model's abstention behavior: it flagged only 2/13 unknowns. More context is not a universal remedy.
Arm C (tuned, specifics in context): halt-and-flag 15% (2/13), average rubric 0.74, 9 hallucinations.
H4 · SUPPORTS
When the Edit Harness Makes a Valid Model Patch Fail
A splice bug affected four of eight patch tasks. A clean post-fix reference still scored only 3/30, so fixing measurement did not fix model logic.
the whole-function configuration recorded 3/30, with the affected tasks now reaching logic failures rather than harness-induced syntax errors.
H4 · SUPPORTS
The Confinement Tax: Measuring Scope Leakage in Agentic Benchmark Harnesses
The sealed harness audited 1,934 calls and denied seven reads. The earlier unsealed run cannot establish a numeric confinement penalty.
All 1,934 tool calls audited; all 7 outside-root read attempts denied.
H4 · SUPPORTS
A Destructive Tool Choice Should Not Disappear Inside an Average
Two models each made destructive requests in 2/2 simulated trials; a third recorded 0/2. The tiny sample is not a deployment rate or ranking.
destructive command choices in 2/2 bait trials for one local coding model and 2/2 for another. A third local model recorded 0/2.
H4 · SUPPORTS
Our Determinism Hash Was Wrong on 74.3% of Steps
A cached checker gave wrong hashes on 22,290/30,000 steps; early golden tests missed the late failure.
Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.
H2 · COMPLICATES
Our Determinism Hash Was Wrong on 74.3% of Steps
The deterministic checker itself was wrong on 74.3% of steps. Code needs a genuinely separate reference check too.
Before fix: 22,290 of 30,000 steps had a wrong state hash (74.3%), the first at step 7,710.
H4 · SUPPORTS
39.7 Million Random Pairs Found Zero Bugs. 144 Targeted Pairs Found Two.
Random fuzz missed a case that a targeted table found twice; large counts did not establish coverage.
Random-float fuzz: 39.7 million pairs, 0 mismatches. Targeted class table: 144 pairs, exactly 2 mismatches, both signed-zero ties.
H4 · SUPPORTS
Our Culling Number Was Right and Still Misleading
Correct kernel timing omitted required maintenance costing 47.9 times as much on a live scene; system timings used a transcription.
Maintenance-to-cull ratio 47.9x; total-to-cull 49.0x.
H4 · COMPLICATES
Five Optimizations That Measured Flat
Five correct optimizations measured flat or near-flat. Checked code and plausible attribution did not establish a useful end-to-end gain; small samples limit tiny-effect claims.
Five correct optimizations that measured flat on whole-frame time
H4 · COMPLICATES
Reaching 120 FPS at Native 4K: The 3.5 ms That Wasn't Where We Looked
Several plausible fixes measured neutral before the actual cost owner was isolated. Instrumentation narrowed the search but did not name the cause.
Candidate fixes in sequence: grid pre-filter neutral; static/dynamic partition neutral; flat cache 8%; supplied bounds zero; a static-prop premise falsified.
H4 · COMPLICATES
Asset Decode Was 83% of Boot. Parallelizing Textures Bought 5.6%.
The first fix moved its phase only 5.6%; the after-fix boot comparison was inadmissible. A correct phase attribution did not establish the expected overall win.
First version (texture decode only, byte-identical): -5.6%. Model loaders: about 66% of the decode phase, not yet parallelized.
H4 · SUPPORTS
Fixed Cost vs Pixel Cost: A Resolution Ladder With R^2 = 0.99993
A controlled ladder found a 48% reduction rather than the earlier roughly 20% spot estimate. The proposed cause of the discrepancy was not established.
4K to 1080p: a 48% reduction.
H4 · SUPPORTS
Frame Budget at 120 FPS: What We Cut and What It Bought
Capture readback dominated a headless total and was absent in windowed execution; measurement paths answered different questions.
Headless capture telemetry initially reported 16.6 ms total due to a 14.1 ms GPU to host readback artifact; windowed gameplay bypasses readback entirely.
H4 · SUPPORTS
Automated 30-Minute Engine Soak Testing: Measuring Frame Latency and Memory Drift
Longer automated runs exposed a 14 MB/minute leak in an early baseline. A later flat 30-minute run cannot rule out all leaks.
In early baseline tests, an unclipped texture streamer caused private bytes to expand at 14 MB/minute; automated soak tests caught the leak before integration.
H2 · SUPPORTS
The Hidden Infrastructure Tax in Multi-Agent Coding: An Estimated Replay of 39,526 Calls and an RTX 5090 Lab
Hardened regex and a local two-stage cascade matched 100/100 on a 100-case synthetic PHI set (rules tuned in-sample) at 0.03 to 0.04 ms negative p50, and hybrid repo-filtered extraction reached 99.46% in-sample Micro F1 on 50 briefs. The CEO approved the local cascade on 2026-10-15 and it is now the default PHI admission check, with Claude Haiku as the fallback; an out-of-sample test is still planned.
Arm 4 Hybrid (Arm 0+ + repo-filtered qwen3:14b) matched 367/370 items (99.73% precision, 99.19% recall, Macro F1 99.40%, McNemar p = 2.65e-22 vs baseline Arm 0 at 59.98% F1 and p = 3.73e-09 vs Arm 0+ at 95.35% F1).
H4 · COMPLICATES
The Hidden Infrastructure Tax in Multi-Agent Coding: An Estimated Replay of 39,526 Calls and an RTX 5090 Lab
Roughly 99% of the 121.82-hour summed concurrent call-time reduction came from software fixes and CPU worker sharding rather than the GPU, and the 100.0% PHI and 99.46% checklist rules were refined in-sample on the evaluation corpora.
roughly 99% of the time reduction comes from software and algorithmic fixes
Article qualification: prospective out-of-sample holdout testing is required to establish unseen generalization.
Outside H1 to H4 · Not used
First Vulkan Wins: Four Fixes Worth 2x to 900x
Rendering placement and build settings show engineering gains, but do not compare agent controls or verification methods.
Enabling /O2 cut startup load from 13.4 s to 5.8 s (2.3x)
Outside H1 to H4 · Not used
Culling at Scale: Eliminating Edge Pop and Shadow Desync
Rendering speed and visual correctness are domain results without a distinct measured agent-workflow comparison.
Whole-frame execution dropped by 64% after hoisting crowd light-fit calculations.
Outside H1 to H4 · Not used
Why We Optimize Natively: Hardware Block Compression Over Neural Upscaling
Texture compression is not an agent control decision. This comparison does not test H2.
Payload reduced from 85.3 MiB (RGBA8 level-0+mips) to 13.34 MiB (.ktx2).
Outside H1 to H4 · Not used
Falsifying Transformer Activation Cache Reuse for Code Edits
A useful negative inference result, but activation reuse does not directly test the four workflow claims.
Downstream token agreement dropped below the 0.99 correctness threshold required for code generation.
Outside H1 to H4 · Not used
Measured: Local MoE Inference on a Single GPU
Tensor placement changed throughput, not measured agent reliability. Relevant to the resource question, not evidence for H2 control decisions.
Saturated GPU VRAM (31.7 GB); 31x speedup over naive layer splitting.
Outside H1 to H4 · Not used
Local Model Serving Traps: IPv6 Stalls and Context Ceiling Reloads
Serving configuration affected latency; this does not compare prompting, context quality, or agent decisions.
Explicit 127.0.0.1 loopback reduced embedding requests from 2,040 ms down to 5.8 ms.
Outside H1 to H4 · Not used
Local Neural 3D Mesh Generation: GPU Flow vs the CPU Topology Bottleneck
CPU cleanup dominated a generation pipeline; this informs the open resource question without measuring orchestration policy.
Single-threaded CPU manifold repair and hole-filling consumed 73.2% of pipeline latency.
Outside H1 to H4 · Not used
Running Agent Memory in Under 500 MB RAM on ARM64
A 440 MB tooling footprint informs resource feasibility, but does not measure concurrency or reliability gains.
Combined runtime consumes 0.44 GB, occupying under 11% of a standard 4 GB RAM budget.
Outside H1 to H4 · Not used
Multi-Lane Memory Saturation: Measuring the Limits of Parallel Local Agents
Nine of eleven lane shells were reaped. Resource limits remain an open question because stability after the adopted caps was not measured.
Running 11 parallel agent lanes saturated 62 GB host RAM; memory guards reaped 9 lane shells mid-work.

Changelog

v0.1, October 2026: Initial thesis with four testable claims, a complete article mapping, contrary and null results, and explicit conditions for changing our minds. No fifth claim added.

v0.1, October 15, 2026: Token estimate added, with measured usage, extrapolation method and limits. Added why the studies range so widely.

v0.1: planned upgrade studies EXP-3 to EXP-6 listed