Published by Sean Carr
Eight Review Passes on an AI That Reads Strangers' Email
What a second model caught, and why six of its findings were caused by an earlier fix.
At a Glance
Second-model P1 findings
17
5 → 3 → 4 → 1 → 3 → 1 → 0 across seven passes. The count did not fall steadily.
Resolution
16 fixed, 1 ruled
One finding (escalation labels applied without an approval gate) was ruled intended: those labels only hand mail to a human. The exemption is now explicit in code and tested.
Caused by a prior fix
6 of 17
A link filter built on a fixed domain list; a per-sender budget keyed on a forgeable From address; schema-error logs that printed model-generated keys; a global budget that unauthenticated mail could drain; a database transaction that fails in production; a new fallback path that skipped an approval gate.
From the orchestrator's own instructions
3 of those 6
The logging rule, the "count rejected mail against the global cap only" ruling, and the fallback-path wording all came from the orchestrator.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Second-model P1 findings | 17 | 5 → 3 → 4 → 1 → 3 → 1 → 0 across seven passes. The count did not fall steadily. |
| Resolution | 16 fixed, 1 ruled | One finding (escalation labels applied without an approval gate) was ruled intended: those labels only hand mail to a human. The exemption is now explicit in code and tested. |
| Caused by a prior fix | 6 of 17 | A link filter built on a fixed domain list; a per-sender budget keyed on a forgeable From address; schema-error logs that printed model-generated keys; a global budget that unauthenticated mail could drain; a database transaction that fails in production; a new fallback path that skipped an approval gate. |
| From the orchestrator's own instructions | 3 of those 6 | The logging rule, the "count rejected mail against the global cap only" ruling, and the fallback-path wording all came from the orchestrator. |
| Orchestrator-only findings | 4 | Includes the first P1: sender authentication accepted any "pass" string, including headers the attacker wrote. |
| Time from first commit to clean pass | 3h 47m | 10 commits. |
| Tests | 102 passing | Each fix shipped with a test that fails on the code before it. The suite once passed over a real transaction bug, because the in-memory fake database did not enforce the real database's read-before-write rule. The fake now enforces it. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
We built an AI agent that triages our public sales inbox. Anyone on the internet can send it text, so every email counts as hostile input. The agent went through eight review passes in one day: one by the orchestrating model, then seven by a model from a different vendor, until the reviewer came back with no high-severity findings. The second model raised 17 high-severity (P1) findings along the way. The useful result is not the number. Six of those 17 were created by the fix for an earlier finding, and three of the six came straight from the orchestrator's own fix instructions. Patching findings one at a time kept moving the hole. One structural rule closed whole classes of them: mail that can't prove who sent it does nothing.
Method
The design treats every inbound email as untrusted. The model gets no tools, no retrieval and no secrets. It may classify and draft. Plain code decides every action.
One build lane wrote the service. The orchestrating model read the diff first, then ran a different-vendor model as an adversarial reviewer on each fix, scoped to that fix's diff.
Each round's findings went back to the builder as a numbered fix brief. Every item needed a test that fails on the old code.
A circuit breaker capped fix-then-re-review loops on the same class of finding. Past the cap, the orchestrator stopped asking for patches and issued binding design rulings.
We counted findings from the saved review outputs and fix briefs, not from memory.
What We Measured
One service, one day, eight review passes, last one clean
Second-model P1 findings
17
5 → 3 → 4 → 1 → 3 → 1 → 0 across seven passes. The count did not fall steadily.
Resolution
16 fixed, 1 ruled
One finding (escalation labels applied without an approval gate) was ruled intended: those labels only hand mail to a human. The exemption is now explicit in code and tested.
Caused by a prior fix
6 of 17
A link filter built on a fixed domain list; a per-sender budget keyed on a forgeable From address; schema-error logs that printed model-generated keys; a global budget that unauthenticated mail could drain; a database transaction that fails in production; a new fallback path that skipped an approval gate.
From the orchestrator's own instructions
3 of those 6
The logging rule, the "count rejected mail against the global cap only" ruling, and the fallback-path wording all came from the orchestrator.
Orchestrator-only findings
4
Includes the first P1: sender authentication accepted any "pass" string, including headers the attacker wrote.
Time from first commit to clean pass
3h 47m
10 commits.
Tests
102 passing
Each fix shipped with a test that fails on the code before it. The suite once passed over a real transaction bug, because the in-memory fake database did not enforce the real database's read-before-write rule. The fake now enforces it.
Experimental Notes & Observability Details
- The finding classes were spoofed sender identity, a parser mismatch in link filtering, budget exhaustion and flooding, a kill switch that still called the model, an auth fallback that accepted unsigned tokens, model output reaching logs, and actions that ran without their own approval gate.
- The service is not deployed. None of these issues were exploited; every finding was caught before release.
What We Think It Suggests
- On an AI surface fed by untrusted text, a narrow patch tends to move the hole. Rounds of link and budget patches each opened a sibling hole. The ruling that unauthenticated mail is inert (no model call, no lead, no draft) closed both the budget-poisoning and the lead-poisoning classes at once.
- The reviewer has to review the orchestrator's rulings too. Half of the fix-caused findings came from instructions written by the model running the process, not from the builder.
- A second model from a different vendor earns its cost even after the first model has read the diff: all 17 P1s came after the orchestrator's own read.
- A green test suite built on fakes proves less than it looks. A fake that is more permissive than the real dependency hides exactly the bugs a fix introduces.
- A capped fix loop is a feature. It forces the switch from "patch this line" to "change the rule."
Limits
- One service, one day, one reviewer model. This is a case study, not a rate.
- Severity is the reviewing model's grade. Nothing was exploited against a live system.
- Later passes reviewed each fix's diff, not the whole service from scratch, so the clean final pass is not a full audit.
- The service has not taken production traffic, so we can't say how the design holds up under real abuse.
Detailed Analysis & Field Observations
We built an AI agent that triages our public sales inbox. Anyone on the internet can send it text, so every email counts as hostile input. The agent went through eight review passes in one day: one by the orchestrating model, then seven by a model from a different vendor, until the reviewer came back with no high-severity findings.
The second model raised 17 high-severity (P1) findings along the way. The useful result is not the number. Six of those 17 were created by the fix for an earlier finding, and three of the six came straight from the orchestrator's own fix instructions.
Patching findings one at a time kept moving the hole. One structural rule closed whole classes of them: mail that can't prove who sent it does nothing.