Eight Review Passes on an AI That Reads Strangers' Email

What a second model caught, and why six of its findings were caused by an earlier fix.

Core Finding:We built an AI agent that triages our public sales inbox. Anyone on the internet can send it text, so every email counts as hostile input. The agent went through eight review passes in one day: one by the orchestrating model, then seven by a model from a different vendor, until the reviewer came back with no high-severity findings. The second model raised 17 high-severity (P1) findings along the way. The useful result is not the number. Six of those 17 were created by the fix for an earlier finding, and three of the six came straight from the orchestrator's own fix instructions. Patching findings one at a time kept moving the hole. One structural rule closed whole classes of them: mail that can't prove who sent it does nothing.

Second-model P1 findings

5 → 3 → 4 → 1 → 3 → 1 → 0 across seven passes. The count did not fall steadily.

Resolution

16 fixed, 1 ruled

One finding (escalation labels applied without an approval gate) was ruled intended: those labels only hand mail to a human. The exemption is now explicit in code and tested.

Caused by a prior fix

6 of 17

A link filter built on a fixed domain list; a per-sender budget keyed on a forgeable From address; schema-error logs that printed model-generated keys; a global budget that unauthenticated mail could drain; a database transaction that fails in production; a new fallback path that skipped an approval gate.

From the orchestrator's own instructions

3 of those 6

The logging rule, the "count rejected mail against the global cap only" ruling, and the fallback-path wording all came from the orchestrator.

Empirical Findings Table

One service, one day, eight review passes, last one clean

Telemetry Metric	Measured Value	Context & Observability
Second-model P1 findings	17	5 → 3 → 4 → 1 → 3 → 1 → 0 across seven passes. The count did not fall steadily.
Resolution	16 fixed, 1 ruled	One finding (escalation labels applied without an approval gate) was ruled intended: those labels only hand mail to a human. The exemption is now explicit in code and tested.
Caused by a prior fix	6 of 17	A link filter built on a fixed domain list; a per-sender budget keyed on a forgeable From address; schema-error logs that printed model-generated keys; a global budget that unauthenticated mail could drain; a database transaction that fails in production; a new fallback path that skipped an approval gate.
From the orchestrator's own instructions	3 of those 6	The logging rule, the "count rejected mail against the global cap only" ruling, and the fallback-path wording all came from the orchestrator.
Orchestrator-only findings	4	Includes the first P1: sender authentication accepted any "pass" string, including headers the attacker wrote.
Time from first commit to clean pass	3h 47m	10 commits.
Tests	102 passing	Each fix shipped with a test that fails on the code before it. The suite once passed over a real transaction bug, because the in-memory fake database did not enforce the real database's read-before-write rule. The fake now enforces it.

Engineering Action Directives

On an AI surface fed by untrusted text, a narrow patch tends to move the hole. Rounds of link and budget patches each opened a sibling hole. The ruling that unauthenticated mail is inert (no model call, no lead, no draft) closed both the budget-poisoning and the lead-poisoning classes at once.
The reviewer has to review the orchestrator's rulings too. Half of the fix-caused findings came from instructions written by the model running the process, not from the builder.
A second model from a different vendor earns its cost even after the first model has read the diff: all 17 P1s came after the orchestrator's own read.
A green test suite built on fakes proves less than it looks. A fake that is more permissive than the real dependency hides exactly the bugs a fix introduces.

Methodology Boundaries & Limits

One service, one day, one reviewer model. This is a case study, not a rate.
Severity is the reviewing model's grade. Nothing was exploited against a live system.
Later passes reviewed each fix's diff, not the whole service from scratch, so the clean final pass is not a full audit.
The service has not taken production traffic, so we can't say how the design holds up under real abuse.