Published by Sean Carr
A Second AI Audit Found Critical Findings, Then Verification Lowered One
A component review recorded no highest-severity findings. A different model's review raised them, and a separate verification pass checked the claims.
At a Glance
| Telemetry Metric | Measured Value | Context & Observability |
|---|
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
An AI audit can contain useful findings while missing others that deserve more urgent attention. It can also overstate the severity of something it finds. Our component audit records contain an example of both problems. A Gemini audit of an application component recorded no P0 findings, our highest severity. A subsequent Codex audit recorded 3 P0 findings. A separate Codex verification pass confirmed 2 of those at P0 and downgraded the remaining claim to P2. The useful result was neither the initial absence of critical findings nor the larger count in the later report. It was the set of claims that survived verification. Data from Oct 2026, analyzed Oct 2026. This is a retrospective case study, with registry status tentative. It extends our published case study of independent review blocking a proposed change: here, we compare audits of an existing component and examine the reviewer as another source of fallible claims.
Method
We selected the component concerned with people and organization records because its original audit, subsequent audit, and verification report were available together. The audit reports identify an identical source revision. That matters: a difference in findings cannot simply be attributed to an intervening source change in these reports.
The original assignment and execution record identify a Gemini audit lane. The later audit and separate verification used Codex. We use those model-family names without assigning an exact model release or reasoning setting that these records do not establish. This was an operational review sequence, not a controlled model benchmark.
We counted severity entries in each report's findings array. For the verification pass, we counted the explicit verdicts and final severities. Verification covered the later audit's high-severity claims; it was not a complete independent inventory of every possible defect or a reassessment of every original finding.
The verification instructions required the reviewer to challenge the claim, trace the relevant behavior, and look for protections the auditor might have overlooked. The resulting report states that it used source inspection and did not check deployed configuration. We do not turn its conclusions into claims about a production incident.
What We Measured
The original audit was not a clean bill of health. It had already recorded serious findings. Nor were all later concerns entirely unrelated to the earlier report: both discussed a change to how records were associated with organizations. Different severity totals alone do not tell us whether auditors found different defects, described related behavior differently, or applied the rubric differently.
Experimental Notes & Observability Details
- The verification report also demonstrates why a later, more severe report cannot be accepted merely because it sounds more cautious. Its downgraded claim overlooked protections relevant to the asserted consequence. The verifier retained a lower-severity concern while rejecting the original severity. That distinction preserved useful information without treating the strongest wording as established fact.
What We Think It Suggests
- Reviewing an AI auditor can add value in both directions. A fresh audit can surface urgent concerns absent from an earlier severity list. A fresh verifier can also reduce an overstatement in the later audit. These are different jobs: searching for defects and checking whether a particular claim follows from the evidence.
- For AI-built software, an audit's polished summary is a poor stopping rule. A report is more useful when its claims are specific enough for another reviewer to accept, qualify, or reject. The disagreement becomes actionable because the reports preserve evidence and explicit judgments.
- This case supports testing review workflows that separate discovery from adjudication. It does not establish a general model ranking. The operational question is whether a proposed finding survives scrutiny, rather than whether its author belongs to a preferred model family.
Limits
- This is a selected component from an operational audit, with small scope and no randomized assignment or controlled effort comparison. It cannot estimate either model's detection rate.
- Verification used the Codex model family that produced the later audit. It was a separate pass, but not a different-family adjudicator or a human ground-truth panel.
- Findings and severity judgments came from source review. Deployed configuration and actual production impact were not established by this verification.
- The verification list covered selected high-severity claims from the later report. Confirmed counts are not a complete measure of remaining defects, and a lower count is not evidence of safety.
Detailed Analysis & Field Observations
| Review stage | Recorded result |
|---|---|
| Original Gemini component audit | 0 P0, 3 P1, 3 P2 |
| Subsequent Codex component audit | 3 P0, 2 P1, 3 P2 |
| Separate verification of the later high-severity claims | 2 P0 and 2 P1 confirmed; remaining P0 downgraded to P2 |