A Second AI Audit Found Critical Findings, Then Verification Lowered One

A component review recorded no highest-severity findings. A different model's review raised them, and a separate verification pass checked the claims.

Core Finding:An AI audit can contain useful findings while missing others that deserve more urgent attention. It can also overstate the severity of something it finds. Our component audit records contain an example of both problems. A Gemini audit of an application component recorded no P0 findings, our highest severity. A subsequent Codex audit recorded 3 P0 findings. A separate Codex verification pass confirmed 2 of those at P0 and downgraded the remaining claim to P2. The useful result was neither the initial absence of critical findings nor the larger count in the later report. It was the set of claims that survived verification. Data from Oct 2026, analyzed Oct 2026. This is a retrospective case study, with registry status tentative. It extends our published case study of independent review blocking a proposed change: here, we compare audits of an existing component and examine the reviewer as another source of fallible claims.

Empirical Findings Table

The original audit was not a clean bill of health. It had already recorded serious findings. Nor were all later concerns entirely unrelated to the earlier report: both discussed a change to how records were associated with organizations. Different severity totals alone do not tell us whether auditors found different defects, described related behavior differently, or applied the rubric differently.

Telemetry Metric	Measured Value	Context & Observability

Engineering Action Directives

Reviewing an AI auditor can add value in both directions. A fresh audit can surface urgent concerns absent from an earlier severity list. A fresh verifier can also reduce an overstatement in the later audit. These are different jobs: searching for defects and checking whether a particular claim follows from the evidence.
For AI-built software, an audit's polished summary is a poor stopping rule. A report is more useful when its claims are specific enough for another reviewer to accept, qualify, or reject. The disagreement becomes actionable because the reports preserve evidence and explicit judgments.
This case supports testing review workflows that separate discovery from adjudication. It does not establish a general model ranking. The operational question is whether a proposed finding survives scrutiny, rather than whether its author belongs to a preferred model family.

Methodology Boundaries & Limits

This is a selected component from an operational audit, with small scope and no randomized assignment or controlled effort comparison. It cannot estimate either model's detection rate.
Verification used the Codex model family that produced the later audit. It was a separate pass, but not a different-family adjudicator or a human ground-truth panel.
Findings and severity judgments came from source review. Deployed configuration and actual production impact were not established by this verification.
The verification list covered selected high-severity claims from the later report. Confirmed counts are not a complete measure of remaining defects, and a lower count is not evidence of safety.