Published by Sean Carr
The AI Proposes, Plain Code Decides: Who Decides What the Rule Means?
An inbox-agent review separated defects in generated code from a disagreement about intended behavior.
At a Glance
| Telemetry Metric | Measured Value | Context & Observability |
|---|
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
Separating a model's proposal from permission to act is a useful design intention. It still leaves an engineering question: does the surrounding software implement the intended policy, and is that policy clear enough to review? An AI-built inbox agent exposed both problems. Independent review identified actions inconsistent with their stated approval requirements. It also raised a finding that the engineering lead later ruled was intended behavior. The reviewer could identify a disagreement, but deciding which behavior the product should permit required an explicit policy decision. This extends our published inbox-agent study. The published inbox-agent study already reported the escalation-policy ruling alongside the repair sequence. Here we retrospectively group the reported policy-related observations and explain why that grouping is not a count of confirmed authority escapes. Data from October 2026, analyzed October 2026.
Method
We reopened the saved adversarial review reports for the inbox agent and the written engineering ruling that resolved the disputed finding. The reports describe candidate code changes, not an experiment in which a deployed agent was attacked. Findings retain the reviewer's severity labels.
We counted explicit high-severity finding entries across the reports. We then grouped entries concerning whether processing or an action should have been permitted. This is our retrospective grouping, rather than a category defined by the reviewer before inspection. We kept the later policy ruling alongside the original allegation instead of silently deleting the disagreement from the record.
We did not rerun tests for this analysis. We also removed claims from the earlier draft about platform-wide behavior, complete protection, and test coverage that this review record cannot establish. The unit of evidence here is a saved review observation and its disposition, rather than an asserted guarantee about all agents.
What We Measured
The reports contain 17 high-severity findings. Four fit the draft's broad policy-enforcement grouping: a shutdown-related processing concern, a drafting concern, an approval-related mailbox action, and a disputed action associated with human escalation. These are results stated at the behavior level; implementation details are omitted.
Experimental Notes & Observability Details
- The phrase "4 of 17" therefore describes a grouping of review findings. It does not describe four confirmed instances of a model escaping its intended authority. The reports concern surrounding application behavior, and the escalation finding was later ruled intended. Calling all four confirmed bypass defects would erase that distinction.
- The written ruling resolved the disagreement in favor of preserving the human escalation behavior. It required the intended exception to be explicit and tested. The final saved review returned no findings in its supplied diff, while stating that important underlying behavior remained outside the available context. We treat that as a scoped review outcome.
- The useful observation is the difference between the model's classification and the decision that followed. The reviewer had identified behavior that needed an answer. Its severity label alone did not settle whether that behavior should be removed. The ruling clarified the intended outcome, and a subsequent review examined the resulting change.
What We Think It Suggests
- An AI reviewer is useful as a source of concrete challenges to an implementation. It can ask whether a permitted action matches the written requirement and whether a change has introduced an exception. Its output remains a proposal for engineering judgment. Automatically converting every finding into a patch risks changing intended product behavior.
- Plain code also needs a clear policy to implement. Moving decisions outside model output does not make policy ambiguity disappear. In this case, the important decision concerned what should happen when work needs human attention. That was a product rule to resolve explicitly, rather than a fact the reviewer could infer from its own preferred design.
- The case also changes how we describe verification. A finding count measures what a reviewer reported under a particular scope. A disposition records what the team decided to do about it. A passing follow-up records what the reviewer saw in the supplied changes. Keeping those statements separate makes the evidence more useful than presenting a single total as a security outcome.
Limits
- This is a retrospective analysis of a single service and its saved review sequence. It is not a benchmark of governance designs or model families.
- The policy grouping was assigned after reading the findings. A different definition could reasonably exclude the shutdown concern or classify the escalation dispute separately.
- The review reports describe visible changes and repeatedly qualify missing context. A clean final diff review does not establish that the whole service is free of defects.
- No live abuse experiment, authority-escape rate, or platform-wide credential audit is measured here. The record cannot support a claim that unauthorized behavior is impossible.