Published by Sean Carr
An Agent Can Understand the Request and Still Fail to Act
A replay separated helpful replies from observable work, then followed the result through later classifier tests.
At a Glance
| Telemetry Metric | Measured Value | Context & Observability |
|---|
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
An orchestrating agent can describe an appropriate next step without taking it. In a replay of real requests, clarification cases passed while autonomous follow-through and resource decisions were much less reliable. Later classifier runs improved direct answers but did not improve every category. The useful finding is about uneven behavior and observable evidence, rather than a single score for whether an agent understands intent.
Method
Data collected October 8 and 12, 2026, analyzed October 2026. We examined a frozen set of 49 real requests and their expected observable behavior. The requests came from engineering sessions, with context needed to interpret the user's authority and ongoing work. The evaluation replayed the request to the orchestrator and recorded its response and calls. Work delivery was simulated rather than sent to real workers.
The scoring rule required an observable action when action was expected. A promise to start work did not count as dispatch. A reassuring statement that work had stopped did not establish that workers had stopped. Some requests instead required a direct answer with restricted tool use. This matters because unnecessary activity can violate intent just as surely as inactivity.
We used the later corrected baseline scorecard rather than treating an earlier scout summary as the final result. We also checked later paired classifier replays so this historical baseline would not be mistaken for the current system's measured ceiling.
What We Measured
The baseline had 48 graded cases and one execution error. Clarification decisions passed 5/5. Direct answers passed 4/7. Autonomous follow-through passed 0/5 graded cases, with the execution error in that category. Resource decisions passed 1/6. Routing passed 2/7, and respecting standing rules passed 2/7. Commit and push handling passed 2/6; stopped-work handling passed 1/5.
Experimental Notes & Observability Details
- These categories test different obligations. Failing a formatting requirement is not equivalent to taking an unauthorized action. Likewise, failing to show a dispatch in the observed turn is evidence of missing follow-through in that turn, not evidence that the agent could never complete the work.
- The baseline also recorded 18 warnings about activity by other terminals. Those warnings are part of the conditions under which it ran. They make a clean causal story about any later change harder to defend.
- In the later paired replay, direct-answer cases passed 9/14 for baseline and 14/14 with a classifier. Resource handling moved in the opposite direction: its reported repeat-mean pass rate was 26.7% for baseline and 0.0% with classification. The later experiment replayed the frozen cases twice per arm and applied a 300-second replay budget. Each arm had six execution errors in its first repeat and nine in its second repeat. All were replay timeouts, and conditional pass rates exclude them.
What We Think It Suggests
- Intent classification can help choose the kind of response without supplying the persistence needed to finish a task. The direct-answer result is encouraging for keeping simple conversations simple. The resource result argues against calling the classifier a general governance improvement.
- For AI-built software, an instruction such as continue autonomously contains several obligations: decide what is authorized, take an appropriate action, preserve evidence, and report the resulting state. Evaluating only the final prose would miss the gap between announcing these obligations and carrying them out.
- A useful evaluation therefore keeps categories separate and distinguishes execution errors from scored failures. It should let a reader see where an agent became more useful and where it became less reliable. That is more actionable than averaging away a weak resource decision behind a strong conversational answer.
Limits
- The cases are a small frozen sample from a particular engineering workflow and orchestrator. They do not establish reliability for other agents or organizations.
- The replay observes a bounded interaction, not the completion of a long autonomous project. Simulated delivery cannot establish successful downstream execution.
- The later run used a different time budget from an earlier uncapped comparison. We do not interpret changes across those runs as a clean measure of classifier benefit.
- Semantic grading and strict response requirements affect the score. Category failures need their recorded reasons, not a blanket label of unsafe behavior.