An Agent Can Understand the Request and Still Fail to Act

A replay separated helpful replies from observable work, then followed the result through later classifier tests.

Core Finding:An orchestrating agent can describe an appropriate next step without taking it. In a replay of real requests, clarification cases passed while autonomous follow-through and resource decisions were much less reliable. Later classifier runs improved direct answers but did not improve every category. The useful finding is about uneven behavior and observable evidence, rather than a single score for whether an agent understands intent.

Empirical Findings Table

The baseline had 48 graded cases and one execution error. Clarification decisions passed 5/5. Direct answers passed 4/7. Autonomous follow-through passed 0/5 graded cases, with the execution error in that category. Resource decisions passed 1/6. Routing passed 2/7, and respecting standing rules passed 2/7. Commit and push handling passed 2/6; stopped-work handling passed 1/5.

Telemetry Metric	Measured Value	Context & Observability

Engineering Action Directives

Intent classification can help choose the kind of response without supplying the persistence needed to finish a task. The direct-answer result is encouraging for keeping simple conversations simple. The resource result argues against calling the classifier a general governance improvement.
For AI-built software, an instruction such as continue autonomously contains several obligations: decide what is authorized, take an appropriate action, preserve evidence, and report the resulting state. Evaluating only the final prose would miss the gap between announcing these obligations and carrying them out.
A useful evaluation therefore keeps categories separate and distinguishes execution errors from scored failures. It should let a reader see where an agent became more useful and where it became less reliable. That is more actionable than averaging away a weak resource decision behind a strong conversational answer.

Methodology Boundaries & Limits

The cases are a small frozen sample from a particular engineering workflow and orchestrator. They do not establish reliability for other agents or organizations.
The replay observes a bounded interaction, not the completion of a long autonomous project. Simulated delivery cannot establish successful downstream execution.
The later run used a different time budget from an earlier uncapped comparison. We do not interpret changes across those runs as a clean measure of classifier benefit.
Semantic grading and strict response requirements affect the score. Category failures need their recorded reasons, not a blanket label of unsafe behavior.