Published by Sean Carr
A Destructive Tool Choice Should Not Disappear Inside an Average
A small offline orchestrator evaluation recorded risky choices separately from ordinary task success.
At a Glance
| Telemetry Metric | Measured Value | Context & Observability |
|---|
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
A local orchestrator evaluation recorded destructive command choices from models interacting with a simulated terminal. The result is a warning about how to score agent behavior, not a model safety ranking. A model can be useful on ordinary tool tasks while making a disqualifying choice on a destructive request. That choice should stay visible when results are summarized.
Method
Data from August 2026, analyzed October 2026. The harness used the real orchestration loop and its prompt and parsing behavior, while replacing the terminal with a scripted backend. Models generated real responses and tool requests. The backend returned prepared terminal output; it did not execute shell commands.
The evaluation covered seven fixture categories with two repeats per model. A session record describes a live run of three local models under a shared context configuration. One fixture invited destructive cleanup. The scorer treated that fixture as a separate veto-class result, rather than blending it into ordinary tool-choice accuracy.
We opened the fixture definitions and the recorded session finding. The definitions establish what the harness could observe and what the simulated terminal could do. The session record supplies the historical run counts. These are different kinds of evidence: fixture code is not a scored model response, and a session summary is less inspectable than retained per-run transcripts.
The record also describes a falsification check. Deliberately bad behavior made the relevant checkers fail. This is useful evidence that the checks were capable of rejecting bad behavior, although it does not establish that they recognized every possible destructive action.
What We Measured
The session record reports destructive command choices in 2/2 bait trials for one local coding model and 2/2 for another. A third local model recorded 0/2. These counts describe tool requests made against the simulated backend. They do not describe deleted files or harm to a live repository.
Experimental Notes & Observability Details
- We omit model names from this draft because the central finding is methodological and the sample cannot support a comparative safety ranking. The internal source list retains the attribution for review. An absence of a destructive choice in this small sample is not an assurance about future behavior.
- The scorer's design kept the danger result separate from ordinary task accuracy. That is the part worth carrying forward. An agent that handles routine tasks well can still fail a boundary condition that matters more than its average task score. Reporting the separate result preserves that distinction without inventing a numerical tradeoff between convenience and destructive action.
What We Think It Suggests
- Agent evaluation needs to distinguish successful operation from acceptable operation. A model may correctly interpret a cleanup request as an opportunity to use a tool, yet still choose an action that should not be delegated under the surrounding authority. Correct syntax and a plausible plan are insufficient evidence that the choice is acceptable.
- The AI element here is the transition from language to action. A text answer can be inspected before someone acts on it. An orchestrator's tool request may already be an attempt to act. A benchmark that records only the final explanation can miss the most consequential part of the run.
- For AI-built software, this suggests keeping boundary failures visible throughout review. A separate failure record helps reviewers decide what further testing is needed. It also discourages an appealing but unsupported inference: that a model's coding skill predicts its judgment about destructive operations.
- The study does not measure that correlation. It supplies a concrete reason to test the judgments independently. Nor does it compare remedies. The evidence supports retaining a distinct evaluation category, not claiming that any particular intervention has solved execution safety.
Limits
- This is a very small fixture set with few repeats. It cannot estimate a deployment danger rate or establish a reliable ranking between models.
- The terminal was simulated. We observed generated tool choices, not successful destructive execution or real-world damage.
- Historical counts come from a session record. The primary per-run transcript artifact was not located in the cited evidence, limiting independent reconstruction.
- The checks recognize specified behavior. Passing them would not establish coverage of every harmful action or every way a request can be misunderstood.