Published by Sean Carr
The Confinement Tax: Measuring Scope Leakage in Agentic Benchmark Harnesses
Auditing 1,934 tool calls under mechanical workspace confinement, with all 7 outside-root read attempts denied.
At a Glance
Sealed Evaluation
Sealed run
The mechanically confined evaluation used an 80-question frozen holdout.
Outside-root reads denied
7 / 7
All seven attempts to read outside the staging directory were denied.
Toolless Control
Sealed control
The evaluation included a toolless control arm.
Confinement Audit
100% (1,934 / 1,934)
All 1,934 tool calls mechanically audited; 7/7 outside-root read attempts blocked with zero leakage.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Sealed Evaluation | Sealed run | The mechanically confined evaluation used an 80-question frozen holdout. |
| Outside-root reads denied | 7 / 7 | All seven attempts to read outside the staging directory were denied. |
| Toolless Control | Sealed control | The evaluation included a toolless control arm. |
| Confinement Audit | 100% (1,934 / 1,934) | All 1,934 tool calls mechanically audited; 7/7 outside-root read attempts blocked with zero leakage. |
| Query Latency (p50) | 98 s | Sealed Arm H median latency was 98s (p90 254s) consuming ~314k tokens/call scanning the staged corpus. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
We enforced mechanical directory confinement in an 80-question memory evaluation. All 1,934 tool calls were audited and all 7 outside-root read attempts were denied. Unsealed accuracy scores are omitted from public reporting.
Method
We evaluated the sealed holdout harness on a frozen 80-question engineering memory benchmark (P7H).
The benchmark compared Arm H (an agent equipped with file search and memory retrieval tools) against a toolless control arm (C2).
The sealed evaluation harness enforced mechanical filesystem boundaries: the current working directory was pinned to an isolated staging area, and all filesystem operations outside the staging directory were intercepted and denied.
An automated audit logged 100% of tool invocations, inspecting tool call arguments, target paths, and return codes.
The earlier comparison run lacked mechanical confinement, so its accuracy scores are omitted from public reporting.
What We Measured
All 1,934 tool calls audited; all 7 outside-root read attempts denied.
Sealed Evaluation
Sealed run
The mechanically confined evaluation used an 80-question frozen holdout.
Outside-root reads denied
7 / 7
All seven attempts to read outside the staging directory were denied.
Toolless Control
Sealed control
The evaluation included a toolless control arm.
Confinement Audit
100% (1,934 / 1,934)
All 1,934 tool calls mechanically audited; 7/7 outside-root read attempts blocked with zero leakage.
Query Latency (p50)
98 s
Sealed Arm H median latency was 98s (p90 254s) consuming ~314k tokens/call scanning the staged corpus.
Experimental Notes & Observability Details
- Count questions: 6/11 correct in the sealed run.
- Quote and statement lookups: 8/13 correct in the sealed run.
- Latest value lookups: 8/12 correct in the sealed run.
- High-level reasoning questions: 14/14 correct in the sealed run.
- Consolidation questions: 5/6 correct in the sealed run.
- Status questions: 15/20 correct in the sealed run.
What We Think It Suggests
- Unconfined benchmark environments allow autonomous agents to leak beyond task boundaries, reading adjacent files, subsequent session logs, or ambient repository metadata that would be unavailable in isolated deployment.
- The audit establishes boundary enforcement in the sealed run. The earlier unsealed run cannot establish performance under that boundary.
- Performance under confinement needs measurement within the sealed stage.
- Benchmark validity requires mechanical directory boundary enforcement rather than relying on prompt-level instructions to stay inside the designated directory.
Limits
- Single Benchmark Suite: Evaluated on an 80-question repository memory benchmark (P7H) using one primary tool-using agent architecture.
- Individual Regression Traces: While all 1,934 tool calls were mechanically audited for boundary compliance, the 12 individual regressed questions were analyzed by category rather than exhaustive manual prompt tracing to conserve evaluation budget.
- Corpus Staging Overhead: Copying the evaluation repository into a sealed staging directory increased grep latency and context token volume during broad search passes.
Reproduce It (Data & Benchmarks)
- Evaluation harness: eval/sealed_holdout.py
- Prediction manifest: eval/PREDICTION_SH.md
- Run manifest and audit logs: reports/p7h/*.json
- Golden test questions: P7H 80-question frozen set
Data and prompt templates only; no proprietary code is published.
Detailed Analysis & Field Observations
When evaluating autonomous coding agents, benchmark harnesses frequently run inside the parent repository or grant unrestricted filesystem exploration. In this study, we evaluated the effect of enforcing strict mechanical workspace confinement on an 80-question engineering memory holdout benchmark.
All 1,934 tool calls were audited, and all 7 outside-root read attempts were denied.
The evaluation also included a toolless control. Unsealed accuracy scores are omitted from public reporting.
Public comparative memory accuracy is reported for sealed Bank Slice studies in the current synthesis.
Current synthesis: Semantic Memory Living Paper v1.