The Confinement Tax: Measuring Scope Leakage in Agentic Benchmark Harnesses

Auditing 1,934 tool calls under mechanical workspace confinement, with all 7 outside-root read attempts denied.

Core Finding:We enforced mechanical directory confinement in an 80-question memory evaluation. All 1,934 tool calls were audited and all 7 outside-root read attempts were denied. Unsealed accuracy scores are omitted from public reporting.

Sealed Evaluation

Sealed run

The mechanically confined evaluation used an 80-question frozen holdout.

Outside-root reads denied

7 / 7

All seven attempts to read outside the staging directory were denied.

Toolless Control

Sealed control

The evaluation included a toolless control arm.

Confinement Audit

100% (1,934 / 1,934)

All 1,934 tool calls mechanically audited; 7/7 outside-root read attempts blocked with zero leakage.

Empirical Findings Table

All 1,934 tool calls audited; all 7 outside-root read attempts denied.

Telemetry Metric	Measured Value	Context & Observability
Sealed Evaluation	Sealed run	The mechanically confined evaluation used an 80-question frozen holdout.
Outside-root reads denied	7 / 7	All seven attempts to read outside the staging directory were denied.
Toolless Control	Sealed control	The evaluation included a toolless control arm.
Confinement Audit	100% (1,934 / 1,934)	All 1,934 tool calls mechanically audited; 7/7 outside-root read attempts blocked with zero leakage.
Query Latency (p50)	98 s	Sealed Arm H median latency was 98s (p90 254s) consuming ~314k tokens/call scanning the staged corpus.

Engineering Action Directives

Unconfined benchmark environments allow autonomous agents to leak beyond task boundaries, reading adjacent files, subsequent session logs, or ambient repository metadata that would be unavailable in isolated deployment.
The audit establishes boundary enforcement in the sealed run. The earlier unsealed run cannot establish performance under that boundary.
Performance under confinement needs measurement within the sealed stage.
Benchmark validity requires mechanical directory boundary enforcement rather than relying on prompt-level instructions to stay inside the designated directory.

Methodology Boundaries & Limits

Single Benchmark Suite: Evaluated on an 80-question repository memory benchmark (P7H) using one primary tool-using agent architecture.
Individual Regression Traces: While all 1,934 tool calls were mechanically audited for boundary compliance, the 12 individual regressed questions were analyzed by category rather than exhaustive manual prompt tracing to conserve evaluation budget.
Corpus Staging Overhead: Copying the evaluation repository into a sealed staging directory increased grep latency and context token volume during broad search passes.