Published by Sean Carr
Preventing Code Slop and Enforcing Task Completion with Mechanical Gates
How active helper cataloging reduces slop to zero and deterministic gates complete 96.6% to 100% of required items on skip-told briefs.
At a Glance
Code Slop Events
4 -> 0 Events
Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs.
Required items completed on skip-told briefs
56/58 Claude, 29/29 Gemini
Hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%). Ungated skip-told runs completed 39/65 and 21/29 items respectively.
False Block Rate
0.0% (0 / 59)
Zero false-positive blocks across 59 valid completion sessions.
Injection Hook Latency
0.27 - 1.16 s
Observed hook latency ranged from 0.27 to 1.16 seconds in the October 2 evaluation.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Code Slop Events | 4 -> 0 Events | Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs. |
| Required items completed on skip-told briefs | 56/58 Claude, 29/29 Gemini | Hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%). Ungated skip-told runs completed 39/65 and 21/29 items respectively. |
| False Block Rate | 0.0% (0 / 59) | Zero false-positive blocks across 59 valid completion sessions. |
| Injection Hook Latency | 0.27 - 1.16 s | Observed hook latency ranged from 0.27 to 1.16 seconds in the October 2 evaluation. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
Autonomous agents introduce two subtle forms of codebase rot: generating duplicate helper functions instead of reusing existing ones ('code slop'), and declaring tasks complete while omitting steps requested in the prompt. Required items completed on skip-told briefs: Hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%). Ungated skip-told runs completed 39/65 and 21/29 items respectively. Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs.
Method
Evaluated active helper injection (TOOLKIT-V2) across 30 runs to detect duplicate utility generation.
Subjected agents to adversarial prompt instructions (GATE-P0) explicitly commanding them to skip tasks or ignore edge cases.
The gate checks specified file changes, required text and test-command results against a task checklist.
Measured false-positive block rates across 59 clean, compliant agent completions.
What We Measured
Slop reduction and required items completed on skip-told briefs under mechanical gating.
Code Slop Events
4 -> 0 Events
Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs.
Required items completed on skip-told briefs
56/58 Claude, 29/29 Gemini
Hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%). Ungated skip-told runs completed 39/65 and 21/29 items respectively.
False Block Rate
0.0% (0 / 59)
Zero false-positive blocks across 59 valid completion sessions.
Injection Hook Latency
0.27 - 1.16 s
Observed hook latency ranged from 0.27 to 1.16 seconds in the October 2 evaluation.
Experimental Notes & Observability Details
- Ungated skip-told runs completed 39/65 items on Claude and 21/29 items on Gemini.
- With completion gates, omitted steps were rejected with structured error feedback, prompting the model to complete the missing tasks in a single follow-up turn.
- Corrected October 11, 2026: Updated run counts from 24 sessions to 30 runs, revised task completion metric to report items completed on skip-told briefs, clarified gate checks inspect file changes, required text and test-command results rather than AST diffs, and updated hook latency range to 0.27 to 1.16 seconds.
What We Think It Suggests
- In this benchmark, active catalog injection eliminated helper duplication across tested runs by surfacing canonical names before execution.
- Deterministic mechanical completion gates effectively counter premature task abandonment under complex prompt constraints without relying on conversational compliance.
Limits
- Adversarial prompt scope: Tested against specific skip-instruction patterns; models may discover novel unmonitored failure modes not covered by current completion rules.
- Single-repository evaluation: Evaluated within MeshTen engineering harnesses across 30 runs; cross-organization workflows with divergent conventions require tailored gate criteria.
- Six incomplete runs passed because generic file-change checks accepted the wrong edit.
Detailed Analysis & Field Observations
When LLMs write code in medium-to-large repositories, they suffer from a subtle architectural hazard: localized optimization. Rather than searching for an existing utility function, an agent will spontaneously re-implement standard primitives, such as atomic file writes, retry wrappers, or path normalizers. Over time, this generates duplicate logic, inconsistent error handling, and bloated codebases.
In our TOOLKIT-V2 evaluation, we measured the incidence of this 'code slop' across controlled implementation tasks. Total measured slop fell from 4 events in 12 Claude OFF runs to zero in 12 Claude ON and 6 Gemini ON runs when active helper catalog injection was enabled.
In parallel, we tackled the inverse failure mode: premature task abandonment. In autonomous workflows, models frequently omit requirements from complex prompts, report 'done', and stop. In our GATE-P0 benchmark, we subjected agents to adversarial prompts, explicitly instructing them to skip required steps or ignore edge-case tests.
Ungated skip-told runs completed 39/65 items on Claude and 21/29 items on Gemini. When guarded by our mechanical completion gate, where the gate checks specified file changes, required text and test-command results against a task checklist, hook-only runs completed 56/58 items on Claude (96.6%) and 29/29 on Gemini (100%), while maintaining a 0.0% false block rate across 59 clean completions. Upfront checklists accounted for most of the improvement; stop hooks recovered remaining blocked items in these runs.
Corrected October 11, 2026: Updated run counts from 24 sessions to 30 runs, revised task completion metric to report items completed on skip-told briefs, clarified gate checks inspect file changes, required text and test-command results rather than AST diffs, and updated hook latency range to 0.27 to 1.16 seconds.