Published by Sean Carr
Auto-Context Prompt Injection: Measuring Code Slop and Helper Duplication Across 36 Controlled Sessions
Injecting repository concordance hints reduced helper re-implementation, but the overall reduction in mechanical slop (paired McNemar test p = 0.062) did not cross the 0.05 significance threshold.
At a Glance
Firing Reliability
17 / 18 Sessions
Context injection fired on 17/18 ON sessions with active symbol hits (0/18 OFF); single miss returned empty in 0.27s on cold startup.
Hook Latency
0.27 - 1.16 s
Hook execution latency ranged from 0.27s to 1.16s, safely within the 2.0-second safety timeout.
Target Helper Duplication
4/4 -> 0/2
On one target helper, duplication went from 4 of 4 runs without injection to 0 of 2 with it, too small a sample to generalize.
Mechanical Code Slop
6 -> 1 Events (p = 0.062)
Mechanical slop dropped from 6 events to 1 event across 18 pairs (paired McNemar p = 0.062, sign test p = 0.062, not significant at alpha = 0.05).
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Firing Reliability | 17 / 18 Sessions | Context injection fired on 17/18 ON sessions with active symbol hits (0/18 OFF); single miss returned empty in 0.27s on cold startup. |
| Hook Latency | 0.27 - 1.16 s | Hook execution latency ranged from 0.27s to 1.16s, safely within the 2.0-second safety timeout. |
| Target Helper Duplication | 4/4 -> 0/2 | On one target helper, duplication went from 4 of 4 runs without injection to 0 of 2 with it, too small a sample to generalize. |
| Mechanical Code Slop | 6 -> 1 Events (p = 0.062) | Mechanical slop dropped from 6 events to 1 event across 18 pairs (paired McNemar p = 0.062, sign test p = 0.062, not significant at alpha = 0.05). |
| Blind Judge Score | 9 -> 6 Events (p = 0.51) | Independent Opus judge recorded 6 slop events ON vs 9 OFF (paired McNemar test p = 0.51, not statistically significant). |
| Context Overhead | 1.06x - 1.29x | Injected context added 6% tokens on compact tooling tasks and 29% tokens on multi-service tasks. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
We evaluated whether automatically injecting symbol concordance hints into agent prompts reduces duplicate helper implementation and code slop across 36 controlled coding sessions. While on one target helper, duplication went from 4 of 4 runs without injection to 0 of 2 with it, too small a sample to generalize, the reduction in mechanical code slop from 6 events to 1 event across 18 task pairs yielded an exact McNemar test p = 0.062, which is not statistically significant at the standard 0.05 alpha level.
Method
We conducted two evaluation runs totaling 36 controlled sessions across 18 task pairs using Claude Sonnet 5 with medium reasoning effort.
The evaluation tested six distinct development tasks: adding a helper function, adding a CLI command, fixing a bug, refactoring names with docstrings, updating configuration, and adding tests.
The ON arm enabled automated context injection (CODEKIT_INJECT=1), where a lightweight prompt hook queried a warm symbol index and injected relevant symbols and drift rules. The OFF arm ran with injection disabled.
Both arms were evaluated in isolated repository clones, ensuring no cross-session state was preserved.
All code modifications were evaluated using automated AST regex oracles for mechanical slop (duplicate helper creation, naming drift, and docstring omissions), alongside an independent blind review by Claude Opus with randomized ordering and scrubbed arm labels.
Hypotheses and evaluation criteria were locked in version control prior to executing scored runs.
What We Measured
Automated context injection reduced utility function re-implementation and showed a directional drop in code slop (6 events to 1 event across 18 task pairs), but this difference (exact McNemar p = 0.062) did not achieve statistical significance at the alpha = 0.05 level.
Firing Reliability
17 / 18 Sessions
Context injection fired on 17/18 ON sessions with active symbol hits (0/18 OFF); single miss returned empty in 0.27s on cold startup.
Hook Latency
0.27 - 1.16 s
Hook execution latency ranged from 0.27s to 1.16s, safely within the 2.0-second safety timeout.
Target Helper Duplication
4/4 -> 0/2
On one target helper, duplication went from 4 of 4 runs without injection to 0 of 2 with it, too small a sample to generalize.
Mechanical Code Slop
6 -> 1 Events (p = 0.062)
Mechanical slop dropped from 6 events to 1 event across 18 pairs (paired McNemar p = 0.062, sign test p = 0.062, not significant at alpha = 0.05).
Blind Judge Score
9 -> 6 Events (p = 0.51)
Independent Opus judge recorded 6 slop events ON vs 9 OFF (paired McNemar test p = 0.51, not statistically significant).
Context Overhead
1.06x - 1.29x
Injected context added 6% tokens on compact tooling tasks and 29% tokens on multi-service tasks.
Experimental Notes & Observability Details
- Task completion rate: 17/18 ON sessions succeeded vs 17/18 OFF sessions.
- One ON session failed when an unexpected string transformation in a shared validation helper caused four custom-role tests to break.
What We Think It Suggests
- Providing models with focused, pre-computed symbol concordance hints at the start of a turn can guide them toward existing utility reuse rather than writing duplicate code.
- Injected context introduces a tangible token overhead (+6% to +29%), meaning injection should be reserved for build and implementation prompts rather than general conversational turns.
- Prompt hooks must execute against an already warm index service. Cold startup races during the first prompt can cause the hook to fail open, as seen in the single missed session.
- While on one target helper, duplication went from 4 of 4 runs without injection to 0 of 2 with it, too small a sample to generalize, establishing statistical significance across broad categories of code slop will require a larger sample size.
Limits
- Statistical Insignificance: The overall reduction in mechanical slop yielded a p-value of 0.062, which exceeds the standard 0.05 threshold for statistical significance. The findings represent a directional trend, not a confirmed effect.
- Blind Judge Disparity: The independent blind judge identified qualitative inconsistencies (such as manual catalog parsing) that resulted in a p-value of 0.51, reflecting substantial variance in subjective code quality evaluations.
- Token and Latency Overhead: Context injection added between 6% and 29% additional tokens to session context and introduced up to 1.16 seconds of pre-turn hook latency.
- Sample Size: The evaluation was limited to 18 matched task pairs (36 total sessions), which constrained statistical power for rare slop events.
Reproduce It (Data & Benchmarks)
- Evaluation harness: inj/harness/run_eval.py and inj2/harness/run_inj2.py
- Pre-registered prediction files: PREDICTION_INJ_EVAL.md and PREDICTION_INJ_EVAL_2.md
- Output ledgers and diffs: inj2/harness/out/
Data and prompt templates only; no proprietary code is published.
Detailed Analysis & Field Observations
When coding agents modify unfamiliar repositories, they frequently re-implement existing helper functions and introduce naming inconsistencies because they lack awareness of existing project conventions.
We evaluated an automated context injection hook that intercepts user prompts, identifies relevant symbols from a resident index, and injects concise concordance hints into the prompt context before model execution.
Across 36 controlled sessions spanning 18 task pairs on Claude Sonnet 5, on one target helper, duplication went from 4 of 4 runs without injection to 0 of 2 with it, too small a sample to generalize. However, across all 18 pairs, mechanical code slop decreased from 6 events to 1 event, yielding a paired McNemar test p = 0.062.
Because p = 0.062 does not cross the standard 0.05 alpha threshold, this reduction cannot be claimed as a statistically significant finding. A blind evaluation by an independent judge similarly showed an inconclusive difference (6 events ON vs 9 events OFF, paired McNemar test p = 0.51).