Published by Sean Carr
A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample
Rules mined from an agent's own search traces matched it on the questions they came from and collapsed on new ones.
At a Glance
Out-of-sample recall@5, compiled rules
0.45
Out of sample: E recall@5 0.45 and recall@10 0.45; hybrid F recall@5 0.45 on both runs vs the agent's 0.975 mean.
In-sample recall@5, compiled rules
0.94
Phase 1: compiled arm E in-sample recall@5 0.94 (reported as a development number, since E was tuned on these questions).
In-sample recall@5, agent
0.94
Phase 0 (50 in-sample questions): agent A recall@5 0.94 and 0.96 over 2 samples; BM25 B 0.54; router C 0.72.
Hybrid F tokens vs agent
12.7%
F used 11,169 tokens per question out of sample, 12.7% of the agent's 88,075.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Out-of-sample recall@5, compiled rules | 0.45 | Out of sample: E recall@5 0.45 and recall@10 0.45; hybrid F recall@5 0.45 on both runs vs the agent's 0.975 mean. |
| In-sample recall@5, compiled rules | 0.94 | Phase 1: compiled arm E in-sample recall@5 0.94 (reported as a development number, since E was tuned on these questions). |
| In-sample recall@5, agent | 0.94 | Phase 0 (50 in-sample questions): agent A recall@5 0.94 and 0.96 over 2 samples; BM25 B 0.54; router C 0.72. |
| Hybrid F tokens vs agent | 12.7% | F used 11,169 tokens per question out of sample, 12.7% of the agent's 88,075. |
| Router C recall@10 | 0.84 | C recall@10 0.84 vs A recall@5 0.85: the locked prediction missed by 0.01. C's injected payload was 7.9k tokens vs about 109k for A (7%). |
| Agent median latency | 16 s | A's median latency was 16 s (p95 42-52 s), against a locked prediction of over 20 s, so that prediction failed. |
| Who-calls out of sample | 0 of 4 | Who-calls questions out of sample: 0 of 4 for E. |
| Hybrid recall@1 within tool cap | 1.00 | When F answered within its tool cap, recall@1 was 1.00 in every run (post-hoc observation, not locked). |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
We tried to replace an agent's grep-and-read code search with deterministic retrieval rules mined from that agent's own traces. On the 50 questions the rules were built from, symbol recall@5 was 0.94. On 20 out-of-sample questions written by an independent pass, it fell to 0.45, while the agentic baseline scored 0.975. We had pre-registered an overfit gap of at least 0.10; the observed gap was 0.49.
Method
Data collected Sep 28, 2026; written up October 2026.
September 28, 2026, against a frozen snapshot of our own repository (Python and C++).
Gold sets were frozen in git before any method run: 50 in-sample questions across six types (where-defined, who-calls, how-tested, what-writes, config-origin, hybrid) and 20 out-of-sample questions from an independent author the method designer had not read.
Arms: A, an agent (Claude Sonnet via headless CLI, Grep/Glob/Read only); B, BM25 over symbols; C, B plus a rule router and one call-graph hop; E, the agent's recurring search moves compiled into deterministic patterns, no model; F, E's top 10 handed to one model call with a capped extra tool round.
Scoring was symbol-level recall@k by a single scorer that alone reads gold.
Predictions for both phases were committed before their scored runs.
What We Measured
In-sample versus out-of-sample code retrieval, compiled rules versus agent
Out-of-sample recall@5, compiled rules
0.45
Out of sample: E recall@5 0.45 and recall@10 0.45; hybrid F recall@5 0.45 on both runs vs the agent's 0.975 mean.
In-sample recall@5, compiled rules
0.94
Phase 1: compiled arm E in-sample recall@5 0.94 (reported as a development number, since E was tuned on these questions).
In-sample recall@5, agent
0.94
Phase 0 (50 in-sample questions): agent A recall@5 0.94 and 0.96 over 2 samples; BM25 B 0.54; router C 0.72.
Hybrid F tokens vs agent
12.7%
F used 11,169 tokens per question out of sample, 12.7% of the agent's 88,075.
Router C recall@10
0.84
C recall@10 0.84 vs A recall@5 0.85: the locked prediction missed by 0.01. C's injected payload was 7.9k tokens vs about 109k for A (7%).
Agent median latency
16 s
A's median latency was 16 s (p95 42-52 s), against a locked prediction of over 20 s, so that prediction failed.
Who-calls out of sample
0 of 4
Who-calls questions out of sample: 0 of 4 for E.
Hybrid recall@1 within tool cap
1.00
When F answered within its tool cap, recall@1 was 1.00 in every run (post-hoc observation, not locked).
What We Think It Suggests
- Rules mined from an agent's traces on a question set appear to encode that set's vocabulary. The out-of-sample questions were phrased vaguely, without code identifiers, so the compiled seeds had nothing to match and the call-graph hop had nothing to hop from.
- The step the compiled rules dropped was the one these questions needed: read a result, find a name in it, and search for that name. This failure shaped the next round of the work, which used independent fresh question sets and is reported separately on the Lab.
Limits
- Small sets: 50 in-sample and 20 out-of-sample questions, with only 4 who-calls questions out of sample.
- One repository, one agent model; two gold answers were flagged as plausibly incomplete and left frozen, which may understate the agent's in-sample score.
- The router rules in Phase 0 were written by the gold author, so they are generic but not independent of the gold set.
Detailed Analysis & Field Observations
We tried to replace an agent's grep-and-read code search with deterministic retrieval rules mined from that agent's own traces. On the 50 questions the rules were built from, symbol recall@5 was 0.94. On 20 out-of-sample questions written by an independent pass, it fell to 0.45, while the agentic baseline scored 0.975. We had pre-registered an overfit gap of at least 0.10; the observed gap was 0.49.
Data collected Sep 28, 2026; written up October 2026.