A Code-Retrieval Index That Scored 0.94 In-Sample Scored 0.45 Out of Sample

Rules mined from an agent's own search traces matched it on the questions they came from and collapsed on new ones.

Core Finding:We tried to replace an agent's grep-and-read code search with deterministic retrieval rules mined from that agent's own traces. On the 50 questions the rules were built from, symbol recall@5 was 0.94. On 20 out-of-sample questions written by an independent pass, it fell to 0.45, while the agentic baseline scored 0.975. We had pre-registered an overfit gap of at least 0.10; the observed gap was 0.49.

Out-of-sample recall@5, compiled rules

0.45

Out of sample: E recall@5 0.45 and recall@10 0.45; hybrid F recall@5 0.45 on both runs vs the agent's 0.975 mean.

In-sample recall@5, compiled rules

0.94

Phase 1: compiled arm E in-sample recall@5 0.94 (reported as a development number, since E was tuned on these questions).

In-sample recall@5, agent

0.94

Phase 0 (50 in-sample questions): agent A recall@5 0.94 and 0.96 over 2 samples; BM25 B 0.54; router C 0.72.

Hybrid F tokens vs agent

12.7%

F used 11,169 tokens per question out of sample, 12.7% of the agent's 88,075.

Empirical Findings Table

In-sample versus out-of-sample code retrieval, compiled rules versus agent

Telemetry Metric	Measured Value	Context & Observability
Out-of-sample recall@5, compiled rules	0.45	Out of sample: E recall@5 0.45 and recall@10 0.45; hybrid F recall@5 0.45 on both runs vs the agent's 0.975 mean.
In-sample recall@5, compiled rules	0.94	Phase 1: compiled arm E in-sample recall@5 0.94 (reported as a development number, since E was tuned on these questions).
In-sample recall@5, agent	0.94	Phase 0 (50 in-sample questions): agent A recall@5 0.94 and 0.96 over 2 samples; BM25 B 0.54; router C 0.72.
Hybrid F tokens vs agent	12.7%	F used 11,169 tokens per question out of sample, 12.7% of the agent's 88,075.
Router C recall@10	0.84	C recall@10 0.84 vs A recall@5 0.85: the locked prediction missed by 0.01. C's injected payload was 7.9k tokens vs about 109k for A (7%).
Agent median latency	16 s	A's median latency was 16 s (p95 42-52 s), against a locked prediction of over 20 s, so that prediction failed.
Who-calls out of sample	0 of 4	Who-calls questions out of sample: 0 of 4 for E.
Hybrid recall@1 within tool cap	1.00	When F answered within its tool cap, recall@1 was 1.00 in every run (post-hoc observation, not locked).

Engineering Action Directives

Rules mined from an agent's traces on a question set appear to encode that set's vocabulary. The out-of-sample questions were phrased vaguely, without code identifiers, so the compiled seeds had nothing to match and the call-graph hop had nothing to hop from.
The step the compiled rules dropped was the one these questions needed: read a result, find a name in it, and search for that name. This failure shaped the next round of the work, which used independent fresh question sets and is reported separately on the Lab.

Methodology Boundaries & Limits

Small sets: 50 in-sample and 20 out-of-sample questions, with only 4 who-calls questions out of sample.
One repository, one agent model; two gold answers were flagged as plausibly incomplete and left frozen, which may understate the agent's in-sample score.
The router rules in Phase 0 were written by the gold author, so they are generic but not independent of the gold set.