Published by Sean Carr
Half the Tokens: Code-Search Hints Cut Agent Token Use 51% on a 30-Question Test
Overnight benchmark on real engineering queries: pre-computed symbol concordance hints and inlined snippets cut agent exploration tokens in half.
At a Glance
Token Reduction
51% - 55%
Arm H consumed 57,438 tokens vs 118,105 for agentic grep on CODERET-NIGHT; 40,737 vs 91,844 on CODERET-E3.
Top-5 Recall
0.94 - 1.00
Arm H achieved 1.00 recall@5 on the 30-question suite (vs 0.97 for grep and 0.67 for pure AST); holdout recall was 0.94-0.98.
File Open Reduction
76% Fewer Opens
Inlined snippets reduced whole-file opens from 1.55 to 0.38 files per question.
Zero-Call Answers
52.5% of Queries
52.5% of questions were answered directly from inlined search snippets without follow-up tool calls.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Token Reduction | 51% - 55% | Arm H consumed 57,438 tokens vs 118,105 for agentic grep on CODERET-NIGHT; 40,737 vs 91,844 on CODERET-E3. |
| Top-5 Recall | 0.94 - 1.00 | Arm H achieved 1.00 recall@5 on the 30-question suite (vs 0.97 for grep and 0.67 for pure AST); holdout recall was 0.94-0.98. |
| File Open Reduction | 76% Fewer Opens | Inlined snippets reduced whole-file opens from 1.55 to 0.38 files per question. |
| Zero-Call Answers | 52.5% of Queries | 52.5% of questions were answered directly from inlined search snippets without follow-up tool calls. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
When coding agents explore an unfamiliar codebase, they typically spend 50k to 100k+ tokens running grep searches and opening whole files. In overnight benchmarks across fresh out-of-sample and real-ask holdouts, pre-computed symbol concordance hints and inlined snippets cut agent token expenditure by 51% with 1.00 recall@5 on the 30-question test (0.94-0.98 on holdout).
Method
Evaluated three search architectures: Arm A (Agentic Grep), Arm E2 (Pure AST Index), and Arm H (Hybrid Concordance Hints + Snippets).
Tested on two frozen benchmark sets: CODERET-NIGHT (30 out-of-sample technical questions) and CODERET-E3 (real-ask holdout set).
Evaluated snippet inlining in CODERET-SNIP (40 questions), tracking file open rates, token expenditure, and follow-up tool calls.
Index generation mapped 32,649 symbols across 1,619 repository files in 8.4 seconds.
What We Measured
Token expenditure, latency, and recall across code retrieval strategies
Token Reduction
51% - 55%
Arm H consumed 57,438 tokens vs 118,105 for agentic grep on CODERET-NIGHT; 40,737 vs 91,844 on CODERET-E3.
Top-5 Recall
0.94 - 1.00
Arm H achieved 1.00 recall@5 on the 30-question suite (vs 0.97 for grep and 0.67 for pure AST); holdout recall was 0.94-0.98.
File Open Reduction
76% Fewer Opens
Inlined snippets reduced whole-file opens from 1.55 to 0.38 files per question.
Zero-Call Answers
52.5% of Queries
52.5% of questions were answered directly from inlined search snippets without follow-up tool calls.
Experimental Notes & Observability Details
- Pure AST search (Arm E2) consumed zero prompt tokens but had low top-1 recall (0.33) because it failed on informal keyword queries.
- Hybrid search (Arm H) combines compact AST symbols with contextual line snippets, giving the model immediate evidence to satisfy the prompt.
What We Think It Suggests
- Providing pre-computed symbol concordance hints and compact inlined code snippets allows agents to locate relevant definitions with substantially fewer exploratory tool calls.
- Directly displaying target code spans in initial search outputs eliminates over half of follow-up whole-file read operations.
Limits
- Evaluated on Python and TypeScript codebases: AST indexing relies on clean grammar parsing; highly dynamic languages or broken syntax trees degrade concordance accuracy.
- Index overhead: Building symbol indices requires background CPU indexing time (8.4s on 1,619 files); very large million-line monorepos will require partitioned incremental indexing.
Reproduce It (Data & Benchmarks)
- Evaluation suites: CODERET-NIGHT (30 out-of-sample questions), CODERET-E3 (real-ask holdout), CODERET-SNIP (40 inlined snippet questions).
- Comparison arms: Agentic grep (Arm A), pure AST index (Arm E2), hybrid hints + snippets (Arm H).
- Codebase: 32,649 symbols across 1,619 files.
- Raw metrics: Arm A 118,105 tokens / 15.9s, Arm H 57,438 tokens / 10.9s (51% reduction, 1.00 recall@5) (reproducibility package coming).
Data and prompt templates only; no proprietary code is published.
Detailed Analysis & Field Observations
In autonomous coding agents, exploration is the largest hidden source of token burn. Before writing a single line of code, an agent routinely issues dozens of grep commands, opens 500-line files to find a single method signature, and exhausts its context window on irrelevant implementation details.
To solve this, we developed a hybrid code retrieval architecture combining pre-computed Abstract Syntax Tree (AST) concordance hints with inlined source snippets. When an agent searches for a concept, the tool returns the exact symbol name, file path, line numbers, and a compact 5-to-10 line snippet of the definition.
We benchmarked this system against plain agentic grep (where the model formulates regex queries and inspects whole files) across two frozen evaluation suites.
On the fresh 30-question suite, plain agentic grep (A) achieved s@1 0.90 and s@5 0.97 while consuming 118,105 tokens at 15.9s p50 latency. Arm H matched s@1 at 0.90, raised s@5 to a perfect 1.00 (100%), and burned only 57,438 tokens (49% of A, a 51% token reduction) in 10.9s.
On the real-ask holdout set (CODERET-E3), the pattern held: agentic grep consumed 91,844 tokens, while Arm H consumed 40,737 tokens (a 55% reduction) with 0.98 recall@5.
Evaluating inlined snippets directly (CODERET-SNIP, 40 questions) showed that agents answered 52.5% of questions immediately without issuing any follow-up file read commands. Whole-file opens dropped from 1.55 to 0.38 files per query, reducing p50 retrieval latency from 8.75s to 6.0s.
In this test, providing structured symbol hints cuts search token burn in half while maintaining top recall.