Half the Tokens: Code-Search Hints Cut Agent Token Use 51% on a 30-Question Test

Overnight benchmark on real engineering queries: pre-computed symbol concordance hints and inlined snippets cut agent exploration tokens in half.

Core Finding:When coding agents explore an unfamiliar codebase, they typically spend 50k to 100k+ tokens running grep searches and opening whole files. In overnight benchmarks across fresh out-of-sample and real-ask holdouts, pre-computed symbol concordance hints and inlined snippets cut agent token expenditure by 51% with 1.00 recall@5 on the 30-question test (0.94-0.98 on holdout).

Token Reduction

51% - 55%

Arm H consumed 57,438 tokens vs 118,105 for agentic grep on CODERET-NIGHT; 40,737 vs 91,844 on CODERET-E3.

Top-5 Recall

0.94 - 1.00

Arm H achieved 1.00 recall@5 on the 30-question suite (vs 0.97 for grep and 0.67 for pure AST); holdout recall was 0.94-0.98.

File Open Reduction

76% Fewer Opens

Inlined snippets reduced whole-file opens from 1.55 to 0.38 files per question.

Zero-Call Answers

52.5% of Queries

52.5% of questions were answered directly from inlined search snippets without follow-up tool calls.

Empirical Findings Table

Token expenditure, latency, and recall across code retrieval strategies

Telemetry Metric	Measured Value	Context & Observability
Token Reduction	51% - 55%	Arm H consumed 57,438 tokens vs 118,105 for agentic grep on CODERET-NIGHT; 40,737 vs 91,844 on CODERET-E3.
Top-5 Recall	0.94 - 1.00	Arm H achieved 1.00 recall@5 on the 30-question suite (vs 0.97 for grep and 0.67 for pure AST); holdout recall was 0.94-0.98.
File Open Reduction	76% Fewer Opens	Inlined snippets reduced whole-file opens from 1.55 to 0.38 files per question.
Zero-Call Answers	52.5% of Queries	52.5% of questions were answered directly from inlined search snippets without follow-up tool calls.

Engineering Action Directives

Providing pre-computed symbol concordance hints and compact inlined code snippets allows agents to locate relevant definitions with substantially fewer exploratory tool calls.
Directly displaying target code spans in initial search outputs eliminates over half of follow-up whole-file read operations.

Methodology Boundaries & Limits

Evaluated on Python and TypeScript codebases: AST indexing relies on clean grammar parsing; highly dynamic languages or broken syntax trees degrade concordance accuracy.
Index overhead: Building symbol indices requires background CPU indexing time (8.4s on 1,619 files); very large million-line monorepos will require partitioned incremental indexing.