Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference
Deterministic routing removed planner tokens. Memory accuracy comparisons use sealed Bank Slice results.
Planner Tokens
45,348 -> 0
In bank_s2 (50 Qs), deterministic routing eliminated all 45,348 planner tokens (152,239 tokens across 180 benchmark Qs).
Pointer Latency
8.69s -> 0.68s
p50 pointer lookup latency accelerated 12.8x with deterministic query planning with no detected accuracy difference (McNemar exact test, p = 1.0).
Sealed Bank Slice 2 Accuracy
46/50 vs 35/50
On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability.
Paired Significance
p = 0.0034
On sealed Bank Slice 2, the exact McNemar test recorded 12 H-only correct versus 1 B0-only correct.
Empirical Findings Table
Telemetry from deterministic planning (180 Qs) and holdout memory evaluation (80 Qs)| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Planner Tokens | 45,348 -> 0 | In bank_s2 (50 Qs), deterministic routing eliminated all 45,348 planner tokens (152,239 tokens across 180 benchmark Qs). |
| Pointer Latency | 8.69s -> 0.68s | p50 pointer lookup latency accelerated 12.8x with deterministic query planning with no detected accuracy difference (McNemar exact test, p = 1.0). |
| Sealed Bank Slice 2 Accuracy | 46/50 vs 35/50 | On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability. |
| Paired Significance | p = 0.0034 | On sealed Bank Slice 2, the exact McNemar test recorded 12 H-only correct versus 1 B0-only correct. |
Engineering Action Directives
- Deterministic rule-based query planning can eliminate LLM planner token overhead with no detected accuracy difference on developer session questions.
- Enabling persistent memory indexes significantly improves retrieval accuracy over naive log searching, but uses its token budget reading structured turns rather than reducing per-query token counts.
Methodology Boundaries & Limits
- Benchmark domain: Evaluated specifically on developer engineering session histories and command logs; may not generalize to open-domain casual chat.
- Fixed question banks: Evaluated across 180 deterministic planning questions and an 80-question holdout set; larger longitudinal datasets are required for full lifespan modeling.