Zero-Token Planning: Removing All Planner Tokens With No Detected Accuracy Difference

Detailed Analysis & Field Observations

Cutting planner tokens to zero

In typical agent memory retrieval pipelines, an LLM call is invoked simply to plan the query, parsing user intents, determining temporal bounds, and selecting index filters. This creates a hidden latency and token tax on every lookup before retrieval even starts.

We tested whether this planning step could be replaced entirely. By replacing the model that plans each memory lookup with deterministic rules, we evaluated 180 questions across three frozen benchmark suites (bank_s2, bank_s1, and p7h). The deterministic planner routed 84% to 98% of queries without an LLM call, dropping planner tokens to zero while cutting pointer lookup latency by ~13x (8.69 s down to 0.68 s median in bank_s2) at identical retrieval accuracy (McNemar exact test, p = 1.0 across all three suites).

Note that these figures isolate the prompt tokens and execution latency of the query planning step itself. We do not claim platform-wide figures for total context tokens saved by memory retention, as continuous longitudinal token savings remain the subject of active production measurement.

Memory on vs off: accuracy, not token savings

For public memory accuracy comparisons, we use sealed Bank Slice results. B0 had notes, handoff logs and memory files, without raw transcripts.

On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability.

On sealed Bank Slice 1, H scored 43/50 versus B0 47/50 (McNemar p = 0.22); 43/50 gold answers cited handoff logs available to B0.

In the earlier P7H token measurement, H consumed about 12% more median tokens (129k versus 115k). Its accuracy scores are omitted because that run lacked mechanical confinement.

Pre-registration records predictions before scoring. Earlier unsealed accuracy results are omitted from public reporting.

Longitudinal recall across 771 development sessions

As AI coding agents take on multi-day software engineering projects, their ability to accurately recall past decisions, architectural rulings, and intermediate test results becomes critical. In practice, off-the-shelf vector embedding search performs poorly on technical session histories. Simple similarity search frequently confuses an early rejected proposal with the final ratified architecture, or retrieves an abandoned worktree thread instead of the active mainline commit.

To benchmark and evaluate our internal memory system (selfmem), we constructed a formal evaluation harness across 771 real development sessions. The benchmark comprised a 60-question development set and a frozen 40-question holdout set with IDs fixed in version control before the run, testing six distinct question archetypes: state counts, latest values, explicit supervisor instructions, task status, session attribution, and architectural rationale.

The development and holdout evaluations explored model backend differences. Their accuracy scores are omitted because these exploratory runs lacked mechanical confinement.

More important than raw scores was our systematic forensic breakdown of all 40 recorded errors. Interestingly, zero misses were attributable to store staleness, as all relevant session transcripts were present in the indexing database. Instead, the largest error category was reasoning failure (15 cases, 37.5%), where the agent retrieved the exact transcript turn containing the answer but misread the status or conflated separate subagent tasks.

The second major failure mode was retrieval omission (10 cases, 25.0%), where keyword search failed to bridge terminology shifts across sessions. Wrong-thread attribution accounted for 5 misses (12.5%), where the system retrieved an exploratory scaffold branch rather than the finalized implementation thread. Furthermore, 9 cases (22.5%) were uncovered where the 'ground truth' benchmark itself had grown stale due to downstream commits.

These findings indicate on this sample that multi-agent memory cannot rely on flat semantic embeddings alone. High-precision engineering recall requires explicit thread lineage tracking, timestamp-anchored state progression, and independent verification of ground-truth citations.

Current synthesis: Semantic Memory Living Paper v1.