Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations

Detailed Analysis & Field Observations

In autonomous software engineering, agents frequently need to know what happened in earlier sessions: which design was selected, why a library was replaced, what instruction the operator gave, or what error occurred three weeks ago. When context windows span millions of tokens, a common assumption is that agents can simply grep through raw session logs on demand. In empirical testing across eight session-memory question sets, diagnostic reanalyses and planner reruns, we measured where that assumption holds, where it fails, and what structured semantic memory actually accomplishes.

This living paper (version 1) consolidates our experimental campaign across selfmem, integrating findings from our five previously published memory studies: semantic-memory-83-percent-fewer-tokens, session-memory-recall, benchmark-confinement-tax, speculative-prefetch-agent-recall, and deconstructing-agent-retrieval-latency. Data collected September 28 to October 3, 2026; synthesized October 2026.

Our earliest baseline, P0, indexed 778 sessions into 6,758 events. These exploratory evaluations lacked mechanical confinement; their accuracy scores are omitted from public reporting.

In P1, we introduced hybrid hints-to-grep (Arm H), where memory surfaces candidate session pointers, timestamps and line excerpts to guide a targeted grep pass.

In P2 and Holdout-80, we observed hint anchoring: the answering model sometimes relied on an intermediate narrative summary instead of inspecting subsequent turns. A pooled Pareto analysis attributed 28.6% of 42 misses to hint anchoring. The analysis estimated that pointer-only hints could recover 8 to 9 misses; that estimate was not measured on the original questions.

In Holdout-P7, memory was restricted to factual pointers rather than narrative conclusions. G lacked access to the Codex source store. These exploratory results do not provide a confined accuracy comparison.

Later evaluations used mechanically sealed stages; earlier exploratory runs lacked those guarantees. In session memory, H consumed about 12% more median tokens than the baseline. Unlike single-document benchmarks where semantic memory reduces prompt tokens by ~83%, session memory recall is an accuracy intervention, not a token reduction mechanism.

In sealed Bank Slice 1, H scored 43/50 versus B0 47/50 (McNemar p = 0.22). Of 50 gold answers, 43 cited structured session handoff logs available to B0. In sealed Bank Slice 2, every gold answer cites a raw transcript, and 17/50 cite only transcripts. On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability. B0 performed poorly on these categories: 0/6 which-session, 1/6 operator statements and 1/5 cross-model.

Finally, in DETPLAN-EVAL-2 (180 questions pooled across P7H, Bank Slice 1, and Bank Slice 2), we tested whether an unconstrained LLM query planner could surpass deterministic query routing. The October 2 run recorded 152,239 planner tokens versus zero, with both arms scoring 135/180. The October 3 unlimited-planner rerun scored 141/180 versus 142/180 and measured pointer p50 latency of 8.8s versus 0.67s. On these reused question sets, deterministic routing reduced pointer latency, with no detected accuracy difference.