Semantic Memory Living Paper: An Empirical Synthesis Across Session Memory Evaluations

A synthesis of eight session-memory question sets, diagnostic reanalyses and planner reruns: structured event extraction, retrieval anchoring, sealed holdouts with source-access limitations, and zero-token deterministic routing.

Core Finding:This living paper (v1) synthesizes eight session-memory question sets, diagnostic reanalyses and planner reruns. Data collected September 28 to October 3, 2026; synthesized October 2026. On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability. The October 2 run recorded 152,239 planner tokens versus zero, with both arms scoring 135/180. The October 3 unlimited-planner rerun scored 141/180 versus 142/180 and measured pointer p50 latency of 8.8s versus 0.67s. On these reused question sets, deterministic routing reduced pointer latency, with no detected accuracy difference.

LongMemEval Token Drop

82.8% Fewer Tokens

On 100 questions over ~115k-token histories, structured events matched full context (88/100) using 19.5k vs 113.6k tokens.

Bank Slice 2 Accuracy Gain

46/50 vs 35/50

On Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034).

Planner Tokens Eliminated

152,239 -> 0 Tokens

The October 2 run recorded 152,239 planner tokens versus zero, with both arms scoring 135/180. The October 3 unlimited-planner rerun scored 141/180 versus 142/180 and measured pointer p50 latency of 8.8s versus 0.67s.

Pointer Lookup Latency

8.8s -> 0.67s

Deterministic routing accelerated p50 pointer lookup from 8.8 s to 0.67 s compared to a Gemini planner with its time limit lifted.

Empirical Findings Table

Telemetry across eight session-memory question sets, diagnostic reanalyses, and planner reruns

Telemetry Metric	Measured Value	Context & Observability
LongMemEval Token Drop	82.8% Fewer Tokens	On 100 questions over ~115k-token histories, structured events matched full context (88/100) using 19.5k vs 113.6k tokens.
Bank Slice 2 Accuracy Gain	46/50 vs 35/50	On Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034).
Planner Tokens Eliminated	152,239 -> 0 Tokens	The October 2 run recorded 152,239 planner tokens versus zero, with both arms scoring 135/180. The October 3 unlimited-planner rerun scored 141/180 versus 142/180 and measured pointer p50 latency of 8.8s versus 0.67s.
Pointer Lookup Latency	8.8s -> 0.67s	Deterministic routing accelerated p50 pointer lookup from 8.8 s to 0.67 s compared to a Gemini planner with its time limit lifted.
Transcript Retrieval Gap	92% vs 70%	On Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034).
Confounded Log Slice 1	86% vs 94%	In Bank Slice 1, raw grep beat memory (47/50 vs 43/50, McNemar p = 0.22) because 43/50 gold answers lived in handoff logs.
Error Noise Floor	8.0% (16 / 200)	Pareto analysis of 200 holdout questions identified 10 bad gold answers and 6 ambiguous questions forming an unreachable noise floor.
Pre-Retrieval LLM Triage	0 / 6 Recall	A lightweight model caught 0 of 6 ambiguous questions. Rules written after inspecting the benchmark flagged all 6 ambiguous questions and falsely flagged 19/158 H-correct questions (12%). This was an in-sample result.

Engineering Action Directives

Semantic event extraction solves different problems depending on conversational structure. On long conversation histories (LongMemEval), structured events yield massive prompt token reductions (82.8%). Across hundreds of disparate developer sessions, structured memory functions primarily as a precision anchor; memory reduced some historical-recall errors in these evaluations.
A confident incorrect memory hint is worse than no hint. In our Pareto evaluation, memory hints contributed to 28.6% of misses because the answering model anchored on the hint rather than verifying the source. The Pareto analysis estimated that pointer-only hints could recover 8 to 9 misses; that estimate was not measured on the original questions.
Benchmark design and source access affect interpretation. In sealed Bank Slice 1, B0 scored 47/50 versus H 43/50 (McNemar p = 0.22); 43/50 gold answers cited handoff logs. On sealed Bank Slice 2, H scored 46/50 versus B0 35/50 (exact McNemar test, p = 0.0034). B0 was a notes, handoff-log and memory-file grep baseline without raw transcripts. The arms had different source access, so observed gains include corpus availability.
On these reused question sets, deterministic routing reduced pointer latency, with no detected accuracy difference.

Methodology Boundaries & Limits

Single developer organization corpus: Benchmarks reflect MeshTen software engineering sessions; organizations with different project naming or logging conventions may observe differing noise floors.
Model family specificity: Answering agents utilized Gemini and Claude models; smaller open-weight models may exhibit different sensitivity to memory hint formats.
Ingestion compute investment: Structured event extraction requires upfront compute during session finalization; it is inefficient for ephemeral conversations queried only once.