Published by Sean Carr
Sub-Second Agent Memory Recall via Asynchronous Speculative Prefetch
Spawning a detached background prefetch worker on prompt submission dropped memory recall latency from 15-25 seconds to 0.29 seconds.
At a Glance
Baseline Memory Recall Latency
15 - 25 s
Sequential end-to-end memory retrieval took 15 to 25 seconds for query planning, search, and answer synthesis.
Prefetched Memory Recall Latency
0.29 s
Cache-served responses returned in 0.29s wall time (annotated [prefetched 11s ago]).
Latency Reduction Factor
Up to 86x
Prefetch cache hits accelerated interactive CLI memory queries by up to 86x.
Cloud API Expenditure
$0.00
Zero dollars spent on cloud model APIs during prefetch execution and unit testing, as embeddings ran through local Ollama.
| Telemetry Metric | Measured Value | Context & Observability |
|---|---|---|
| Baseline Memory Recall Latency | 15 - 25 s | Sequential end-to-end memory retrieval took 15 to 25 seconds for query planning, search, and answer synthesis. |
| Prefetched Memory Recall Latency | 0.29 s | Cache-served responses returned in 0.29s wall time (annotated [prefetched 11s ago]). |
| Latency Reduction Factor | Up to 86x | Prefetch cache hits accelerated interactive CLI memory queries by up to 86x. |
| Cloud API Expenditure | $0.00 | Zero dollars spent on cloud model APIs during prefetch execution and unit testing, as embeddings ran through local Ollama. |
| Test Suite Verification | 24 / 24 Tests Passed | Prefetch unit suite passed in 2.30s; full repository suite passed with 154 tests and zero regressions in 100.58s. |
Published by Sean Carr
View & print executive brief version →Executive Summary & Core Finding
Retrieving historical engineering context in interactive agent sessions typically requires 15 to 25 seconds for query planning, database filtering, and response synthesis. By implementing an asynchronous, stdlib-only prompt interception hook that spawns a detached background prefetch worker with local vector embeddings, we reduced memory recall latency to 0.29 seconds on past-work queries with zero cloud model spend.
Method
We implemented a prompt submission hook (prefetch_hook.py) using only Python standard library components to keep execution overhead minimal.
The hook evaluates incoming user prompts against regex patterns for historical queries (such as 'where did we leave', 'what was the decision on', or project names) while verifying rate limits (maximum 30 per hour) and daily budget headroom.
For qualifying prompts, the hook spawns a detached background worker process (prefetch_worker.py) that executes concurrently with other terminal activity.
The worker computes a query embedding using a resident local Ollama embedding service, executes the full retrieval pipeline, and stores the structured output in a session-keyed cache file.
When the user or agent invokes the memory command (selfmem.cmd ask2), the CLI computes the prompt embedding and performs a cosine similarity lookup against the cache.
If the cosine similarity is at or above 0.85, the CLI returns the cached answer immediately. If specialized flags (--json, --answer, --pointers) are passed, the cache is bypassed to avoid format mismatches.
We verified the architecture across 24 unit tests covering regex classification, detached process spawning, rate limits, cosine thresholds, cache invalidation, and full CLI execution.
What We Measured
Asynchronous speculative prefetch reduced interactive memory recall latency from 15-25 seconds to 0.29 seconds (up to 86x speedup) on past-work queries with zero cloud API spend.
Baseline Memory Recall Latency
15 - 25 s
Sequential end-to-end memory retrieval took 15 to 25 seconds for query planning, search, and answer synthesis.
Prefetched Memory Recall Latency
0.29 s
Cache-served responses returned in 0.29s wall time (annotated [prefetched 11s ago]).
Latency Reduction Factor
Up to 86x
Prefetch cache hits accelerated interactive CLI memory queries by up to 86x.
Cloud API Expenditure
$0.00
Zero dollars spent on cloud model APIs during prefetch execution and unit testing, as embeddings ran through local Ollama.
Test Suite Verification
24 / 24 Tests Passed
Prefetch unit suite passed in 2.30s; full repository suite passed with 154 tests and zero regressions in 100.58s.
Experimental Notes & Observability Details
- Prompt interception hook uses standard-library Python only to minimize startup overhead.
- Enforces safety through strict cap of 30 speculative prefetch executions per hour and daily budget headroom checks.
- Cosine similarity threshold of >= 0.85 on query embedding serves cached answers.
What We Think It Suggests
- The perceived delay in agent memory systems is largely an artifact of serial request execution. Speculatively running retrieval while the user interacts with the terminal eliminates waiting time without altering query results.
- Local embedding models running on resident hardware are sufficient to calculate semantic similarity for cache matching, removing both the latency and financial cost of cloud API calls for prefetch operations.
- Decoupling speculative retrieval into a detached process allows developers to retain full execution safety through rate limiters and budget checks that fail closed without interrupting the primary coding session.
Limits
- Pattern Matching Dependency: Speculative prefetch relies on heuristics that identify past-work query shapes; queries with non-standard phrasing will not trigger prefetch and fall back to the standard 15-25 second latency.
- Local Resource Utilization: Spawning background embedding and retrieval workers consumes local CPU and GPU cycles, which may cause contention on resource-constrained systems during heavy build operations.
- Specialized Flag Exclusion: Queries that require specialized output formats (such as raw JSON exports or explicit pointer listings) bypass the prefetch cache by design, requiring a fresh synchronous run.
- Hook Availability: The prefetch architecture requires terminal or agent harnesses that expose prompt submission lifecycle events.
Reproduce It (Data & Benchmarks)
- Prefetch hook implementation: prefetch_hook.py
- Background worker script: prefetch_worker.py
- Test suite: tests/test_prefetch.py (24 tests)
- Integration entrypoint: cli.py (lookup function in prefetch.py)
Data and prompt templates only; no proprietary code is published.
Detailed Analysis & Field Observations
In agentic pair programming, developers frequently ask historical context questions such as where a feature was left off or why a particular decision was made. Standard retrieval pipelines take 15 to 25 seconds because they sequentially plan queries, search session logs, and synthesize relevant context.
We built an asynchronous prefetch architecture that intercepts prompt submission events before model execution begins. A lightweight, standard-library hook identifies past-work prompts, spawns a detached background worker, embeds the query using a local Ollama model, and caches candidate memory answers.
When the developer or agent executes the corresponding memory CLI call, matching queries (cosine similarity >= 0.85) resolve immediately from cache in 0.29 seconds, achieving up to an 86x speedup with zero cloud API spend.
Current synthesis: Semantic Memory Living Paper v1.