The Hidden Infrastructure Tax in Multi-Agent Coding: An Estimated Replay of 39,526 Calls and an RTX 5090 Lab

An estimated 39,526-call telemetry replay and a 6-arm RTX 5090 laboratory study showing how hook overhead, serial test gates, and background prefetch storms took more summed call time than the frontier models.

Core Finding:When multi-agent engineering sessions across Claude Code, Codex, and Gemini feel slow, the default assumption is that the frontier reasoning model is the bottleneck. To test that assumption, we audited 39,526 raw telemetry records (121.82 hours of cumulative summed call duration across concurrent agent lanes and background workers over 14 days) and ran a 6-arm controlled laboratory study (WF-GPU-LAB-001 v3_optimized) on an NVIDIA GeForce RTX 5090 (32 GB VRAM, Blackwell compute capability sm_120) and AMD Ryzen 7 9800X3D workstation. The frontier model was not the primary bottleneck: interactive velocity was being strangled by per-turn and per-tool hook spawns, serial test gates, cold interpreter imports, and an invisible 31.9-hour/week background cloud-LLM prefetch storm. Separating attribution cleanly: roughly 99% of the 121.82-hour 14-day cumulative time reduction comes from software and algorithmic fixes (filtering subagent transcripts out of background ingest, switching prefetch to ask2 --pointers, reordering leftoff ahead of temporal_truth, deterministic regex/YAML rules, and the CPU-resident 4-worker WarmPytestDaemon on the Ryzen 7 9800X3D), while roughly 1 hour of logged time (plus interactive tail latency) is directly accelerated by the RTX 5090 GPU (pinned nomic-embed-text embeddings + BLAS search at 8.42x, local qwen3:4b Stage-2 PHI screening, and qwen3:14b hybrid freetext checklist extraction). Together, non-coding session overhead dropped by 10x to 19x while in-sample micro-task accuracy reached 99.46% Micro F1 on brief-to-checklist extraction and 100.0% on a 100-case synthetic HIPAA PHI set, where the two-stage local cascade matched 100/100 with rules tuned in-sample. The CEO approved the local path on 2026-10-15, so the two-stage cascade (regex, then warm qwen3:4b) is now the default PHI admission check, with Claude Haiku as the fallback when the local model is unavailable or its answer does not parse. On the live code path, warm qwen3:4b answered 96 of the same 100 synthetic cases with 0 false negatives and 0 false positives at a 393 ms median; the 100/100 lab result was on a synthetic, in-sample-tuned set, and an out-of-sample check is still planned.

14-Day Telemetry Replay (Estimated)

121.82 hrs → 36.59 min

ESTIMATED TIME: 39,526 logged calls across 14 days, replayed at measured local latencies, model to under an hour of execution (55.36 min in Hybrid 14B mode)

Warm Pytest Worker Pool

10.36x speedup

Parallel-sharded 4-worker daemon cut median unit test runs from 664.07 ms to 63.15 ms (15.16x on single files)

Brief-to-Checklist F1

99.46% Micro F1

Hybrid repo-grounded deterministic + filtered qwen3:14b (367/370 oracle items vs 59.98% baseline)

5090 Canary Routing & PHI

100.0% accuracy

1,350/1,350 multi-terminal canary isolation (0 cross-talk) and 100/100 on a 100-case synthetic PHI set at 0.03 ms (rules tuned in-sample)

Empirical Findings Table

Across 39,526 replayed telemetry records and 6 laboratory arms, software/algorithmic fixes (~99% of cumulative time saved) plus warm RTX 5090 GPU acceleration (~1 hour of logged time plus interactive tail latency) cut 121.82 hours of summed concurrent call duration to an estimated 36.59 minutes (Fast Mode) or 55.36 minutes (Hybrid 14B Mode).

Telemetry Metric	Measured Value	Context & Observability
H1 Concurrent Routing (N=1,350)	1,350/1,350 (100.0%)	Zero cross-talk (Clopper-Pearson 95% CI [0.0, 0.2729]%) across C in {1, 2, 4, 8} on qwen3-coder:30b, qwen3:14b, and qwen3:4b after disabling qwen3 thinking preambles.
H2 Embed + 5,600-Vec Search (N=30)	8.42x	End-to-end speedup (77.42 ms vs 656.82 ms p50, 95% CI [8.19, 8.65]x, Wilcoxon W = 0.0, p = 1.86e-09): 7.38x embedding speedup + 25.44x BLAS search speedup with 98.75% top-5 parity.
H3A HIPAA PHI Screen (N=100, Synthetic)	100.0% accuracy	Arm 0+ hardened regex achieved 100/100 accuracy (F1 = 1.0000) at 0.03 ms p50; Arm 4 two-stage cascade (Regex -> Warm qwen3:4b) achieved 100/100 with 0.04 ms negative p50 (88% Stage-1 fast exit, McNemar p = 0.015625 vs baseline Arm 0 at 93.0%, in-sample refined rules).
H3B Brief-to-Checklist (N=50, 370 Oracle)	99.46% Micro F1	Arm 4 Hybrid (Arm 0+ + repo-filtered qwen3:14b) matched 367/370 items (99.73% precision, 99.19% recall, Macro F1 99.40%, McNemar p = 2.65e-22 vs baseline Arm 0 at 59.98% F1 and p = 3.73e-09 vs Arm 0+ at 95.35% F1).
H4 WarmPytestDaemon v3 (N=90)	10.36x speedup	CPU-resident Ryzen 7 9800X3D 4-worker parallel-sharded speedup vs cold pytest (63.15 ms vs 664.07 ms p50, 95% CI [9.91, 11.27]x, p = 1.74e-16): 15.16x on 1 file (42.29 ms), 10.18x on 3 files (63.15 ms), and 9.43x on 5 files (80.80 ms, 2.85x faster than v2 serial warm).
14-Day Telemetry Replay, Estimated (39,526 Rows)	121.82 hrs → 36.59 min	Row-by-row replay across 217 hourly ledger files (Oct 2-15, 2026) cut 121.82 hrs of cumulative summed concurrent call duration (including 29.92 hrs on Oct 12 across parallel lanes) to 36.59 min (0.61 hrs) in Arm 0+ Fast Mode or 55.36 min (0.92 hrs) in Arm 4 Hybrid 14B Mode (ESTIMATED TIME), modeling both eliminated/bypassed calls and accelerated local calls and removing 19,305 calls >= 29.5s (48.8% of all records).
20-Turn Session Non-Coding Overhead	10x-19x faster	Estimated non-coding wait time per 20-turn coding session dropped from 2.9-5.0 minutes down to 10-19 seconds (saving ~69.14 hrs/week combined foreground + background summed compute, ESTIMATED TIME).

Engineering Action Directives

Consolidate per-turn and per-tool hooks into a single process, debounce high-frequency shell touches, and keep local embedding and helper models resident in VRAM during active work.

DON'T

Spawn multiple cold Python interpreters, unbatched git status calls, or unthrottled background cloud LLM syntheses on every prompt and tool invocation.

Pair deterministic repo-existence and AST filters with a local helper model (Arm 4 Hybrid) or fast-exit regex cascade when structured contracts are absent.

DON'T

Use an unconstrained small or mid-sized LLM as a standalone path extractor without checking predicted paths and symbols against the repository index.

Keep a persistent, mtime_ns-aware, parallel-sharded pytest worker pool warm on loopback for iterative 1-to-5 file unit test runs.

DON'T

Pay a 664 ms cold Python + FastAPI + Pydantic import tax or pass -p no:xdist on every iterative edit verification.

Tie local GPU model residency and background test daemons to the developer console lifecycle so 100% of VRAM is freed automatically when work ends.

DON'T

Leave 20+ GB of models permanently pinned in VRAM after the coding session closes on a shared workstation.

Methodology Boundaries & Limits

In-sample rule tuning: The 100.0% PHI accuracy (Arm 0+ and Arm 4) and 99.46% checklist Micro F1 (Arm 0+ and Arm 4) include regex and AST rules refined after inspecting failure modes on the 100-case synthetic PHI set and the 50-brief repository corpus; prospective out-of-sample holdout testing is required to establish unseen generalization.
Software vs GPU attribution and summed concurrent time: The 121.82 hours of 14-day telemetry (including 29.92 hours on October 12) is cumulative summed call duration across concurrent agent lanes and background workers, not calendar wall-clock time; roughly 99% of the replayed reduction comes from software/algorithmic fixes and CPU-resident WarmPytestDaemon sharding, while roughly 1 hour of logged time (plus interactive tail latency) is directly accelerated by the RTX 5090 GPU.
Production status: the CEO approved the local qwen3:4b PHI classifier on 2026-10-15, and the two-stage local cascade (regex, then warm qwen3:4b) is now the default PHI admission check; the local supersede judge remains opt-in. Claude Haiku is the fallback when the local model is unavailable, times out (3 s), or returns an answer that does not parse, and Anthropic is covered under our BAA. Live-path check on the same 100 synthetic cases (production prompt with structured JSON output): the live regex stage flagged 49 of 50 positives and 0 of 50 negatives (the miss was an email address with a diagnosis and no name); warm qwen3:4b answered 96 of 100 calls locally with 0 false negatives, 0 false positives and 0 UNSURE at a 393 ms median (1.04 s p90), and the other 4 hit the 3 s timeout under shared-workstation load and would have gone to Haiku. Haiku on the same 100 marked 0 negatives as PHI but judged 24 of the 50 positives NOT_PHI and 10 UNSURE, consistent with the live prompt telling it to admit synthetic test data. The 100/100 lab result was on a synthetic, in-sample-tuned set, and an out-of-sample check is still planned. The PHI rows in the replay and weekly estimates price the local path; calls that fall back to Haiku are not modeled separately.
Weekly time-saved figures (~69.14 hrs/wk across the Oct 8-15 7-day window) and 14-day telemetry replay totals (36.59 min Fast / 55.36 min Hybrid across the Oct 2-15 39,526-row ledger) are ESTIMATED TIME projections that model both eliminated/bypassed calls and measured p50 latencies on a single RTX 5090 (32 GB VRAM, Blackwell sm_120) + Ryzen 7 9800X3D workstation, not a prospective A/B randomized trial.