Context Platform Engineering to Reduce Token Anxiety — Val Bercovici, WEKA
Nov 24, 2025 · 23:52
WEKA's Val Bercovici and Callan Fox present Context Platform Engineering, open-sourcing a toolkit that maximizes KV cache hit rates—called the single most important metric for production-grade AI agents by Manus AI. They argue that without this engineering, users resort to Context Financial Engineering, a clairvoyant prompt cache arbitrage against time-to-live (5 minutes to 1 hour) and cache read/write pricing. The toolkit includes a load generator that configures agent swarms with specific SLOs, cycling through deterministic and random prompts. Callan's WEKA Labs research shows that 1-minute TTL causes 15-16x token repetition, while 1-hour TTL approaches 1x, but requires larger cache capacity. Benchmarks compare HBM+WEKA (purple) vs HBM+DRAM (orange) vs HBM+DRAM+slow storage (pink); WEKA's NVMe-backed augmented memory grid maintains higher output token rates under increasing concurrent users—up to the point where DRAM tiers drop off sharply due to insufficient speed. The episode covers how SLA requirements translate into SLOs via memory tiering and KV cache offloading, and emphasizes that subscription users effectively purchase cache allotments to keep inference providers in…
