User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch
Jun 28, 2026 · 15:37
Sonam Pankaj, CEO of StarlightSearch, argues that production agents fail because retrieval is static and eval signals never cross into runtime context. She introduces utility-ranked memory, where memories are re-ranked by a utility score combining semantic similarity with outcome history—passing runs raise a memory's score, failing runs lower it. In a product SQL agent demo, a search for 'gaming mouse' initially failed, but after marking that output as a failure and noting a wireless mouse was relevant, the agent's trajectory updated in real time to find the correct product. Benchmarks on Tao Bench show reflect memory improves performance from 66% to 76% without skills and 80% with skills; on agentic tasks, reflect achieves 61.3% versus 35.7% baseline and 58.2% with other memory systems. Pankaj explains that after enough reviews, memories can be baked into skills, allowing continuous improvement without manual prompt rewriting or model fine-tuning.
