# User Signal Dies at the Retrieval Boundary - Sonam Pankaj, StarlightSearch AI Engineer · 2026-06-28 Sonam Pankaj, CEO of StarlightSearch, argues that production agents fail because retrieval is static and eval signals never cross into runtime context. She introduces utility-ranked memory, where memories are re-ranked by a utility score combining semantic similarity with outcome history—passing runs raise a memory's score, failing runs lower it. In a product SQL agent demo, a search for 'gaming mouse' initially failed, but after marking that output as a failure and noting a wireless mouse was relevant, the agent's trajectory updated in real time to find the correct product. Benchmarks on Tao Bench show reflect memory improves performance from 66% to 76% without skills and 80% with skills; on agentic tasks, reflect achieves 61.3% versus 35.7% baseline and 58.2% with other memory systems. Pankaj explains that after enough reviews, memories can be baked into skills, allowing continuous improvement without manual prompt rewriting or model fine-tuning. ## Questions this episode answers ### How does utility-ranked memory work in AI agents? Sonam Pankaj explains that utility score combines semantic similarity with outcome history, akin to a credit score. Memories are retrieved by relevance to the current task, then re-ranked based on whether they historically helped or hurt execution. Unlike static fact storage, these memories capture reasoning—like a policy to check settlement before refunds—so the agent learns from past successes and failures without manual retraining. [4:22](https://aie.addtry.com/60c9ce5b-1415-4624-b281-5030ef6fe646?t=262000) ### What performance improvements did StarlightSearch's reflect memory system show on the Tao Bench? Sonam Pankaj reports that on Tau Bench, which measures policy adherence, reflect improved performance from 66% to 76% without baked-in skills. When reasoning was baked into skills after accumulating around 10 memories, it reached 80%. On agentic tasks like the Human Last Exam, reflect achieved 61.3%, up from a baseline of 35.7% and 58.2% with other memory systems. [6:56](https://aie.addtry.com/60c9ce5b-1415-4624-b281-5030ef6fe646?t=416000) ### How did the product SQL agent improve after receiving failure feedback in the reflect memory system demo? In Sonam Pankaj's demo, a product SQL agent initially failed to find a ‘gaming mouse.’ After marking the run as failed and providing feedback that a wireless mouse existed, the agent’s trajectory changed: on the next query for a gaming mouse, it searched for a wireless mouse instead and successfully retrieved a product, demonstrating how utility re-ranking updated the tool call without manual prompt changes. [9:45](https://aie.addtry.com/60c9ce5b-1415-4624-b281-5030ef6fe646?t=585000) ## Key moments - **[0:00] Intro** - **[0:29] Why Agents Fail** - [0:29] An agent executes loops of reasoning and tool use but lacks a learning mechanism from past outcomes, says Sonam Pankaj. - [1:14] Gartner reports 85% of AI agents fail in production, and McKinsey 2025 finds retrieval is the main cause. - [1:35] Ram Srinagar Shah: 'We have been optimizing for the wrong thing.' - [2:17] Eval signals die in the dashboard; agents lack access to why past runs passed or failed, says Sonam Pankaj. - **[3:37] Memory Shortcomings** - **[4:22] Utility Score** - [4:22] Utility score is similarity weighted by usefulness, incorporating past traces and outcomes. - **[5:41] Reasoning Memory** - **[6:31] Benchmarks** - [6:56] On Tao Bench, Reflect memory improves policy adherence from 66% to 76% without skills, and to 80% with baked-in skills. - [8:26] On the Human Last Exam agentic benchmark, Reflect memory lifts the baseline from 35.7 to 61.3%. - **[8:56] Limitations** - **[9:45] Demo** - [9:49] Demo: Sonam Pankaj shows a product SQL agent that initially fails to find a gaming mouse. - [10:31] Product SQL agent query 'Find me a gaming mouse' initially returns zero memories and fails. - [13:17] After feedback, the agent's trajectory changes: previously empty product search now returns a wireless mouse. - **[14:06] Baking Skills** - [14:26] After accumulating positive reviews, memories are baked into skills, allowing agent updates without changing system prompts. - **[15:16] Outro** ## Speakers - **Sonam Pankaj** (guest) ## Topics Agent Skill Design ## Mentioned Elastic (company), Milvus (company), Pinecone (company), Qdrant (company), StarlightSearch (company), Big Code Bench (product), DSPY (product), LangChain (product), Memzero (product), Reflect (product), Tao Bench (product), embedanything (product) ## Transcript ### Intro **Sonam Pankaj** [0:01] Hey everyone, I'm Soyim, I'm the CEO and Co-Founder of StarlightSearch, and today my talk is on user signals die after retrieval boundaries. So we'll look into what are agents, essentially why agent fails, what is the cause of fails in retrieval particularly, and how to make actually signals cross the retrieval boundary, and how to make your agent basically outcome-fair. ### Why Agents Fail **Sonam Pankaj** [0:29] So let's get started. What is an agent? An agent is an LLM that has agency to reason, invoke tools, interact with the real world, retrieve the memory to complete the task. One major loop here is missing is learning. It should also learn from what worked and what didn't work. Suppose if I have to explain what is agent, I can explain with React agent. So if I have to explain what agent is, I'll explain it with React agent. So basically, user prompt the agent, LLM execute it in a loop, client tool retrieval search, and then pause when the task is complete. This is very basic React architecture. One thing that is missing is how to make agent learn from the outcome. So agent keeps failing at the same task. Gartner reports that 85% of AI objects fail in production and source in McKinsey's 2025 report. The problem came out to be most of the time is that retrieval is static. 73% of RAG pipeline fails because of retrieval not generation and context stuffing. So a recent post from Ram Srinagar Shah, the ex-CTO of Pinecone, said, "We have been optimizing for the wrong thing." You are paying a note for your agent's memory. This is probably broken. And we have been optimizing for the wrong things. We made wrong answers appear faster and cheaper, but we forgot to make retrieval learn. So why does this does it matter? Again, the third problem is agents are not outcome-informed. So there's a missing layer between evals and action. Your observability has all the traces, all the stack that capture observability is the stack that capture every tool call, every LM completion, every exceptions. Your eval suite judges whether the final output was correct or wrong, basically pass or fail. But these evals are not reflected in agent context skills, MD files, or agent action in any ways,right? So the agent doesn't has any access to why yesterday's runs passed or fail. The eval signal dies in the dashboard. This is a missing layer: a system that consumed traces, absorb eval, and convert both into retrieval guidance for future runs. So there's a manual improvement tax, and engineer actually has to sit and see if the eval and observability perform well. Rewrite the prompt, redeploy it, either upgrade to expensive model, restructure tool, or harness, or fine-tune the custom models. ### Memory Shortcomings **Sonam Pankaj** [3:37] Why are current memories failing? Why? Memories was designed to actually address this, but it's not. So let's see what we have as a current system and current memory is that they basically store user preferences, profile, conversational history, or long-lived personalization. So chat experience is not self-improving learning systems for production. If you see the already existing approach in the market, there's LangChain, there's Memzero, which does extracted fact preferences. The user's retrieval signal is embedding similarity. Does it learn from output? No. So we have come up with something called utility score, which is a similarity weighted by how useful it is for the agent to execute the task. ### Utility Score **Sonam Pankaj** [4:41] It has actually the history of past traces and past outcomes. So we came up with agent RTX, and that is agents with runtime experience. It's a runtime layer that let production agent improve from experiences without retraining, fine-tuning, or manual prompt training. It's a bit different from compile time like DSPY because you bake in all the lessons in the prompt. Here it's actually improving while it is executing the task. So which again introduces utility score. So you do not retrieve by keyword. You retrieve by semantic similarity to the current task, weighted by whether those memories have historically helped or hurt the execution or the outcome. The eval outcome becomes a first-class signal in the retrieval re-ranking and not just for employee. ### Reasoning Memory **Sonam Pankaj** [5:41] One of the key things is it treat memory as reasoning, not as facts. Static fact with no context and no history, but reasoning. Like suppose if there's a if there is a customer support bot looking for a refund, it will not only say, "Hey, user prefers a dark theme," or "user prefers to be called by a shorter name." It actually reason about a query. Like if someone asks for a refund, you should check the settlement before refunding it so that the customer doesn't get paid the refund twice. So re-rank based on usefulness. Context is updated based on task. ### Benchmarks **Sonam Pankaj** [6:31] So this is a very big thing because most of the agents fail with context stuffing, and this has been brought up in the past and learned from history and reasoning,right? Talking about benchmarks. So we have benchmarked our memory system reflect with on Tao Bench, which essentially measured if agents have followed the policy well or not. So we have seen the performance improve from 66 to 76% without baking in skills, and with skills, reflect performs at 80. So once there are enough memory, like 10 memories, what we do is we bake in the reasoning and the understanding into skills so that your agent always remains updated. What happens most of the time we have seen, suppose you have a product SQL agent and there's a system, there's a column in system prompt. Even though that column is no useful anymore, it remains in the system prompt. So there's no systemright now that can update that. Hey, there's no columnright now called this. So maybe probably you shouldn't entertain it in the future. And this is possible with skills that because agents always uses calls that skill updated skill all the time. And the similar behavior has been seen in GPT-5.4. We have also benchmarked on agentic task, which essentially test a model's ability to reason, plan, and use tools over extended multi-step workflows rather than measuring a static Q&A. So you can see here with the suppose the human last exam with RAG, you get 47.5. If it is starting from the baseline 35.7, with the other memory system it gets to 58.2, but with the refilled memory system it gets to 61.3%. So this kind of trend is shown in another agentic benchmarks as well, like Big Code Bench, like Long TV, etc. So of course there are limitations to this approach. First of all, there's a cool start. So in the beginning, it's pure semantic search until enough reviews have been accumulated. There's a utility drift. Maybe sometimes similar memories could come. There are a lot of problems that could come at scale with this experience, but we have come back most of them. ### Limitations **Sonam Pankaj** [9:20] There's a review quality. So noisy labels can make the utility noisy as well. And there is a hyperparameter call Lambda that is associated with credit and re-ranking. So we have built reflect in such a way that most of these problems and most of these limits are now reduced except for cool start, which we cannot do much about it. ### Demo **Sonam Pankaj** [9:45] Let's now get into the demo. So let's check this demo. So it's basically a product SQL demo. I'll give ask it to search for some product in a SQL database and then see if it is able to find it out. So it gave it find me a gaming mouse. Zero memories retrieved. I couldn't find a gaming mouse. Okay. So maybe let's just go and see what's happening in the dashboard. Okay, it came out that So coming on another benchmark that with this agent bench with essentially measures, if agent has actually done theright reasoning planning and have followed which essentially test a model's ability to reason plan and use tools over extended multi-step workflows rather than measuring a static Q&A. So you can see here with the suppose the human last exam with RAG, you get 47.5. If it is starting from the baseline 35.7, with the other memory system it gets to 58.2, but with the refilled memory system it gets to 61.3%. So this kind of trend is shown in another agentic benchmarks as well, like Big Code Bench, like Long TV, etc. Wait, let's check. It didn't find I couldn't find any gaming mouse in the product catalog. So suppose I want to mark it fail and tell it give any relevant because there is a wireless mouse in the database and I have submitted the failure. This was the input. This was the response. I couldn't find any. And this was a trajectory and tool collector. So you can see the product is empty with this product or with this query. Let's check again what happens now. Let's see now what happens. I'll see. So it searched a wireless mouse. Let's see what happens in the dashboard now. Find me a gaming mouse was input. The response was, "I found a product related to mouse as we asked it to find any kind of relevant." But the most important thing is how the tool call evolved. You can see that previously the tool calls or the trajectory used to look just with one search, that is search products it called, and the product was empty. Now the trajectory has changed in production and it found something in the product and it is still calling search product tool call, but it is getting some answer that we fed in as a feedback. So that's the demo. And the most important thing that is happening over here is it's forming memories, which is retrieved based on the utility score. That is the score which basically keeps improving, keeps re-ranking itself on the basis of how useful that memory was. ### Baking Skills **Sonam Pankaj** [14:26] So you can see from these traces, past traces, which memory. So this memory scores keep changing and after a certain while, after suppose five reviews need, you can bake in these findings or these new updates in a skill without so without changing any system prompt, you can actually update certain things that agent draws a lot from. So you can update the skill, which is very cool. So I hope everything was I hope you get to try this new feature, this new runtime experience that we have built. ### Outro **Sonam Pankaj** [15:26] If you want more details, you can visit our website and you can also contact me at sonam@starlitesearch.com or visit me. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.