# The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica AI Engineer · 2026-06-28 Sohail Shaikh and Ankush Rastogi of Prosodica argue that the common 'Fat Agent' architecture—loading every tool schema into every prompt—causes latency, cost, and accuracy collapses as tool catalogs grow. Their benchmarks show tool selection accuracy dropping from 78% at 10 tools to 13.6% at 741 tools, while their Semantic Tool Router pattern stays above 83% across the same range. The router uses Just-in-Time Context Injection, embedding tool descriptions offline, retrieving 3–5 relevant tools per query at runtime, and injecting only those schemas into the model call. This reduces input tokens by up to 99% (from 127k to ~1k tokens) and cuts time-to-first-token latency by up to 90% in high-tool-density scenarios. They present a production checklist and emphasize that routing pays off once beyond 50 tools, with K=5 as a strong default, but warn of router misses and weak descriptions needing fallbacks and monitoring. ## Questions this episode answers ### What is the "100-Tool Agent Trap" and why does it cause AI agents to fail? Sohail Shaikh explains that the "100-Tool Agent Trap" occurs when every tool definition is loaded into each prompt. As the tool count grows, the model's accuracy plummets from 78% at 10 tools to 13.6% at 741, due to confusion. Latency also spikes, with Time-to-First-Token exceeding 5 seconds for 500 tools, and token costs soar to 127,000 per request just for tool schemas. [0:02](https://aie.addtry.com/04f2aa5f-c002-4d33-abc2-fd657a149c63?t=2000) ### How does Semantic Routing with Just-in-Time Context Injection improve AI tool selection? Sohail Shaikh describes Semantic Routing as retrieving tools instead of documents. Tool descriptions are embedded and stored in a vector index. For each query, the router retrieves only the top-K relevant tools (typically 3–5), injecting just those schemas into the model call. This reduces token usage by 99% (from 127k to ~1k), maintains selection accuracy above 83% even with 741 tools, and keeps latency stable. [9:20](https://aie.addtry.com/04f2aa5f-c002-4d33-abc2-fd657a149c63?t=560000) ### What benchmark results demonstrate the performance of Semantic Routing versus a Fat Agent? Sohail Shaikh and Ankush Rastogi report benchmarks where a Fat Agent's accuracy drops from 78% to 13.6% as tools increase from 10 to 741, while the Semantic Router stays above 83%. Time-to-First-Token also rises sharply for the Fat Agent, exceeding 5 seconds with large catalogs, but the Router’s controlled prompt keeps latency nearly flat. [14:27](https://aie.addtry.com/04f2aa5f-c002-4d33-abc2-fd657a149c63?t=867000) ### What are the steps to implement a Semantic Router for tool selection in an AI agent? Ankush Rastogi outlines a three-step process: offline, embed tool descriptions and store them in a vector database. At runtime, embed the user query, run a similarity search to retrieve the top K tools (start with K=5). Then, only those schemas are passed to the model. Sohail Shaikh adds a code loop: index tools with descriptions, and at call time, inject only the routed tools. Logging decisions helps improve descriptions. [15:44](https://aie.addtry.com/04f2aa5f-c002-4d33-abc2-fd657a149c63?t=944000) ## Key moments - **[0:00] Intro** - [0:02] The '100-Tool Agent Trap' causes slower, costlier, and less accurate agents as tool catalogs grow, says Sohail Shaikh. - **[2:16] Fat Agent** - [3:29] 741 tools consume 127,000 tokens in tool descriptions before the user query is even considered. - **[4:16] Failure Modes** - [4:16] Fat Agent tool-selection accuracy drops to 13.6% at 741 tools, while Semantic Router stays above 83%. - [5:45] With 500 tools, Fat Agent Time-to-First-Token latency exceeds 5 seconds in real-time products. - **[7:33] Semantic Router** - [7:33] Q: How many tools before a Semantic Router is necessary? - [9:20] Semantic Routing is essentially RAG for tools, leveraging the same embedding and vector search infrastructure. - **[11:10] Just-in-Time** - [11:10] Anthropic's MCP on-demand tool loading reduced tokens from 150k to 2k, a 98.7% reduction. - **[12:36] Benchmarks** - [12:57] Q: What is the recommended K for Semantic Router tool retrieval? - [14:27] Semantic Router's Time-to-First-Token latency stays almost flat as tool catalog grows to hundreds. - **[15:44] Implementation** - [15:44] Core lesson: the tool catalog can grow, but the model's working set should stay small. - [18:57] A 'find me a flight to New York' query injects only 3-5 relevant tools, not the entire 200-tool catalog. - **[20:28] Checklist** - **[21:48] Community** - [22:12] Developers report tool confusion even with few tools; open-source project MCP0 explores routing at scale. - [24:19] Router miss risk: fallback by widening K, running a second retrieval, or routing to a broader tool group. - **[24:25] Trade-offs** - **[25:52] Takeaways** - [25:52] Start with K=5, log every decision, and improve tool descriptions over time, advises Sohail Shaikh. - **[27:21] Closing** ## Speakers - **Ankush Rastogi** (guest) - **Sohail Shaikh** (guest) ## Topics Agent Skill Design ## Mentioned Anthropic (company), Aurelio Labs (company), Berkeley function calling leaderboard (product), ChromaDB (product), MCP (product), MCP0 (product), Pinecone (product), Semantic Router (product), Tool Bench (product), quadrant (product) ## Transcript ### Intro **Sohail Shaikh** [0:02] Hi everyone, thanks for being here. Uh, I'm Sohail here, and along with me is Ankush. So today we'll be talking about a mistake that looks harmless at first: giving— which is basically giving an AI agent every tool access it might ever need, all at once. So basically that approach works well in a demo. It might even work with a small number of tools, like say for example 10 tools, but once the catalog grows, the agent gets slower, it might become more expensive, and less accurate as well. That is why we're calling it the "100-Tool Agent Trap." In the next half an hour or so we'll show why it breaks, what the numbers look like, and how Semantic Routing with Just-in-Time Context can help us fix this problem. So a quick introduction about myself: I'm Sohail Shaikh. I'm currently working as a data scientist with Prosodica. My background spans across AI, NLP, marketing, analytics, and even engineering. My current focus is on Applied AI, NLP, and Conversational Intelligence, along with RAG systems. I'm especially interested in making AI systems more reliable, measurable, and even scalable beyond the demos in production. **Ankush Rastogi** [1:31] And I'm Ankush Rastogi. I work as a Senior Data Solutions Engineer at Prosodica. I have spent more than a decade in AI, data engineering, and production systems. My focus is the engineering side, so it's not about what's going to work in a notebook, but whether it's going to survive in real load, real user, and real failures. So that is the angle we're taking today. Sohail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production tradeoffs. **Sohail Shaikh** [2:16] Awesome. Thank you, Ankush. Um, so let's get into it. So let's imagine a common design. You tend to build a system, and it can do many things. Say for example, querying a database, sending an email, even checking a calendar, or looking up an order, calling an API, and so on and so forth. ### Fat Agent **Sohail Shaikh** [2:39] The simplest approach over here would be to give a model every tool definition on every request. Every function name, every description, and even every JSON schema will go into the prompt, whether the user might need it or not. So that, we're calling that as a Fat Agent. At small scale it feels fine. With 10 tools as well, the model might usually pick theright one. The demo looks good. Then the product grows. 10 tools might become 30, or it will keep on increasing, and eventually the model starts calling the wrong function, starts confusing similar tools, may invent tool names, and even take longer to respond. The important point is basically: the design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog. So let's look here. There are, say for example, 741 tools in your entire schema, but— and it will basically take up to 127,000 tokens just to have all those tool descriptions in it. And this is even before the user's actual question is even considered. So basically this will lead to context overload, and we need to manage that properly. So on this slide we see why it is failing, and why the accuracy collapses beyond a point. So when you look at the accuracy curve, with the 10 tools, Fat Agent will get the toolsright almost 78% of the times. ### Failure Modes **Sohail Shaikh** [4:32] That is not perfect, but it's usable. At almost 100 tools, the accuracy drops to around 40%. Less than half of the tools that are called are the correct tools. And if it grows beyond that, like say for example in over here, at 741 tools, the accuracy will be a mere 13.6%. So in short, it's roughly 1 correct tool out of 8 tools. So when we compare it with the Semantic Router, Semantic Router behaves very differently. It stays above 83% across the same catalog sizes. That is because the model is not choosing from hundreds of tools, it's choosing from a small and relevant set. One reason the Fat Agent fails is because it's lost in the middle problem. Model pays stronger attention to the beginning and end of the long context. When hundreds of tool schemas are packed into the middle, the model does not reliably use them. So we end up paying for a huge prompt, and that prompt makes the decisions even harder. So two more reasons over here. First is the latency, and the second is the cost. So we saw in the earlier slide that accuracy was a big problem. Another issue is with latency and cost over here, as we can see. So like say for example, if we have 741 tools, we saw that it requires almost 127,000 tokens. That will include the tool description and the schema text. So that cost is basically being paid on every request. At 100,000 requests a day, if you push this into production, you are sending in billions of tokens just to describe those tools. And with Just-in-Time Routing, the prompt may include only 3 to 5 relevant schemas, closer to about 1,000 tokens. That is roughly a 99% reduction in tool context tokens. Another issue is latency. With Fat Agent approach, Time-to-First-Token grows as the tool catalog grows, because the model has to process a larger prompt before it can answer the user question. So say for example, if you have 500 tools in your agent, the Fat Agent path can push first token latency past 5 seconds. So this is specifically important if your product is real-time, because this will take a longer time to respond, which will make the design feel slow and unpredictable. And with Semantic Routing, it might feel more responsive, and it will be more real-time in a way. ### Semantic Router **Ankush Rastogi** [7:33] Here is a clean comparison. So on the left side we have a Fat Agent design, and on theright side we have a Semantic Router design. So in the Fat Agent design, every schema is loaded for every request, no matter what the user asked. And because of that, the catalog will grow, the prompt will grow, due to which the latency will rise and accuracy will drop. The agent also becomes one large monolithic, which makes it hard to test, risky to update, and even painful to debug. But in the case of Semantic Routing design, the agent does not start with every tool. So in this case, the router first looks at the user's query, it retrieves the 3 to 5 most relevant tools, and then it injects only those into the model call. So the context stays small, latency stays stable, and the accuracy holds because the model is choosing from a focused list instead of a giant catalog. This is okay, so there is one important caveat here. If you have fewer than 20 tools, a router may be unnecessary. Just load the tools directly. But once you passed 50 tools in the production system, then justifying router-based schemas makes more sense. **Sohail Shaikh** [9:20] Thanks, Ankush. So let's now look into how Semantic Routing works. If you have built a RAG system previously, this will feel pretty similar to that. The difference is that we will retrieve tools instead of the documents. So at the first stage, say for example over here, each tool will have a clear description. For example, a search flights tool, a check calendar availability tool, or a retrieve customer order status, say for example. The second step would be those descriptions need to get embedded and stored in a vector index. This is usually done offline when the tool catalog is created or updated. At the runtime, the user will ask a question. We embed the user query with the same embedding model. Then we search that vector index for the tools with the description that are closest to the query. So the router will then return the top K tools. Often K would be 3 or 5 tools that matches the user query. And only those selected schemas will get injected into the model call. So this is the whole pattern. At the first step, you need to index tool descriptions offline, then retrieve relevant tools at runtime, and keep the model's context focused. So in short, the idea is pretty simple. Semantic Routing is basically a RAG for tools. If you already have an embedding model and vector database in your stack, most of the infrastructure is already familiar to you. ### Just-in-Time **Ankush Rastogi** [11:10] Thank you, Sohail. Now we will deep dive into Just-in-Time Context Injection. So Semantic Routing is the retrieval layer. And Just-in-Time Context Injection is the context management strategy. So in the case of Fat Agent approach, it loads everything before the query is understood, due to which the model receives a huge list of tools first, and then it tries to reason through it. Just-in-Time Context does the reverse. It will wait until the query is known, then it will inject only the context needed for that request. I mean, this is not a new software idea. We have used lazy loading, Just-in-Time compilation, and on-demand resource loading from years. We are just applying the simple and same principle to the LLM context. Anthropic has written about this pattern with on-demand tooling loading through MCP. Their report, token usage went from 150k tokens down to 2,000, which is 98.7% token reduction. ### Benchmarks **Ankush Rastogi** [12:36] That is the signal. Large tool catalogs should not be dumped into every prompt. They should be retrieved when needed. So moving to the next slide, let's talk about how to evaluate this fairly. We measure four things here: tool selection accuracy, time-to-first-token, input tokens per request, and the estimated cost per 1,000 call. So here, for datasets, we used Berkeley function calling leaderboard, skills bench style scenarios, and synthetic tool pools that allow us to scale the number of tools. So we can test at 10, 50, 100, 200, or even 741 tools. So we ran the same queries in two modes: the Fat Agent one and the Semantic Routing. Same model, same answer key, same tool catalog. The only difference was whether the model saw every tool or only the routed tool. We also swept K at 3, 5, 10 to understand the tradeoff. So the smaller K is the faster and cheaper. Larger K can recover more edge cases. In practice, K equals 5 is a strong default starting point. So if you want to try this, you can default it to 5. Here, K means the number of tools the Semantic Router retrieves and gives to the LLM for each user query. Here are the benchmark results that we got. So here is the result. On the left side, we see the Fat Agent accuracy line drop sharply as the tool count grows. It started around 78% at 10 tools and then falls to around 13% at 741 tools. But in the case of Router, the router line stays above 83% because the model stays stable. From the model point of view, it's always choosing the handful of tools, even if the real catalog contains 100. On theright side, Time-to-First-Token shows the same story. The Fat Agent path gets slower as more tool schemas are added. At large catalog sizes, the model spends noticeable time just processing the prompt. But in case of Router, it stays almost flat because the prompt size is under control. This is the core lesson from the benchmark. The catalog can grow, but the model's working set should stay small. Now, in the next slide, we'll be discussing about the implementation pattern. There are three steps in it. Step one is offline. ### Implementation **Ankush Rastogi** [16:06] So build a catalog of tools. For each tool, store the name, description, schema. Then embed the description and store it into a vector database. You can use ChromaDB, you can use Pinecone, you can use Quadrant, whichever vector database you are using. Then step two will happen on every request. Embed the user's query, then run the nearest neighbor search and return the top K tools. In the step three, it will also happen on every request. Fetch the schema for those selected tools and then pass only those schemas into the model call and log which tool was selected. So here, logging really matters. So let's say if you miss something, and if you want to improve the description, or you want to tune the K, you need a good logging system that you can use it. So the runtime overhead is small: one embedding call and one vector search. The payoff is much smaller prompt and more stable tool selection problem. **Sohail Shaikh** [17:34] Awesome. So now that you have learned about the implementation pattern, I'll take you through the code version of it, which is pretty straightforward. So initially, you need to loop through the tool catalog. And for each tool, as Ankush mentioned, embed the description, store that in the vector database with the tool name and other details. At runtime, embed that user query, compare that query vector against the stored tool vectors using cosine similarity or vector search, and then retrieve the top most relevant tools. And then call the model only with those schemas in the function or tool parameter. That final line is the most important part because the model does not receive all the tools, it only receives the routed tools. This works with any embedding model and with any vector database. You can start, as Ankush mentioned, you can start with Quadrant or any other vector database locally, and then move to a more managed vector store later if needed. If you already have an existing RAG infrastructure, this is not a new infrastructure, as we mentioned earlier. It is the same retrieval pattern applied to the tool selection. **Ankush Rastogi** [18:57] Thank you, Sohail. Now we will discuss Semantic Routing in action. So imagine you have a 200 tool agent. A user asks, "Find me a flight to New York next Wednesday." The router embeds that query, returns tools like search flights, book flight, calendar check. The model sees only those relevant schemas, so it is much more likely to call the correct tool. But if we load all 200 tools: portals tool, weather tool, emails tool, SQL tool, or any unrelated workflow tool, it would all be competing for attention. That is when the model starts choosing the wrong function. Now let's take a second query. What is the weather in Parisright now? The router returns GET Weather and GET Forecast. Flights tools are not injected. Portals tools are not injected. They are simply not available at this request. This is an underrated benefit. But the router does not only add theright tool, it also removes the wrong tool from the model's choice set. Now, in the next slide, we will discuss implementation checklist. ### Checklist **Ankush Rastogi** [20:28] So here is the production checklist if somebody wants to follow this along. So first, catalog the tool in one place. So add name, description, schema, owner, and the version. Second is to build the index. Embed each description and store the vectors. The third one is to write the router. So in this, embed the query, then search the index, and take top K and fetch schema. The next one is the fourth one, in which a user needs to wire it up into the agent loop so the model tool list should come from the router, not from a hard-coded full catalog. Fifth is to evaluate. Run your test set at K equals to 3, 5, and 10, and then pick the smallest K that meets your accuracy target. Sixth is to monitor production. Log selected tools, final tool calls, failure, and fallback usage. Then re-embed tools when descriptions or schema change. This is not the six-month platform rewrite. For most teams, it is a focused sprint. So in the next slide, we will discuss what the community already has confirmed. This is not just the observation. ### Community **Ankush Rastogi** [22:12] So Anthropic has published on-demand tool loading with MCP and reported a drop of 150k tokens to 2k, which is a huge reduction in tokens. Developers have raised similar issues in mainstream agent and SDK projects, pointing out that sending all tools' definitions on every request can increase latency, token usage, and tool confusion. Open-source projects like MCP0 have also explored routing at very large scale, including thousands of tools across many servers. And you can find forums posts from builders who hit tool confusion with only a few tools long before they reach 100. So if your agents start failing as tools are added, it doesn't automatically mean your prompts are bad. It may mean the architecture is asking the model to solve the wrong problem. **Sohail Shaikh** [23:31] So here we are just trying to say about the we are going to talk about the open-source projects that we came across. So one of them is from Aurelio Labs. They have an open-source Semantic Router project that you can easily use to test locally for benchmarking. Tool Bench and the Berkeley function calling leaderboard are also useful starting points. And for practical guidance, you could read the Anthropic MCP write-up. It explains why on-demand tool loading matters and gives a strong real-world signal for this architecture. The point is that you don't need to invent the ecosystem from scratch. Those pieces already exist. Next slide. **Ankush Rastogi** [24:25] Sorry. Here it is. ### Trade-offs **Sohail Shaikh** [24:27] Thank you. So there are a few trade-offs with this. The first risk could be a router miss. So if the router may fail to retrieve the tool the models need and handle that, you could handle that with a fallback. If the model cannot complete the task, you could widen K, run a second retrieval pass, or even route it to a broader tool group. The second risk would be weak tool descriptions. If your descriptions are vague, embeddings will end up being weak. Write descriptions in the words users actually use and include intent, action, and key entities along with it. The third risk over here would be rare tools. So some tools may never score high unless their descriptions include theright language. So you need to monitor misses and rewrite those descriptions accordingly. And finally, just to conclude, do not over-engineer small systems. If you have 10 or 15 tools, static loading may still be fine. Routing generally pays off when the catalog is large enough that the prompt size, latency, or tool confusion becomes a real production problem. So a few takeaways. First thing is, as we discussed, tool overload hurts accuracy. Loading every schema into every prompt makes the model's decision harder as the catalog grows on. The second one would be ### Takeaways **Sohail Shaikh** [26:11] about tokens. It's like tokens are both cost and latency. Larger tool catalogs can add tens of thousands of tokens before the user's actual request is handled. Semantic routing can help fix that problem. When the catalog is large, the model only sees the tools relevant to the current query. This is basically a RAG for tools. Index those descriptions properly, retrieve at runtime, and inject only what is needed. And finally, start simple. Use K as 5 as a starting point, log every decision, and evaluate against a real test set. And improve the tools' descriptions over time in case you get incorrect tools being picked up. And the goal is not to make these agents more complicated. It is basically to stop forcing the model to reason over irrelevant tools. **Ankush Rastogi** [27:21] So there are some references on this slide. So if you're building something, you can use these references like Semantic Routing Libraries, Tool Bench, Berkeley function calling leaderboard, Anthropic MCP write-up, and vector stores like Pinecone, Quadrant. Thank you so much for watching this. ### Closing **Ankush Rastogi** [27:46] If this was useful, please share it with someone building agents. They have either already hit this wall or they're about to. You can scan our LinkedIn QR codes on this slide to connect with us. We are happy to continue the conversation, share resources, or hear how others are approaching this problem in their own system. Thank you so much for watching this. I'm Ankush. **Sohail Shaikh** [28:16] This is Sohail here. Thank you so much. **Ankush Rastogi** [28:19] Yeah. Thanks, everyone. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.