MCPs are Boring (or: Why we are losing the Sparkle of LLMs) - Manuel Odendahl

I'm Manuel, and I'm going to be talking about "MCP is boring" and why we're losing the sparkle of LLMs. So, just a little bit about me: I'm Manuel, I'm "Program with AI" on Twitter, my GitHub's "vesun," and I've been a software engineer for 25 years, probably a veteran coder at this point, I guess.

And I've been writing a lot of comment lists, and I'm bringing that up because it's going to be relevant for this talk. But I've been an embedded engineer: I love a search engine, I love databases, I love backend coding, I've done all the older coding, basically.

That's like boilerplate and backend and those kinds of things. I've been coding with LLMs since 2022, since the alpha of Copilot, I think, and then when ChatGPT came out and decided to do all my coding with LLMs. And I've been pretty much obsessed with it since then.

That's my "Program with AI" Twitter account, where I share all my tips and all the things I've discovered. And we've been using tool calling and LLMs for, I guess it came out like 2 years ago. We've been using it increasingly.

They're great,right? Because LLMs are language producers. So what that means: we've written a lot about tools, so per se, LLMs know a lot about calling tools. And I mean, we've written a lot about tool using in our literature, in our code, and they've been reinforced, so we have much content to talk about tool calling.

And what that means is that LLMs can produce language that calls tools. In structured format is what we usually do, and then we map the structured format that comes out to just basically call a function, call an API, call an MCP server.

And it's magical: the text comes back, we don't really need a schema, the LLM can continue working on it, can answer questions about it. And what that means is that now we can, instead of writing code, having to parse schemas, having to validate them, having to be really careful, we can just talk to the machine, say like, "Please, you know, please call the API and then give me that information," and we're done.

Pretty amazing,right? I think everybody here uses agents and knows pretty much what I mean. What does that look like in practice? I think this is something that is fairly straightforward. It's the standard example of "What's the weather like in San Francisco?"

The assistant can do some chain-of-thought, for example, and checks what tools it has available, which are provided as a schema in the context that's being sent to the LLM. It basically says, in a special token format, "These are your tools, this is how you call them, this is what it means to call them."

And the LLM then decides that, you know, getting the current weather might be a pretty good tool to use to get the weather from in San Francisco,right? And the tool then, which is a deterministic piece of code, is going to call whatever weather API you have, fetch your current conditions, return that as JSON maybe.

And the LLM, receiving this JSON, does a new inference and outputs an answer to the user's request. No magic there. Well, there's no magic in how it works, but as soon as you've wired up a few tools, especially your own tools, you realize how magical that is.

So, great. We have MCP now, which allows any kind of LLM app to interact with tools that other people have built. So you can download an MCP to interact with GitHub, you can download one to interact with Blender, you can download one to interact with your own files, you can download one to interact with the temperature in your room if you have built such an MCP.

It's absolutely mind-boggling possibilities. The problem is, when you give an MCP too many tools,right? So I'm just going through, like, we—it's absolutely magic. It's amazing. But if you give an LLM too many tools, if you give it like 100 tools, it's going to be, "Well, should I call Get Weather or should I call, like, search internet for weather in San Francisco?"

It's kind of hard to decide. And the LLM, like, ends up calling the wrong tool, or uses the wrong schema, or, like, doesn't really understand what parameters to give to a certain tool. So it's not great. And the other problem is that when you call a tool, that means the arguments that the LLM is going to give that tool, and then, per induction, those that we are going to provide to that tool, have to be inferenced by the LLM.

So if we want to call the tool and say, like, "You know, we have this, like, fragment of document, please search for something similar somewhere else," you have to emit this part of document, which probably is already in the context higher above anyway, and repeat that to give it to the tool, which is pretty expensive, it's pretty slow.

I think you all know that, where it's going to be like, spinny, spinny, spinny, spinny. And actually, it's just copy-pasting something that's literally just above. If you go back to the San Francisco example, it's like a lower, less impressive example of that, but it repeats the San Francisco,right?

So the problem is that all the tokens that come back from that tool also have to be put in the LLM. So if you do something like getting the weather and you only want the wind speed, you'll still have the whole JSON that gives you the date and gives you all kinds of information about the sun and the moon, but you actually only care about, like, a single number that's like a single token.

So you've wasted 2,000 tokens on something that could have been answered much more quickly. What does that look like? Imagine if you have a CRM and you ask for the contact information of OpenAI. What it's going to do is going to call its tool that's called GetCRMCompany.

That's like the tool that exists. And this is a massive response because there's 36 companies in your database. So you end up having this, like, insane list of companies. And then at the end, you basically just look at the email of OpenAI.

And you could say, "Oh, bad tool,"right? Like, "Get a tool that's called GetCRMCompany." But then if you misspell OpenAI, maybe that won't work,right? Like, you have all these, like, little—you're starting to optimize for a certain case where you, like, only want one company and maybe you only want the contact information for the company, so you add a flag like contact is equal to true, and you end up with, like, something crazy like GraphQL at the end of the day.

If you want to cover all your bases and then the model's going to create wrong GraphQL, it's going to go downhill. So tools have limitations,right? It's great to have all the information of the companies because suddenly I can ask questions like, "Who's in San Francisco?"

And it's going to be able to answer these, like, fuzzy queries that I maybe never asked for. The problem is you end up having 20,000 tokens, it costs you like 50 cents, you're waiting for 5 minutes before you get the answer to your query.

And that's not—it's not great. So we can engineer around it, but really what we're doing is, why are MCPs basically so restricted? And I think it's because they're boring. Like, we are not leveraging all the things we could actually do with LLM, with code generated by LLM, with coding the tool calling around it, for example.

Why don't we pass the chat history along with LLM calls? The LLM call can decide to—the tool can just decide to discard it,right? But we already have all this information in the context of an LLM application. Why not also pass it to the tool call so that we can say, "You know, we've called this thing like 15 times, let's reuse the arguments from before," for example, or use this as, you know, a search query.

So that would be one example. It's pretty simple. It's like, you know, "Please just give me the chat history that is present as metadata to your tool call." You could also give me your memories if you have an LLM application that has memories and you have persistent knowledge.

Like, why not give that to my tool to say, like, "Well, you've already called it 15 times," or, like, "Maybe my location is San Francisco, then I can just call Get Weather," and Get Weather will look at my memories and say, like, "Oh, there's a location memory.

Here you go." Pretty easy extension that allows you to do so, so, so much more at a very, very low cost. And another thing is, like, if you can attach files in your LLM application, like, why not pass these files along?

Or at least pass, like, at least their metadata. Dangerous,right? Like, the security implications. And the problem is also there's many, many, many different LLM apps that all have different modalities. So suddenly you have to design a protocol that covers all these different cases.

Maybe I have an application where people can, like, draw little images. So suddenly, how do I pass that as an attached file? Like, maybe I have a GraphRAG, how do I pass my Graph structure? Lots of questions, a lot of boring engineering.

We've done that in the '90s and the 2000s with ontologies and semantic web and graphs and scriptlets and XML schemas. Could go back to that.

But one very easy thing we could also do is, you know, before you call the tool, why not give the user a little UI on what they want to attach and how they want to attach and maybe edit the arguments?

Because why not? Very, very easy to at least, you know, if I have the LLM write a whole tool call and it opens up and it says, like, "Well, I'm searching for the company, like,

OpenAI," and it's misspelled, then the user can go in and say, like, "Nah, this is OpenAI, it's different." Or maybe it's, like, suddenly it's searching for Oracle for some reason. You can say, like, "No, no, no, this is not the tool call that I want."

Currently, the only interaction we have is, like, allow, which is boring, is square. It's like, why not think wilder? The next thing we can do is, like, well, let the user edit the tool result before we paste all of this stuff back,right?

Like, if I get 30 pages of results, as a user, it's actually faster for me to just edit down these results, and maybe the UI is even nice to do it, instead of waiting 10 minutes and spending 50 cents to have the LLM do it.

So to show you an example of what that can look like is, you can say, "I want to find all the customers with overdue invoices,"right? We get a tool call. Imagine we have a SQLite or like a SQL tool, so it's able to write SQL queries, and it's going to have this query CRM thing with a filter.

And before it gets called, we have this little approve/reject, which we know, and a little edit dialogue, which, and here's a very raw way of editing it, but it allows you to edit the arguments before they go, before they go further.

So suddenly, when the call comes back, you can actually edit the results as well and tune your query. Or, like, realize, "Oh, it's the wrong call," like, "Let me go back to the edit step, let me do the call again."

I don't need the LLM to do that kind of stuff,right? And that, to me, especially when I'm doing a lot of, you know, database queries, those kinds of things, if I do the wrong query and I have only the option allow and I get back 10,000 things, I'm like, "Damn, I'm, like, messed up."

And if it's an agent thing, I have to rewind a whole agent run instead of being able to edit this one tool. I've got a whole set of thoughts around context editing, but this goes beyond the scope of this talk, and you're welcome to come talk to me to get more info.

So

LLMs are so much more,right? They're language producers. They can create every word under the sun. They've been trained under every word under the sun, and you can do the weirdest crap with them. You can do poems about Oracle, you can do,right, like, those are even boring things you can do if you look at everything like AI red teams are doing.

And if you go into starting to talk to it like a terminal, it's a terminal. Like, basically, everything you tell the LLM is going to be what the LLM's going to pretend to be. So this gives us, like, a lot of leverage because they've been trained so hard and really reinforced to learn about code,right?

Like, great at coding these days, you can, like, one-shot entire applications. You can call, like, 15,000 APIs that have been, like, recently built. Like, Sonnet 4 is amazing at knowing stuff that was just, like, built two months ago.

It one-shots things that I never thought would be possible before. So why are we stuck with, like, tools that don't even work that well,right? Like, I know a tool call is a function call. Why are they so bad at function calling when at the same time they can generate code that is so much better?

So code is a tool,right? There's many things that are called, like, even if I have just a SQL tool, I'm basically giving it code because a SQL query is code. If I have an edit file tool, I can give it code, and you can write code that calls tools, that calls code, that runs an agent.

You can do all this, like, infinite recursion stuff at the inference time. And basically, you can tell an LLM, "Please create the tool that I want,"right? If I give it a database schema, I can say, like, "Well, please create the tool GetCompanyContact."

No problem. Every LLM by now can do that. It can probably do it with, like, a 3 billion model, 3 billion parameter model locally. And so there are these kind of, like, magical genies that can just, like, create whatever you want at the moment you need it in the way you want it and modify it,right?

So why don't we leverage that instead of being stuck with this, like, "You can only call functions, you can only call functions with this schema that we've given you, and it's, like, static, and you can't even modify it"?

So the only prompt engineering you actually kind of need to do agents is, like, "Write the code to do X,"right? And there's a couple of papers around it. I haven't linked them, but there's the Voyager paper from two years ago already.

And there's the, I think, "Code Elicits Better Tool Actions," something like that, which is a very short paper that basically says, like, "You know, just, like, write code to do tools." And I've been on the LLM stuff pretty early to do code, like, once the instruct versions came out.

And

writing my little tools was the first thing I did. I was like, "I want to write a shell script to do this XYZ, and I want to do the shell script to do XYZ." I would run it, I would paste the result back into ChatGPT, which is basically like a, like, "I'm the tool caller" at that point.

And that was really, really useful. So the whole time before I read these papers, before MCP came out, before tool calling came out, I was, like, generating these shell scripts or generating little applications to do these kinds of queries or generating SQL queries that would push back.

And so it was already crappy back then, but it hasn't really gotten significantly better,right? Like, it feels that tool calling is still stuck at this, like, GPT-3.5 kind of intelligence. I don't know why, but they're not that great.

However, they're so good at writing code that the only MCP I think I need is, like, Eval,right? Instead of copy-pasting things, putting it in the shell, copy-pasting it back, all I need is, like, Eval around it. And it can be bash, and you can look at it.

Every coding agent does most of its work with actually bash calls. They call grep, they call find, they call ls, they call sed when they're, like, struggling with their edit file tool,right? It's like just editing a file; they actually don't know how to do it.

And when they really fail at calling their tool, they're just like, "Ah, fuck it, I'm going to write sed code," which is an insane tool anyway as well.

So you can realize, like, why do we even need MCPs when we just have Eval? And so here's an example where Eval is actually SQL,right? And I just ask it, like, "How many orders did customer John Smith place the last month?"

And I don't even tell it what the database is. I just say, like, "You have a SQL evaluation tool." So the first thing it's going to do is going to be like, "Well, what tables do I have?" Right?

And it says like, "Oh, well, I found customers and orders, so, you know, let me look at their schema." And it says like, "Oh, well, customers, you know, they have like a field called name, and then there's like a field customer ID in the table orders with a date."

Like, yeah, I know how to do a SQL query to do that,right? And so it's doing the structure, loading, it looks at the schema of customers and orders and says like, "Oh, look, there's customer ID." And now it can just write a SQL query that does the result of the orders, the join that it needs with the aggregation that it needs, and just returns the result.

And the crazy thing here is like, if you had a thing that's called like orders or customer placed last month, maybe the LLM won't realize that actually it has to pass, you know, first name, last name, for example.

So the first tool call fails, and then it like repeats its tool call and says like, "Well, oh, the date format is wrong. You have to do like minus 30 days." So it tries with minus 30 days, and then suddenly it gets like a huge table with all the invoice items.

It's like, "Okay, cool, now I have the information. I'm going to aggregate as an LLM, get the wrong number because they can't do addition, and you wasted 5,000 tokens, and you get like a wrong response." While this actually probably takes like, you know, 500 tokens, and you get a deterministic, repeatable kind of query that can reuse,right?

So this is why Eval is such a nice tool is that, oops, why did this go so fast? But if we take it to the next step, it's like, once it works, you know, why not store this query?

Why not say like, "Oh, now we have like a get customer order amount query." And so the way you can do that, for example, with SQL, is that you can just create a view to do it, and then suddenly you don't even need to look at the tables, you don't need to do complicated SQL, you just do like, "Oh, select amount from view."

And so this is what it looks like, is this thing is going to run. It's like a couple of queries. Like, maybe it even needs to do things in sequence where it's like going to be like, "Oh, I'm going to select the orders," and then I see like, "Oh, okay, I have to join this table.

Okay, let me do it again." And then I have to do this as this complicated code. But then being good coders, it knows how to turn like 15 queries into a single view. And at that point, and this looks like not the example of the view that I was trying to show.

So

I can show you that live. I used the wrong screenshots. But that shows you that you can easily create tools and functions and views and whatever. And so when you create a lot of functions and views and make them nice to reuse, that's called a library,right?

So instead of like exposing a GitHub tool, you just say like, "Well, your Eval now has access to the GitHub library." And it's pre-configured. You don't need to deal with like OAuth tokens and whatever. You just have like this whole API to do interactions with GitHub.

And finally, is like, if you put 10,000 functions into an API, the LLM actually knows how to use them. If you put them as tool calls, it doesn't. Which, I don't know why they get so bad once you add tool calls.

I mean, they've gotten better,right? Like Sonnet 4, GPT-4.1, like all these newer models have been trained to be a little bit better at tool calling, but you still like very, very quickly run into like weird things where it doesn't understand the parameters.

However, they're so good at writing code these days that I rarely have to fix anything in code, even for my own libraries,right? Like I just point them at my set of functions in a header file, and then it works.

So why, instead of doing the CRM MCP, you know, just do like import star from CRM, and then you're done. You can not only do you have all the tools that you used to have as an MCP, but now you can create your own tools that are really rich.

So, you know, just build the GetCompanyInfo tool.

Because you don't need to build an MCP GetCompanyInfo tool because you can actually just generate the code to call CRM ListInfo. And then you have like, you know, it's able to put a for loop around it, it's able to put any kind of code around it.

And so if you think a step further is that these tools and the code that's generated is like not just for the LLM, but like a lot of it is for us as well. So why don't we use the fact that we can use code now to build tools that are much richer than just a function call with like a little JSON window that you have to click in to edit it, if you can even edit it, but instead have tools that build UIs for us,right?

So instead of just incorporating a JavaScript interpreter that has like under the hood access to libraries, but then still on the surface just calls like functions, why not have something in the LLM host application that allows us to do UIs very easily?

And so I've built a couple of prototypes around it, but just to show you what this would look like if we go back to editing, you know, the tool input and the tool output, if you give the LLM the opportunity to say like, "Well, if the user wants to edit my input, you know, give it like a good UI instead of a little text window where you can edit JSON."

And so this could, if we go back to the previous example, it could like output some kind of UIDSL that's rendered by the LLM host. And suddenly, instead of having to edit like JSON fields, you get a slider, you get like dropdowns, you get all kinds of things, which the user can validate or tweak or, you know, attach a file, say like, "No, I don't want it to know about my memories," and then call the tool.

And maybe there's like preferences you can save. Like there's all kinds of things you can do around it,right? And then similarly, you can have a UI that allows you to edit the output with like maybe a scroll view and a filter.

You can say like, "Well, remove this, add this, the LLM shouldn't know about this, recall the tool by modifying my previous inputs," and you get this like rich UI to do your work.

And so what I built, and I can show you that live, is like a very simple MCP, which has a sandbox.js, it's written in Go, and it has two libraries. It has SQLite library, which is loaded, and it has a web server that basically has a single function that's called registerHandler.

And then you just write JavaScript for the handler,right? So there's a single call, it's called Eval, and then the LLM, when it calls it, can also register REST handlers. So what this looks like,right? Like there's execute.js, and if you want to load a file that you've already written, if you're like saying cursor, you can use execute.js file.

What this means, if I use the same query and,right, it's a prototype, so I'm a little bit aggressive on the prompting, it will suddenly write JavaScript where it's going to be like, you know, "Let me look at the tables."

So exactly the stuff from before. But it's already clever because it's Sonnet 4, so it will actually look for the company table in code already. It won't even wait for the result,right? Like it won't stream back all the tables in my database.

It will actually already filter them. So it saves on tokens and whatever once they return these tables in their schema. So you save two tool calls,right? And you save a lot of tokens just by the virtue of having Eval.

Just by the virtue of having it. It's the same tool. It has like dbSQLite query, but suddenly we're already like saving money.

And then in the same call, actually, I forgot about it. In the same call, it actually already does the querying of it. And it's well possible that, you know, you already get like some stuff at the beginning. Suddenly you get like the first 10 companies, and then it calls it again with all the companies, and then the LLM is able to show the result,right?

And you can see the schema is super ugly. It like actually doesn't even take care of printing it as JSON, apparently. It just like literally logs it out with the standard like Go syntax. And then you get the query response.

And this took two seconds,right? Or like three seconds. Not 15 tool calls. It took one tool call.

And so what I can do now is like just save it as a global function. It being Sonnet and being kind of on crack cocaine and just like deciding to do 15 things, it like generated 15 functions, sure.

So now I have all my tools,right? I've got companies info. I can get a company by its ID. I can search companies. I can do all kinds of things by the virtue of just like two tool calls. And one of them was just because of a syntax error because my prompting's bad because it's a prototype.

So why not create a REST API endpoint,right? Because it has registerHandler. So why not hook up all of these tools to all of these functions to a REST API, which like, allright, here you go,right? Like just like not really hard.

It just like calls the thing. Because if Sonnet, again, it like generated like even more.

And then I asked it to generate like a website,right? So it's registering a handler for the JavaScript. It's registering a handler for the CSS. And it's registering a handler for the HTML. It already has the REST endpoints. And boom, now I have a whole CRM.

I don't even need the LLM anymore. I can just start working with it. And that was a single tool, Eval,right? So I think we're leaving so much on the table by focusing on tool calling and saying like, "Oh, there's like agents.

There are these little widgets with these little creatures with tools." And that's our mental model instead of being like, "No, this is like a magical genie that can create anything I want when I want it," without even needing any big information because it's all in the training corpus.

You can edit the companies. You can add new ones, which will be stored in the database. Like this is a real, real CRM. This is just, I don't even, but I don't need cursor. I don't need anything. I can just say like, "Do it."

So to close this off, LLMs are absolute magic, and I think you should think, get used to thinking recursively,right? It's like if you ask the LLM to do something, ask it to do the code to do something, and then once you have something that writes the code to do a certain task, ask it to write the code to write the code,right?

Which is kind of what I did with the JS sandbox. I didn't just give it like JavaScript with loaded libraries, is that suddenly I have a JavaScript sandbox that you can use to create libraries that can then be loaded later on.

And those are APIs that I can then reuse in like different systems, and it's like all very circular. I can create everything I want all the time. And,right, they create words. And those words create more words. And then I can create words that create an LLM that create words that create words.

It's infinite.

And all of these words ultimately are going to make things happen in the real world. But if you focus on just the thing that you need to make happen in the real world, you tend to forget that it's not just tool calling.

It's actually we are engineers, the LLMs are engineers. Everything's this engineering in this case. So just write the code to solve the problem as we have been used to instead of saying like, "We have an agent," and suddenly it does everything.

That's not true. The LLM writes code that does everything.

So yeah, infinite loops of creation. I hope you enjoyed this talk, and I hope that you are able to bring back the magic into LLMs,right? Like the sparkle, sparkle, sparkle, because there's so much more than what we're trying, than what we're thinking of them these days.