MCP = Mega Context Problem - Matt Carey

Hello everyone. Welcome. Quiet down, quiet on. Hi. Awesome.

How is everyone? Yeah? Good? Thanks. Um, yeah. Want to hear some MCP versus CLI debates? Yeah, is that why you all came? Anyway. Um, hello. My name is Matt. I work on MCP and agents at Cloudflare. And welcome to my talk.

It's all about how we can make every API a tool for agents. APIs exist in the wild. How can we connect them to agents and make them do things?

So I really love my job because every day I get to decide, like, if an agent looked like this, would he do this? Or would he do this? And I think it's kind of fun. And we often fluctuate between the two of them.

Someone does something that you think is slightly funny, and then six months later we're all doing it and claiming it was the best thing in the world. So yeah, it's a really good crack. But the main part of the role, I guess, and what I end up doing day to day is, like, how do we give agents hands?

How do we let them interact with the outside world? And you're probably familiar with something like this. This is tool calling, function calling. It's been around for a while now. The LLM writes a function. You execute the function.

Bash, bash, bash. Weather in London is 18 degrees. It's not. It's like 8 and it's freezing. Um, sad times. And then from there we went from bundled tools to something like shared tools. Like, people made tools in their agents.

And you probably this is all recent history, so everyone's probably aware of this. But before MCP we had, like, people would bundle all their tools in their agents, and then they would keep them bundled in their agents. And then if I was, like, trying to interact with Gmail or something, I would make loads of tools for Gmail, bundle them with my agents, and that would be it.

And the next person would have to do exactly the same thing. And then we ended up with this big explosion of MCP and remote MCP about April last year. And the service providers were like, oh, we can, like, give everyone MCP tools, and then everyone can use the same standardized tools, and we just make it once.

And we provide it as another surface for people to consume our API. Maybe there's a CLI. There's an API. Maybe there's, like, I don't know, GraphQL API. And there's now MCP. There's, like, another surface. But this got a little bit fun because it was okay with, like, eight tools.

But then what happens if you added, like, a few more? Or a few more? Or a few more? Or a few more? Or a few more? And now you're like, I want to give an agent access to our whole API surface.

And, well, that ain't going to happen. Why is it not going to happen? You've exploded a context window of the agent. You've, like, completely annihilated it. This is 1. something million tokens. And this was the problem that we, like, came across a few, well, around a year ago now.

We were trying to give access to the whole of the Cloudflare API to agents. You put all of the you try and make naive tools out of every single API endpoint, and you fully explode a context window. Our OpenAPI spec is 2.3 million tokens.

Into tools, that's something like 1.1 million tokens. And that's, like, never going to fly, even with, like, the biggest foundational models. And in that time we were like, we know this is not necessarily an MCP problem, but it's how everyone else is doing it.

So we're going to adapt. We're going to adapt. We're going to improvise. And we're going to split up our API into lots of different product-based MCP services. So you've probably seen this. Like, a company that publishes 16 MCP services potentially, and then users have to interact with the one that they want to use when they want to use it.

There's much less context, but the user has to select. And most of the time there's kind of incomplete coverage. So, like, for instance, like, one of our product suites, we might have, like, six tools in our MCP server, but the total API maybe has, like, 30 endpoints.

Like, you've completely missed some coverage there. And this is not, like, fulfilling the goal of, like, how do we make every API a tool for agents. It's actually kind of annoying. So I think we did it all a little bit wrong.

Well, in Cloudflare we had 16 servers very, very quickly. We were hovering around 2,500 endpoints. I think we're actually at, like, 2,600 API endpoints now. But we basically couldn't split up all of these into all of our servers, and the users had to pick the ones that we wanted.

What we really needed was progressive discovery of tools. Who's heard of progressive discovery? Anyone heard? Yeah. Cool. And that brings us to the crux of the debates that everyone has online. And that is, like, how do we do progressive discovery?

And, like, is MCP dead? Like, was MCP like a really bad idea? And I'm going to say, like, I don't think it was. MCP is a protocol. All of these can be exposed over MCP. We just shouldn't be dumping loads of tools into context.

That's, like, the main thing. We shouldn't be dumping tools in context. And all capabilities. Like, in the future we might have prompts and resources more. Skills are basically resources. And we just shouldn't be, like, loading all of those at once.

So there's, like, sort of three ways you can get around that problem. There's a CLI, which people really like. There's tool search. Or there's a third one that we're going to come to a little bit later.

But, like, how would a CLI work for agents? So this is a sandbox in the background. And if I use our CLI and I do something like I just call Wrangler, we get a bunch of commands. The agent can, like, read these commands, parse these commands, and be like, oh, I want to interact with the database.

Let's do Wrangler D1. And maybe we want to list our databases. Whatever. And then after some period of time and some interactive process, apparently, we get the databases I have on my account. And, like, an agent can kind of do this, mostly.

And it can call --help to get, like, introspection on, like, which parameters it needs. And this mostly works. This mostly works. It's used very popular by things like OpenClaw. And, like, people generally really like CLIs. But you need shell access.

This is, like, the main thing. This is, I guess, the crux of it. It's like, you have to have shell access. And that's kind of annoying. So for things like Claude Code, they wanted a bit more of a structured way of doing things.

So they have, like, tool search. They have a search tool which loads the tools that they need when they need them into context. So say I want to, like, create a worker. What it would do is it would take the user question, it would do some sort of keyword matching, and then it would add k equals, say, 8 tools to context.

And then at some point the LLM is going to look at, oh, actually workers create this is the one we need. And so we're going to use that one. But the rest of them stay in context. Maybe it's not 8.

Maybe it's 6. It changes. But, yeah. You end up with, like, 2,100 tokens and only 500 of them are being used. But, like, it works. It works quite well. You only load the tools that are relevant. And then this last thing is a blog post that Cloudflare published in the summer of, well, last summer.

And it's like, how can we instead of doing, like, a static search tool, or instead of, like, enforcing an agent to need a CLI, how can we do something where we just let the agent write code? And we let the agent write code against our API.

And it turns out that TypeScript is actually, well, types are a very concise way of representing inputs and outputs in a way that an agent can reason about. So say you have all of these endpoints. You have, like, get worker scripts, or create a worker, or something like that.

We generate these types. And then we let the model, given these types, write some code against these types. So here we're doing codemode list workers. I hope you guys can see that. And we're going to try and list some workers.

So this might be, like, a user request to list workers. The model generates this code against a typed SDK that we generate from our API. You can generate them from OpenAPI specs. And then we can run that. And we can, like, list the workers that we have on our account.

And we could deploy a worker. That would be fun. Hello, world. And we could put it behind one of the hardest things to do at Cloudflare, which is so weird because it's such a powerful product. But we can add access, which is like our managed IdP.

And now this worker is secure behind access. Kind of cool. With, like, an access policy to only allow me into it and all of this sort of good stuff. Super, super easy. And an agent can generate all of this code given our types.

So this feels like a step in theright direction. We just let the model write code. We benefit from the model getting better. We benefit from, I don't know, like, improving our OpenAPI spec. It's like, that should be the source of truth.

But we had this, like, kind of weird thing where we thought this was awesome, and we were pretty stoked about it. But the clients didn't implement it. And then and when I mean clients, I've gone into, like, MCP terms now.

So the client is the agent. So we'll be referring to the agent as a client from now on. So the clients didn't really implement it. And we were, like, a little bit confused about why this is the case.

Like, this was sort of 8, 9 months ago now. And it's a better way of interacting with APIs. Just let the model write code against the API. But they didn't implement it. And why not? And that's because, like, running untrusted code is mega, mega scary.

Like, if I had said to you a few years ago, oh, we're just going to let a language model write some code that we're going to execute for our users without looking at it, without reading it, without seeing what it does, that might have potentially, like, secrets access.

Ideally it has some secret access. You'd be like, that's crazy. That's a CVE,right? It's a CVE. Like, it's a vulnerability. That's a problem. Like, and now we're proposing you to do this. So it is quite scary. Loads of things can go wrong.

It could read a file system, read some secrets that you don't want it to read. It could exfiltrate those secrets into a network request, run infinite loops, consume all your resources, do, like, really scary stuff. Run a crypto miner, you know?

That would be bad. And in the past people have tried loads of things to let people run code-like solutions. So if anyone's ever written a DSL, some sort of, like, JSON spec about how to interpret that as code, that is basically this.

If you ever used one of those integration softwares where you have to do that, that is this. They just don't trust you to write code on their servers. VMs also. People spinning up sandboxes to run code. Big sandboxes, big VMs.

That is this. And also Code Review.

But it's kind of lucky because we have a pretty cool primitive that solves this. And there will be other primitives that solve this. I just think this is the first. And so it's worth, like, shouting about, really. And this is, like, how do you run untrusted code in a way that's super safe for you and your infrastructure?

And it's kind of like this. So we can execute a worker from a string. And a worker is just, like, a little it's like an isolate in V8. There's many blogs about how all this works. I'm not going to go into it super deeply.

I'm just going to show you what it can do. So, for instance, we have this, like, this piece of code that was generated. And we're going to run this piece of code that was generated. And this ran on the backend.

It didn't run in my browser. It ran in a dynamic worker that's fully isolated. And, I guess, how can I prove that to you? If we do this one, we're trying to get some secrets here. Process.env. And if we print them, there are no secrets.

And we also have this weird Cloudflare global. Ooh, interesting.

If we turn that was with Node Compat on. If we turn Node Compatibility off, we don't even have process.env there. And it all errors out. So we can, like, influence we have this, like, programmable sandbox. It's not quite a sandbox.

It's like a very lightweight thing that you can load code into it and then run it. And I'll show some other options later. Like, it's not just us that has this, but we have one that we host for you and goes to, like, Cloudflare-level scale.

If you want to do billions of requests, knock yourself out. And now, like, here's one where the agent's written some code that accesses, like, an external API. And if we run this one, it's like, this worker is not permitted to access the internet of our global functions.

Or maybe we want it to access the internet. And now we can give it access. So it's a programmable sandbox with programmable guardrails. And all we're doing here is, like, flicking a boolean in the server. That's, like, all that's happening here.

But you can provide, like, a more in-depth function to be, like, only access things to these domains. And that's what we do on the Cloudflare MCP.

If we go to the next oh, speaking of the Cloudflare MCP, this is where I really hope the demo works.

So this is an MCP client in this slide. And if we ask it a question, we're going to get some we're going to get, like, an auth screen pop up. And then hopefully all this works.

Oh, insane. So now we have, like, complete well, we have read-only access to the whole of the Cloudflare API. All of my Cloudflare infrastructure I have read-only access to.

Which is pretty cool. These account IDs, don't worry about them. They're not secrets in Cloudflare world.

Cool.

So we just listed a worker. But you could do many more things here. Like, you can deploy workers from your command line. You can do what we did earlier and add access to something. You could introspect your DNS.

You could send emails soon. You can do loads and loads of other stuff. Like, it's very, very cool what you can do here. Because you have access to the whole of the Cloudflare API. All 2,000 and something endpoints.

And I guess, like,

this kind of brings up the question, like, where are we going with letting agents access external tools? Like, what does this look like? Like, you have people installing CLIs for everything and running it on their own running it on their own machine.

Maybe running it on a VM. That's kind of cool.

You have us being like, oh, you could just run untrusted code in this, like, other in this other place that's, like, really isolated. You have people doing tool search. You have people rendering UIs JSON. I don't know.

And I guess, like, my main thought is that, like, we're going to have so many isolated environments on the web. And there's going to be loads of infrastructure primitives that allow you to run this type of untrusted code on the web.

Because code is actually a very compact plan. Instead of doing tool calls, you can have one tool called code where the model generates the code of your choice and then you run it. And that code has so many more degrees of freedom than, like, an individual tool call.

So it makes sense to me that as the models get smarter, this is what we will do. And people will adapt their infrastructure primitives to do this. So there will be so many more of this. And you see this starting with, like, Pydantic Monty, Deno also.

And, like, we also have it with WorkerD, the dynamic workers I showed earlier. Like, people more people are going to build these primitives because they're going to become more and more useful. So, like, just, like, a little explanation.

This is WorkerD, like, spawning a dynamic worker in this sandbox and running some code to get a FIB sequence. You can do the same thing with Deno, with Deno run with some questionable checking. I have no idea what that does.

And then you can also kind of do the same thing with Pydantic Monty, the new code interpreter for running untrusted Python. Because it's Python, we have to download Python. Sucks.

This might never work. I actually have no idea. Oh, there we go. Great. So maybe you can see, like, where we're trying to go with this. There was a previous time where no one would ever run untrusted code.

That was a CVE. Like, you were just immediately, like, you have to, like, stop allowing that. And then it seems that LLMs, it's actually really good for them to run for them to write code that you can run.

And so now we're building the primitives to actually enable us to do that. And it feels like we missed out on this whole part of the tech scene that, like, we've never tried before. Like, in the 1950s when you wanted to run something on a computer in your local town, you printed out some punch cards and you stamped them and you gave them to the guy.

And that was kind of like running untrusted code,right? Like, that was kind of it. And then when we went to the cloud, we got away from that. And now I think we're going to go much more back to that where your users can write code.

Because your users are AI. And AI is very good at writing code. And that is how they're going to interact with your platform, whether through MCP, whether even through, like, Bash and CLI. Like, I don't mind. I think they're just going to write code against your services.

And your services have to be ready for this. Like, your APIs have to be ready to take a beating. Because they have to have good rate limiting. Because I can run this in a for loop on multiple sandboxes at once and just hammer your API.

Like, you have to have some way of protecting against that. Like, this is the new world that we're now going to be living in.

And that's, like, on the server side, on the services side. Like, what's going to happen on the client side? Because I think that's almost even more interesting. Because that's the user-facing side of things. Like, the user's not going to see the server.

The user doesn't care. The user just, why is my agent not getting my Gmail emails? Or why has it deleted my whole inbox box? They're not going to, like they're not going to see that. But on the client side, like, there's a lot of innovation that's going to happen here.

And I think we've stalled a little bit recently because actually building an MCP client in particular got really, really hard. Like, to actually build a client that was performative, that worked, you needed to manage stateful connections. You needed to manage resumability between those connections.

There's plenty of other reasons why building an MCP client was hard. But, like, it was a pain. An absolute pain. And so people had, like, the most stripped down clients they possibly could. They mostly offloaded to the MCP SDKs, which are quite bare bones.

And no one was building these, like, more unique UI experiences on top of that. And I think that is going to come, like, very, very soon. So the most obvious thing is we're going to have programmatic tool calling in the clients.

The previous slide we just did showing those sandboxes with WorkerD, Deno, and Pydantic, that is, like, just running untrusted code in a client. People are going to do that. If your client is remote, you're going to do it like that.

If your client is locally, well, just YOLO it, whatever. Just eval it, you know? It's going to be fine. But more people are going to do this programmatic tool calling. It's going to happen. And because you're generating code, people are going to save this code.

And they're going to save it in these, like, mini scripts. And users might be able to decide, oh, this action that I just did that the LLM generated for me, I want to keep that for later. And then it will be much faster.

So you can see things for things like cron jobs. A user might set up some web scraping job, like, without any knowledge of how web scraping works. And then it generates a script. And that script is ran, like, every day, every two days.

And whenever it breaks, because web scraping's, like, pretty brittle, the agent will fix it and re-save the script. Like, this stuff is going to happen. And I think, like, these saved mini scripts, they only work when you embrace, like, programmatic tool calling.

But they really do work. And then the last thing is we're probably going to have many, many more clients. Because they've been so hard to make up until now. And it is going to get easier. There's actually only there's not a huge amount of really well-used MCP clients.

That's going to change. And with that change, like, more people are going to be able to make them. More people are going to deploy agents to the cloud that end up being an MCP client. And I think more people are going to try and do this, like, stateless agent loop thing.

Like, it was fine to have sandboxes for every agent, like, running Claude code locally if there were a million agents in this world. I think when there are, like, 100 agents for each person oh, hello. Let's not do that.

That's going to be that's going to start getting really tough. And you're going to have to, like, embrace a cloud-native way of doing things, which means that state has to be something you can turn on or off.

And this is, I think we're nearing the end. But this is my last thing. It's like, I work a lot on MCP servers and on the SDK. And this is where I think that bit's going. I think we're going to see MCP as a middleware in an when you build an MCP server, when you build an API and you build an API service, it will be a flag that you can flag on in your favorite framework.

The SDK itself is getting super, super lightweight. And I think by the end of this year, we'll be, like, natively in every single at least TypeScript big full-stack framework. It will just be there natively. Because it will be so small, it will literally just express the protocol in itself.

And it will be silly for them not to have it. They'll just have a native integration. And they'll be able to do MCP is true on all of your APIs. And because all of the clients will be doing programmatic tool calling, you can express, like, your 1,000 APIs from one Next.js app and just do MCP equals true and expose it over at the most tools over MCP as well.

And I think that will happen. I mean, I think that's going to happen for a while. But I think we're pretty, really close there. And the last blocker is, like, fixing the SDK really so that it's capable of doing that.

It's capable of fitting in every single front-end, every single bundle really. And that's the plan.

You can find out more at the we have a codemode blog post that came out pretty recently. It's how we gave agents an entire API in 1,000 tokens. If you have a big API, you should probably do this.

Any observability providers, please just do this. Because it's really, really good for people to access your data. And thank you. Try it out. NPMI agents. Thank you very much.