Skills are the New SDKs - Elvin Aghammadzada, DataRobot

We'll talk about skills today,

the current challenges about context, and then lastly about the eco-ecosystem that's being built around skills lately, especially with OpenClaw and all of those stuff getting a lot of attention in the industry. So let's get into it. Every AI or agentic app typically has 3 layers: the one that user sees, which is typically the UI, the user interface; the one that model sees, which is the system prompt as well as the tool descriptions; and the one that data sees, which is the schema of data or the input and output of the tool calls.

And these little bugs typically live on the second and third one, which is slightly hidden from the user. And so let's talk about these problems. The first one is that beautiful lie that we've been told, that the latest frontier models have infinite context windows.

They typically promise that each one has 1 million, 5 million, and even latest models, they promise that infinite context window, which incorrectly shapes our thinking perspectives about RAG and MCP in a sense. Because if we consider that there's infinite context window, we just think that we can dump the whole list of documents to the context and expect it to magically work.

This applies to MCP as well. Let's say we can write 100 tools and expect the LLM to use or pick theright tool at theright time, which is typically not true. Because in reality, the longer context typically doesn't mean better performance.

Because each time you put more context into LLM, that's one more place where the LLM might be misled, or its context might be poisoned. In fact, there was a paper called "Context Rot," and it proves that after 25% usage of the context window, so for example for 1 million tokens, if you used 256k of it, the performance starts to degrade.

So I'm pretty sure it starts really well as day in the life, but as time goes and as the conversation grows, let's say there are 100 prompt input and output responses in your one chat, and add it to 100 chats, and then each one of them kind of calls MCPs or tools each time.

Out of them, in the middle fails. And at the end of the day, the context is almost eating itself, in a sense that the latest frontier model can perform like a really poor model. I'm pretty sure we all have seen this, "You are absolutelyright," from Claude Code, even though it feels assuring that you're absolutelyright.

At one point, it makes you feel bad that Claude Code is making the same mistake it made 5 minutes ago.

The second problem is mostly about documentation websites. This is an interesting statistic that just from last year to this year, the traffic to documentation websites increased from 10% to 50%. That's coming from coding agents. The challenge here is the fact that docs are reaching for humans, but models are expected in a different format.

Because docs typically require some kind of intuition, asking for a follow-up, or if you don't understand it well, you can just Google it quickly and then find the answer for it. But models typically don't have those capabilities until you actually build a context engineering around it.

I want to add one more strategic point about the future of enterprise AI, and why we think that skills might be a huge contributor to where the enterprise agents or enterprise AI is heading towards. Let's talk a little bit about modes these days.

Now, if you want to understand modes today or tomorrow, we can go back in time to the history of software to see how has it been in the past. Typically, the things that have been the hardest to replicate have been the modes for the software companies, or even hardware ones.

So for example, whoever controls the hardware, the data, or especially the integrations in the SaaS era would be the ones that would control the market most of the time. So the real mode here would be friction, where you would make it genuinely hard to leave, to make someone leave your platform.

So the core idea here is the switching cost was too much that it was just hard to switch. Now, these days we're hearing news where Claude Code would rewrite 100,000 or a million lines of code from Python to Rust.

So basically, we're getting really, really good at switching costs and reducing it. Now we can have better tools, better APIs, just better languages, and then switching it in a few days.

Now, skills are creating an entirely different ecosystem, where instead of creating a friction mode, they're actually creating fluency mode. So the core idea is switching the layer a little bit so that each time your skill makes the experience better for whoever is the one that's using your platform.

And that fluency compounds by experience. The real result that the users of your platform are getting from here is the experience. If you would look at the definition of experience, you can realize that it's satisfaction or minimum amount of friction between intent and the outcome.

So basically, whatever the user is trying to achieve through your platform, achieving it in maybe minimum amount of time, or trying to optimize the whole satisfaction layer, or whole experience layer of it. If you look at the equation, you will see that overall friction modes and fluency modes are the opposite.

Friction modes are typically defensive, so you're trying to make your experience in a sense that someone wouldn't switch. Now, fluency modes are offensive, in a sense that you're making the whole experience really nice, in a sense that you're making it genuinely hard for someone to switch anyways because they like the platform.

So we think that the feel of the thing, or the reliability of the outcome from the start, which is your intent, to the result, is the whole experience layer. And the skills are one of the best tools that can actually, quote-unquote, "commoditize" that experience layer for agents.

So the real mode in SaaS era might be the core idea of skills, where you would commoditize the feeling of your platform, or in other words, how trustful or reliable you are in terms of intent and outcome equation.

When someone evaluates the platform, they typically have a checklist of things that they take a look,right? They typically look at how secure your platform is, are you compliant with all sorts of new standards that's coming up every week, how is my data governance layer, where is my data stored, are your SLA guarantees, can I see the logs of my agent, how about tracing, how is the integration layer, are you able to integrate to my auth layer that's connected to my internal on-prem systems.

So now we think that checklist really has a new item, a new guy in the room, which is teachability. Again, coming back to the experience layer for agents. So here the core idea is, if someone is new, or if my agent harness is new to a platform, how easy is it for that agent harness to pick up that operational knowledge that you've encoded to the skill and apply it directly, and then get into the result in less than a few seconds.

So the core idea of teachability is to make that experience easy as possible whenever someone wants to use your platform for the first time or for subsequent times. So if you wrap it up, we think that the skills might play a huge role in the enterprise AI or enterprise agent world, in a sense that it might define what the mode is.

If the mode these days is about fluency, skills will help it, in a sense that the value that you are giving will get compounded by each skill you add to your platform.

Why think about standards? I feel like I just need to skip this slide for this audience. Because standards are good,right? For example, React came about recently, but if you go back in time to 20 years ago, it almost feels like LLM and agent ecosystemright now.

So 20 years ago we didn't have React. It introduces a philosophy on reactivity and modularity to the way we build apps. And same for LLMs and agent ecosystem, in a sense that we don't yet have that React moment.

It's still up on here in terms of how to systematically develop really long-running agent systems.

And as I said, context engineering, in a sense, is kind of a one attempt to solve that problem in an organized way. So if you would summarize what context engineering is, it's almost an idea of minimizing the amount of information that flows to the LLM at theright time, so that we can filter, so that we can fill the context with theright information at theright flow, at theright moment.

So let's look at this example on theright. So if we consider a typical agent, whether it's Claude Code or our own agent that we built for a customer, it typically has system instructions, Cloud.md, coupled built-in tools, maybe also connection to MCP,right?

And on top of that, a user or a customer is kind of prompting the LLM. Typically this 40% is the smart zone, where the LLM can potentially perform well. But if you are getting past this 40% context used, it typically goes into a zone called dumb zone.

The core idea here is the fact that even though you have 100% context available for you, but as you go past 40%, then you are actually getting really dumb responses. So the core idea here is to make sure that you're not passing 40% even without starting to work with your agent.

So imagine you have all sorts of different instructions, a lot of MCPs here, and then you add one more block here for another MCP, one more block here for another MCP that's connected to internet, one more for web search, and things like that.

And then even you haven't ever chatted to your agent yet, now you've filled your 40% context window. And then when you just start to chat for the first time, you start from dumb zone from the first call.

So typically creating a great context has pieces to it,right? Let's cover each one of them briefly. The first one is instructions that kind of comes by default. You put it to the system prompt. The second one is external data.

Maybe the user will upload some PDFs or any kind of RAG. And the execute history. Obviously, LLM needs to have access to what happened on that session, what's the past full call results, and where are we in terms of state.

And the latest agentic systems has memory, in a sense that they can remember things across conversations. And then the output schema, the correct way to output all of these stuff to the user. So as you see, each one kind of adds one more level of complexity to the things that agent needs to handle.

And it almost creates end-times and problem in a sense that one agent or one LLM needs to be connected to all of these stuff at once, at the same time, at any time of the conversation, which is a pretty hard task to do, actually, if we would emphasize LLMs.

The core idea is, instead of hard-coding everything into either one of these items, basically, and then loading it to the LLM, we would progressively load whenever those instructions are needed to the LLM. And then LLM would have access to a piece of it at runtime, and then it can load it to its memory whenever it's needed.

The problem that skills typically solves is it enables a rich context access. In a traditional context, everything goes into the context which would lead to context rot problem that we covered. But with skills, those capabilities, whether it's about web search or whether it's about some capability that doesn't exist in the main agent, that leaves externally, and then LLM or agent can have access to it.

So if we would give an example and compare with MCP, typically if an agent is connected to an MCP server, even this number is really small for 15 MCP servers. If it's connected to 15 MCP server, I'm pretty sure it's consuming over 100,000 tokens per session, just in tool definitions itself, without starting the conversation yet.

But same operational headache becomes really small, probably at least 10 times small with skills, thanks to progressive disclosure. So this one is only expose metadata of skills to the agent or to the LLM at runtime so that it can act like an index in a database.

Just like how indexes work in a DB so that you can quickly search for stuff, same for the metadata for skills. Even if skills might have thousands of tokens in it, LLM has access to the metadata of it, which most of the time leaves at the top of the file, which is being loaded to the context on demand.

This creates mental model shift on developers as well as the industry, in a sense. Because the skills are kind of becoming almost like a framework across platforms, like Cloud Code, Codex, and then all of them supports this perspective.

Why you would ask long-running agents, even coding agents are acting like a general purpose agents,right? So when a customer asks us to build a supply chain agent or an agent that's in a manufacturing space, we don't really have to build a separate agent for that specific use case.

It's the fact that we can still take the main general purpose agent as the engine and then put skills on top of it. So you see how skills creates a shift in terms of perspective about how to build solutions, in a sense thatright now you don't have to actually build the agent itself, the engine itself, but actually build skills for it.

And then that's a way to switch agents between industries, between different domains.

Allright, moving on to part two, where we'll cover briefly on skills versus MCP. So they kind of solve slightly different problems, in a sense. So I just want to clarify this. Both of them are from Anthropic, both MCP and skills.

So they're not competing for a thing. So in a sense that MCP kind of definitions for what actions the agent should take, in a sense that it doesn't change the agent definition and then what the agent is, what capabilities it has.

And for skills, it's different in a sense that it helps to think about problem and then change. You can basically customize the way agent acts through skills, how to think about this problem. And then skills introduces you a way to self-modify, in a sense that agent can modify or update its own skills based on the experience that it had with that skill.

On the other hand, for MCP, it's sort of a server that sits in a separate environment where the agent has access to use that tool, but it doesn't have access to the code of that MCP tool, in a sense.

Anything to add here, Carson and others?

I think this is a good opportunity to answer to Joji's question. A thing a skill can do is expose an MCP server in its skills MD that says, here's your MCP server. It can also just actually, you can dump the tools that were in your MCP server into your skill directly, and the agent can then execute them.

So it can actually service and create its own MCP server as part of a skill. This is why we get so excited about the robustness of skills versus MCP. It's not just self-modification, which you could replicate with MCP,right?

MCP with memory and caching and user customization and using tool calls that are actually agents themselves. Those agents may even have their own skills that you could specify as part of the tool is load these skills and execute this thing,right?

Like there's this kind of incestuous relationship, but still skills live on their own as a distinct thing that can operate, templatize code, execute code, execute MCP servers, build and create and run, and then therefore add to their root context window new MCPs, just all via the skill.

So it's like an entire complete package of what an agent can become. Essentially, Copilot could be a skill.

So the core idea is use skills when the hard part is just about reasoning or thinking in theright ways. But MCP solves harder problem for us in terms of authentication, access, or the horsepower. If you want to actually isolate the process for some of the execution that really requires a lot of resources, then that's an easy way to let that have its own server.

That can interoperate,right? Skills run with your agent's machine. So there's a little bit of a loop dedupe, but they're still sufficiently different that MCP gets to live in its own world and be a production piece of code, and skills get to be kind of vibey.

I think MCP is Intelligent Apps, which it is actively working to adapt,right? It's the older standard. And skills, we didn't know quite how powerful we were until we had 85,000 of them out there. You still don't want that hosting capability,right?

I don't have access to a GPU on my computer that is sufficient in order to do significant operations or to do semantic search across 400 terabytes of documents. In those cases, I still need a robust hosted MCP server.

The way I get my agent to use that MCP server is you can use skills to do progressive disclosure for that MCP server that then has that 500 terabyte disk or the access to 75 GPUs or whatever it is that that tool needs.

So they're still interconnected in a way that they both have their uses and are kind of inescapable. Is that MCP server going to access data in a restricted environment in your medical records or whatever that your personal computer can't access?

Does it need resources that are unachievable on the agent that you're executing on? It's essentially a way to execute remotely, and that still always has a benefit.

And one last argument against MCP, which might turn out to be interesting, is the fact that it turns out that current agents don't need a lot of tools. If you go into the way Codex or Cloud Code is developed, you will see that there's only like a handful of tools that's being run.

It's not like there's hundreds of tools that's being executed in a separate MCP server, which also proves the fact that a typical skills folders or capabilities folders with different options can replace MCP as long as you have a good base reasoning model.

Correct.

Skills has two different readers. The first one is the front matter, the top of the skill, which is here. If you see an example of one of our skills, you see that this is almost part of that MD file, but this acts like an index for that skill.

So at runtime, this is the only thing that's loaded to the agent's context, agent system prompt. But this one is loaded to the agent's context, which is the body of it. So when I say there's two different readers, the first reader is the agent's runtime, which is the system prompt.

So the reader for this one isn't the context itself. It doesn't really burn a lot of context because it's just a small XML that has probably less than 100 tokens typically, which agent do have access to prior to the execution.

And then second one is the markdown body, which is read by LLM if activated. If the LLM thinks that we need that skill, then it will be loaded. So this is the first one, less than 100 tokens. The second one, on activation, which is typically less than 5K tokens.

The level three is what third one is, which is scripts. The scripts can be in two ways. For Cloud Code, for example, they're executable in a sense that the Cloud SDK or Cloud Code can actually execute this getDeploymentFeatures.py and then only put the output of that execution back to the context.

But a lot of other examples, a lot of other platforms support this one, which is also code, but not executable in a sense. Just serves as a purpose of example for the agent. So three levels of managing a simple skill.

First one is the front matter, second one is the body, and third one is extra scripts or context. Part four, talking a little bit about ecosystem and what's going on around there. So there is 26 plus platforms that support thisright now, like Cloud Code, Codex, Copilot, Gemini CLI, and others.

There is probably at this point hundred thousands of skills out there. And then there is marketplaces where people sell skills for money. Evan Jensen talked about Clouhub, which is a skills registry for them. That being said, we are also part of ecosystem, and we also published our open source agent skills repository.

This is a really good demonstration of both the power and the risks of agent skills. OpenClawright now has become the repository that has the most stars beating Linux and React. And I feel like there is a lot of marvelous ways they've done the context engineering in a sense that works across different apps and chat applications.

But I feel like probably a couple details regarding their sub-agents, which kind of serves the purpose of keeping the context clean, as well as self-evolving skills, made it really impressive for whoever looked at the code. The fact that it can kind of self-heal based on the experience of it with skills, as well as write its own skill to extend its capabilities, in a sense, scary, but what made a good impression, the fact that this is just a coding agent that has access to about 10 tools, which can do anything for you as long as you have skills on top of it.

So that kind of changes the way we think about customers or enterprise solutions in a sense that we just need to develop one good agent, which is base for everything, and then you can just build skills on top, which solves your domain-specific problems or whatever your customers are asking for.

So if OpenClaw can write skills, then let it write it,right? But this touches the fact that typically we need humans to write skills. There is a lot of research on this that's recently published, but they published that actually the LLM generated skills hurts the performance of LLM in a sense that it uses more tokens, it spends more time to go and actually reason up about the problem rather than helping it to be faster or to use less context, in a sense.

There's a lot of risks to consider here. If the LLM is writing and executing its own skills, then obviously there's a ton of risks in terms of prompt injection because everyone is writing skills these days and then publishing it out there.

Pretty sure you have seen on the news, there is bad stuff that happened with OpenClaw. And there is no isolation. One of the advantages of MCP in the sense that you isolate the process for your MCP and your agent, but everything happens on your laptop, on your agent, on your environment.

And last one here is about the fact that these marketplaces still lack the verification control. Just like how NPM was scary 10 years ago, in a sense that you would still check the number of stars or who developed this before you downloaded the package, same for skills.

Closing thoughts, context is a budget. Context is almost like a limited resource that we need to carefully filter information. Definitely longer context doesn't mean better. Every irrelevant file, every irrelevant web search or error message really drags agents' attention, and then you lose a lot of reasoning power.

Skills are complementing MCPs, at leastright now. A skill is only as good as the human who wrote it based on our experience. And lastly, skills are software, which can take weeks to build, so that we should actually start versioning them, evaluating and testing them, and actually writing good skills.

These are a couple resources that we've used. Thank you.