Keynote: Why people think "agent" is a buzzword but it isn't

Hello, my name is Chip. I started an AI infrastructure startup a few years ago, and after selling it last year I have been happily unemployed. Before that I worked with NVIDIA, Snowflake AI, and also taught a couple courses at Stanford.

I have a mic. For today I want to talk about the challenges in building agents, or why people think "agent" is a buzzword, and why I think that it's not. I have been wanting to come to the AI Engineering Summit for a long time, but SWIX never invited me.

Until very recently I shared the section of agents from my book "AI Engineering." It's a very long section, it's like 8,000 words, and people seem to like it, so SWIX invited me here. And I actually prepared another talk for the summit, but then after watching a lot of talks yesterday I realized that people have covered a lot of ground.

So I created a new talk hoping to cover newer, more exciting topics. So this is a new talk created specially for this conference, and I hope you like it. I heard that if you are to give an agent talk today, you're obligated to define what an agent is.

I know that a lot of people think there are a lot of talk about agents is just hype, but I don't think so. I think there's a lot of exciting use cases for agents, but I guess I'm preaching to the choir.

So agent is not a new term. When I was working on my book, I decided to look at a lot of books, AI books from the '80s and the '90s, and trying to understand how people define agents back then.

And a definition that really resonated with me is from the book by Stewart Russell's and Peter Novik. So they defined an agent as anything that can perceive the environment and that acts on the environment. So let's say that you have an agent that plays chess.

That the chessboard is its environment, and its actions are the chess moves. ChatGPT,right? ChatGPT can interact with the internet. So the internet is its environment, and it can do actions like web browsing, it can also use a calculator, it can also generate text and images.

So one of the most popular use cases of agents nowadays is coding agents. And here is from the paper "Swe Agent" paper. And as you can see here, the environment for Swe Agent is a computer with a terminal and a file system.

And the list of actions it can perform includes navigate repos, search files, view files, edit lines.

So the environment determines the kind of actions that the model can perform. So if you are in a game, if an agent is in the game, it can only perform the actions that the game allows. At the same time, giving the model more actions can also help expand its environment.

So if you give the model the ability to browse the web, then now the internet becomes its environment. There are many reasons why we would want to give a model access to actions. So first, actions can help address a model's limitations.

So all the models have the cut-off date, and that makes it pretty hard to answer questions that require new information. By giving the model access to newer APIs, such as news or weather web browsers, the model now can get relevant recent information to answer questions.

A very common limitation that people discover very early on with AI is that AI is pretty bad with math. So instead of trying to train a model to be really, really good with numbers, you can simply give the model access to a calculator.

Another thing is a very exciting use case that you can turn text-only or image-only models into a multi-model model by giving it access to tools or actions. So for example, give it a language model,right? A language model can only process text and output text.

So if you want a model to also be able to process images, I can give you access to, say, an image captioning model. So I give it an image, it can use this tool to generate captions, and then use a caption to generate a response.

Now that model can process both text and image. It's very cool. But sometimes it's even more cool, so I think it's why agents are so exciting, is that actions allow you to embed models into the workflow. So now, for example, you can give the model access to your inbox, your Slack, your calendars, or the code editors, so that the models so that you can use a model instead of data in the workflow, instead of having to open, say, a web browser, so that you can use AI.

So when we are talking about agents, people will always ask me, "Okay, if agents are so cool, so why isn't everyone using it?" Tell me. Everyone is asking me, "Give me one good use case of agents." So why isn't everyone using it?

It's because doing agents is really, really hard. So for the rest of the talk, I will cover a few reasons why doing agents is so hard. So we're going to start with the curve of complexity.

So we know that task failure rate increases as the task complexity increases. This is true not just for AI, but also for humans as well. If you are given more complex tasks, we will be more likely to fail.

So let's say that you're building an application for your company, and you're okay with the failure rate of, say, 1 or 2 percent,right? And so a model makes mistakes 2 percent of the time for one step. So over 10 steps, the model will make mistakes about 18 percent of the time.

And that is a lot. That might be unacceptable. And if you increase your number of steps to 100 steps, the model becomes almost worthless. It will make mistakes most of the time. For a lot of agent use cases, a lot of agent use cases are pretty complex and might require multiple steps to solve them.

So it's not that you don't want to use agents for simple tasks, but simple tasks just don't usually need agents to do. And also simple tasks might have lower economic value, so they are less exciting for people to solve.

Let's go through a very, very simple example. Let's say that you want to ask the agent how many people bought product from company X last week. So this is a very, very simple query. But the agent might need to break it down in several steps.

It might first get the product list of company X, and then for each product in this list, it will want to get the number of orders from last week. And then given all these number of order counts, it will have to sum it up.

And then given these numbers, it has to generate a response to the users. So even with this very, very simple query, very simple task, there are only 4 steps. And the more steps there are, the more complex queries, the higher the number of steps, and the more likely the agent is going to fail.

So in the vast majority of agent use cases I'm seeingright now, it's very, very rare to see them consistently being to solve tasks that involve more than 5 steps. And I do believe that enabling agents to handle more complexity will unlock many, many new use cases.

So this is a tricky question. How do you know what complexity your agent can solve? Because you want to give the agent the task that has theright level of complexity it can solve, so that it doesn't fail and cause catastrophic business failure.

So different kinds of tasks, different use cases have different definitions of complexity. A very, very common way to define complexity is by the number of steps needed to solve the task. So this is

a synthetic planning benchmark that I'm working on, and I'm hoping to publish very soon. So I use a synthetic data set, a synthetic benchmark, because it allows me to control the level of complexity to study model behavior. So now I can ask the model to generate tasks that require 5 steps to solve.

So with that, in my benchmark, most models don't perform quite well. Most models can only solve tasks that have at most 5, that require at most 5 steps. And after 10 steps, most models fail. And this is consistent with another study that I have seen.

It's an older study now. It was from 2021. So it's in coding. So the results, so the actual pass rates for the task for models must have increased a lot by now. However, the learning, the insight, is still, I think, still very relevant.

So in this paper, they try to construct different doctrines, and then they ask the model, the agent, the model to generate code based on the doctrine. So they count the complexity, they measure complexity of the task based on how many steps needed in the doctrine.

So for example, for this task, first you ask the model to write code to convert the string into lowercase, and then you ask the model to write code to remove half of the characters in the string. So this I consider two building blocks or two steps in the doctrine.

And they found out the same result, as I did, is that the success rate, the pass rate, decreased rapidly as the number of steps increased.

But the good news is that with newer models, they are actually getting a lot better with planning. So here in the same results, you can see this. Here are these three very nice curves. They come from a DeepSeek R1, Gemini 2.0 Flash Thinking, and R1 Preview.

I didn't test on R1 and R3 because I didn't have access to this model when I ran this test. And you can see that the curves are being pushing upward. The newer models are able to solve tasks with more complexity, and I do believe that this is going to increase over time, allowing us to use an agent for more practical, complex, real-world tasks.

So here's another result from my benchmark. So as you can see here, it shows the number of tasks that each model was able to solve. And in overall, you can see that there's a pretty big difference between newer reasoning models, such as R1 Preview, DeepSeek R1, and Gemini Flash Thinking, and non-reasoning models, such as Sonnet 3.5, Gemini 2.0 Pro, or GPT-4.0.

Different use cases might define the complexity differently. So here is a paper from ZebraLogix that came out just last month, in which it's a logic task. So they define each problem's complexity by its number of Z3 conflicts. So you can see that they also got the same result.

The model's success rates decreased rapidly

as the number of Z3 conflicts increased.

So I think there are several tips to get the agent to handle more complexity. First, we might want to break tasks into subtasks that the agent can solve. So you don't want to give an agent a task more than it can handle.

So let's say that your task consistently requires something like 5 or 6 steps to solve, and the agent can maybe do at most 3 steps. Then you might want to break the task into 2 subtasks. Another way to have the model deal with more complexity is to test time compute scaling.

But

I think that in the last few years, people have been talking a lot about test time compute scaling. So it's one of the newer, very exciting concepts that give rise to reasoning models, and I'm very excited about it.

So the idea is that you can give the model more compute during inference, so that it can either generate using more thinking tokens, so it can think more. Or you can also use the compute budget to generate more output.

So for example, given a math problem, it can maybe output 10 different samples, 10 different solutions, and then pick the one that the model, most of this, pick the one that is most common, that the model thinks output most of the time.

So yeah, so it's test time compute scaling. You can also use stronger models. So using stronger models can also cause chain time compute scaling, because now you need to invest more compute into training a bigger model. Okay, so we finished the first challenge, which is the curve of complexity.

The next part we're talking about is a challenge related to tool use. So tool use is basically natural language to API translations. And what does this mean? So a lot of time for agents,right? We have humans using agents, and the humans give the agent instructions in natural language.

So for example, a human might give the agent a task like, "Hey, given this customer email, create an order." So the agent will need to translate that into functions that can perform this task. So it might first need to call a function to extract the customer ID from the email address.

And then it might call another function to extract the order ID from the content of the email. And then given this customer ID and the order, it would need to actually create the order. So now you can see that it needs to translate from this natural language to just a set of API calls.

The challenge with this is that the challenge comes from both sides of the translations. So for natural language, it can be extremely ambiguous, and at the same time, on the API side, you can have very bad API and very bad documentations.

So let's go into the first example of ambiguous natural language. Consider this agent with access to very, very simple functions, like fetch top products and fetch product info, so that fetch product info can return you the product price.

So let's say that the fetch top products take in three arguments, like start date, end date, and number of products,right? And then the user has this query: find best-selling products under $10. So now the agent knows that it needs to call the fetch top products.

But what would the start date be? What would the number of products be? How many products should it query? And what start date, what end date should it be? Would it be from does the user want best-selling products from yesterday, from last week, or from last month?

So this is very ambiguous. Okay, so now we talk about very, very bad API or bad documentations. In my coding career, I have been pretty fortunate or unfortunate to have seen really, really, really bad comments. So

as an engineer myself, I know that people don't usually like writing documentations. And if you can't explain the function to the agent, it's going to be really, really hard for the agent to know how to use itright. So I do think that when you give an agent access to a tool, it will need to provide necessary documentation.

At least you need to explain what the function does, what parameters it takes in, what is the type of the parameter, what does the parameter stand for. You also need to show different error codes for the functions, and also expected returned values.

And the more detail, the better. And that's not all. Because with error code,right, you don't just want, "Okay, this model returns this error like stat 99." It doesn't mean much for the model. You might want to explain to the agent, "Okay, this error is usually caused by this.

And if you enter this error, if you encounter this error, maybe this is how you should address it." And one company told me that one of the very, very one of the biggest improvements they got for their agents is after they explained and add to the documentation how to interpret returned values of the functions.

So let's say that the function returns a value of 1. What does that mean? So if you help the model interpret the result, the agent can actually be able to perform calling the functions a lot better and be able to plan a lot better.

And something very important thing to think about is that it's very tool use for agents can be counterintuitive for us, because humans and AIs have fundamentally different ways of using tools. So for example, humans and AI have different preferences.

So humans might prefer working with visual things like with GUIs, whereas AI might work better with APIs. So if you ask a human to use Salesforce, they might go to Salesforce website. But if you assign a task for AI, it will perform much better, not having to deal with a lot of visual gear and just calling the shit forward API instead.

And also, humans and AI operate in a different way. Humans, at least for me, I find it impossible to perform multiple tasks at once. So I could perform different steps sequentially, whereas AI can perform tasks in parallel. So for example, if you need an agent to perform, to browse, if you need to browse 100 websites, it could be very, very boring for humans.

I did that for my book. I browsed thousands of websites, but it was not fun at all. However, for AI, browsing 100 websites or 1,000 websites is extremely easy. You can just send out an open query like this, 1,000 websites, and get back the summaries.

And it's pretty straightforward. So that is actually a challenge for training or creating examples for the model to do planning or tool use. Because given a task, what the human annotator does might not be optimal for AI. So that's the reason why the reinforcement learning is so exciting, because with supervised fine-tuning, you are teaching AI to clone human behaviors, which might not be optimal for AI.

Whereas with reinforcement learning, you let the model figure it out with trial and error, and it might find ways to do it that is optimal for AI.

So there are several tips on how to make agents better at tool use. So the first is that you should create very, very good documentations. With everything, not just the function descriptions, parameters, but error, return values, like we just talked about, you should also give agents very narrow and well-defined functions.

So I just caught up with a friend working for a very big company. I wouldn't say the name, but if you say the search engine, you probably know what it is. And he was saying that for their use cases, they give their agents only 3 or 4 very narrow and small, well-defined tools.

For their agents, for their tasks, the agent just did not work at all with more than 5 tools. You should also, because of the ambiguity of natural languages, you can help the models understand the task or the user query better by using techniques like query rewriting or using intent classifier to help classify the user intent.

You can also instruct, or you should definitely instruct your agent to ask for clarifications when it's unsure of what users want. So for example, if you ask 5 best-selling products until under $10, it can make some random guess to fetch products from yesterday or from last year.

Or it can also ask users, "Hey, do you want top products, best-selling products from yesterday or from last week?" One pretty exciting or interesting direction I'm seeing is that a lot of companies are building specialized action models for specific types of queries and APIs.

So we already have specialized action models for coding,right? We have some models trained specially for, let's say, VS Code or for coding. So why not have specialized action models for different environments as well? So for example, I've seen Salesforce might be interested in building maybe I shouldn't say Salesforce, but say General.

Different companies with very complex ecosystems might want to train action models for their environment. Okay, cool down. We got the curve of complexity, the tool use issue with natural language and API translations. The last one is context. And it's really funny, because we have been talking about context for a long time.

First for RAG, and now for agents. We still talk about context. So models, AI has always required a lot of information. So before agents, a model has already had to work system instructions, which can be pretty long if you really want the model, really want the application to perform well and secure.

So you might want to instruct the model what kind of queries it should respond to, what kind of queries it should not respond to, what kind of tool it should carry. And they also use instructions and examples. But with agents, you see a lot more information.

So first, you might need to pass documentation about the tool to the agent. And the more tools there are, the more documentation will be needed. Of course, after you call a tool, there will be tool outputs that the model will need to keep track of as well.

And this will grow with more execution steps. And after getting back tool outputs,right, the model may need to reason, "Okay, now I got this result. What do I do next?" Or after a model generated a plan, the agent might also want to reason, "Hey, is this plan reasonable?

Should I execute it?" And all these reasoning tokens take a lot of time take a lot of take a lot of input tokens. And this also grows with more complex tasks. So the information that an agent can work with can grow very, very, very quickly.

And I haven't even mentioned other kind of information, such as table schemas for tasks like text to SQL. Let's say that you want to do a text to SQL task,right? And you don't have you have not just one table, but like 1,000 of tables.

So when you translate a SQL query, you might need to figure out what table to apply the SQL query to. And for the model to be able to pick theright table, you might need to pass in all the table schemas.

And if you have 1,000 table schemas, that's going to be a lot of information for the model to process. So one thing that I have experienced when I was working with agents that I would love to have more research on is how to make a model that works well with both planning and long context.

Because in my experience, the models that are good with planning are not necessarily the models that work with long context. And the reason is that plannings are like reasoning, usually require a lot of reasoning. It requires a lot of generate a lot of thinking or reasoning tokens.

So this kind of tasks are output-heavy, whereas for long context, it's input-heavy. And in my benchmark, my personal benchmark, I've seen that models that perform well on my long context benchmarks don't perform as well on the planning benchmark and vice versa.

So okay, so we talk about an agent has to deal with a lot of information. And that information might not fit inside a model's efficient context. So I want to highlight the word efficient here, because a model might have very long context, but then it might not use that context efficiently.

So a model might be able to fit in a million tokens. But if you give it anything more than 30,000 tokens, it might get really, really, really funky and hallucinate all the time. So at least in my personal experience, I have yeah, so I have done a lot of benchmarks evaluations just to see at what point of my documentation does the model start hallucinate and making up things.

So if you can't fit all your information into the model context, efficient context, you might need to realize on other form of information persistence or information storage. So context, you can think of it as a short-term memory. You should use this for it should be used to store information relevant to the task at hand.

And then you can also supplement it with long-term memory, for example, external databases or storage. And it's very common with use case like RAG,right? So if you connect a model to your external databases, then you're connecting it to long-term memory.

So it can also it's a kind of agent,right? You can store less immediately relevant information in external files. So let's say that your task requires 10 steps. So maybe the output from all these 10 steps don't quite fit into the context.

So you might want to store the output of the first few steps into external files and only retrieve the output when necessary. And of course, there's also short-term memory, long-term memory. And another level of memory system is internal knowledge, which is the knowledge that the model already has.

So if you have some information that models that is essential for the model to perform multiple tasks, you might want to include that in the training data and fine-tune the model on it so that the model can just use this as part of internal knowledge instead of having to waste context tokens.

Okay, so that is pretty much for today. So I think we talked about what is an agent, different challenges to building agents, including

first, including trying to get the model to handle theright the task ofright complexity. And we talked about tips like how to make the model handle more complexity. We talked about tool use challenges, like how to translate between natural language and API.

And we talked about how to help get models to handle longer context with a memory system. So thank you so much, everyone. I do have a website. And if you have any questions or if you want to talk about the agent planning benchmark I'm working on, feel free to reach out.

Bye.