Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil

The theme of this conference today is "Agents at Work." Unfortunately, for the next 18 minutes you'll be stuck with me talking about how agents don't work very well today, and how we can do better when it comes to AI engineering.

So there is so much interest in agents from all fronts: in the product world, and in the industry world, in academic labs, in research. If you're someone who doesn't think that companies will be able to scale language models all the way to AGI, then what we are going to see more and more of in the near future is agents that are not really deployed directly, but function as small parts of larger products and systems.

And this is what AI is probably going to look like in the near future. Six came up with a few dozen definitions of AI agents. This is one of them, where language models control the flow of a particular system.

In fact, even when people naively think of ChatGPT and Claude as models, these tools are actually examples of rudimentary agents at some level. They have input and output filters, they can carry out certain tasks, they can call these tools, and so on.

So in some sense, agents are already widely used as well as successful. We've now seen mainstream product offerings that can do a lot more. OpenAI's Operator can carry out open-ended tasks on the internet, the Deep Research tool can carry out 30-minute-long report-writing tasks on any conceivable topic.

So that's the first reason I think today's conference is timely. But the second reason is that on the flip side, the more ambitious visions of what agents can do are far from being realized. So on the left here is a vision for what agents can do, something out of science fiction films like the film Her, and on theright-hand side are how these ambitious products have failed in the real world so far.

Now, I'm pointing this out not to criticize the specific products on the slide, but to genuinely challenge the audience into the challenge of building AI agents that really work for the people who are about to use them. And so over the course of this talk, I'll talk about three main reasons why agents don't yet work, and what we can do to realize the potential of agents to get past some of these failures.

The first one is that evaluating agents is genuinely hard. So to begin, let's see some examples of how, when people have tried to productionize agents, these agents have sort of failed in the real world. Do Not Pay is a US startup that claimed to automate the entire work of a lawyer.

The startup co-founder even offered $1 million for any lawyer who would be willing to argue in front of the US Supreme Court using Do Not Pay in an earpiece.

In reality, a couple of years later, in fact, very recently, the FTC fined Do Not Pay hundreds of thousands of dollars. The reason for the fine was that the performance claims that Do Not Pay seemed to be making were actually entirely false.

Now, you might consider this a case of rushed invention, of a small startup making claims that it cannot back. So let's look at some of the work from more well-established companies. Law firm LexisNexis, as well as Westlaw, are widely regarded to be some of the leading law tech firms in the US.

A couple of years ago, LexisNexis launched this product, which it claimed was hallucination-free in its ability to generate legal reports and reasoning. But when Stanford researchers evaluated LexisNexis and Westlaw products, they found that in up to a third of cases, and at least a sixth of cases, these language models hallucinated.

In particular, in some cases, the hallucinations basically completely reversed the intentions of the original legal text. In others, the paragraphs were made up. They have about 200 examples of such errors in leading law tech products. We've also heard examples of AI agents soon automating all of scientific research.

So this is an example from startupsakana.ai. Sakana claimed they had built a research scientist that could fully automate open-ended scientific research. Now, our team at Princeton wanted to test this claim in the real world, in part because automating scientific research is one of our main research interests.

So we built a benchmark. We created this benchmark called CoreBench. The tasks in this benchmark are way simpler than what you might expect from open-ended real-world scientific research. They just try to reproduce a paper's result, even providing the agent with the code and the data needed to reproduce it.

So as you can imagine, this is far simpler than automating all of science. What we found is that the best agents as of today cannot even automate scientific research reliably. Less than 40% of the papers can be reproduced by the leading agents.

Now, of course, you can see these models getting better. And even if an agent can automate only 40% of reproducibility, that is a huge boost because researchers spent a lot of time reproducing baselines from past results. But on this basis, to argue that AI can soon automate all of science, or that agents will render scientific research as obsolete, is way too premature.

In fact, when people actually looked at how well Sakana AI's AI scientists worked, they found that it was deployed on toy problems, that it was evaluated using an LLM as a judge rather than human peer review, and that in fact, once you start looking at the results, they turn out to be very minor tweaks on top of other papers.

Think undergrad research projects rather than fully automating all of science. Now, a couple of days ago, as I was preparing the slides for this talk, I came up with another claim, or Sakana came up with another claim, where they claimed to build an agent for optimizing CUDA kernels.

The claims were indeed very impressive. They could lead to 150x improvement over the standard CUDA kernels that PyTorch comes with. The issue, though, was that if you sort of analyzed their claims one level deeper, you would see that they were claiming to outperform the theoretical maximum of the H100 by 30 times.

Clearly, this claim was false, and once again, it was because of the lack of rigorous evaluation. It turned out that the agent was simply hacking the reward function rather than actually improving the CUDA kernels. Once again, the point is not to call out a single company, but rather to flag that evaluating agents is genuinely a very hard problem.

It needs to be treated as a first-class citizen in the AI engineering toolkit, or else we continue risking failures like the ones on this slide. The second reason why building agents that work in the real world is hard is because static benchmarks can be quite misleading when it comes to the actual performance of agents.

And that's because for the longest time, we focused on building evaluations that might work pretty well for evaluating how well language models do. But agents are not the same as models. For example, for most language model evaluations, all you need to do is to consider an input string and an output string.

Those are the domains where language models work. It's really enough to construct evaluation. On the other hand, when you're thinking about agents, these agents need to take actions in the real world. They need to interact with an environment.

And so building this sort of evaluation that makes these changes possible, that creates the virtual environments within which these agents operate, is a way harder problem. A second difficulty in evaluating agents is that for LLMs, the cost of evaluating a model is bounded to the context window length of these language models.

You can basically look at these evaluations as having a fixed ceiling. But when you have agents that can take open-ended actions in the real world, there isn't any such ceiling. You can imagine these agents calling other sub-agents. There can be recursions.

There can be all sorts of systems, maybe just LLM calls in a for loop. And because of this, cost needs to be, once again, a first-class citizen in all evaluations of agents. If you don't have cost as an axis alongside accuracy or performance, you're not going to really understand how well your agent works.

And finally, when you build a new benchmark for a language model, you can basically assume that you can evaluate every single language model on this benchmark. But when it comes to evaluating agents, these agents are often purpose-built. So for instance, if there is a coding agent you want to evaluate, you can't really use a web agent benchmark to evaluate it on.

And this leads to a second hurdle, which is, how do you construct these meaningful multi-dimensional metrics to evaluate your agents rather than relying on a single benchmark to evaluate how well it works? Now, all of these concerns might be thought of as theoretical.

You could reasonably ask, why do we care if static evaluations don't really work well for agents? The reason is that because of these differences with the cost and the accuracy, because of the singular focus on optimizing for a single benchmark, we are basically unable to get a coherent picture of how well an agent works.

So at Princeton, we developed this agent leaderboard that tries to solve some of these issues. In particular, for example, for the CoreBench leaderboard I mentioned earlier, you can have multiple agents which are evaluated with cost alongside accuracy. So here on this Pareto frontier, you can see agents like Claude 3.5 scoring about as much as the OpenAI's O1 models.

But a Claude model actually costs $57 to run, whereas O1 costs $664. Even if the performance of OpenAI's O1 was a couple of percentage points higher, which it wasn't in this case, by the way, but even if it were, for most AI engineers, the choice here is obvious.

You would, any day of the week, take a model that costs 10 times lesser while performing about as well. Now, in response to this sort of two-dimensional Pareto, I've often been asked, are LLMs becoming too cheap to meter?

In other words, why do we even need to care about the cost of running an agent if the cost of creating these models is dropping drastically? And it is indeed true that costs have dropped drastically in the last few years.

If you compare Text DaVinci 003, which was OpenAI's model back in 2022, to today's GPT-4.0 Mini, which in most cases outperforms this older model, the cost has dropped by over two orders of magnitude. But at the same time, if you're building applications that need to scale, this type of approach is still quite costly.

And especially from the point of view of releasing prototypes, one of the barriers for AI engineers is you really need to sort of iterate in the open. And so if you don't account for cost, your prototype might soon end up costing you thousands of dollars.

And finally, even if the cost of scaling inference time LLM calls continues to drop, what is known as the Jevons Paradox, I suspect, will keep increasing the overall cost of running agents. So Jevons Paradox is this theory from a 19th-century British economist who figured out that as the cost of mining coal reduced, the overall usage of coal increased, not decreased, along several industries.

The same happened when the ATM machines were introduced all over the US. People expected a loss of jobs for bank tellers. But what happened was the opposite. Because ATMs were so easy to install, the number of bank branches actually drastically increased, leading to an increase in the number of bank tellers employed.

This is also what I expect will happen as the costs for language models keep dropping drastically. And that's why, for the foreseeable future at least, we do need to account for cost when it comes to agent evaluations. So how do we do all of this in an automated way?

Well, with the Holistic Agent Leaderboard, or HAL, we've come up with a way to automatically run agent evaluations on these 11 different benchmarks already, and many more are on the way. Beyond that, though, even if we come up with these multi-dimensional benchmarks, even if we do come up with cost-controlled evaluations, there are still certain issues with this type of evaluation.

And that's because agent benchmarks have sort of become the metric against which VCs fund companies. An example is Cosign, which raised its seed round of funding based on its results on Swebench. In fact, agent developer Cognition raised $175 million at a valuation of $2 billion, driven primarily by the fact that the agent did very well on Swebench.

Unfortunately, benchmark performance very rarely translates into the real world. So this is an excellent analysis of how well Devin works. Devin is the agent developed by Cognition from the very impressive folks at Answer.ai. Instead of relying on standard benchmarks, they actually tried to incorporate Devin into the real world.

And what they found was that over a month of use, they tried it for 20 different tasks, and it was only successful at three of them. So this is the other reason why this over-reliance on static benchmarks can be really misleading.

How do we get over this? One of my favorite frameworks to think through this is the work by folks at Berkeley called Who Validates the Validators? On the top is the typical evaluation pipeline, which consists of singular LLM calls against static metrics, which is the sort of broken paradigm for AI evaluations that we just discussed.

And at the bottom is what they propose. They propose having humans in the loop who are domain experts who proactively edit the criteria based on which these LLM evaluations are constructed. And that can lead to much better evaluation results overall.

This brings me to the last key takeaway for why agent performance does not really translate into the real world, which is the confusion between what capability is and what reliability is. So very roughly speaking, capability means what a model could do at certain points of time.

For those of you who are technically minded, this means the pass at k accuracy of a model for a very high k. That means that one of the k answers that the model outputs are correct. On the other hand, reliability means consistently getting the answerright each and every single time.

When agents are deployed for consequential decisions in the real world, what you really need to focus on is reliability rather than capability. That's because language models are already capable of very many things. But if you trick yourself into believing this means a reliable experience for the end user, that's when products in the real world go wrong.

So in particular, I think the methods for training models that get us to the 90% of it, what in Swig's term would be the job of a machine learning engineer, don't necessarily get us to the 99.999%, what is often known as the five nines of reliability.

And closing this gap between the 90% and the 99.9% is the job of an AI engineer. And I think this is what has led to the failures of products like Humane Pin and Rabbit R1. It's because the developers did not anticipate that not having reliability in products like these would lead to a product failure.

In other words, if your personal assistant only offers your orders, your DoorDash food correctly 80% of the times, that is a catastrophic failure from the point of view of a product. Now, one thing people have proposed to fix this sort of issue, to improve reliability, is to create a verifier, something like a unit test.

And on this basis, there have been several claims that we could improve the inference scaling capabilities of these tools and get to very reliable models. Unfortunately, what we found is that verifiers can also be imperfect in practice. For instance, two of the leading coding benchmarks, Human Eval and MBPP, both have false positives in their unit tests.

That is, a model could output incorrect code and still pass the unit test. And once we account for these false positives, what we have are these inference scaling curves bending downwards. So rather than model performance continuing to improve, if there are false positives in your verifiers, the model performance sort of bends downwards, simply because the more you try, the more likely it is you'll get a wrong answer.

And so this is also not a perfect solution to the problem of reliability. So what is the solution? I think the challenge for AI engineers is to figure out what sorts of software optimizations and abstractions are needed for working with inherently stochastic components like LLMs.

In other words, it's a system design problem rather than just a modeling problem, where you need to work around the constraints of an inherently stochastic system. And I want to argue in the last one minute of my talk that this means looking at AI engineering as more of a reliability engineering field than a software or a machine learning engineering field.

And this also brings me to the clear mindset shift that is needed to become successful from the perspective of being an AI engineer. If you look at the title slide of my talk, this title slide sort of pointed to one such area where we've already overcome certain types of limitations of stochastic systems.

And that is with the birth of computing. The 1946 ENIAC computer used over 17,000 vacuum tubes, many of which at the beginning of this process used to fail so often that the computer was unavailable half the time. And the engineers who built this product knew that this is a failure from the point of view of the end users.

So their primary job in the first two years of this computer was to fix the reliability issues, to reduce it to a point where it becomes well enough, it works well enough to become usable by the end user.

And I say that this is precisely what AI engineers need to be thinking about as their real job. It is not to create excellent products, though that is important, but rather to fix the reliability issues that plague every single agent that uses inherently stochastic models as its basis.

So this is what I'll leave you here with today. To become successful engineers, you need a reliability shift in your mindset, to think of yourselves as the people who are ensuring that this next wave of computing is as reliable for end users as possible.

And there's a lot of precedent for this type of thing happening in the past. Allright, with this, I'll leave you with the three key takeaways. It was a pleasure being here. Thank you.