Why Agent Hype can fall short of reality – Joel Becker, METR

Hey guys, thank you so much for having me. My name is Joel Becker. I work as a researcher, or member of technical staff, at METR, which stands for Model Evaluation and Threat Research, as we'll see in a second.

I'm going to be talking about AI capabilities: how do we know how performance AIs are today, how— how performant they might be in the near future, from these two different sources of evidence that seem to give somewhat conflicting answers.

You know, I could have done this whole talk without reference to METR papers in particular, but we'll look at two papers I've been involved with as examples of benchmark-style evidence, and then more economics-style evidence. On the benchmark side, measuring AI ability to complete long tasks— this is the paper that comes with the chart that many of you would have seen on Twitter and so on, that METR is well known for.

And then the second, this RCT, measuring how allowing AI affects developer productivity. And then we'll be talking about how to reconcile the gap that's implied between these two different kinds of measurements.

As I mentioned, METR stands for Model Evaluation and Threat Research. We are an independent research nonprofit that seeks to inform the public, policymakers, labs, about the degree to which AIs might pose catastrophic risks to society. The model evaluation part means that we seek to understand AI capabilities and propensities, and the threat research part means we try to connect those capabilities and propensities to potential catastrophic risks.

Okay, the first paper we're going to talk about, associated with this chart that many of you, I think, might have seen.

Taking a step back first before we dive into the paper, you know, how usually do we think about measuring AI capabilities using benchmarks on a SWE-bench or a GPQA, so on and so forth? There's some notion of 0% performance, or random performance.

So for GPQA, that's 25%, which corresponds to this floor, the worst you can possibly do. Perhaps there's a human baseline that's below 100%. For GPQA, I think this is something like 75% that represents maybe expert human performance. And then, of course, you can go all the way up to 100%, potentially, on these kinds of benchmarks.

But what does it mean? You know, if I'm getting 50% on GPQA, if I'm like half the way from

the floor to the expert baseline, you know, what does that really mean about how performant the AIs are? If I meet the human baselines, does that mean that the AIs are now as performant or even more performant than expert humans, in a relevant sense that I care about?

It's hard to interpret. You know, another thing that you see from this graph is that benchmarks seem to have less and less time between coming online, sort of giving any signal at all, and being fully saturated. It's harder and harder to create benchmarks that have plenty of signal that, you know, might be informative to us about how capable models are for an extended period of time.

So we're going to go about this a different way. First, we're going to gather human baseline data for diverse tasks spanning a range of difficulties. You should think of these humans as, you know, experienced experts, but on their first day or first week on the job.

These are not people with context on the tasks in particular. It's not exactly the kind of thing that's come up in their work before. But if it's a software engineering task, you know, they're a relevantly skilled general software engineer.

Same for the machine learning tasks and the cybersecurity tasks here that we'll talk about. The type of tasks come from these three buckets or task distributions: HTAST, which is a collection of software-based tasks seemingly requiring autonomy, you know, interacting with tools, interacting with the environment, thinking through the problem, not just this kind of Q&A-style dataset.

The SWAR suite, which are these atomic problems. These are problems that, you know, maybe GPT-2 can do, maybe it can't. Problems like, here are four files, one of them is called passwords.txt, which file contains the passwords? And then on the other end of difficulty, we have ARRI-bench, which are challenging, novel, open-ended machine learning research engineering challenges, which are very difficult even for top human experts.

In addition to gathering the human baseline data, we'll also, under as close to identical conditions as possible, measure AI performance for the AIs that we're interested in on the same set of tasks. And then we're going to convert the time it takes for humans to complete these tasks into an estimate of AI autonomous capabilities, as I'll show you in a second.

Here's an illustrative diagram, in this case for Claude 3.7 Sonnet, which was the frontier model at the time that this paper came out. You can see that, you know, for the very short tasks, something like 4 minutes or below, Sonnet is getting the answers correct, you know, essentially 100% of the time, or maybe even here, literally 100% of the time.

For the very hardest tasks, it's struggling. And then there's some range where we're kind of in the middle, you know, we're somewhere between 10 and 90%. I'll say that this empirical pattern, where models are less performant at tasks that take humans longer, is, you know, it's not a fact of nature, but it's something that we see pretty commonly, pretty robustly across models, at least on this task distribution, and I'd conjecture for other task distributions as well.

So we try and fit this dark purple line to something like this data on how long it took humans to complete the relevant tasks that the models are attempting. And then we call the point on the x-axis, this horizontal axis, this human time-to-complete axis, at which we predict the models will succeed 50% of the time, the time horizon of those models.

There's much debate in the 50% number. I can talk later about the reasons why we chose that. And then we'll do the same exercise for the other models. So here I have Claude 3 Opus, has a time horizon of something like 4 minutes.

That's where we're predicting that it has a success probability on this task distribution of 50%. For o1-preview, I'm seeing something like 15 minutes, so on and so forth. And then, of course, all these models, you know, they come out over calendar time.

So if we plot the time horizon, the x-coordinate, on this set of plots against calendar time, we find something like this. It looks, you know, kind of like an exponential trend that's going up at some constant rate. In fact, it doesn't just look like an exponential trend.

If we had a perfectly straight line here, it would indicate a perfectly exponential trend. We see something really remarkably steady, actually much more steady than we were anticipating when we went about doing this research project. And that's continued to be the case.

So many of you will have seen updates that we've made of this graph on Twitter. This is going all the way up to GPT-5.1 Codex Max, so extremely recent. The predictions from this, you know, shockingly straight line have held up very well, I think.

Taking a quick step back. What are benchmarks telling us, or here, kind of benchmark-like evidence? Well, one thing is that AIs can succeed at what for humans would be exceedingly difficult tasks. The tasks in ARRI-bench are, you know, really far beyond my capabilities personally, and, you know, the AIs are having a good crack at them some decent percentage of the time.

And the second, you know, kind of obvious, is that progress is rapid. On the other hand, you know, how much stock should we put in the evidence suggested by benchmarks? What limitations might they have? Lots, but here are three that I'll note.

One is, as I mentioned, these are humans who are, you know, expert in some relevant sense, but they're low context. It's something like their first week on the job. They haven't seen tasks exactly like this previously. They just have some relevant experience, presumably people who were more sort of, you know, not just having the relevant experience, but also highly familiar

with the set of tasks, would perform the tasks even sooner. And then we think, relative to those people, the AIs were more performant. The second is that benchmarks can be low ceiling. Even, you know, GPQA, I'll use that example again.

We're beginning to get to the point where that benchmark is totally saturated, not providing additional information for marginal models. Whereas time horizon is providing this nice way to sort of chain benchmarks together, in some sense, over time. But, you know, nonetheless, it's still very hard to create these ever-harder tasks when the time horizon of models is doubling every something like 6 to 7 months.

So even time horizon might be saturated in not too long, or the benchmark's underlying time horizon. And the next one is, you know, not a concern that's limited to the METR tasks, to the tasks behind time horizon. It's also true for SWE-bench.

It's also true for many of your favorite agentic benchmarks. The problems aren't very messy, in some sense. They don't require a ton of coordination with humans. They're often in relatively small, contained environments where not much can go wrong, you know, not these sort of massive open-source code bases, or other ways in which the problems can involve more interaction with the real world, or be messy, in some sense.

So we did this project. And then early this year, we were, you know, we were trying to think about how can we attack some of these limitations? What's a different source of evidence that might have its own pros and cons, but, you know, importantly be more externally valid in the scientific jargon?

Perhaps field experiments are the answer, some more economics-style evidence. So here, we might be interested in very high-context developers who are expert on the kind of tasks they're already doing. Speed up, or some notion of productivity boost, you know, it seems to have more signal through even some superhuman, according to benchmarks' range.

You know, perhaps GPQA is fully saturated and you're getting a 1.5x, 2x speed up, something like that. But you can still achieve a 3x, 4x, 5x speed up even after that. We maintain more signal. And the last is that, you know, the tasks are messier.

They are tasks that are coming up in people's real work. They're not synthetic. They're not small and contained. This is a real deployment scenario.

Here's what we're going to do. For this paper, we're going to gather 16 experienced developers on large, mature open-source projects that we'll go through in a second. Each of these developers will, on average, complete about 16 tasks from their real work.

These are issues on the relevant GitHub repositories, the kind of thing that they might otherwise have completed, with the caveat that the very longest issues we're not going to include. The tasks will be randomly assigned to AI disallowed or AI allowed.

AI disallowed, you know, it means what you think it means. It means software development in 2019. It means no AI-powered tab autocomplete. It means no cursor agentic coding tools. It means no LLMs via the web UI. Or they can be randomly assigned to AI allowed, in which case everything's on the table, you know, any of the AI tools I just mentioned, or not using the AI tools.

If you're in the AI allowed condition, you're not compelled to use AI. You just have the option. And we buy these developers cursor pro. So for the most part, that's the tool that they're using, with typically 3.6 or 3.7 Sonnet at the time, which was the frontier model when we conducted this work.

And then we're going to record the time it takes for the developers to complete each task and see the degree to which they might save time when AI is allowed versus when it's not.

These are some of the repositories. Many of you will be familiar with them. We've got the Haskell compiler represented. We have scikit-learn. We have Hugging Face transformers. These are, on average, a million lines of code plus. They've been around for 10-plus years.

The developers who are going to be working on these repositories as part of this study are, on average, the third top contributor out of hundreds, or even in some cases, thousands of contributors to these repositories. They personally have been contributing to the repository for something like five years, on average.

These are the top experts.

Some of you might have seen this graph too, and so the punchline's been spoiled. For the rest of you, we asked economics experts, machine learning experts, you know, these are people at major AI companies and labs, top academics, some graduate students, so on and so forth, you know, how much they expect developers to save time when they're using AI.

They say something like 40% or a little bit less. We asked the developers themselves, the study participants, how much they expect to be sped up ahead of time, and they say something like 24, 25%. Then we asked the developers, after the study has been completed, how much they think they were sped up in the past by AI being allowed on the issues they completed as part of this study.

And they say that it will have sped them up by something like 20%. And the punchline is that we find that developers are slowed down by 19%. They take 19% more time when AI is allowed relative to when AI is not allowed.

You know, when I first saw the data coming in, saw sort of early versions of this plot, I thought, presumably the same thing that many of you might be thinkingright now, that we've messed something up, that, you know, something's gone wrong.

There's some issue in how we've set up the experiments. How could it possibly be the case? You know, at least

these developers have access to the zero points because they can not use AI at any time. So we poured over, you know, many, many, many, many, many hours of screen recordings from these developers working on issues as part of the study.

We looked to dive into a bunch of hypotheses that might explain what's going on and try to categorize, you know, the things that we think are going on versus not. Many of this is listed in the paper. I'll just quickly go through some of the things that we think are contributing.

First, over-optimism about AI usefulness. That seems like an obvious one. You know, the developers, even after the study's completed, they think that AI's going to be helpful to their work. It makes sense they might overuse AI on that basis.

Two more, implicit repository context and high developer familiarity. You know, these developers are coming to these problems already knowing the solution to the problem. They don't, they're so expert in this work, you know, I imagine them as not trying to spend a bunch of time thinking through the solution that the AI can work through.

Instead, they're just limited by how fast they can type, which means that, you know, using AIs, instructing AIs to do it, comes with some significant time cost versus how they might otherwise have spent their time. I think many of us have the sense that AIs might be less performant on large and complex repositories, which is a difference from this benchmark-style evidence or from some previous work.

And then low AI reliability. You know, maybe the AIs are very performant on these kinds of tasks, but, you know, they're only performant 50% of the time, or 80% of the time, 20% of the time. And so, at the very least, you need to check their work afterwards.

And perhaps even you need to spend time correcting their work afterwards, which is something we see quite a lot on these issues.

One thing from the factors with an unclear effect that I'll mention briefly, I'll have to talk to people about later, is below-average use of AI tools, which came up in the public discussion. This is in the unclear column because it's sort of evidence for and against.

That's true for many of the things here. We don't have anything so conclusive to say we're still working on this line of work.

Here are some caveats, all important. First, you know, obviously, we do not provide evidence for all software developers or tasks. These are extremely experienced developers working on extremely complex, long-lived open-source repositories. I, in my own work, you know, I'm not as expert in the relevant sense as these people are.

I'm working on much smaller repositories. I feel more comfortable saying that even at this time I was sped up by AI tools, even if the developers weren't. This setting is weird. It's weird for the same reasons that it's interesting, this unusual developer population.

Second, the experiment is concentrated in March 2025, as I mentioned. We know that AI progress is rapid. Perhaps this result will have already changed by the time I'm giving you this talk.

So there's kind of puzzle suggested,right? The benchmark-style evidence is giving a very impressive sense of what AI capabilities look like today. Whereas the more economics-style, you know, I include labor market impacts working here too, in addition to our field experiments, look somewhat more bearish or unimpressive.

You know, why is the former not translating to the latter? At least naively, there seems to be a clash. How might we go about resolving this puzzle?

So one possibility is that, in fact, we messed something up. This is still live and on the table. You know, maybe the developers really are not very capable at using AI. And if we continue to run these experiments, as in fact we are, they'll, you know, learn more familiarity with the tools and so get productivity benefits that they weren't getting at the time.

I'm a little skeptical of that story, but that's one possibility. Another that economists like to bring up is that we're not incentivizing these developers to finish quickly. We're paying them per the hour, which we do for external validity reasons.

You know, looking through their videos, I really do not think they're developing differently in accordance with these incentives. But that certainly is one possibility that's on the table. You know, another more statistical in nature possibility is, you know, this is a small study.

You shouldn't over-update so much. From small studies, we are doing bigger things that I'm excited to release at some point. Okay, but let's assume we haven't messed something up and this is a result

that we think does hold up. How could we resolve the puzzle? So one possibility, you know, as I alluded to briefly, is that reliability needs to be very high to save time. That you need to be getting the answers to these problems that developers are putting in correct.

You know, something like 95, 99% of the time in order for developers to tab, tab, tab through and, you know, not spend lots of time verifying the AI's work, which, of course, is pretty costly from a time perspective.

Another possibility is SWE-bench-like or algorithmic costless scoring at the margin versus mergeability-like scoring. SWE-bench scores are not trying to account for, you know, whether the code is build-onable by other people in future, or whether it's matching quality considerations that aren't considered by the unit tests.

You know, perhaps AIs really are performant according to SWE-bench-like scoring, but not performant according to this kind of more holistic scoring that we might care about. Low versus high-context baseliners, I mentioned previously, these are just much more skilled humans.

You know, relative to those humans, perhaps the AIs are less capable. Task distribution, maybe these are just different kinds of tasks, you know, in particular less messy than the benchmark-style tasks. Maybe that's explaining what's going on here. Suboptimal capability elicitation, a huge amount of work has gone in at METR to making the agents as performant as possible given the underlying models on all kinds of tasks.

And, you know, that involves churning through a load of AI tokens. Perhaps that's less the case for cursor in particular at the time when we completed the study. And then interdependence across tasks. Maybe it's the case that, you know, if humans can complete task A and task B, AIs can only complete task A but not task B, and of course can do task A faster, then it still makes sense for humans to do task A and task B, not delegate task A because, you know, they need to know the outputs.

They need to know how task A was completed in order to reliably complete task B. I think that's part of what's going on. You need to maintain context as you're working through these subtasks.

Lastly, I will say that we are hiring, not just for this kind of work that you've seen being extended, you know, ever longer tasks, ever more ambitious RCTs, even more sources of evidence from which we can triangulate the truth about AI capabilities, but also for much more.

Besides, you can find this at meter.org/careers. In particular, I'm excited about research engineers, research scientists who might be hiding in the current audience. We're excited not just, you know, for research types with academic experience, but very much for scrappy startup people as well.

And we're also hiring for a director of operations. And with that, thank you very much for listening.