Evals Are Broken, Use Them Anyway — Ara Khan, Cline

Allright, allright. First of all, thank you so much for coming. I'm actually rather surprised. A lot of times you're working on this stuff and you're, like, cooped up in a room, and you're like, "No one cares." And then it's like, so many people showed up, so I suppose someone cares.

So anyway, so the title of my talk today is "Evals Are Broken, and You Should Use Them Anyway." And a lot of this talk is just like a straight-up critique of, like, the way we do evals these days.

And I kind of want to help you out. I kind of want to give you a way out of this. It's like, you have this, like, interesting technology, and you can use it, but there's, like, so many ways to, like, mess it up.

So I want to help you out. So my first claim is that people are wrong about evals, actually. Let me, uh, let me correct myself. Most people are wrong about evals. And I want you to beright about evals.

I want you to, I want you to use them. I want you to, like, I want you to be able to build with them, interpret them, use evals in your own agentic flows, leverage them in any way since you can.

So that's, that's, that's basically the point of the conversation. So to beright, to beright about something that has, like, a lot of nuances that can go in, like, many different directions, the fundamental question is, like, how are people wrong about that thing?

Right? So there's basically two camps of people who are wrong about things,right? So there's two camps of wrong on evals. The first camp is a camp of objective metrics. So the objective metrics camp is this. Like, there are people who would, like, look at this dashboard,right?

And they will interpret this as something akin to, like, like, it means something. As in, like, GPT-5.4 is effectively the same as, like, Gemini 3.1 Pro preview. Believe me, they're not the same. There's a lot of these models which feel like show up, like, with similar numbers.

And these numbers, at a certain point, it's like, this is, the whole thing is a hoax. Like, you won't believe it at all. So there was this tweet that came out, like, just this morning. And it was a critique of Meta, where Meta came out and it was just, like, classic benchmark maxing.

Just, like, we're doing best on the benchmark. Everything's great. And I assure you, if you try a lot of these models, like, you just, just won't hold the, hold the test of, like, actual real-world evidence. The other one, the other camp, the other way where people are wrong is that they go too far the other way.

So this is basically, like, the test camp. So people in the test camp are kind of like this. They just, like, they're, they're an archetype. They're, they just, they think that it's, like, it's everything's about, like, vibes and everything's about, like, you know.

Like, it's like, if you ask them, like, "Why do you like Opus?" and they'll say things like, "I like talking to her." Like, they'll, they'll, like, anthropomorphize it. And, and that's also notright either,right? So I think the truth is somewhere in the middle, that the evals are not the end-all, be-all.

They're also not completely useless. There areright ways to use them. There are wrong ways to use them. So in order to do that, I'll give you, like, three stages that will help you, like, use them really well. So the first stage is that you can, like, leverage evals from other people.

The second stage is to, like, use evals to improve your own agents. And then the third stage is, like, build, actually build your own evals for specific use cases. In the interest of time, I could personally talk about evals for hours, but, like, I can only talk about level one and two, and I think those would be most helpful for most people in the audience.

So I'm going to give you, like, a few heuristics to interpret eval. So the first heuristic is that whenever a model apps comes out with, like, a number, just don't believe them. Just don't. Just, like, these are approximations.

Coming back to the tweet. Like, just, like, just don't believe the model app eval numbers. Like, they're somewhat of an approximation. Sometimes they're good, sometimes they're not. There was this tweet that's, like, a pretty cool one where Nixon said that, like, a lot of, like, AI researchers and engineers routinely dismiss evals.

They don't really, they don't really think of it as, like, something that's, like, the numbers to be taken that seriously. And to some extent, it's a matter of, like, actual trying and preferences. And I think, I think that is, like, somewhat more accurate.

Wait a second. Yeah. So the second heuristic is that, like, you want to stay current, but you don't want to be the earliest adopter. And why am I saying this? So this is, like, EPOCH index is basically, like, the aggregate score of, like, different models on, like, evals.

And if you notice, like, in the last, like, two years, every single couple months, like, the frontier, the frontier model is changing. And it's changing so fast. Like, it's just, like, it's so hard to keep up with this stuff.

And I've worked for, I've worked on this. I've been doing this for a living for years at this point. And I, even, even me, I have, like, preferences changing so fast. So, like, I think that, like, when you're, when you're working through these things, like, the way I would recommend is that, like, let the thing come out first.

Let things set on fire for, like, a couple weeks. And then if the, if the thing still stands the test of time, I think at that point you should, like, do your model switch and, like, try something rather than, like, always trying to be on the cutting edge.

Like, the people who have to always try the new model and try the new things, like, those will be me. But, like, I do this for a living. And you don't have to. The third, the third heuristic for evals is that, like, you need to look for very, very new and very precise evals.

And the reason this is necessary is that, like, a lot of evals are, like, at this point, they've become, like, standardized. They're actually kind of old. Like, they're, they're not useful for you. So, like, this is a, this is a blog post from OpenAI where they straight up said, "SVE Bench verified no longer measures frontier coding capabilities."

And I think, like, to a lot of people in the AI research community, that was, like, very obvious. It was very obvious that SVE Bench doesn't measure frontier coding capabilities because it had, like, it would have problems, like, solve the Fibonacci sequence.

It would have problems, like, you know, do matrix multiplication or something. And it's just, like, it doesn't apply to, like, real-world software engineering. So you want to have something that's, like, very new, but also, like, actually legit. And it takes some discernment to figure that out.

So that's, like, the first part. But the second part is, like, okay, now that we know that, like, okay, that we have a few heuristics of, like, how to, like, use evals, like, how do you use evals to improve your agent upon them?

And I think this is the part where I kind of lean into, like, the core philosophy of this conversation where, like, you want to think of evals as, like, an engineering problem, but also as a philosophy problem,right? So the engineering problem is obviously hard, but the philosophy problem is also very hard.

The philosophy problem is that you, you want to, you want to, you have a problem and you can't exactly approximate the, the search space of, like, where the problem could go, where the problems could fail. It's, it's sort of somewhat easier-ish to do it for coding problems.

But even then, coding problems have, like, an infinite search space. They can go in any direction. So you want to, you want to build evals that, like, are somewhat more approximate representation of the actual thing that you're dealing with.

And for us, like, to give some context in Cline's journey. So I work at Cline. Cline is an open-source coding agent company. We have a, we, we have a very interesting product. I encourage you to try it out.

So in Cline's journey, one of the things that we dealt with is, like, in the last year, one of the things we found is that, like, there were, like, a few evals available. At the time, it was, like, we were very rudimentary.

Every, every other company was very rudimentary as well. And our thinking was, like, okay, like, if, if there's, like, if there's, like, so few standardized evals available and also they're not a factor, like, they really are not measuring what it is that you're trying to do in your day-to-day programming job, like, what do you do?

So our stance was, and this was the stance of the Codex team and a lot of other teams that we've talked to, vibes, vibes or understand. Just, like, this, this evals just completely ignore them. They're completely unnecessary. You should probably, probably be wasting your time and it's just, like, I don't know who, who will be appeased by them.

And then last year we, we, we came to this idea that, like, okay, like, listen, I think, I think, I think we, we got to up the ante and we got to, like, we got to have some measure.

We got to try evals. And if no one else is doing it, we'll do it ourselves. We'll build actual evals from scratch that would, like, actually test real-world programming problems of users. So we got, like, we went through a lot of, like, our massive data sets of, like, people who had opted in to share their coding usage of Cline with us.

And we offered them money and we, we got a lot of this data set of, like, okay, this is what the problems that people are actually doing. Then spent a lot of time parsing through that, figured out, like, an actual data set of, like, these are the problems that people are solving.

And then just, like, completely cleaning it all up, like, doing a lot of, like, really hard manual labor, trying to make, like, very decent problems that can be solved with, say, Cline or any other coding agent. The hardest part for us when we were building evals is that, like, if you're building evals for anything that's, like, rudimentary, like, if you're, if you're building evals for, say, an LLM model, you have a very simple, like, one-shot use case of, like, how many toes does a cat have?

And then the LLM can just be, like, I don't know, 11 or whatever. I don't, I don't know how many toes a cat has. But a single-turn eval is very easy to do because it has, like, a binary answer.

It has just, like, a very limited search space of what the answer could be. But when you're working with an agent, that can't be the case. You're working with an agent. You can give an agent a problem like, "Hey, I have this new MCP server.

It's probably not working. Like, how do you, how do you, like, make it work for me?" And that's usually how a lot of you guys talk to Claude Code or whatever agent you're using and myself as well. So in this, like, it's very hard to gauge because, like, the agent, like, reads through files, searches through docs, installs the environment, sets things up, runs Python scripts, does all of that, and then in the end, like, runs some tests and then maybe the whole thing works.

Like, so we're, we're trying to grade the second thing. We're trying to get grade, like, all these things that will take a lot of time and then figure out, like, oh, did it actually work or did it not?

Did it work, but, like, broke other things? Like, so that's why it was, like, harder. So in the same time, some very, very awesome, smart, bright people from Stanford University came up with Terminal Bench, which does the same thing, where they came up with, like, 89 coding problems, which are, which are just, like, very approximate, decent representation of, like, real-world programming problems.

So these could be things like, you know, race conditions, database issues, like, other stuff, like, it's, like, figure out this infra issue. And I think that that was built, the Terminal Bench was built to, like, use with any coding agent CLI.

So you can, like, you can actually test and run things, like, really fast with, like, CLI. And then it will take, like, a couple minutes to run. So some of these tasks will take up to, like, 30 to 40 minutes.

And that's how you know they're legit because, like, the agent does a lot of things and just, like, runs in circles and sometimes just goes crazy and, and yeah. So, so we started using that. And the way to use that, like, the way to use Terminal Bench is that, like, you think of an eval problem as, like, an evaluation suite, which has a set of problems.

This one has 89 tasks. Some others would have more. And what you want to do is, like, you want to give it an environment. You want to give it an isolated environment where you just, like, you, let's say you have a task, like, "Hey, figure out this race condition for me in this repo."

And the race condition is that, like, this thing is not working. So you want to be able to give the eval run an isolated environment, like a virtual machine. In that virtual machine, it has the whole setup. It has the repo.

It has everything. And then you install whatever agent you have, in our case, Cline, Claude Code, Codex, whatever you want to use, you can do that. To do that, it's, it's not that it's, like, hard, but it is also not trivial.

And Harbor is another software that came from Lodd Institute where they made this thing where, let's say you have 89 tasks. One way to do evals is that run, like, each of these tasks in sequence and then, like, do all the setups.

Another way to do is, like, have a very standardized configuration defined in infrastructure where each of these 89 tasks have, like, the proper Linux machine, the proper RAM, CPU usage, and then, like, being able to, like, isolate those environments and then run those 89 tasks in parallel on infrastructure.

So you could use a couple different things for the infrastructure here. You could use Daytona. You could run it on your Docker machine if you have, like, very powerful machines. I'm sure if you can handle those, that much compute, sure, but I wouldn't.

We use Model. Model, we're very thankful to Model. They've helped us a lot. So shout out to them. And yeah, so in this case, like, Harbor basically lets you split up, like, the 89 tasks and then they all run in parallel.

So that way your limit, the limiting factor is basically the slowest task. Yeah. So, yeah, so the slowest task is the limiting factor. So the process is this. You, you, you get a score. You first do a run on the 89 tasks.

You get a score. You evaluate all the failures. So let's say you get, like, say, 50 failures,right, out of the 89 tasks. What you want to be able to do is you want to portfolio allocate those failures. You want to say, you want to run, like, another agent which goes through the traces of all the failures.

So the trace would be, like, this massive file which has, like, every single LLM call that the, that the, that the agent did. And then be like, okay, this one, this specific problem failed because it didn't run tests.

This failed because the retrial tool was broken. And once you portfolio allocate those failures, you figure out, okay, these are the small levers that I can pull. If I pull those levers, like, I can make, like, massive improvements to my AI agent.

So what you're testing is, like, you're basically testing, like, three things. You are testing the model itself. Like, if you have a very decent model, like, somehow, like, you could have a horrible harness, you could have a horrible agent, but, like, the model just, like, overshoots so hard that just, like, you, you, you know, you, you get a great score.

You're testing the harness. You're testing your coding harness. So, like, you, you're testing, say, if you're using Claude Code, Codex. So sometimes you'll find, I'm sure I guarantee you some of you have noticed that, like, let's say Anthropic's models could potentially work with Cursor, could potentially work with Droid, would work with other coding agents.

But for some reason, it just seems to work so much better with Claude Code,right? And I think that, that, that is, like, the testing the harness that, like, is the harness actually really leveraging the best, best of the model?

And the third problem, whether the problem is sane. If you're solving stupid problems, it doesn't matter if you score 100% all the time. So you really got to make sure that, like, the, the problems are sane, which the Lodd Institute has done a pretty great job of.

So for us, it was a case like this. Like, we, we basically, like, had, like, this original score, which was, like, much lower, like 43%. We made changes to, like, CPU. We made changes to memory infra of the containers.

We raised timeouts. We improved the thinking behavior. Sometimes we would ask the model to think more. Sometimes asking the model to think more actually interferes with the quality of the response because it goes in, like, it gets, like, a stroke and it just, like, goes in, like, circles and it's like, it will just, like, I am a model.

I am a model. It would just, like, keep doing it for, like, like, 2,000 tokens. So yeah, so, like, you, you, you got to think through all of that. And yeah, so for us, like, we have, like, a huge, like, internal benchmark for all kinds of models, open-source models.

So, like, we just, like, keep, like, a list of, like, trying different versions and stuff. We encourage other people to try that as well if you pretty helpful. So whenever, whenever you get, like, zones of improvements, you get, like, basically three zones of improvements when you get an original score.

The first one is the obvious flaws. Like, sometimes your harness really has, like, very obvious flaws of, like, there's this bug that straight up crashes the harness. And those obvious bugs you got to fix,right? Sometimes you are not, like, you're getting rate limited or whatever.

Fix those. That's fine. I think the zone two is the most critical one where you actually do nuanced improvements. And these nuanced improvements are things like there are certain prompt engineering techniques that apply to Anthropic model families that just straight up would not apply to Codex model family that would be very different from Gemini model family.

And those are the nuance of, like, why is it that this is a model that's so good that so many people are saying it's so good, but for some reason it just isn't working for me? I think those, that's, that is the essence of, like, working with agents and hill climbing that, like, you figure out those nuanced improvements of, like, tweaking your prompt, making it larger, making it smaller.

And then zone three is the danger zone where it's like you're straight up overfitting. So you're overfitting in the sense that, like, you're just straight up cheating to get the highest score and then you can, like, make a tweet about it.

Don't, don't, like, a lot of people have done it. Don't do it. Like, I wouldn't do it. I mean, never mind. Anyway, so yeah, so anyway, so this was, like, this was, like, basically the rough outline. So the final, the final wording for me would be, like, basically regardless of the kind of problem that you have, I want you to, like, find a benchmark and, like, like, build the eval and just, like, hill climb models.

So hill climbing means that, like, you get a score and then you improve the score of your harness on the eval. And you have to do both. Like, you can't just, like, have, like, a good number and be happy with it.

Like, you, you, you got to both pass the vibe check. Like, does it actually feel good to use this product in this model? And at the same time, you also have, like, a very, very, very decent score, hopefully.

If a new thing comes out, you, you, you do your absolute best to, like, give it theright judgment. For us, like, one of the things that we learned was that, like, we were very decent on Anthropic model families, not so much on, say, Gemini model family, not so much on, say, Gemini model family, which again are very decent models.

So when we, when we started hill climbing, we learned that, like, oh, if we support these models, we have this, like, entire swaths of people who love these models and they can start using us. And I think that in some refraction of that would also apply with you.

So my final, my final note to you guys is that, like, you know, I've, I've done some hot takes or whatever. And if you work for, for some of the companies that I've said not-so-nice things about, I still love you and everything.

And it was, it was, I work at Cline. So if you find these problems fascinating, if you want to learn more about these, like, this is my Twitter. So, like, you can feel free to reach out to me, DM me about, like, if you want to work on problems like these, like, by all means, like, I can put a word for you.

If you want to learn more about evals, if you want to learn, like, how to, like, I have a problem that's, like, completelyorthogonal to everything you're defining for coding agents, like, how do we work on that? So feel free to reach out to me and I'll, I'll respond to you.

And once again, it's very, very kind of you to give me your time. Thank you so much.