The maturity phases of running evals — Phil Hetzel, Braintrust

Welcome, everyone. Um, it's always a challenge to be a presenter directly after lunch, because that's typically when the energy level goes fromright around here to around here. But I'm going to try to make this session worth your while, uh, today.

We've got 18 very quick minutes together, and during that time I'm going to be talking about the different maturity levels that I see people go through as they perform evals for their agents. Before we get into that, just roughly, quick agenda today.

I'll explain a little bit about myself, the company that I work for. We'll spend most of the time today on more theoretical concepts, not product concepts. And then we'll talk about where I think this field is going in the future.

I'll also make sure to leave enough time, hopefully a couple minutes, for questions as well. I didn't over-prepare the content in hopes that we could have a little bit more of a discussion at the end of this. First of all, this is me.

My name is Phil Hetzel. I lead solutions engineering for a company called Braintrust. Effectively, what that means is that it is me and my team's job to make sure that people are getting the most value out of the platform as quickly as possible.

Prior to Braintrust, I spent 12 years in consulting and systems implementation. First 4 years with KPMG, last 8 years in consulting with a company called Slalom Consulting. And with Slalom, I led their global Databricks business unit. And I noticed that a lot of my customers were prolific at creating generative AI proofs of concepts.

They were not as prolific at bringing those proofs of concepts to production. So I started using Braintrust first as a user, because I wanted to help bridge that gap for my customers. And I liked the product so much that I ended up joining the company, and I've been here for about a year.

Outside of work, I like to play chess, but I'm not very good at it. And I like to spend time with my wife. And my dachshund is named Pistol Pete. He's the one in brown, not the one in black.

What is Braintrust, the company that I work for? Braintrust is an agent quality company. One of the, I guess, two of the main ways that we contribute to agent quality are evals and observability, which we consider to be very much the same problem from a systems perspective.

Evals, of course, being the thing that you're doing in order to gain confidence in your agent as you want to bring it to production. And then observability being the practice of, once that agent is in production, remaining confident in it.

It's a growing space. It's a very fast-moving space. And when you build an evals platform, you really have to grow with the technology, the underlying technology, as it changes. So it's a very fun place to be in. Let me give, like, a quick overview of the problem.

We talked a little bit about why we do evals in the first place. How many of you all are doing evals today? Hopefully, as you build, every single hand should be up. And certainly, when I give this talk next year at this conference, all you're going to come back, of course, to this session, and every hand is going to be up.

Eval is very important. The reason why we do evals is wholly in service to agent quality. That's the most important thing. We want to make sure that our agents are doing what we expect when confronted with real usage and real users.

This is really important from a risk perspective and a brand perspective. We don't want the reputational risk of an agent being unkind or unhelpful to a customer. We don't want the systems risk of an agent costing us too much money as it operates.

And there could even be compliance and legal risks if your agent goes too far off the rails. So evals are both a defense against those types of risks, but they're also they can play offense with evals in knowing, with each tweak that you make to your agent, how it's improving and how much it's improving your application.

A couple of primitives here. Evals are not unit tests, whereas unit tests are very exhaustive in how you perform them. With evals, you want to make sure that you start very high level with the failure modes of your agent.

Either you or a subject matter expert can educate about the specific failure modes of an agent, and you build evals around those very specifically. What you don't do, like you would with unit tests, is think about exhaustively every single thing that could potentially go wrong with your agent and try to make an eval for it.

Why can't we do that? Because it's infinite. You would spend all of your time writing tests and none of your time shipping, which is not productive. Eval results don't need to be perfect. Sometimes they can be sometimes they can be directional.

Using LLMs to judge other LLMs, LLM as judge techniques, you're probably not going to get 100% every time. That's OK. As long as you're trending in theright directions with those more non-deterministic techniques, that is completely fine.

Different primitives with the eval itself, how it's constructed. You have three things. You have a task. That's the agent under test, or the prompt under test. You have some data set of examples that initiate that task. How do you invoke that task?

You use some example that you give to an LLM or give to an agent to start that workflow. And then you have certain scoring functions, which you're using to judge the utility or the quality of that task.

There are a couple of different maturity areas that I've noticed some of our customers go through. I've listed four here. This is probably more of a continuum than being very discreet. But suffice to say that these stages, you know, you will traverse these stages as and when you create more complexity within your agent, just by necessity.

The more complex agent that you're building, the more vectors there are for failure, the more failure modes that you may need to account for. We're only going to be focusing on the, like, eval theory itself today. We're not going to really talk about the platform surrounding evals.

We've got a booth for that downstairs. If you're interested, you can come find me. So we'll go through these four: just getting started, measuring to manage, accounting for complexity, and then some advanced eval techniques.

OK, just getting started. It's not wrong to just get started with vibes. I know, like, vibe checking is a very nasty phrase here at this conference. I actually think it's OK. It's certainly better than nothing. When you're first starting out, you can't help but start with vibes.

I think the only thing that I would really recommend is that as you are vibe checking, you're also documenting. So when you have an agent under test, you give that agent maybe 10 different example inputs and loop through those inputs to see what the output is.

You should probably have some human, whether it's the person who built the agent or, even better, a subject matter expert that really knows what a quality response would look like. You should really have them analyze these outputs and give two pieces of information.

You should give a thumbs up or thumbs down. Was this response good? Was it bad? But more importantly, you should make that human annotator perform a justification for why they chose that thumbs up or thumbs down. Reason being is that you need to extract a lot of this domain-specific knowledge out of that human annotator's head so that eventually you can scale that type of knowledge through a technique like LLM as judge.

But this is a great first step, performing human annotation. Who in this room is, like, at this step?

OK, this is a way more advanced group. That's OK. That's a good place to start. Are you using, like, human expert annotators?

I just got, like, a I mean, I haven't got any infrastructure set up.

Yeah.

Just, yeah, it's pure run an agent on my data.

You're going to see how it looks.

And then repeat.

Yeah, totally. You have to start somewhere. It's a great place to start. This is how, like, that workflow is going to look. You have a trace come in, thumbs up or thumbs down. And then you add some justification to that so that eventually you can use it as an LLM as judge score down the line.

This is, like, what this might look like in a platform like Braintrust. We have, like, a human annotator view built into the platform. We actually let you vibe code your own annotation views. Important point: don't give a generic annotation platform to users.

Really make it very specific to them. They're going to have an idea of how these agent traces should look. So you should deliver that to them, and it'll encourage them to evaluate these appropriately.

OK, the next part is expanding upon that a bit, where now I just don't I don't have only some human grader giving thumbs up and thumbs down and justification. Now I'm starting to use those justifications. And I'm probably running those justifications through a cursor, Claude Code, or Codex to try to derive the actual failure modes of why, when they gave a thumbs down, why they delivered a thumbs down.

You now know and understand the failure modes of your agent. Now that you understand the failure modes of your agent, you want to be able to scale that human knowledge and be able to automate it so that you're not dependent on a few people with expertise to judge agent outputs.

A couple of ways to do this, one of which is using LLMs to judge other LLMs. LLM as judge, that concept's been around for quite some time. Very effective. Important here is that whenever you use an LLM judge, just because you put a robe and a cloak on an LLM, that doesn't make it inherently more trustworthy.

You should be evaluating LLMs as judge outputs as well. That's not really covered in this presentation, but you should not just judge LLM judges blindly in that regard. There also might be some objective failure modes where you can deterministically encounter them just through code.

That's OK, too. You don't have to use LLMs to judge other LLMs. You can use code to understand. If you're using too many tool calls, you might want to fail that eval. As an example, if you're using too many tokens, you might want to fail that eval.

I think the most important point here is that this data set that's on theright-hand side of this slide, at this point, you should probably be gathering production traces or at least UAT-level traces into that evaluation data set. We want it to be like, don't think about evals as running tests.

Think about evals like rerunning production, because ultimately we want to be confident as we run these workloads in production. A great way to do that is just to capture production data.

Most important point is this. We call it the flywheel internally. We want to be able to capture these agent traces in production, understand what's going wrong with them, either through a human or through automated tooling, and then bring those examples back to some offline experimentation environment, rerun production through an eval, and then use that to guide us to which direction we should be improving our agent.

So evals, that's like more playing offense with your evals. This is just, like, an example of setting up an LLM as judge scoring function to expand your ability to evaluate at scale, rather than using just a human.

OK, level two. Now we're starting to not just do simple model calls. We might be performing work with external systems. I think of tool calls in two different ways. There's context gathering tools that are just gathering data and injecting that into the LLM.

And then there's CRUD-based tools where you're creating, reading, updating, or deleting information from a database or an external system. Both of these can have a lot of lift in terms of whether your agent is quality or not. It also means that there's a lot of other things that can go wrong with your agent when you're starting to interact with external systems.

Often now, instead of just having one

of evaluating one specific part, i.e., the output of an agent, now you might be having to evaluate the entire trace of an agent. So in that sense, like, this is where tooling starts to come into play. You'll need some way to capture these large traces, understand each and every step that an agent took to be able to introspect and eventually target evals towards maybe even individual tool or MCP calls that your agent is creating.

The other problem here that we might have is when you're performing CRUD on a system. You don't really want to do that when you're offline, of course. There might not be a way to do that when you're offline.

So when you run an eval, there's two things that are problem areas. One, really challenging to represent the state that

other external systems were in at the time that eval input was created. And then two, it makes it really challenging to interact with those systems that the agent could be interacting with, because you don't want to overwrite any production data.

These are real challenges that we have to solve for. I would say it's not completely solvedright now. However,

there are some ways where you can represent external system state and interact with, like, mock-level APIs so that you can approximate a real production environment

as you're running evals. The idea for this is that these agent traces can be arbitrarily large. In that sense, it's a lot different than application tracing. So if a trace can be arbitrarily large, you can actually cram in a ton of context, i.e., system state, the state that the external systems were in at the time, into these traces and inject that into the task that you're running the eval upon.

In that way, instead of having to

create entire test structures and infrastructure, you can represent a lot of that stuff within the trace itself and encapsulate it there. The other thing that you can do is you can use really specific querying techniques to perform timestamp queries to systems that support them.

So if an input came and you added it to your data set at a certain point in time, perhaps the way that you've set up your vector database, you can run a version query to query the vector database at a certain point in time.

So that way you're adequately representing the state of when that task ran originally. These are more complex techniques, but ones that are a little bit more emerging.

I only have about two minutes left to go. What's next? Performing topic modeling at scale to make sure that you're uncovering those failure modes automatically in production. That's something that I'm more than happy to talk about at the booth downstairs.

And then, of course, performing evals in a way where you're using Claude Code and the eval provider CLI to be able to do this in an automated way. These are two other patterns that I see emerging in this space.

I want to be conscious of time. I probably have time for, like, one question before I have to jump here. Is anyone curious about anything specifically? Otherwise, you can find me at the booth. Yes, sir?

In our sphere, like, it's kind of normal to put a bit more respect on deterministic evaluation, deterministic graders. Do you agree with it? Do you think that we should push for more deterministic graders in our eval platforms? Or do we embrace LLM as a judge as a whole?

Some things are subjective. That's why we love agents so much. I would embrace LLM as judge, but also perform a lot of evals on the LLM as judge so that it's very aligned with what a human would decide in the same circumstance.

You would eval the eval.

Yeah. It's easier to do that because LLM judge outputs are going to be discrete. So you can create a ground truth data set for that. Yeah. Allright, everyone. I have to jump. I'm at my time. It was a pleasure to be with you all.

And yeah, feel free to find me in the booth downstairs.