Evals 101 — Doug Guthrie, Braintrust

Hey everybody, my name is Doug Guthrie. I'm a solutions engineer at Braintrust. As you can see here, we're an end-to-end developer platform for building AI products. We do evals—if you watched the keynote this morning, you saw our founding engineer jumping up and down on stage yelling "evals!"

I am not going to do that. I'm not as funny or cool as him. But we should all be very excited about evals here.

Very brief agenda of what we'll cover today in this sort of, uh, this intro. Very, very brief company overview. Give you an intro to evals, why you—why would you even start thinking about using them, what are they, what are the different components that you need to create an eval.

Some more Braintrust-specific things via running evals via our SDK. You'll see in the examples that you can run evals both in the platform itself as well as the SDK. I think it's a really kind of cool thing that we can connect maybe the local development that we're doing with what we're doing within the platform.

And then how we then move to production. This is maybe human review, this is online scoring. So how well is our application performing in production? And then lastly, a little bit of human-in-the-loop. Getting user feedback from your users.

How do we now take some of these production logs and feed them into the datasets that we're using in our evals, creating this really great flywheel effect.

Cool. Quick, quick company overview. You see up there, maybe in the topright, some of our—excuse me—some of our investors. Maybe a quick callout on the leadership side. Ankur Goyal is our CEO. Maybe the reason to call out that is Ankur, in the last two stops, he essentially built Braintrust from scratch at the last two places.

And this is really where he found the idea to, like, maybe this is actually a thing that people need. And really the origination story of Braintrust. The other thing to call out here is, like, we already have a lot of companies using Braintrust in production today.

This is just a few of the companies that are utilizing us for evals, for observability of their GenAI applications. So I won't bore you too much with that. But let's jump into this. If you were at the keynote, you probably saw a similar slide here.

I didn't take it out. But you see, like, the tech luminaries here, I think as Manu referenced them, talking about evals and the importance of them. I think this is a, you know, obviously this is an intro to evals track.

And what better way to start this out with some really, you know, influential people in the space talking about evals and why they are so important. So why would you think about evals,right? They help you answer questions. So here's a few of them.

When I change the underlying model to my application, is it getting better or is it getting worse? When I change my prompt,right? When I change certain things about the application, is it getting better or is it getting worse?

We want to get away from not having a rigorous sort of process around building with large language models, which, as you all know, non-deterministic outputs create somewhat of a challenge. And without evals in place, it becomes really, really hard to create a good application that we can put into production.

Maybe some other ones, like obviously being able to detect regressions within the code. I think the other thing that Ankur mentioned to me when I first started, which I didn't really mention this, but this is my third week at Braintrust as a solutions engineer.

But one of the things that he mentioned to me that I thought really resonated was, evals are a really great way. I think people think of them as almost like unit tests for

our applications. But he kind of described it in another way of, like, this is a really great way for us to play offense as opposed to just plain defense, where I think maybe unit tests are kind of used for.

This can actually be used as a tool to really help create a lot of rigor around us building and developing these applications and ensuring that we actually build things that we can put into production.

Maybe from, like, a business perspective, why would you think about running evals or using evals? Here's a few here. If you have evals running both offline and online, you create this feedback loop or this flywheel effect that I think Manu mentioned in the keynote.

Excuse me. And this flywheel effect allows us to, you know, cut dev time, allows us to enhance the quality of this application that we're putting out into production. If you're able to connect the things that are happening in real life with your users in production and being able to filter those logs down, add those spans to datasets, inform what you're doing in an offline way, creates that flywheel effect that becomes really powerful from a development perspective.

Again, a little bit more on the customer side. Here's a few of our customers and some of the outcomes that they've seen using Braintrust, whether it's moving a little bit faster, pushing more AI features into production, or just increasing the quality of the applications that they have.

Here's a few of the outcomes that we've seen or that they have seen. So let's start talking a little bit about the core concepts of Braintrust. Obviously, we're here to talk about evals.

The things that I think you see, like these arrows going one way and the other. This again is that flywheel effect that I described earlier. There's the prompt engineering aspect of this. In Braintrust, think of this playground that we have as an IDE for LLM outputs.

The playgrounds allow for that rapid prototyping. As we make those changes, as we change the underlying model, what is the impact to that particular task or that application? Then those evals allow us to understand the improvement or the regression of those changes.

And then the observability aspect,right? This is the logs that we're generating in production, the ability to have a human review those logs in a really easy, intuitive interface, and then have user feedback from actual users be logged into the application as well.

So what is an eval? You probably heard this several times today throughout the week if you stopped by the booth. But sort of our definition here is that structured test that checks how well your AI system performs,right? It helps you measure these things that are important: quality, reliability, correctness.

So what are the ingredients in an eval,right? I've been talking a little bit about tasks,right? This is the thing, the code or the prompt that we want to evaluate. The really cool thing about Braintrust is that this can be as simple as a, excuse me, a single prompt, or it could be this full sort of agentic workflow where we're calling out the tools.

There's no sort of limit onto the complexity that we put into this task. The only thing it requires is an input and an output. The second thing is a dataset. This is our real-world examples. This is essentially what we're going to run the task against to understand how well our application is performing.

And how we do that is via scores. So the score is really the logic behind your evals. There's a couple different ways to think about this. There's the LLM-as-a-judge type score. So you give it the output and some criteria, and it is able to assess, like, say, I want, based on this output, is this excellent, is this fair, is this poor?

And then those outputs then correspond to, you know, 0, 5, or 1. You also have code-based scores,right? These are maybe a little bit more heuristic or binary. But we can use both of these to really aid in the development of that eval and ensuring that we're building a really good application.

I think I just sort of mentioned this here as well, but like the two mental models here of evals, there's offline and online. Offline is pre-production. This is us actually doing that iteration. It's identifying and resolving issues before deployment.

This is where we're defining those tasks. It's where we're defining those scores. Online evals, this is that real-time tracing of the application in production. It's logging the model inputs and the outputs, the intermediate steps, the tool calls, everything that's happening.

It allows us to diagnose performance and reliability issues, latency. Based on how you instrument your application with Braintrust, we can pull back lots of different metrics related to cost and tokens and duration. And all of these things help inform how we build this application.

I'll jump into a little bit more of like how we can instrument our app for online evals. We're going to first, I think, talk a little bit more of the offline. Before doing that, maybe just like level set on how to improve.

I think one of the things that I've seen in the last few days here, the conversations that I've had, it's like almost how do I get started? Or what do I do if X? Or, you know, those types of questions.

Another thing I heard from Ankur very early on is that, like, just get started. Create that baseline that you can then iterate and build from. I think a lot of people get caught in, like, creating this golden dataset of test cases that they can then, like, iterate from.

Start. You don't necessarily have to do that. Start and build that baseline. Establish that foundation that you can then improve upon. But this is a really good sort of matrix of, like, if I have good output but a low score, what do I do,right?

Improve your evals. If I have bad output and a high score, improve your evals or your scoring. But really good kind of like high-level understanding of where to start to target your efforts when you are building these apps and you're creating evals.

So let's jump into the actual components that I just talked about. So within the Braintrust platform, we have a task. Again, this could be a prompt. This could be like this full agentic workflow. Very basic. You see that GIF running.

This is a prompt within the platform. You specify an underlying model that you want it to use. You give it a system prompt. You can also give it access to tools. It has access to mustache templating, so you can pass in variables like user questions or, you know, the input from the user or its chat history or metadata,right?

So when we actually go and want to parse through these logs, the metadata becomes actually beneficial, enabling us to do that in a really easy way. Going forward, maybe we have a multi-turn sort of chat type scenario where we want to add additional messages for the system and the assistant and the user and our tool calls as well.

The platform allows for that just via this messages button, and then you're able to add those different messages to the prompt.

Also, we can add tools. So oftentimes the prompt will need access to something,right? Maybe it's a RAG type workflow. Maybe it's doing web search. Whatever it is, we can now use those tools as part of that prompt,right? And so when you sort of encode in that prompt, like make sure you use X tool, this prompt has access to that tool while it's running.

The last one, this is actually a feature that is in betaright now. It's actually creating more of that agentic type workflow within the Braintrust platform itself. So it's a wayright now, at least, to chain together prompts where the output of one now becomes the input of the other.

But if you think back maybe to this slide, if I have sort of this prompt that has access to tools, you create a pretty powerful system here where you're able to go from maybe that first step that has access to a certain tool, and we get some output from that.

We can then go to that next step that has access to maybe some other tools,right? This sort of like maybe multi-agent type of workflow we can create with the underlying tools within those prompts.

The second thing that I talked about is datasets,right? These are our test cases that we want to give to the task to run. So we can sort of iterate over that. We can get the output and then going down a little bit further, actually score that.

But this is obviously really important when we're running our evals and then when we are trying to pull from production,right? The actual logs that are happening, we can add those spans, those traces to the datasets in a really easy way.

And I can show you what that looks like. But if you look there at the bottom, the only thing that's required is the input. You also have the ability to add expected. So what is the expected output for that input,right?

You can sort of like create some sort of score that looks at the output with the expected. There is a score called Levenstein that allows you to measure the difference between those two. So you can do some different things based on what you provide to that dataset.

You also have metadata as well, again, being able to filter down different things, pulling the dataset maybe into your own code base. And I want to filter by, again, X, Y, or Z via the metadata. That's all possible.

I mentioned this a little bit ago, but just start small and iterate,right? You don't have to create this golden dataset to get started here. Just start and then continue to iterate and build from that baseline. The human review portion also becomes really powerful.

Again, when we have stuff being logged within production, having humans actually go through those logs. And there's lots of different ways to filter it down to the things that they should be looking at. And then we can now decide to add those things to certain datasets that then inform the offline evals that we're running.

The last thing, the last ingredient here that we need for our, excuse me, for our evals are our scores. We have both code-based scores,right? Again, this is like more of those binary type conditions, but you can actually code TypeScript or Python.

You can do that within the UI, as you see over there on the bottom left, or you can within your own code base create that score and then push it into Braintrust so we can use it in the platform.

Other users who maybe aren't in the code base can use that score as well. The other score that we have access to is called LLM-as-a-judge. So this allows us to use an LLM to sort of judge the output.

We can give it the set of criteria that indicates what a good or a fair or a bad score or whatever it is. You get to decide what that looks like. So you give it that criteria and it says, if it's good, I want to do a 1.

If it's bad, I want to do a 0. But this starts to create the scores that we can use in that offline and that online sense. The other thing to call out here is that we have internally built a package called AutoEvals.

So this is something that you can now pull into your project. These are out-of-the-box scores that are both LLM-as-a-judge as well as code base. And so it just allows you to get started very, very quickly. Another thing I heard Ankur mention is maybe starting with Levenstein, maybe not the best score in a lot of cases, but again, it establishes a baseline.

Very, very little development work for our users, but it creates that thing that we can then build from. And now you have a direction, a direction to go in to go build maybe that more custom score.

Some of the things that we've heard from our customers, some tips that, you know, important to think about. A lot of our customers are using higher quality models for scoring, even if the prompt uses a cheaper model. Just makes a lot of sense, like while we're running that application, to use the cheaper model, but use the more expensive one to actually go out and score it.

Also, break your scoring into very focused areas. So the example that I'll show is an application that generates a change log from a series of commits. So I could create a score that says assess my accuracy, my formatting, and my correctness.

Or I could create three different scores that assess accuracy and then formatting and then correctness. So have your scores be very targeted to the thing that they're supposed to be doing. Test your score prompts in the playground before use.

And then avoid overloading the score prompt with context. Focus it on the relevant input and the output.

A couple things here. Here's where over on the left we have our playgrounds. This is where we do that sort of like rapid iteration where we can pull in those prompts, we can pull in those agents, add our datasets and add our scores.

And we can click run and it'll go out and sort of churn through that dataset that we've defined. And it will give you a sense for how well your task is performing against the dataset with the scores that we define.

But this is the place where developers, PMs, we even have a healthcare company that has doctors coming into the platform and interacting with the playground and even doing human review as well. Depends a little bit, obviously, on the organization.

The thing on theright is our experiments. This is our sort of like snapshot in time of those evals. So imagine now, like as we are doing this development and we're trying to understand, like the last month or so, are we getting better,right?

The changes that we are making, the model changes, whatever it is, are we improving our application? And the experiments is a really great way to understand that. Really important maybe to call out as well, you can see on the bottomright, the evals can happen from the application,right?

The Braintrust platform, as well as via the SDK.

Cool. Maybe just really quick, because nobody likes looking at slides all the time. I certainly don't. Maybe if you haven't seen Braintrust yet, this is maybe a good quick demo. So again, like the idea here is I have this application.

I'll just give you, I'll show you over here. You give it a GitHub repository URL. It grabs the most recent commits and then creates a change log from there. And then once this completes, you can even provide some user feedback.

But this is the thing that we want to evaluate. So what I can do, I'll go into my playground,right? This is the place where I can start to run those experiments or I can start to iterate on that prompt that I have.

From my project, I've actually loaded in two different prompts. So before going into the playground, I've actually created these two prompts within my code base and I've pushed them into the Braintrust platform. I've also created a dataset in that code base and I've also created some scores,right?

These are the ingredients that we need to run our evals. So now when we have those, we have those different components, now we're able to start to iterate here. So I'm going to actually create a net new playground and I will load in one of these prompts.

So again, here's my first prompt. My first prompt has a model associated with it. What becomes really cool here is the ability to iterate on the underlying model,right? I think a lot of us are, we have access to a lot of underlying providers and we want to be able to understand if I change this or if a provider adds a new model, what is the impact to my application?

So I can duplicate this prompt and maybe change this to GPT-4.1. I can run this.

Before I run it, I have to add all of my components. So I'll add my dataset and then I can add my different scores that I've configured here for my change log. And so I can click run. And then now we'll understand here what is the effect of changing the model for this particular task with the scores that I've configured against this dataset.

So this will churn through all of these in parallel and we'll start to get some results back. Lots of different ways to actually start to look at this data. I always like coming over to the summary layout because I can understand, like you can see over here, this is my base taskright here and then my comparison task.

So I can understand, it looks like on average the base is performing a little bit better than my comparison task on my completeness score. It's faring a little bit worse on my accuracy score. Both of them are 0% on my formatting, so probably have some work to do there.

But you can start to see how you can use this type of interface to iterate very quickly,right? Now, the other thing that maybe shouldn't do, but I can't resist because we just released this today, is this new loop feature.

So imagine you are a user within Braintrust. And before this, you would sort of manually iterate here, creating net new prompts, making modifications, changing the model. What if you could now utilize AI to go and do that for you?

So any sort of like cursor-like interface, we can ask it to optimize a prompt. And I think the really unique thing here is it has access to those evaluation results. And so when it goes to go change that prompt, it understands that, it changes the prompt, it runs the evaluation, it understands if it got better relative to the scores that we defined.

So you can see it's going to go through here. It'll fetch some eval results. You'll probably see a diff here very, very soon. If we don't, I won't hang out here too long, but I do want to highlight one of the things that we are releasing that really enables our users to iterate in a really fast way.

So here's my change. We can click accept. And then it'll actually go out and run that eval again, or it would. I think I have an issue with my Anthropic API keys. But the idea here again is like we can create that very rapid iterative feedback loop here within the playground.

The other thing here is we can run these as experiments. So this

is where we can start to create those snapshots in time of that eval. And again, see as I make these changes to that application, how is it sort of performed over time? I want to make sure I don't want to go down.

I don't want to decrease the performance of my scores relative to obviously the last time it ran, but looking out over the last month, six months, whatever it is that we're tracking.

Cool.

So that was very, very brief sort of intro to evals via the UI,right? Again, like just to summarize, we need a task, we need a dataset, and we need at least one score. We can pull those into the playground and now we can start to iterate.

We can save these via experiments. And now we have a way in which we can understand how well this application is performing,right? This is no longer like qualitative,right? This isn't like, hey, I think this got better. That output looks better.

There is actual rigor behind this now. Customers oftentimes ask though, like I don't really want to use, or I'm not going to use the platform as much. I'd rather use this from my code base. Is that possible? And it is.

So we have Python SDK, we have a TypeScript SDK. There's some other ones as well, Go, Java, Kotlin. For the most part, most of our users are using Python or TypeScript. Here's just a couple examples of what this might look like from an SDK perspective.

Actually, if you all aren't opposed to looking at some code, here's just a really basic example of defining a prompt within my code base and then pushing it into Braintrust. So just leveraging that Python SDK. Another example, I should come over here.

Creating a score. So you give it sort of like the things that it's looking for. But now I've sort of defined this score within my code base. It's version controlled. Also, the prompts that you create within the UI are version controlled as well.

But this is just another way to start to interact with Braintrust. So again, scores, we could do datasets, and then you can even do prompts up here as well, I believe. So here's my eval dataset. Here's my change log to prompt.

Again, being able to start from the code base and actually push them into the platform is possible. Just depends on the organization where they want to start. The other thing here is like this is more on the components of the eval side.

So that's that top portion. Define those assets in code, run that Braintrust push, and now you have access to that in that Braintrust library. The other one is actually like defining the evals in code,right? So what that looks like is slightly different.

Come over here. So we have our eval, but this is just a class that's coming from our Braintrust SDK. Again, it's looking for the exact same things that I just described,right? A dataset that we can use from Braintrust itself, the task that we want to invoke, and then the scores.

So again, defining this here within your code base, certainly possible. And then I can run a command

that actually runs that eval within Braintrust. So from here, go into Braintrust, see the eval running. This is now an experiment that I can view over time. So again, like you saw two different types of workflows here. Again, catering to maybe two different personas or again, the way in which organizations want to work, it's up to them.

Braintrust is very flexible in how we allow our users to consume or use the platform.

Probably jumped ahead a little bit, but this is sort of a recap of what I just showed you. Again, from your code, you create your prompts, your scores, your datasets. You can push them in there. Maybe just important to highlight here of like why you would do this.

You want to source control your prompts. The big one here to call out is the online scoring. I have a section in a little bit diving a little bit deeper into that. But if you want to use those scores that we define in the dataset, we should push them into Braintrust so that we can create online scores.

We can understand how our application is performing in production relative to those scores that we want or that we're using within our offline evals. What I just showed you, maybe another variation of that, that eval within our code, again, defining that dataset, defining that task, defining those scores becomes very, very easy to now connect these two things.

It's just again, up to you to decide where you want to do this. The other thing to call out here is that this can be run via CI/CD. We do have some customers that want to run their evals as part of the CI process.

So understanding in a more automated way,right? The score for A, B, and C, whatever they've configured, has it gotten better? Has it gotten worse? This becomes maybe a check as part of CI. If you look within our documentation, there's a GitHub Action example that shows you how you could set this up.

Cool. Let's move to production.

Moving to production entails setting up logging,right? It entails instrumenting our application with Braintrust code. Being able to like say, I want to wrap this LLM client. I want to wrap this particular function when it goes to call that tool.

It becomes very, very easy to do that. But so why should you do it? I think I've probably said it numerous times here, but we want to measure quality on live traffic,right? We actually want to understand how well our application is performing with those scores.

Really great to use during offline evals. It becomes our aid in ensuring that we build really good applications that we're not creating regressions, but also really important to monitor that live traffic. The other really important thing to call out, I think, is that flywheel effect that it creates.

So we have these datasets that we use to inform our offline evals. It's very, very easy now to take the logs that are generated within production and add those back to datasets. This also speaks to some of that human review component where we want to now bring those humans in.

They can start to review some of the logs that are relevant. Like maybe there's user feedback equals zero. Maybe there's a comment or whatever it is, but like they can filter down to those particular things. And as they find really interesting, maybe test cases, it's very, very easy to add those back to the dataset that we use in our offline evals.

So I think the feedback loop or the flywheel effect that this creates is one of the really fundamental value props of the platform. So how do we do this? There's a couple different ways. We're first going to initialize a logger.

This is just going to authenticate us into Braintrust and point us to a project. You may have seen when I open up the platform, I had numerous projects inside of there. You can almost think of a project as a container for that feature,right?

So you probably have multiple AI features that you're building. I want to have a container for feature A, for those prompts, those scores, those datasets. You could certainly utilize those things across projects, but it becomes a really good sort of way to containerize the things that are important for that feature.

Then you can start really basic,right? You can wrap an LLM client. So when you saw some of those metrics with like tokens and duration and costs, just very basically within the script or excuse me, the code here, I just wrapped that OpenAI client and now I'm just sort of ingesting all of those metrics into my logs.

That's the easiest way to get started. You obviously probably want to do a little bit more. Again, maybe you want to understand when that LLM invokes a tool. So I want to trace, I can add a trace decorator on top of a function.

I can even use some of the Braintrust low-level like span elements to create custom logs. And I want to customize the input and I want to customize the output and the metadata that we log to that span. So again, you can start very basic with wrapping a client and then go down to like the individual span itself, specifying that input and that output.

This leads us to online scoring,right? I talked a little bit about this,right? This is where like when our logs are coming in, we can actually configure within the platform those scores that we want to run and we can specify sort of a sampling rate.

So we don't necessarily run that score across every single log that comes in. Maybe it's 10%, 20%, so on. But it creates that really tight feedback loop that I've been talking about. Also maybe just important to mention the early regression alerts.

So we can create automations within the Braintrust platform. If my score drops below a certain threshold, let's create an alert with our automation feature.

This is just, and I can maybe walk through what this looks like instead of showing you here. The custom views, this is where like there's a lot of really rich information within these logs. And it becomes really important, I think, again, for the human review component to like filter these down to the things that they care about or the things that anybody cares about.

So we can create custom views within Braintrust with the appropriate filters. And then it's very easy for that human to go into what we call human review mode within Braintrust and sort of parse through those logs, the ones that are going to be most meaningful to them.

Let me connect some of those dots there. So

again, showing you some code may be good, may be bad, but I'm guessing there's some technical people in the room that don't mind here. So if I look for the, you may have seen in one of those slides, there is the Vercel AI SDK.

I want to wrap this AI SDK model. Again, this allows us to just create all of those metrics within Braintrust with just zero lift from us as a developer. This becomes really easy to do. You can also see where I have specified that span itself,right?

I actually want to define the inputs and the outputs of that. The reason you would do that is because you have a specific dataset with a structure that you want to ensure maps to that. So like when you are within those logs parsing through them, it becomes really easy to add those spans back to that dataset.

So ensuring that that data structure is sort of consistent across offline and online becomes really important, again, to create that feedback loop. So this is very high level of like how we can start to create those spans. Now that we do,right, we can now go in the platform and start to configure our online scoring.

So this is here just within this configuration pane. I can click online scoring. I'll just delete. I'll create a new rule. So my new rule. And here's where we can add different scores,right? Obviously, I have a few here that I've been using for the offline evals.

I don't necessarily need to select all of them, but I certainly can. And then I want to apply a sampling rate. So I want to actually give you an example of what this looks like. So I'm going to do 100%.

The other thing to call out here is that you can apply these to the individual spans themselves and not the entire root span. So where this becomes beneficial is like when you are invoking maybe tool calls, you're invoking like a RAG workflow and you actually want to create a score on whether or not the thing that it gave back is actually relevant to the user's query.

So we can actually create a score specifically for that and highlight what that span is here. So again, very, very flexible in how you apply these scores to the things that are happening online.

Now when I come back here to the application and we'll just run this again, creating that change log, you'll now start to see here within the logs, this will start to show up and then you'll start to see these scores be generated,right?

Again, this is where like you can now start to understand over time in production, how are these things doing? How are they faring? Where can we get better? Again, now we can connect again like the things that are happening in our offline evals with the things that are happening with online.

The other thing to call out here is the feedback mechanism,right? We certainly have the ability to do like human review, but oftentimes you want your users to provide feedback as well. And so this is just a basic example of a thumbs up, thumbs down, and you can even provide a comment here.

This can now be logged to Braintrust. So I should see over here my user feedback. So here's my comment. And then I have my user feedback score. But now I can also do something like this. So again, maybe I want to filter my logs down to where user feedback is zero.

So click that button. I'm going to change this to zero.

I don't have any rows yet like that, but now I can save this as a view and people who are now using this as human review can filter this down to where user feedback equals zero and we can figure out what's going on,right?

What are the things that fell down here within this application that we need to go fix? The other thing I'll highlight here is our sort of human review component. Actually, you click that button or you can hit just R and it opens up this different pane of your log.

So it's a pared down version of what you just saw there. It's a little bit easier for a human to go through and actually look at that input and that output. But you as a user of Braintrust can configure the human review scores that you would like to use.

So I have this add something here. So maybe this is a little bit more free text. I have a better score. Again, these are the things that you can add to your platform that map to the, excuse me, the scores that you want your humans to add to those logs.

Just really quick, I'll highlight some of these things here. This is what it starts to look like when you instrument your application with those different wrappers or those different trace functions. I'm able to understand at a very granular level, excuse me, granular level, the things that it's doing,right?

So I essentially have these tool calls where it's going out and it's grabbing the commits from GitHub. It's understanding what the latest release is and it's fetching the commits from that latest release. And now I can generate that change log.

But again, the really unique thing here, and maybe a different example of this, is I can start to score those individual things that are happening. So this is a different application

with this example. So if I open this up, I have this conversational analytics application. So a user can ask a question and can return back some data. But this application goes through these various steps. Like the first step is to rephrase the question that the user asked.

So imagine like there's this chat history that we can load in as input and the LLM needs to rephrase that user question. If the LLM does a really bad job of rephrasing this question, everything as a result of this will fall down.

Probably not going to get aright answer. So what I can do is create a score specifically for that span to understand how well the LLM did in rephrasing that question. I can also understand the intent that I was able to derive or the LLM was able to derive from that question.

Is thatright? But you start to think of like these more complex type of applications that you build. You need to be able to understand the individual steps that are happening and Braintrust allows for that very, very easily via these scores and then being able to apply them not only again while you're in offline eval kind of mode, but also online,right?

We want to understand these logs and be able to apply these scores at the individual span level. This becomes pretty powerful as well.

I think I actually stole from my next section, my human in the loop. Kind of walk through this a little bit. Maybe just another callout. If you happen to be at one of our workshops on Tuesday, Sarah from Notion, who's a Braintrust customer, talked a little bit about how they think about human in the loop.

And I think it's important to consider like the size of her organization and what they're doing. She mentioned that like she has a special type of role that they use for human in the loop type of interaction,right? It's almost like a product manager mixed with an LLM specialist.

They're the people that are going through and doing those human reviews. Smaller organizations, she made a comment that was, it actually makes a lot of sense for the engineers, some engineers to actually go through and do this as well.

It becomes really powerful to pair like the automation with the human component of this. Like this is not going to go away. I think it adds value to the process.

Again, I think I just stole for myself like why this matters,right? This is really critical for the quality and the reliability of your application. It provides that ground truth for what you're doing.

Two types of human in the loop interactions here. I walked you through that human review. Give me one second and I'll call you. Yeah. The two types, the human review, being able to like create that interface within Braintrust that allows that user to kind of parse through the logs in a really easy manner, as well as configuring scores that allow them to add the relevant scores to that particular log.

And then the user feedback, this is actually coming from our users in the application. Again, being able to create sort of views on top of that feedback that then power maybe the human review and then creates that flywheel effect that we want.

That's all I have today. Appreciate you all coming out here and listening to me. But yeah, you had a question. Thanks.

The human in the loop aspect, user feedback as well as human eval, how are you handling or what feedback have you heard from your customers or even internally when it comes to taking that log data and identifying or extracting anything and then using that to modify your calls or get a different model or the actual model?

Yeah, the question is around like how are we using human review and like some of the logs and informing the offline eval portion of this, largely that. Cool. Yeah, one thing I maybe I didn't highlight here is, so maybe back within Braintrust, I'm going to go back to my initial project.

So imagine now like we have all of these logs. We filtered it down to a particular, oh,

are we still showing? On the screen?

Awesome. Thank you. Yeah, so imagine like we have this process now,right, where we're doing that human review. We filtered it down to the records that are meaningful for whatever reason. It becomes really easy again to connect what's happening within production.

So maybe I select all of these rows or I select individual rows, but I can add these back to the dataset that we're using within those offline evals. I think I've said this like a hundred times over this conference, this flywheel effect.

This is like I think what's missing oftentimes when we're building these AI applications and what Braintrust allows for really seamlessly. Yeah.

I have two questions. The first one is about production. Is it possible to have multiple models in production and compare how they behave?

Yeah, I don't see why not. Like my guess is in the underlying application, you're swapping them out.

Like having like A/B test, you know, I can have like two or three or four and easily compare.

Absolutely. Yeah. Let's see if I have an example here. You're able to group some of these scores. Maybe this is sort of an example of what you're talking about. So like maybe within production we have different models running.

This sort of view here allows us to understand like the models that we're using under the hood. And this is just, you know, you could do this within production as well and sort of do that A/B testing.

Cool. My second question is about humans in the loop,right?

Let's suppose that I have multiple humans and they behave slightly different as scorers. Do you have anything or what is the vision to do with that? Like is there a way that I can actually compare how they're scoring or something like that or not really?

So different users can maybe have different sort of criteria for scoring. Maybe the first thing I would say to that is like there should be like maybe a rubric for your users who are interacting with human review. So you're not creating that.

You certainly have the ability to see like who is scoring different things within the platform. I'm not sure if you're able to pull that as like a dataset to like assess the differences there, but maybe like before it gets to that place, like have a rubric, have a guideline of what scoring looks like for your humans.

Okay. Thank you.

Yeah, of course.

Hi. So the scores I'm used to working with for like LLM as a judge are like they're relativistic,right? So they can't tell you is the answer relevancy good or bad for a single run, but it can tell you how it compares to previous iteration of like the same test set, for example.

Do you guys use LLM as a judge scores for online or is, and like

are they relativistic like that or do you have some way to be like this is a good answer, you know, in and of itself for this sample or because it's all, you have new data coming in,right?

Yeah, I think a lot of our customers who are thinking about this are like almost doing evals on their evals, like trying to understand did the LLM as a judge actually do a good job there? So like when that actually runs, there's a rationale behind it.

And so you can sort of run an eval of those LLMs as a judge. I think Sarah from Notion in our workshop described sort of a process like that within Notion, but I think that's sort of like where I would aim you.

Okay. Cool. Thanks.

Cool. Awesome.

Are any of your customers doing evals before they launch? Like I'm working with a government. They don't want to launch until we show some accuracy levels.

Yeah.

So we're getting our subject matter experts to enter in all the questions that they have,right? They have huge datasets of thousands of questions, believe me, as a government.

And then we're using measures like you're talking about. Do you have a way to do that? I guess it's the same, is it?

So what you're describing is what we call offline evals,right? This is development. We can actually do this testing before we get into production. This is what I was talking about, like establish that baseline using those scores, using that dataset that you've already created.

But this all happens before we get into production,right? And then you can like one of the things that I heard from somebody earlier is like one of my challenging things of building this AI application is establishing trust or creating that trust in this thing.

That's part of what this is,right? It's like it's showing those people the scores of that application. So you start to iterate on this thing. Maybe it starts at 20%, then it goes to 30, then at 40, and so on.

That to me is the thing that you use to create that trust and create that like groundswell to push it into production.

Okay. Yeah. That is what we're trying to do. But I wondered if I can see the tool does that. Thank you.

Yeah, of course.

Time for one more.

Thanks. Quick question. I love the CICD competence. We're trying to build a law. We're trying to build like ML as a platform for our team. So we get into evals and stuff like that. So how much of the monitoring dashboard you have in Braintrust can actually be like take the data taken out and post it in a unified dashboard somewhere else?

Yeah, all of this is available via SDK,right? You can pull down experiments, you can pull down datasets. So you're able to pull this down. Like we have a customer that is actually building their own UI on top of like the SDK itself.

Like so they built their own sort of like components utilizing the SDK and pulling the sort of things that we've logged, the experiments that we have in the application into their own UI. So certainly possible.

Awesome. Oh, great. Thanks.

Allright, cool. Thanks, everybody.