[Evals Workshop] Mastering AI Evaluation: From Playground to Production

Hey everyone, thanks for joining us for the Evals session today. This is the first workshop that we'll be leading; there's another one at 3:30, so you get to be the first people to go through it. Very exciting stuff.

If you've gotten a chance to sign up for Braintrust, please do that now. If not, we also have some workshop materials and a Slack channel for you to follow along. In the Slack channel, we also sent out a poll.

If you'd like to respond with a little emoji underneath the message, that'd be great.

In the Slack channel also, there is the workshop guide, so in case you're not able to get the QR code for whatever reason, go into the Slack channel and you'll be able to pull up that document.

Before we jump in, obviously, maybe just a quick intro of Carlos and I. My name is Doug. I am a Solutions Engineer at Braintrust. I have a background in data and finance, actually my third week here at Braintrust, but looking forward to kind of leading you all through the platform and giving you a sense for how you can master Evals with Braintrust.

Yeah, and my name's Carlos Esteban. I'm also a Solutions Engineer, helping out with some of our great customers at Braintrust. I'm a little bit more tenured, been here six weeks, and before I was in the infra world working at HashiCorp, doing some stuff with Terraform and Vault.

But yeah, super exciting to be here today at the AI World Fair. We have a lot of exciting things to go over with you.

So just to go over the high-level agenda, we're going to be alternating between lectures with slides and hands-on activities. So we're going to start with understanding why even Eval, what is an Eval, go over the different ingredients or components, if you will, there.

Then we'll jump into the actual Braintrust UI, you'll go through some activity tasks there, and then move back into the lecture, talk about the SDK, how you can do the same thing via the SDK. It can be a bit more powerful in certain situations as well.

Then go into production, like logging, so day two stuff, how are you observing your users interacting with your production app or production feature. And then finally, we're going to be incorporating some human in the loop, so trying to establish some ground truth for the ideal responses, improve your data sets, and overall improve the performance of your app.

If you've gotten a chance to check out the poll in the Slack, feel free to submit a response. Really curious to see how everybody is currently evaluating your AI systems.

Just as a question that I'm out of curiosity, could I ask for a show of hands? How many people have seen Braintrust before, gone to braintrust.dev, and interacted with it?

Cool, that's great. So we have some pupils that have already gone and explored a little bit. That's exciting, and a lot of people that are brand new.

So starting off with just an introduction, what are Evals? How do you get started? First, I wanted to just show off some mentions of Evals in the public space. You may recognize some of these names. They see the importance of Evals, which may or not point you that this is something important that we should be thinking about when pushing changes into production, when we're developing these AI features.

So why even do Evals? Well,

they help you answer questions. That's ultimately what they're for. What type of model should I use? What's the best cost for my use case? What's going to perform best in all of the edge cases that my users will be interacting with?

Is it going to be consistent with my brand? Is it going to

talk to the end customer, to the end user, in the same voice that I would want a human? Am I improving the system over time? Am I able to catch bugs? Am I able to troubleshoot effectively? So all of this can be answered with the help of Evals, which is what we'll be discussing today.

The best LLMs don't always guarantee consistent performance, so this is why you need to have a testing framework in place. We have hallucinations occurring at a pretty high rate. Performance is also degrading when you make changes. It's difficult to guarantee that the change that you're putting through isn't going to regress the application.

And

changing a prompt, even if it may seem like it's improving it, will actually regress it. So you need to have some scientific, empirical way of testing these changes and making sure that your AI feature is performing at the level that your users expect.

So how do Evals help your business? Well, they cut dev time. You'll be able to push changes into production a lot faster. Evals will live at the center of your development lifecycle. They will reduce cost. Due to the automated nature of Evals, you'll replace manual review.

It will then lead to faster iteration, faster releases. You'll also be able to optimize the model that you're using, make sure that it's the best bang for buck. Your quality will go up, and you'll be able to scale your teams.

It will enable non-technical users and technical users to also have a say in the prompt choice, in the model choice, and just the overall management of the performance in the production traffic.

These are some of Braintrust's customer outcomes. So we've been able to help some of these great companies move a lot faster, increase their team productivity, and increase their AI product quality.

So now moving into some of the core concepts of Braintrust. So we're really targeting three things: prompt engineering, so we're thinking about how we're writing the prompts, what's the best way to provide context on our specific use case to the prompt so that we are optimizing its response.

The middle piece, Evals, are we measuring improvements? Are we measuring regressions? Is this being done in a statistical way that's easy to review, easy to understand? And then finally, AI observability. Are we capturing what's happening in production? Do we know if our users are happy with the outputs?

Unhappy? Are we able to prioritize certain responses so that we can keep iterating, keep improving?

Great, so now moving to the Eval section. So what is an Eval? So the definition we've come up with is that it's a structured test that checks how well your AI systems perform. It helps you measure quality, reliability, and correctness across various scenarios.

Ideally, you're capturing every scenario that a user will live through when interacting with your AI feature.

When it comes to Braintrust and writing Evals, there's really three ingredients that you need to understand to be able to work effectively. The first is a task. So this is the thing that you're testing. This is the code or prompt that you want to evaluate.

It can be a single prompt or a full agentic workflow. The complexity is really up to you. The one requirement is that it has an input and an output. Then we have our data set. So this is the set of real-world examples, our test cases that we want to push through the task to see how it performs.

And then the score, that's the logic behind the Evals. So how are we grading the output of our prompt on our data set? These can be LLM as a judge, scores, or they can be full code functions. And the caveat is that they need to output a score from zero to one, which will then be converted into a percentage.

Can I have a question?

Yeah.

What are the test cases? Is it also agent-generated?

It can be at first. The question was, is the data set synthetic? Can it be synthetic? And the answer is, it's a great way to get started quickly, is having an AI generate those initial use cases. But as you progress, as you mature, it's great to ground those in logs.

So you're capturing the real user traffic, the real interactions that users are having, and integrating those into your data sets.

OK.

Great, so now I wanted to talk about offline Evals and online Evals. So there's two mental models to think through. Offline Evals are what you're doing in development. So this is the structured testing of the AI model, of the prompt that you are going to then eventually show off to customers in production.

So this is for proactive identification of issues. This is what we'll be doing today in the playground in Braintrust and also via the SDK. But then on the other side is online Evals. So this is in production, real traffic is being captured and being measured.

It's being graded just like your offline Evals are being graded. And this is going to allow you to diagnose problems, monitor the overall performance, and capture user feedback in real time so that you can understand, oh, this edge case isn't included in my data set.

This is a weak point in my current AI product. I need to spend some time attacking it and improving it.

A big question that we get asked is, what should I improve? I have my prompt. I have my Evals. How do I know what's wrong? And I think this matrix really helps simplify this question. So if you have a good output from your own judgment, looking at what the LLM is giving you, and it's a high score, then great.

You've verified yourself that the output is high quality, and also the scores, the Evals have also come to the same conclusion. If you think it's a good output, but it's a low score, then that's a signal that you need to improve your Evals.

Maybe the score isn't actually representing what a human would think. If it's a bad output but a high score, same thing. It doesn't match what a human would think looking at the output, so you need to improve your Evals.

And then finally, if it's a bad output and a low score, your Evals are working correctly. That's good. And now you need to focus on improving your AI app. So I hope this helps explain how you should be thinking through your scores and what to tackle in which moment.

So now we're going to zoom into each of those ingredients or components, starting off with the task. So as I mentioned, a task is really just an input and an output. It can be a single LLM call or a whole agentic workflow.

It's really up to you what you want to test. So in this pattern, we're just going to be creating a simple prompt. This is what the activity today is going to encompass. And you can use dynamic templating with Mustache.

So you can provide your data set rows as part of the prompt, and that will be tested. And you'll get to see that in action soon.

What if you have more than just a prompt? What if you have a multi-turn chat, a whole conversation that you want to evaluate? So you can do that in Braintrust today. You can provide the whole conversation as extra messages.

So providing that whole chain of messages back and forth with the user and the assistant. You can include tool calls as well to simulate those tool calls and evaluate that big chunk, that context, that whole conversation at once.

Tools are also something supported in Braintrust, so that's oftentimes something that your AI applications will leverage, talking to external services or grabbing information from somewhere else. So you can add tools to Braintrust and have the tool available for your prompt to use.

And just to mention, that's great for RAG use cases. So I know that's a hot wordright now, so if you have that in mind, Braintrust can handle it. We support tools. We support RAG. Agents, another hot wordright now.

So we also allow you to chain your prompts. So you can have three prompts chained together. The output of the first prompt will become the input of the next, and so on. So you can start testing end to end

all these prompts back and forth and do the same thing that you would with a single prompt.

Great, so now moving into data sets. So this is the test cases. You're going to keep iterating over time, but initially, maybe you're using something synthetic. There are three fields that you need to understand for a data set.

Only one of them is required, though, and that's the input. So that is the prompt, the user-provided

use case. The prompt that would be provided by the user would be the input. You could think of it that way. And then you have the expected column, which is optional, which is the anticipated output or the ideal response of that prompt.

And then finally, you have your metadata, which can allow you to capture any additional information that you may want to associate to that specific row in the data set.

Some tips for data sets is to start small and iterate. It doesn't need to be the largest data set of all time. It doesn't need to include all of your use cases. Just get started. Use synthetic data at first.

And the important piece is to keep improving, keep iterating. So if you start logging your real user interactions, even if it's just in staging or internally in your organization, you can start to increase the scope of the data set, and it will start to become closer to the overall domain of use cases that users will interact with.

And then finally, you want to start implementing human review. This will allow you to establish ground truth, improve your data set, improve the expected column, which will be great for your Evals.

And zooming into scores, so this is you have two options here in the type of score that you want to use. LLM is a judge. This is great for more subjective or contextual feedback. What would a human need to understand when looking at the output?

What criteria would they consider? This is more of a qualitative question that you want an answer to, using that LLM as a judge. On the code-based score, this is deterministic. So you would want exact or binary conditions. This is more of an objective question.

And the important piece is to try to use both. So you want some LLM as a judge scores, but you also would like some code-based scores, and they'll help you meet in the middle and understand the quality.

So some tips here, if you're using an LLM as a judge, maybe use a higher quality model, a more expensive model to grade the cheaper model. Make sure that the LLM as a judge has a focus. So don't give it four or five criteria to consider.

Zoom into one specific piece and expand and explain the steps it should think through to come to its conclusion. If you're writing LLM as a judge, maybe you should eval the judge and make sure that the prompt that you're using is matching what a human would think.

So that's another great way of improving your scores. And just make sure that it's confined and you're not overloading it with all the context in the world. You want it to be focused on the relevant input and output for consistency.

Great, almost at the end here. So there's two things to understand about the Braintrust UI specifically. So there's the playgrounds, and this is for quick iteration of your prompts, agent scores, data sets. It's really effective for comparing. You can do A/B testing with prompts.

You can do A/B testing with models. And then you can save a snapshot of the playground to your experiments view. And the experiments is for comparison over time. So you'll be able to track how your scores change over weeks, months.

And everything that your team is doing across the UI, across the SDK, will also aggregate in the experiments view. So you can analyze everything and understand, OK, this new model came out today. How is it performing to the prompt from two weeks ago?

Great, so now we've reached the first activity. So if you could please go to the activity document, and it will take you through the journey of running your first eval in the Braintrust UI. Please raise your hand if you have any questions or run into any issues.

We'll be walking around and just making sure there's no blockers.

If the registered?

Yeah, if you check Slack, we'll also go back to the QR code.

So did everybody have a chance to get these QR codes?

The middle one is going to be the most important. This is where you're going to access the materials for the workshop.

Have we uploaded the slide deck to.

Yeah, I'll repeat the question. The question was around extra messages in the prompt. And if you are overseeing agents and multiple types of users, multiple different roles are all talking back and forth, and you want to distinguish their roles and all within the playground UI.

So you canright now, there is no additional delineation between the assistant, the user, the tool call, and

I believe that's it. So having the user be branched and play different roles is something that you would need to rely on the SDK for that additional flexibility.

That was supposed to be my other question. You have the API,right?

Right.

You can do things out of the UI programmatically.

Yeah, and we'll cover the SDK in the next section. I think maybe the biggest takeaway is that there is no limit really on the complexity that you feed to that as that task. The only requirement is that input and that output.

Maybe something is a little bit more tailored to the Braintrust playground in the UI, where some things are actually a little bit more tailored to that SDK. So we can jump into that in that next section. Maybe it makes sense as we're going through the workshop, I'll kind of walk through this as well, just so you can all kind of see me go through it.

But feel free to raise your hand. We can walk around and answer questions.

For Slack, are you able to access the document via the.

Is the slide deck not public? Is that.

It's not public.

Oh, for the slide deck? Is this.

Yeah, the question was, this is all in the UI. That's the only place we've been thus far, just talking a little bit about the different components of the Braintrust platform.

So let me walk through that and give you a sense for what we just kind of showed to you in slides.

So you can't access the slide deck?

We can update that.

Yeah.

Yeah. Just let's kind of walk through this so we can get a sense for what we're building here, some of the things that Carlos just walked through. I have a lot of this stuff already installed. I hope that you kind of walked through this.

We need certain things on our system to actually go and run this. So we have Node. We have Get. We're going to sign up for a Braintrust account. Creating a Braintrust org, already done this, so I'm not going to kind of bore you through that stepright there.

This project, unreleased AI. If you don't do that, you'll see two projects in your account. But this is where we're going to actually create our prompts and our scores and our data set from the repo that we're going to clone into our local machine.

Part of this demo requires an OpenAI API key. That's just what we're using under the hood. That's certainly not a limitation of Braintrust. You can use, and maybe just to kind of highlight something here, you can use really any AI provider out there.

So if you've gone into Braintrust account, you've probably seen this. You've entered your OpenAI API key here. This is what is going to allow you to run those prompts in the playground. You can see you have access to many other providers.

You have access to cloud providers like Bedrock and so on. And then you can even use your own custom provider. But for this workshopright now, we are using OpenAI. Question. Are you able to do this local?

Sorry? Are you able to run evals of Braintrust locally with local models? Yes. Yeah, the question was, can you run Braintrust locally using local models? Yeah. If you look a little bit further out, there's a section for what we call remote evals.

Might not have time to get to it in this particular section, but know that you can go to that and play with that feature as well.

Sorry, coming back down here.

So we're going to clone this repo. This is the application that we're creating. The idea is to give it a GitHub URL and look for most recent commits since the last release and then summarize those for us as developers.

So that's the application that we're going to use. We're going to create some different API keys locally. So if you've cloned your repo, you'll have a .env.local file. I'll show you my example. You're going to also input your Braintrust API key here and then your OpenAI API key.

This is optional down here. It's just if you don't want to get rate limited by GitHub. Probably not going to create a lot of requestsright now, so you probably don't need this.

Really important step here. So I'm going to come back into Braintrust.

So as part of our install, we're actually going to go create some of these resources within our Braintrust project that we just created. So I'm going to run pnpm install. This will actually go push some of these resources, and you'll find these in the Braintrust folder, the resources.

And we'll jump into that, but just wanted to highlight that. So now if I look back into my project, I should see that unreleased AI and the different things that we've created. We have two different prompts now. These are the prompts that we used to generate the changelog, as well as the test cases, the data set that we'll use as part of this.

Yeah. Real quick, can you make the font a little bit bigger?

Yeah, yeah, of course.

Code or here within the doc? Both?

Yeah.

Allright, let me stop there. Anybody have any issues just kind of going through that initial setup phase?

How are the slides?

Excuse me?

Where are the slides?

We haven't made those public yet. I think we're.

Yeah, I'm trying to do that now.

Yeah.

So am I supposed to just pick the QR code on my phone and then send it linked over?

Do you have Slack?

I do have Slack.

Were you able to join the workshop evals channel?

My Wi-Fi is just terrible. I can't really I don't know if these functions.

Yeah, the Wi-Fi is not working.

It's really slow.

OK. Well, I'll walk you through. Yeah.

How are we connecting the cloned repo to the project UI?

Yeah, so when we ran pnpm install, we ran a script in the background called just Braintrust push. And if I look at that file here, there's different things that we've configured. We've configured my changelog. So this is actually the Braintrust SDK under the hood.

This is where we're creating that prompt in that project, unreleased AI. And so there's a couple of things that we can do here from an SDK perspective. This is like you think about version controlling all of these different things and actually pushing them into the Braintrust UI.

So there's a lot of different ways to work with Braintrust, I think we mentioned earlier, either via just like the UI or actually via the SDK. But that's how a lot of this stuff got created.

Cool. Let's kind of walk through this first activity. We're going to access the unreleased AI project. So if we go to that prompts, so this is what we just created. We created two different prompts. This is essentially what we can start to play around with.

We have this one. And Carlos mentioned earlier, there's this mustache syntax. We can actually input variables here into our prompts. And this is going to actually map to the different data sets that we can actually use as part of this project.

So here's our first prompt.

That's about impossible to see.

That was impossible? OK.

It was.

No, no, that's fine. I appreciate that. Thank you.

And the lighting is hard.

Yeah.

Oh, maybe we can.

Change the appearance to light mode. I don't know.

Yeah, where's that?

Let the user appear.

How's that?

Better.

Thank you. No, I appreciate that.

How's this?

Looks good.

OK, cool. So really just reviewing the stuff that we created. We created these two prompts. Here's our data set that we're going to use when we run our evals and our experiments. You can get a sense for this.

Here's my input. I have a series of commits. And then I have a repository URL. And I have when the last release was. That's that since field. So this is, again, the thing that we'll use inside of that playground to create evals and to use to iterate from.

And then the last thing that I'll call out here

is the scores. So we created a few different scores that we'll want to use to actually score these prompts. So we have an accuracy, formatting, and completeness score. And again, this is just in that repo in that resources.ts file.

We have, maybe just to point out, linking a little bit of what Carlos was talking about to the actual code that you're seeing, we have LLM as judge scores, as you can see here. Really, again, trying to pinpoint here the accuracy.

We're not overloading a single LLM as a judge score with accuracy, completeness, and formatting. Going to be very

sort of detailed or scoped down to that particular thing. And then last one, we have a code-based score. So this is a little bit more binary. Is the formatting of this changelog that the LLM generated, does it map to what we expect?

And so we can use some code to do that. So that's what we created via that script.

Question.

Yeah.

How do we get that Sandbox project to our Braintrust project?

Yeah, so if you go back to the lab setup, when you run that install, so I'm using pnpm. I don't know if you use that. If you're using that locally, you can also use npm.

What is the key for OpenAI API key?

What is the key? Do you not have an OpenAI API key?

And the instruction says that reach out to us if you do not have the key. Please say so. In case you don't have API keys, please let us know.

OK. I don't know if we have one to distribute at the moment.

Yeah, if you don't have.

I'll do this later.

Well, if you go like the part of the setup here,right when we go to if you come in here to a playground as an example.

Right, and we're going to pull in one of those prompts so we can pull in both of these prompts to do, again, like what Carlos was talking about, that sort of like A/B testing.

It's going to ask for some OpenAI models. If you don't configure an OpenAI API key inside of your Braintrust account, you don't have a provider to actually run this task against.

But this is the playground. This is what Carlos was talking a little bit about earlier, about being able to do some A/B testing. I have my two prompts that I've loaded in. The idea here is to load in those different ingredients, our tasks, our data set, and then our scores.

So you'll look down here. We're going to select that data set. And then we're going to select the different scores that we want to score this task against. And I'll load in my accuracy, formatting, and completeness. Can do a couple of things here.

I can click run. This will actually, in parallel, go through that data set and use each task that we've defined. And then it will score those. So the idea here is to, again, this provides that sort of rapid iterative feedback loop that we oftentimes need to build these types of products.

So here are my example rows. Again, these could be synthetic. These could be a small subset of rows that are coming back from my application. But now I can get a sense for prompt A, prompt B, how are these performing with my scores relative to the things that I have here.

But then I'm able to do a lot of different things here. I can look at maybe a summary layout, get a sense for the scores. So at the top, this is sort of like my baseline. Up here is my base task, and this is my comparison task.

So you can get a very high-level look at these different scores and how they fared with the different prompts that we've loaded in here. Now, the other thing that we can do that Carlos mentioned is experiments. So oftentimes, we'll want to capture these scores over time.

So when we do make changes, we understand how those scores fared a week ago, a month ago, or whatever it is. So I can click this experiments button. And you'll see the different things that we've loaded up into this playground are already here within this modal.

We'll click create. And this will actually create the experiment that you can go to here. And again, this will, if I click maybe one out, this will allow us to track this over time. This is what we can also lay in our CI kind of workflow.

So we go make a change to that prompt. We make a change to that model. What are the impacts to the scores relative to what we had over history?

What's the completeness score?

Sorry?

What is the completeness score?

What is the completeness score? Yeah, we can dig into that a little bit. This is an LLM as a judge score. So the idea, we're just going to give it instructions. The LLM is going to score the output based on what we've provided in this prompt.

You'll also note I'm just really pulling in the structure of my data set. And so you obviously can write another thing that Carlos mentioned is like scoring the score, evaluating the score. So how well is this thing actually doing based on the output that we are seeing within our application?

OK, that's really activity one. It's reviewing some of that stuff. And then it's creating that playground and showing you all the sort of way that we can iterate here within Braintrust to create better AI or GenAI products. This allows me to now, well, OK, so maybe this isn't theright model.

Maybe if I do, maybe I want to see this new GPT model. I can run this. And now I can see how the model changed for that particular score, how the scores changed when I changed the underlying model.

But now I have these, you have all of these different inputs that could happen to these applications. It's a way for us to track and understand when I do go and tweak this thing, there's actual data behind it.

This isn't like vibe check. This isn't, yep, I think that looks good. I looked at this row. It seems like it's better output. This is data behind it. And we can actually understand as we tweak that prompt, tweak that model, how does that impact our scoring?

And then again, you can overlay this within CI and so on.

Question?

Yeah.

I pulled the project down. And I have the Braintrust account with the OpenAI key and everything. I just don't know where should I beright now with the project. How do I get the GitHub project that is on my machine to the.

Yeah, OK, so you cloned the repo.

Yeah.

Can we get a sense of where people are?

Yeah, please. We can back up considerably. I can just, with you, we can fix this. You've cloned the repo, correct?

Yeah.

OK.

In your .env.local, do you have a .env.local file?

I should have, yeah.

There is a .env.local.example file. So you can copy that into the .env.local. Those are the keys that we want to fill in. Have you filled those in?

No, I did not.

OK. So if you haven't within your Braintrust org created an API key.

Do all of the people already use that? Am I the only one?

I'm guessing no.

Yeah, I don't think so. I think the internet connection is probably the biggest thing we're finding here.

Yeah.

Luckily, all the instructions will be available after the workshop. Same with the slides. I'm about to share the slide link. So tough to update with the connection. But at least seeing Doug go through the same process will give you an understanding of what we were hoping you'd have that hands-on experience doing.

We don't have too much time to wait on this specific activity. So I think in a few minutes, we'll keep going. And hopefully, you'll be able to set up your key so that you can catch up when you have some time.

Yeah, just in the interest of time, we'll probably move forward. But just to complete that, if you go into your settings within your project, or excuse me, within your Braintrust org, you'll see API keys. This is where you're going to create this API key.

I did.

You did that. OK, perfect. Create that. You put that in that .env.local file. And then you run pnpm install.

So that is in my home directory,right?

It's wherever you cloned that repo.

Yeah.

So at the root of that, you put the Braintrust API key in your .env.local file.

Allright.

And you should have a spot for it already if you're using sort of the template from the example. And then when you run pnpm install, again, just to highlight this, it's actually running this command, Braintrust push. So we're taking the resources that we've configured in our project and is pushing it via our API key to the Braintrust org.

Allright.

Cool.

Allright.

Maybe going forward a little bit here to talk about the flip side of this. We've been kind of in the UI for the most part, in our playground, doing that iteration, changing prompts, changing models, and so on. I think it's important to understand that we can do a lot of this via the SDK as well.

We also have a Python SDK, if that's the kind of flavor you're most used to using or the language. But that top portion here is essentially what we did in that install, that post-install script. We defined our assets in code.

We defined scores. We defined prompts. We defined a data set even. And then we pushed that into our Braintrust org. The benefit here is that, again, we can use our repo. We can leverage version control to ensure that the things that we want to change are actually version controlled alongside of everything else that we're building within that application.

So there are really two modes to actually work with the Braintrust platform. It's its UI or its SDK. Again, there's really no limits that we place on the user of the platform. It's going to cater to maybe a different persona, a different use case.

But you can use kind of both of these different things. The other thing that we haven't done yet, that we did within the playground when we ran that experiment, we ran those evals. We actually evaluated the task against the data set with our scores.

You can also define evals in code. You can define the evals within your repo. And in a very similar command, Braintrust eval, now we can push that up to the Braintrust platform and essentially run that same thing, track it in that experiment.

I have now an understanding over time how my evals are performing as I go and change things, all those different things. A little bit more insight here you probably saw with some of that code. This is the command.

We're going to run Braintrust push. And you essentially give it either the name of a file that has evals in it, or you can give it the name of a folder as long as your files have .eval.ts as their naming convention.

And we're just going to go run those evals in that parallel fashion that you saw within the UI. A couple of things maybe I mentioned earlier, but just important to highlight. You should do this when you want source-controlled prompt versioning.

You want consistent usage across your different environments. And you also want to leverage online scoring.

Mention this, obviously, the .eval.ts. That's essentially what the SDK is looking for when we go and run those evals. It makes it really easy to run a larger subset of those without specifying each single file that you want to go run those evals for.

But you can see, and let me jump into the actual activity. You can see the eval that we've created. It's what we've been talking about, the three ingredients that we need. We need that task. We need that data set.

And then we need at least one score there as well.

How do you bootstrap a data set with multiple examples? Is there human feedback in the loop? Or is an LLM being used to create the data set as well?

Yeah, so the question was like, how do you bootstrap a data set? I think it's a good question. I think you could certainly start with synthetic data. Or you could even start with

you release a feature. You're going to start logging that feature. This is another thing that we haven't yet covered. But you can actually use the logs from that to add to the data set. So now you have actual real-life data.

The thing to not do is wait until you have 100, 200, like you have this golden data set. If you think back to that matrix that Carlos was showing, there are the different ways in which we could start to think about improving the application based on what we observe.

So start with something small. And again, it could be synthetic. But then you can, once you start to evaluate it, then you have a different you have different inputs on what you need to do to go and improve a score or improve the application.

But it's really, I think, maybe up to you. The best practice or the thing that I would think about is don't stop yourself because you only have a small subset of data.

Got it.

OK. Yeah.

So for the tests you're running, if you're using an LLM as a judge, so like for the percent completeness score, you're using GPT-4.1 as a judge. That's subjectively scoring the test that you're running,right? So for two runs that happen one after the other, you could end up with different scores,right, if you're using LLM as a judge to run the evaluation?

So the question was, would I get different scores

because I'm using an LLM to do this? And it's not really deterministic. I think that's the reason why you would use a better model so you don't see something like that. I don't know, Carlos, you have any other thoughts there?

Yeah, I mean, I think you're speaking to the nature of an LLM being non-deterministic. So yes, there may be some variability. What we see our customers do, especially with the SDK, is do trial evals. So you will run it maybe five times and then take the average of those.

So there are things that you can do to try to beat that. But it is the nature of the beast. And you have to learn to work with it.

And then the other thing is, how are you scoring a percent completeness if the task of the LLM is to judge, put it in categories like excellent, good? Are you mapping those to scores?

Yeah, so I think the question is, and if I come to the score and you look at the completeness, so the LLM here, yeah, it has to decide based again on the criteria that we give it. If it comes up with excellent, that maps to one and so on.

But again, the score that it gives has to be between zero and one. Yeah.

It's really helpful when you're using an LLM as a judge to go through the Braintrust logs and read the rationale. So it will explain why it shows 100% or 75%. And you can use that to tune the LLM as a judge and improve it.

It likely will not you don't want it to say 100% for everything. If that's the case, you need to improve your evals. And even if it's saying 30% for everything, it doesn't necessarily mean that it's performing horribly. What really matters is the baseline that you're comparing it to.

How did it perform yesterday on these scores on this data set? You shouldn't be comparing just it needs to be 80%. Not necessarily. It matters what happened yesterday, what you're comparing to previously.

Yeah, and this is where it becomes really beneficial to be able to actually drill into what happened within that task. So being able to not only understand the calls that the LLM makes, the tools that it invokes, but actually drilling into those scores that we're using.

And then, like Carlos mentioned, what is the rationale that it gave to give it a good score here? Does that make sense? This becomes, again, another way of this is the human review portion or part of building a Gen AI app.

Yeah.

Does Braintrey offer any kind of optimization or prompt optimization features? So assuming you have your eval down, you've got tons of different data points to test from, can you kind of use Braintrey to optimize the prompt itself?

Yeah, that's a good question. And it's something that we're thinking a lot about is how can we add LLMs to optimize this process for you, not just for prompts, like you mentioned, but also for data sets and for scores.

We're one to two weeks away from releasing our first version of this. We're going to call it Loop. And it will do exactly what you're saying. It will help you optimize the prompts and improve your evals.

Just to build on that, imagine this feature, Loop. It has access to previous results as it's doing it. So it understands when it makes that change. Is it better than it was previously? So it's sort of like that agentic type workflow where it has access to tools.

But it's able to iterate on that prompt and run those experiments with you, of course, in the middle to prompt it to do different things. But it has access to those previous experiments and the results of that. So it knows the general direction it needs to go to get improvements.

Yeah.

So how do you eval that?

We use Braintrust. Yeah, exactly. It's a way of dogfooding. So the question was, how do we eval our AI feature? And it's, of course, we have to use Braintrust. And it's honestly really cool to look at the project and see all the logs coming in and looking at the scores that we've chosen to go with.

Yeah, Braintrust is really helping. And it was actually something that Ankur, our CEO, has talked about, the process of actually getting to a point where we were excited to release something like this. Previously, the models were not performing to the level that we were needing them to perform to.

So every few months, he would run a new benchmark on this specific use case. And it wasn't until the last month that a model finally reached that expectation.

Yeah.

They gave me a mic, so hopefully you can hear me.

Yeah.

Cascading off the gentleman in the front's question around the subjectivity of using the LLMs as a judge in these types of cases, do you offer any way to gain access to that rationale programmatically, such that you could evaluate the thought process of the LLM as it's doing it?

It's kind of meta, like one-step review. But adding in that second layer where you could identify weak spots, perhaps, if there's something that's hyper-workflow oriented or has a very strict process you're looking for the LLM to follow.

Yeah, I mean, you probably saw when I highlighted here, thumbs up if you saw this. Yeah, OK. This is all accessible via API. Coming back down here. So the rationale that the LLM gave, we could certainly build something on top of these rationales and then generate, again, like eval the eval type of workflow.

Cool.

Yeah.

Yeah, question. So I'm just curious.

There's a mic if you'd like it.

Yeah, thank you. Yeah. So yeah, I'm just curious, how should we build our confidence around the result of LLM as a judge? Because, I mean, how do we trust the eval result? Because it's after the model evaluates the data set.

And we could get maybe a good result. But actually, maybe the large model is just overconfident or something like that. Is it like we need to evaluate our eval results using humans at the beginning so that we can build the confidence?

Yeah, just curious any experience you have here.

Yeah, definitely. That's a great question. So I guess everybody heard. Don't need to repeat.

I think there's two things that you can do. One that you mentioned, which is involving a human reviewing that LLM as a judge and confirming that it's thinking theright way. It's outputting the correct score. Another approach that you could do as well as the human in the loop is using deterministic scores, so full coding functions that are trying to grade the same type of criteria using regular expressions or some other logic.

And you can approximate. So if the LLM as a judge is giving a zero, but the deterministic code score is giving a way higher score, then you know that there's something that needs attention, needs to be fixed. The matrix as well that we showed at the beginning that pointed to should you improve your evals or improve your AI app, that's also a great resource.

Yes.

Thank you.

How are you guys thinking about the role, if you think there is a role, for traditional machine learning models in evals? I mean, on one hand, you have totally deterministic code. And then on the other, you have LLM as a judge.

Do you think there's kind of a middle ground for things like intent classification models, entity recognition, sentiment classification, and clustering, and kind of more traditional machine learning approaches that kind of sit somewhere in between the totally deterministic versus totally non-deterministic spectrum of code versus LLMs?

Do you think that there's a role for those type of models? And what do you think that looks like?

Yeah, that's a good question.

I think it's still to be determined how this all shakes out. There are some customers, companies that we talk to that are going full deterministic. They don't use LLM as a judge. And then there are others that are very much going in the LLM as a judge route.

And I think the reason that there's a split is because they both work. So I don't know. I don't know how this will eventually shake out, if we'll reach in the middle or if determinism will win. What I can say, though, is that it's highly dependent on the use case, the problem that you're solving, and experiment with both.

And then you can determine which one is working best.

I guess those are also largely code-based, the things that you're talking about. And so maybe they lean a little bit more towards that.

Yeah, I mean, I'd say there's still kind of neural approaches that are still not entirely deterministic.

Thank you. Yeah. No, I got you. It's sort of like that middle ground. Yeah, I don't have a great answer for you. I do think using Braintrust, you do have the ability to at least configure both of these, the LLM as a judge and then the scorer.

And then you can, again, using the human review process, find the ones that actually map to theright output the best. And then that's how you start to build your application. But it's a really good question.

Thank you.

How's the activity going? I

know we are getting the last 25 minutes of the session. We still have two more little slide chunks to go through. So maybe in two minutes or now, we could move to the slides and then keep it going.

Again, this will all be available after. So feel free to keep working on it.

Yeah, maybe just really quick, we could run the eval just from here. You could see

we're actually going to take

the eval defined here. So we have our task. We have our scores. And then we have our data set, essentially what we just did within the UI, pushing that into the Braintrust platform. And then you can even see this is where it's runningright now.

So again, being able to do these in a couple of different ways, either via the SDK or from the playground itself.

I think you explained this already. And maybe I was distracted by the Wi-Fi. But how do I think about the difference between the playground and the experiment?

Yeah, that's a great question. Let's see if we can just quickly go back to the slide.

Playground, you could think of as quick iteration. Experiment, so a playground ephemeral, experiments, long-lived historical analysis, if that helps answer your question. They're becoming more and more the same. Historically, the experiments had a bit more bells and whistles.

So typically, teams would gravitate towards the experiments. But we found that they really liked the quick iteration. They really liked using the UI. And so we started beefing it up. And now they've become pretty much the same. So yeah, playground, more ephemeral, quick iteration.

You want to save the work that you've done to an experiment so that you can later review it and see the scores update and change over time.

I guess that was my opinion. So when I do an eval from the SDK, it always is an experiment. What if I just want to iterate in my text editor? I should use the playground UI for that?

Yes, and remote evals, which will allow you to define via the SDK the eval, but then expose it into the playground. So it's like the bonus activity in the document at the very end. So maybe you should check that out.

And we won't have time in this session. But if you come to the one at 3:30, we will.

Any other questions? OK, cool. So moving into lecture three. So once you've finished development, it's reaching customers. You're in production. Now what do you do? Well, the important thing is logging. You want some observability. You want to understand what's going on.

How are they using it? Are there any gaps? Are they unhappy? It can help you debug a lot faster. It can allow you to measure quality instantly on that live traffic. You can turn those production traces, what you're logging, you can turn it into a data set and bring it back into the playground, keep improving the prompt.

And it allows a lot of non-technical people to understand what the end user is thinking. So you can close this feedback loop. We have a lot of PMs, SMEs using Braintrust and going through the logs and looking at that user feedback to understand what gaps and improvements may exist.

So how do you log into Braintrust? Well, this is done via the SDK. It needs to plug into your production code. So these are some of the steps here, the tools that you can use. So you need to initialize a logger that will authenticate into Braintrust.

It will connect it to a project. So now your logs will go to a specific project in Braintrust. Some great ways to get started with really one line of code is to wrap your LLM client. So you can use wrap OpenAI around that LLM client.

And now any communication will get logged with your prompt, response, also metrics, so how many tokens were sent back and forth, the latency, all errors, everything, just by adding that wrap OpenAI. You can do the same with Vercel AI SDK.

Or you could use Otel. So we also integrate with Otel. So if you want to go that route, it's also available. If you want to log and trace arbitrary functions, we also support that. You can just use a trace decorator around the function,

really helpful for keeping track of any functions that are helpful to understand and keep track of. And then if you need to add additional information, like metadata, you can use span.log. So it's very capable, very flexible. But there's still these one-line code ways to get started.

So now that you're pushing all of your logs into Braintrust, you're capturing, you're observing real user traffic. Now we're going to get into that online scoring piece as opposed to offline. So online is measuring the quality of live traffic.

So you can decide how many logs that are coming in will get evaluated and scored. It could be 100%. It could be 1%. It's up to you. This allows you to set early regression alerts. So if it starts dropping below 70%, 60%, ultimately, it depends on the score that you're using and what you've established as the baseline.

But if it starts dropping below a critical amount, you can set up alerts and notify the correct team. You can also A/B test different prompts. You can set up tagging and understand, oh, this trace coming from this user is from prompt A versus this one from prompt B.

And you can compare the grades, the score results coming back in. So this is great for just improving feedback, moving quickly, and understanding if there's been a drop in quality.

How do you create an online scoring rule? Well, everything is done via the UI. You go to your project configurations, click on online scoring, and then you can add your rule. This is where you'll define what scores you want to be used on that live traffic.

And then crucially, that sampling rate. So maybe at the beginning, you start with a lower sampling rate. And then you can increase it once you trust the metrics coming in. You can also choose what span you want this online score to run on.

So it defaults to the root span. But you can get more granular and specify, I want this nested child span to be scored.

Once you start collecting these logs, collecting these online scores, oftentimes teams want to view them in interesting ways and customize the lenses on those logs. So that's where custom views come in. You can apply filters, sorts. You can customize columns on the logs with whatever information you'd like.

And now you can start saving those views and making them available for the rest of your team to just come to the logs and select, oh, I want to go to the logs under 50% view or their own custom view that they've made that's specific to what they care about.

So it's a great way of collaborating and speeding up the process of viewing the important things to you.

Great. So we went through the slides. Now we would jump back into

the activity document. There, we can look at the actual code, see where the logging is being captured in our files, spin up the application. So you can actually view the application in your dev environment, interact with it. And you'll see those prompts and outputs being logged in Braintrust.

So if you've gone that far and you have the dependencies installed, then I would recommend doing a pnpm dev. And now you'll have your application up and running, interact with it a few times. And you'll see that populate in your project logs.

Once you do that, then you can go to your online scoring settings, set up a rule. And you can keep interacting with the app. And now you'll see it populate with that online score that you just enabled.

Maybe a quick example for those that are still having Wi-Fi issues. Come back down here.

So I'm going to come and spin this up. So as Carlos mentioned, pnpm dev. This is going to now spin up that server on localhost:3000. You should see something that looks like this. There's a few things that you can just click these.

This is sort of like the easy button to get going. This is the GitHub URL. Again, it's looking for the commits that have been made since the last release. And it's going to summarize, again, using the prompts that we've configured here and then start to categorize them.

But now the interesting part of this, if you come back into the Braintrust platform, is if you look at the logs. So this is what just happened on the Braintrust side. So we sort of have this top-level trace, the generate changelog request, and then essentially the tool calls you can think of as we're getting the commits.

We're getting the latest release. We're fetching those commits. We're then loading that prompt. So we're actually loading the prompt from Braintrust. And then you can start to you can click through a lot of these. And as Carlos mentioned, when you use those wrappers, you get all of this sort of goodness out of the box.

So what are the number of prompt tokens? What is the estimated cost? This becomes really helpful as you start to monitor over time. Probably not set up yet because we don't have too much going on. But actually understanding what does that token amount look like over time?

What does the cost look like over time as we change models? And so on. But that's how really easy this is. And maybe just to complete that loop, if you come over to the resource not the resources, the app generate route.ts, you'll start to see some of this stuff.

So I'll just highlight a couple of things. We're wrapping this AI SDK model. So this, again, is how we're really getting all of that metrics. And it's really allowing for us to log a lot of that information with very little lift from ourselves as developers.

But then you also have the ability to configure things in a different way. So maybe we have different inputs or different outputs that we want to actually log in a particular span or actually we want to log metadata.

This becomes really powerful when we want to actually go into those views. And we can actually start to filter these things down. We can even filter by that metadata. So this is where, again, you can hit the easy button.

We're going to wrap our LLM client with those SDKs. Or we can actually get a little bit more detailed and start to log the particular input and output information that we want, metadata. And now we can sort of set these different things.

So if I come out here, I can even create actually, when we add in the scores here, we can create filters based on those scores. So I want to create a view that says, hey, anytime my completeness score goes below 50%, I'm going to create a view for this.

This is going to enable my human reviewers to go in and actually understand that. And then if you look up here in the topright, we can actually add this span to a data set really easily. So we find this thing that will actually add a lot of value in that offline eval type of process.

Click this. Now we have a net new row in that data set that now adds a lot of value. This is sort of like that feedback loop. We've done that offline eval type of work. We have found theright prompt, theright model, all these different things.

It's in production. We are logging it. Now we're sort of understanding that and maybe in a human review type of way. Add that span to the data set. This adds, again, to the offline type of portion. Again, you just see this sort of flywheel effect of creating really powerful GenAI apps.

Yes.

Is there an eval score generated for this online log as well as we ran it? Is there an eval score created for it? And do we add it to the data set based on if the eval score is good?

Because we don't want bad examples in the data set.

That's one way of thinking it. You don't need to run an eval as the user's interacting with it. That's what the online scoring does. So once you set up that online scoring rule, it'll output a score based on the judge that you've chosen as the online scoring rule.

And exactly what you said. You could either filter it and select the good responses, add those to a data set, or vice versa, select the bad responses and understand why are they bad? How do I improve them, make them better?

Just to complete that and kind of how do we configure this? Carlos walked through what this would look like. This is my rule. Obviously, you would call that something a little bit better. But I actually want to add in my scores for those online logs.

And then you probably wouldn't do 100%. But we're just going to do that for this instance. And now when I come back to my application here, and maybe I want to do I'm going to just do a quick refresh.

So now when these logs happen within the Braintrust side, we're actually going to run those scores against that output. So we'll understand based on what happened here, how did it score on formatting? How did it score on correctness?

And then this also can now layer into so you can see here, we have a 25% on the accuracy, 100% on the completeness. So maybe we have a little bit of work to do. But now if I click into this, this is where you can start to now create different things within the view portion here to ensure so this is a filter.

So maybe I want to change this to anything that is less than, let's say, 50%. Now I can save this as a view. And my human reviewers are able to now come in here, open up this view, and look at all of the logs where my accuracy score is less than 50%.

And now we can, again, create that sort of iterative feedback loop.

Any questions on this section?

Yeah. Maybe a good segue to the human in the loop. This becomes really you almost really oh, sorry.

I had a question about using Braintrust and implementing it on existing projects. Is it something that's easy to do with

something like Langsmith? You can just add a couple of lines and it'll trace everything for you. Is it the same in Braintrust? Or do you have to refactor all your prompts to use the Braintrust prompts?

Essentially the same thing. So you have Langsmith as code to wrap an LLM client. Or it has decorators to put on functions. So the exact same thing on the Braintrust side.

Got it.

Yeah.

So then if you do that, is it easy to then

use create data sets from those logs?

Yeah, absolutely. As long as

the logs that you're producing map to the structure of the data set that you've created, then absolutely. It becomes really just you click that button. We're going to add that span to the data set. And it becomes really easy to connect those two things.

Cool. Thanks.

Yeah, of course.

Yeah, good question.

OK, yeah. So let's talk maybe a little bit really quickly about the.

I was going to say there's a question over here.

Oh, go ahead.

Super quick. With the sampling rate, is there a way to just override that where if certain inputs are received, you can force that to be included in your sample set? Like if you have some manual user input, just push that back into the system automatically.

It's not in your sample rate?

The way that you would go about that is changing the span that is targeted. So instead of it applying to the root span, you would specify a span that only happens if a certain criteria is met. So then it could be 100% or 50% of just when that span appears.

OK, so yeah, this is where we could bring sort of like the human in the loop type of workflow. This is where we want to actually maybe bring in subject matter experts. And Carlos mentioned product managers, maybe SMEs.

We also have doctors coming into the platform and actually evaluating some of this stuff. These are the people that actually understand whether or not that output that's created by that large language model is valid, is good. And this is a really powerful thing to have as part of the process to building really powerful AI applications.

We can catch hallucinations. Being able to establish that solid foundation or that ground truth is oftentimes having that human in the review in the loop type of person becomes really beneficial here.

So why does this matter? It's really critical for quality and reliability. Like we were just talking about with LLMs and being able to trust whether or not they can do the same thing over and over again. It's non-deterministic.

Automations can miss that nuance. We want to be able to sort of apply that human type of review to the things that we're doing on the AI side with the LLM as a judge type scorer. We also want to help you make sure the final product meets the actual expectations of the user.

So the user is going to have a much better understanding of how to or what the final output should be. So having that person in the loop to look at those outputs becomes really powerful to ensuring that you build really strong GenAI applications.

Two types of human in the loop. There is the human review. So this is where these people are actually going to go into the Braintrust platform and actually manually label, score, and audit those interactions that the user had with the AI application.

And then there's actual feedback from the user real time. So this is like a thumbs up, thumbs down button within the application saying, hey, you did a really good job. You did a really bad job. But now we can sort of use these together as well.

I can now create a view within Braintrust that filters down to any of my user feedbacks of zero, so thumbs down. And I actually want to review whether or not those things are bad. And if they are bad, I can add those to the data set, again, creating that sort of flywheel effect.

Just really quick here, if you go if you look at that application, you're able to actually click one of these thumbs up or thumbs down, create a comment, really good. And then this is now logged back to the Braintrust application.

So if you look back at our logs,

I'll remove our filter. So we should have a user feedback score now here of 100%. And then we should have a comment over here as well, really good. But then, again, these are the things that we can now if I open this back up, I can create a filter on my user feedback score.

Now I want to understand all of my logs where user feedback is one or zero. And then I can do something from there. But this is done very easily via the log feedback function within Braintrust. You provide it sort of like the span that was already created within that log.

And then you provide that user feedback to it.

You can also enter really quickly here in the platform. You can enter human review mode. So this is a way in which

it's sort of hiding away some of the different some fields that may not be really relevant for those people that are coming in and doing human review. Oh, I haven't actually configured any scores yet. So you can actually see you would come out here.

And you would create different scores for that human to go in and do that review, whether it's sort of like an option-based free-form input slider. So is this maybe thumbs up, thumbs down, sort of like yes or no?

Or maybe you could do something a little bit more verbose, A, B, C, D, whatever it is. But this is where you create these scores. Now they exist as part of those logs. Those humans can go in and now look at the input and the output, give their review of it, again, adding to that, again, that flywheel effect that we need to create.

Yeah, I wanted to add there as well, this is really helpful for evaluating your LLM as a judge. Oftentimes, we see customers use this process to provide some ground truth for their data sets and also for the LLM as a judge.

So you can imagine having a team of SMEs come in. They review. They do thumbs up, thumbs down on maybe five different criteria, qualities that they're measuring. And then they provide that data to a playground where the prompt is the LLM as a judge.

And they go through the playground. And they test to make sure that the LLM as a judge prompt matches what the humans thought. So just something there to think about. But as Doug is saying, it's a great feedback loop.

It's a great flywheel effect that can be created when you add this human to verify and confirm.

Cool. That is it for the workshop. We do have a few minutes left. You can certainly answer a couple more questions.

So for people who are successful with this, how much time are they spending going backwards and forwards, putting the human around it, validating it as themselves before they get into more live testing? And then how often are they going back?

What's the kind of balance of time on task?

Offline versus online evals?

Yeah, and how much do you have to do up front to really get best results? Or can you really just put something down, figure it out later, and optimize on the fly? Because you don't want to get stuck in analysis mode with a bunch of stuff.

Right. Yeah, the question, just to repeat it, is how much time do you have to invest up front to get value? Should you keep going over it to try to optimize? Or better to just start quickly with minimal scores, minimal data set, and then keep improving?

And I would say the latter. You don't want to be fixated on creating a golden data set or 20 scores. If you have one or two scores, and you have 10 rows in a data set, it's going to be tremendously helpful.

And then from there, it's all about iteration. So just going back and improving, adding some more rows, adding another score, tweaking the scores. But you really just want to get started quickly.

Yeah.

So you've mentioned some elements of this scoring. There's the function that you want to test. You have to define the test steps, if you will. One of the challenges that we are finding is our actual application does change.

And it could change biweekly. It could change monthly. Is there a way to look at

trying to automate changing the actual function that you now need to change to match the way that your application logic has just changed this week from two weeks ago?

Would you say oh, go for it. Yeah, I guess I was just going to, again, clarify to make sure I understood. So you're saying that the score, the actual scoring function, is going to

stop being useful. It's going to become obsolete. It's going to become too old to actually gauge the quality?

Not just the scoring function, but the actual steps that you want to test. So this week, there might be only two turns. Just giving a very simple example. In two weeks, in the next sprint, there are now five turns in your app because the logic has changed.

And now you have to update, of course, I think the function element. There's probably no way around it. I'm just curious about whether you guys have thoughts about how that could be improved or made easier.

Well, I think your task will always change. The thing that we're trying to build, that's where Braintrust helps because we're going to understand when we do go make that change. We actually understand whether or not that change improved our application or regressed it.

So we're not going to say stop making changes to the underlying.

Yes. Yes, no, I understand that. So it is inevitable that the application is going to be changing. And you're going to have to constantly update the test, the test step, the function that you're actually wanting to mimic in your test.

It's very similar to traditional software testing. You don't want to write a test that lasts for a day or a week. You want to think of robust tests that will live on for months or years and will actually measure the underlying quality of the application that will be long-lived.

So I think it's more of how do you optimize the scores to measure qualities that will still be around even if you add some additional steps in the task.

No, it's worse than that because unless you have those additional steps in your function, you're not mimicking your application's logic. You're still using the logic from last sprint. So no matter how good your scoring could be, it's no longer reflecting what your application is doing this week or this sprint.

I think regardless of how many steps you have, there's still an input. And there's still an output that we want to score against. Correct?

Yes, but I think one of the things you need to do is to first define how you're going to arrive at the score. The input comes in. And now maybe you have three turns. And then because you're mimicking your app, and then you get your output from these three turns, your app just got upgraded.

There are now five turns or whatever.

Yeah, so when you're writing the evals, you can dynamically call the task. So even as you're working on your application and it's changing, you're still pointing to the changing app. So the idea is that when you are wanting to merge into main, you open a PR.

And then your evals will run on those new changes. You don't need to go and update the .eval.ts files. They will now reference the updated task application that you're trying to understand the underlying quality for. Does that make sense?

So I think the question, again, is are the scores is the underlying logic something that you can trust and that will live on? Again, it's not easy. And it's something that is changing. But that's what we're hearing from customers is investing in that.

At Braintrust, when you send evals to humans, SMEs, what's the name of that role? And how are you managing that? I'm guessing to some extent, it was originally the team. But that can't scale. So how are you managing that?

I think it's like organization specific. I don't know if there's a specific.

I'm saying your organization, using your own tool, how are you managing the SMEs yourself?

I don't think we're using any SMEs at the moment. We're not a health care company or a legal tech company where we heavily rely on specialized knowledge to that degree.

But you're not doing human evals of your own product?

We just now started. We just now branched into having an AI component to our application. So we haven't needed to go there just yet. But we talk to a lot of customers that are working in those specific industries with those use cases.

And they will sometimes hire external services that will do the annotations for them. Or they'll bring them into Braintrust. And they'll be using the platform just to review. So they have a specific role within Braintrust. And there's a specific view that they would operate in that's just for annotation.

Great, yeah, another question over here.

I was just curious because we're using out-of-the-box AI models here and are not really fine-tuning the models as the application progresses. Do we have a way to do some few short example prompting from the data set and the eval scores that we are already using?

So is there some feature like that where I can use the data sets or the online logs that are added to the data sets? If the eval score is good, use it as an example for future prompts to just make the prompt better because the models are out-of-the-box.

Yeah, so question around few-shot prompting, providing examples to the prompt of the ideal response. That's something that you can do today within the data set. In the metadata column is where you can provide the few-shot examples that you want for each row.

And then when you're running that eval or messing around in the playground, it'll reference the few shots in the metadata.

Got it. But what about the online testing stuff or the online logs, whatever you call it? When users are actually using the application and it's hitting the prompt, can the prompt real-time use those examples from the data sets as well?

Right now, it's not something that Braintrust facilitates. Within the SDK and building your own logic, you could come up with a workflow like this. But natively in the platform, we're not facilitating live traffic into few-shot examples.

Got it. Makes sense.

Great. Well, thanks, everyone. I know we're over time. Really great to have you all here for our first workshop of the day. I hope you can walk away with some ideas of how you can improve your eval workflow.

And our team is here. We have a booth just outside of this. So feel free to stop by. We can answer more questions, have a conversation. Yeah, thanks, everyone. Thank you all.