7 Habits of Highly Effective Generative AI Evaluations - Justin Muller

Hello. Welcome to my talk on the 7 habits of highly effective generative AI evaluations. I'm Justin Muller, I'm a Principal Applied AI Architect at AWS. I have degrees in physics, math, law, and business. I've been working in natural language processing for about 15 years, on and off, and generative AI for the last 4 years.

Inside of AWS, my team is a small specialist team that helps customers scale Gen AI workloads, so I've had the opportunity to work on many, many workloads across all different industries, all different sizes of customers. Some small workloads, some of the largest in North America.

I'll also mention that because of that experience, I've seen a lot of workloads fail, and I've seen a lot of workloads succeed. And so one of the things my team does now is that we've gathered up a lot of the best practices we've seen across the successful workloads, as well as some of the common failure points we've seen on the failed workloads, and share them as best practices in talks like this one.

Allright, so let's get going. The first question is: what is the biggest challenge in scaling generative AI? You may be able to guess the solution here, as this is what the talk's all about, but I would argue that it's evaluations.

In my experience, having been called in to help many different customers scale their Gen AI workloads, there's lots of concerns that come up. And when I ask this question with a live audience, there's things like cost, hallucinations, accuracy, capacity—all these different types of concerns come up.

But the number one thing that I see across all workloads is a lack of evaluations. And in particular, I call it the missing piece to scaling Gen AI, because a lot of times when a customer has successfully built a POC and they have a workload that they like, or a chatbot that seems cool, and they're figuring out how to scale it, this is the piece that I come in and most often it's missing.

And when we add evaluations, it unlocks the ability to scale. And we're going to look at how that happens. But it's by far the most common way to unlock scale. Allright, I'll give you a quick customer example. In July 2024, I was called in as an escalation on a workload where a customer was doing document processing, and they'd been working on this document processing workload for about, I think, 6 months, 12 months maybe, and they had 6 or 8 engineers working on it.

So it was a pretty big project at that point. And the

VP of technology that was in charge of the project called me in. He asked everyone else to leave the room, and he was like, "Look, this project, we spent a bunch of money on it. Accuracy is at 22%.

We're thinking about just cutting the project. We're not getting the results we need. So I need you to tell me if we can save this thing." And essentially, I spent a couple of weeks doing discovery and understanding what was going on in the workload.

And the number one thing that I realized was they had zero evaluations,right? They had this big process, and end to end, they had a single number coming out the end that said, "Well, it's 22% accurate." So what I did is design an evaluations framework, and the results were almost funny, because once the evaluation framework was in place and you could see exactly where the problems were, fixing them were trivial,right?

Fixing the problems wasn't really the challenge. Increasing the accuracy wasn't really the challenge. It was knowing where the problems were and what was causing them. So over the course of the next 6 months, they built the evaluation framework, they fixed many of the issues that came up, they were able to achieve 92% accuracy by January, which 90% was their threshold for launching to production at scale.

And they launched and became the single largest document processing workload on AWS in North America at the time. Allright, so evaluations are really, really important. What are they? How do we actually build these things? This is the place I like to start with evaluations, because many times, especially if you're coming from a traditional AI/ML background, you look at evaluations as a way to measure quality.

Because you're familiar with things like, "Well, what's your F1 score? What's your precision and recall?" And you get a score, and it just tells you how well you're doing. And while generative AI evaluations do produce a score, if anything, it's a far third of the top reasons why we do evaluations with Gen AI workloads.

The main goal with any evaluation framework should be to discover problems. Similar to the workload example that I just shared, if your evaluation framework tells you where the problems are, and perhaps even suggests solutions because it includes an element of generative AI reasoning, now we can improve our workloads.

And of course, yes, we need to measure quality. That's very important. But I start this way because when you think about designing an evaluation framework, if you come in with the mindset that it's going to find errors, you design it in a very different way than if you come in with the mindset that it's just going to measure how well you're doing.

Allright, so let's double-click on what that looks like.

And before we do that, though, I do want to add a second aside on why I think evaluations are so important.

My team within AWS is a very small specialist team, and so we have to have very, very strict filters on which projects we say yes to and which ones we support. And my experience has been evaluations are the number one filter that separates a science project from a successful project.

When I talk to a team and the team says, "Can you teach me about multi-agent collaboration?" and I say, "Great, let's sit down and spend 2 hours putting together a gold standard set for our evaluations so that your multi-agent collaboration framework project is successful."

If the team comes back and says, "Oh gosh, 2 hours on eval sounds boring. Can you just give me the toys to play with?" I knowright away that's a science project, and it's not going to go anywhere. And science projects are important.

They're a fun way to learn. They're just not what goes to scale. On the other hand, for my projects that have been wildly successful, where customers have had 100x return on investment or maybe cut their costs by 10,000% or something like that, those customers, when I say, "Hey, can we spend 2 hours building an eval framework?"

they say, "Why don't we spend 4?" Right? Evals are so important, and we recognize that, and this project is so important that we're going to invest the time. So as an aside, I'll just mention it as my number one filter for deciding which projects are going to be successful and which are worth spending time on.

Allright, so evals in the generative AI world come with baggage,right? And especially if you have a traditional AI/ML background, it can be a little scary,right? You look at evals and it's free text coming out, and you're like, "Oh my gosh, it's not going to be a specific number.

I can't mathematically calculate an exact number every time. Let's panic." Right? And it turns out that the human race has been grading and evaluating free text workloads for hundreds of years, if not thousands,right? Can you imagine if my professor in English assigns me an essay and says, "Write me a 3-page essay on the emotional meaning of sunshine"?

And I spend my hours and I write my essay, and I turn it in, and the professor says, "Well, I can't calculate an exact answer to this. I can't calculate your exact source, so I give up. I'm just not going to score it."

No, of course not. The professor's going to give it a score. And we can evaluate generative AI in the same way that we've been evaluating free text for decades and generations,right? But the key understanding is, if you have a bad professor, they just give you a score.

Maybe you've had this experience like me. You've turned in an essay and gotten an F,right? If you just get a score, you're like, "Well, that stinks, but what do I do with it?" Right? With generative AI, we can go a bit deeper.

We can do what the good professors do, which is point out what you did wrong, point out where you can improve,right? This is why evaluations in the context of grade school are very, very valuable, because you can improve.

This is the same for generative AI. Allright, let me give you another example, another unique complexity to the evaluation of generative AI. This is a 2x4, and I gave myself the task of drilling a 1-inch hole into the 2x4 all the way through it.

And when I talk to a lot of audience, I ask for thumbs up, thumbs down, "How did I do on this task?" Right? Very similar to the types of feedback we often have for generative AI workloads. And often I get mostly thumbs up.

There's always that one guy that gives me a thumbs down. And what you've done is evaluated my output. But if the methodology I used to create this hole looks like this,right, suddenly maybe I didn't do such a good job.

Maybe how I got there matters. And the reason is, even though I managed to make a good hole this time, if this is my setup, if this is the methodology I used to achieve it, or in the context of generative AI, if this is the reasoning I took to create the hole, then maybe I need to rethink my system.

So let me give you another example in the context of a real customer use case. This is a meteorology company that was creating summaries of local weather based on sensor data. So I've summarized what their prompt looked like, just so it's easy to read here.

But essentially, they have a prompt that gives sensor data, like it's raining and it's 40 degrees and it's windy, and it's asking for a summary. And if the summary says, "Today it's sunny and bright outside," all of a sudden it's like, "Hmm, something has gone wrong,"right?

If we evaluate this, the score is 0, you know this prompt gets an F, but we don't know why,right? But what if we ask the model to explain its reasoning? And its reasoning is, "It's important to mental health to be happy, so I decided not to talk about the rain."

Now that we've looked behind the scenes and we've seen kind of how the model got there, we suddenly have a lot more insight into how to fix the problem,right? And you can imagine this in the context and why this is such a big problem in the context of evals.

Let's look at the opposite example. If we give it data and it says it's sunny, and then the response comes out in sunny, well, we'll say response is 10 out of 10. Awesome. I've built a brilliant prompt and I can go to scale.

And this is a danger, because what happens is, if you didn't eval the reasoning, and this was the reasoning, all of a sudden you're like, "Oh, okay, maybe my prompt isn't workingright, because even though I got theright answer, I drilled the hole correctly in this one case,"right?

The reasoning is not getting me where I need to be. Okay, the last step before I get into my 7 Habits of Generative AI Evaluations is what I call prompt decomposition. And although this technique is not specific to evaluations, it's often done in the context of evaluations.

Because when it comes to Gen AI, you only can attach an eval to one prompt. It's difficult to evaluate the first half of a prompt versus the second half of a prompt,right? I think of it like a multimeter.

For my electrical engineers out there, you've got your two sensors. You put one sensor at the beginning of a prompt, one sensor at the end of the prompt, and you're sensing what went on in the whole prompt. Many times, if the prompt is very large and it does a lot of complicated things, it's difficult to understand where the error is, because you just kind of get like, "Well, there's something going wrong in this big pile of code or big pile of prompting."

And so this is that same weather company. This is an example from them where they were having to write a summary of today's weather. Inside their prompt, they did a lot of different instructions based on what the sensor data was.

And one of the things they noticed is that for a portion of their prompt, they actually had it say, "Okay, well, if the wind speed is less than 5, then it's not very windy, but if it's more than 5, then it's windy."

And what was happening is, at the POC stage, that worked great. When they tried to scale out, maybe 2 to 3 percent of the time, Claude was coming back and saying, "Oh, the wind speed is 7. 7 is less than 5, so it's not windy,"right?

And just completely messing up the math. So what we did here is a series of prompt decompositions. And if you're interested in it, I've got a blog here you can check out.

The basic idea is take the big prompt and break it into a chaining series of prompts. From an eval standpoint, that means that you can attach evals to each section of the prompt. And what that allows you to do is decide, "Okay, well, this section's working great.

This section isn't, so now I know where to focus my efforts." The other thing it does is allows you to decide, is generative AI even theright tool for that section of the prompt,right? So in the case of this weather prompt, doing a mathematical comparison, is 7 larger than 5?

You don't need Gen AI for that. Python is perfect at that. It gets it accurate, perfectly accurate. And so in this series of chaining steps, one step was a Python mathematical comparison. When they made this change, the accuracy went to 100%.

And they could tell the accuracy of each step. Allright,

I will briefly mention as well, in the context of that prompt decomposition, it's a very, very common pattern, and so I wanted to talk about its impact on evaluations. One of the most common patterns we see is semantic routing.

Some kind of query comes in or an input to a workload, and the first step is, "Well, what kind of task is this? If it's an easy task, go to a small model. If it's a hard task, go to a large model."

And by doing this, what I'm doing is using theright model for the job, not depending on the prompt itself, but depending on the input from the user and the complication level of the task from the user. And the interesting piece here is, again, attaching evals to each step.

And so that means for an eval and a semantic router, often the input that you're evaluating is a query and the output is just the number 1 or the number 2 in this case. And what you'll see when you run the evals is often even breaking this down significantly increases your accuracy because you're removing what I call dead space or dead tokens,right?

If this was all one big prompt, what you'd see is your instructions, your instructions for your easy stuff, your instructions for your hard stuff,right? And if an easy query comes in, it only needed those easy instructions. Those hard instructions are basically dead space.

They weren't necessary, but what they are is extra cost and an opportunity for the model to get confused. So often by breaking it down this way, we see the evaluation showing the accuracy popping up because you're only sending the necessary information for the task.

Allright, so let's jump into the good stuff. I didn't mean to keep you waiting, but here we are. These are the 7 most common trends I see across all generative AI workloads that have successfully scaled. I will say I have never seen a workload go to scale without evaluations, and most of them also include these 7 habits.

The first one is fast. It is the case that I have seen evaluations built with the mindset that getting results within a week is okay.

What happens is I make a prompt, I push it to some kind of test site, I email a team to go test it, they test it whenever they feel like, they email me back whenever they feel like, and a week later I get to make one more change to my prompt.

And you can imagine a team that's working on that mindset that they get to have like 4 changes or 8 changes a month versus a team that's managed to build an evaluation framework that operates in seconds, and they can make hundreds of changes every day and hundreds of tests every day,right?

The pace of innovation and the pace of increasing your accuracy is going to be much, much faster. To the point where I don't think I've ever seen a workload scale unless it's gone through many, many iterations. And so sometimes just knowing that at the outside is helpful for teams,

rather than somehow you're going to craft this beautiful, perfect prompt your first try and never change it again. I will say, and this is what we're going to see an example of in just a minute, often my rule of thumb target is 30 seconds to run your evaluation framework.

And you do that by using generative AI as a judge to evaluate. Or if the output's numeric, like from a semantic router, it writes 1, 2, 3. You're using Python to say, "Is this, did it output a 1?

Yes or no? Is 1 equal 1? Yes. Okay, great. Good job." The reason I say 30 seconds is often what it means is you spend 10 seconds in parallel sending 100 test cases for generation. I've created a test, I've created a prompt template I want to test, I run it across my 100 test cases and get results, 10 seconds.

Then I take those results, and in 100 more parallel calls, I judge them. So I take my results, compare them against my gold standard, and I get 100 judges going for 10 seconds. And in the final 10 seconds, I take my output from my 100 judges and I summarize it.

Often that means breaking it down by categories and by what'sright and what's wrong. And so what I see in my output is not 100 generated responses that I have to go read through and figure junk out. What I see in that last 10 seconds is a summary where it says, "Hey, you know what?

For all of your questions, all of your queries that had to do with taxes, when you got itright, here's what the trends were. When you got it wrong, here's what the trends were." And again, it's going from that mindset of, "How do I figure out what's going wrong and where the errors are and how to fix them?"

Quantifiable is my second one. All effective frameworks also produce numbers,right? And sometimes, depending on the teams, I get a lot of pushback on this because you'reright that they're not always the exact same number every time you run the test,right?

If I turn in an essay, maybe I turned it in on Thursday and I get a score, or I turned it in on Friday. Friday, my professor has a few cups of glasses of wine before grading, and so my score is a little better,right?

This happens in real life. It happens with Gen AI evals. The way that we deal with it is the same in real life as it is with Gen AI evals, which is we make it numerous,right? Even if there's a little bit of jitter in the score, in the quantifiable score that we ask for, if we have enough test cases and we average across those test cases, that jitter goes out.

Just like in grade school, hopefully you didn't turn in one assignment ever. You turned in a bunch of assignments and your final score was the average of all of those. The other reason why numerous is so important is that we want to cover all of our we want to be diverse and cover all of our test cases,right?

So oftentimes when I'm running this as an exercise with customers, even the exercise of building 100 test cases is very valuable for the team to figure out what the scope of the project is. Because inevitably someone will say, "Hey, when someone asks about taxes, we can direct them to this site where they can get their W-2s."

And then someone else will say, "No, no, no. If they ask about taxes, we can't respond at all because we're not allowed to,"right? And the team is debating the product design, nothing to do with Gen AI, but it's a good exercise to understand what is the scope of what you want to answer and make sure you have questions outside of what you want to answer so that you can measure that the model is redirecting those that are outside of scope.

The next habit is explainable. This is that insight where it's like, "Don't just look at the outputs. Look at how you got there,"right? Look at how the model is reasoning. And in particular, I said reasoning for generation and scoring.

Look at how your judge is reasoning as well in some cases where you're asking the judge to do a lot of reasoning. Because just like you're doing prompt engineering for your prompt for your users, you also need to engineer the judge prompt and make sure the judge is scoring correctly.

I'll show a small example of that in just a minute, but the general idea is just like a professor grading a paper,right? It's not necessarily that you just say, "Hey, give me a score,"right? But the professor has a rubric.

The professor says, "Okay, well, it needed to be 3 pages long. If it's not 3 pages long, 100% off. It needed to have 5 different sources cited." You didn't do that. If you only cited 3, well, that's 20% off.

You needed a professional tone. Every time your tone slips out of professional, I take 3% off your score,right? And so you can see that the rules start building up and you give really clear instructions on how the output should be judged.

And asking your judge to explain its reasoning is often a good way to do prompt engineering for the judge. The next is segmented. This is why I showed those slides on prompt decomposition. In practice, almost all scaled workloads are multiple steps,right?

There's very, very few workloads I've ever seen that are a single prompt,right? And what that means is you need to evaluate each step individually. And this actually is powerful for a number of reasons, not the least of which is that it's very rare for a workload to be a single model.

It's much more appropriate for each step because they're evaluated individually. We evaluate which model is appropriate. Many times as semantic router, you want to use something like NovaMicro that's going to give you an instant response. It's very simple and easy.

It's going to give you 1, 2, 3. And so you can, with an evaluation framework, prove to yourself the smallest model that's appropriate to use on each step.

Diverse is what I was mentioning earlier. You want to cover all of your use cases. So sometimes I'll start by listing out what are the use cases and scope, and then make sure we have questions applying to all of them.

I will say the 100 is a rule of thumb. Sometimes if they're use cases that are like edge cases, we don't really care about. I'll put in like 3 or 4 examples for that. But the core use cases, we want a lot of examples for.

And then the last one I'll mention is traditional. And I put this in there because there is a tendency to throw the baby out with the bathwater and say, "Hey, this is Gen AI. We're all going to use Gen AI.

We're going to Gen AI everything." The output of your my semantic router is 1. I'll use Gen AI to see if that 1 is really the number 1,right? And the answer is no. There are a lot of traditional techniques that are very, very powerful.

Obviously, if the output is numeric, just use a numeric evaluation. For things like RAG architecture is one of the most common. There's many, many types of database accuracy evaluations, retrieval and precision, and F1 scores, and all those kinds of things for question answering.

Simple things like measuring cost and latency are still traditional tooling. So I just want to I put that there to highlight the fact that traditional tooling is still very, very powerful and very important in the context of Gen AI evaluations.

Allright, I'm going to give a quick example of what this looks like and just visually. We start with that gold standard set. Again, this is where I think the most important use of your time is, is building this.

Your whole system is going to be pointed towards and designed towards this gold standard set. So if your gold standard set has a bunch of errors, what you've done is designed a whole system that creates errors,right? So invest.

It's always worth time investing in building a good gold standard set. I will, as a brief aside, mention this is a terrible place to use Gen AI. If you use Gen AI to create your gold standard set, if the Gen AI set creates errors, you build a system that generates the same errors that the Gen AI system has, and it can lead to a lot of problems.

It can be helpful to generate what I would call a silver standard set, which is the Gen AI system taking a guess at a gold standard, but it still needs to be reviewed by a human to confirm its accuracy.

Allright, we take an input from that gold standard set, one of those. We put it into our prompt template, put it into our LLM in order to generate an output. We include the answer and the reasoning. Then we take the matching answer from that gold standard input, and we compare it with that generated output into a judge prompt.

And the judge generates a number and the reasoning behind that number.

And then we pull the category. Often, I'll include the category in the gold standard set so that we can then, as that final step, break things down by category and

generate a summary for theright and wrong answers for each category.

Allright, that's it. Thank you so much. I hope this was helpful. Please feel free to reach out with questions and comments and enjoy your day.