Your Evals Are Meaningless (And Here’s How to Fix Them)

Hey everyone, I'm really excited to present today at the AI Engineering Summit. Uh, before we dive into the topic of LLM evaluations, let me set the stage for what we're going to be covering: why your evals might be meaningless, and more importantly, how can we fix them.

Now, some of you might be thinking, "My evals are fine, we've got a robust testing framework." And you know what, I had the exact same thoughts when I was building the ML Systems platform at Amplify. But after working with hundreds of teams, I've seen patterns emerge that completely change how I think about evaluation.

So let me tell you why this matters. So, first of all, a bit about me and why I'm obsessing over this problem. I co-founded HoneyHive in late 2022 to build evaluation tooling for AI engineers. Before that, I was leading the Data and ML Platform team at Amplify, a Series D enterprise startup.

But then something really interesting happened: at HoneyHive, we started working with teams across the entire spectrum of AI. We're talking hundreds of teams, from 2-person startups to Fortune 100 enterprises. And the use cases, they really span from everything you can imagine—multi-agent systems, text-to-sequel, RAG, so on and so forth.

And across all these different teams, we kept seeing the same problems with evaluation coming up again and again. Problems that standard testing frameworks just weren't equipped to handle. Today, I'm going to share what we've learned. We'll look at traditional evaluation approaches, why exactly they fall short, and more importantly, what can you do about it,right?

Because here's the thing, like, getting your evaluationright isn't just about catching bugs, measuring accuracy, etc. I think it's really just about building AI systems that actually deliver ROI in the real world. And they're not just fancy demos. So let's start with the fundamentals,right?

What exactly is an evaluation? The fundamentals date back to unit integration testing in traditional software. Just like you wouldn't want to push changes to production without any tests in traditional software, you wouldn't want to push changes to your AI application without any evals.

And what's really funny is, like, when I talk to teams and ask them about their evaluation strategy, I often get responses like, "Oh yeah, we test things before production," or "We have some automated test setup." But when we dig deeper, there's really a lot of uncertainty about what makes a good evaluation framework.

So let's just break this down piece by piece,right? To test quality before production, we need 3 key components. First, we need an agent.

This is basically whatever you are evaluating. It could be an end-to-end agent, it could be a small function within an agent, it could be just your retrieval pipeline, so on and so forth. Now, an agent itself could be many things.

It could be a customer service chatbot you've built, maybe it's a Q&A agent that is parsing through legal contracts, and so on and so forth. Each of these has its own unique requirements, challenges, and such. Like, for instance, if you're building a document Q&A system, it needs to not just be accurate, but it also needs to be compliant with regulations.

Let's say if it's financial Q&A, you want to be able to explain its reasoning. It needs to have some level of nuance around financial accounting standards, and so on and so forth. Your evaluation needs to account for all these different aspects.

The second component in this sort of piece of evals is a dataset, and it's the most important, in my opinion. Really, this is what you're evaluating against. This is where I see a lot of teams stumble. They'll show me a couple of test cases, maybe they've built 50 queries, all handwritten by the developers, and they say, "Oh well, yeah, this covers all our use cases,"right?

But does it really? I don't think so. Your dataset really needs to include both your inputs, the kinds of queries and requests your system will actually receive in production, and the ideal outputs. What good responses, your ideal responses, should look like.

And these need to cover not just the happy path, but also the tricky edge cases where things might actually go wrong. And these are the sort of examples that really need to be written by domain experts. People who understand the necessary business context to be able to judge quality.

And really just define what should be the requirements for this agent, what exactly is the contract that we're building here. Third, this is really crucial,right? We need evaluators. This is essentially how exactly how you're measuring quality. Now, traditionally this meant human evaluators.

You'd have subject matter experts reviewing outputs, scoring them, providing feedback. This kind of works, but it's very slow and expensive. Then you have code-based evaluators, great for subjective things like response time, latency. It could even be metrics like ROGE-L, which don't really work, but that's a whole other story and a talk in and of itself.

And now we have LLM evaluators, which promise to combine the best of both worlds,right? You have nuanced reasoning that can really understand humans, and the human nuance and context behind these applications, with the speed and scalability of automated systems.

So when you sort of break it down into 3 components, really it's agent, dataset, and evaluators, you can start asking theright questions. What aspects of your system does really matter? How representative is your test set? Are your evaluation methods really measuring what you think they're measuring,right?

And here's the thing, these components aren't static in nature. They need to evolve over time. As your agent improves, your dataset might need to include more challenging cases. As your evaluation criteria becomes more sophisticated, you might need different kinds of evaluators.

It's a very dynamic system. Now, let's talk a little bit about why LLM evaluators have become so popular. And I really mean popular,right? I'm seeing teams who are switching their entire evaluation stack to rely upon LLMs as a judge.

The main promise here is really compelling. They're cheaper, faster, and more scalable than human evaluation. But why does this actually matter, and what does this mean in practice? First off, it's speed. We're seeing evaluations that used to take roughly about 8 to 10 hours with human evaluators.

They can now be completed in under an hour. So imagine 1,000 test cases that you're processing. With human evals, like something like Mechanical Turk, it might take roughly about a full day of work. So we're talking about 8 hours,right?

With something like an LLM evaluator, it could be about 50 to 60 minutes, assuming you're executing these evals sequentially and not paralyzing them. That's just an incremental improvement,right? It's a huge improvement. Cost is another factor. Now, just to throw some real numbers here, a traditional human eval through Mechanical Turk, you're looking at several hundred dollars for about 1,000 ratings.

LLM evaluators, we're seeing costs roughly in the $30 to $120 range, depending on what model you choose. Now, that's a 10x reduction in cost, huge ROI. And here's where things get really interesting. It's the consistency. We're seeing over 80% consistency with human judgments.

And I know some of you are thinking, "Okay, it's only 80%," but here's the thing,right? When we measure agreement between different human evaluators, we often see similar response rates and similar levels of consistency. Humans don't necessarily agree with each other 100% of the time either.

And we see that with LLMs as a judge as well. The research backing this is substantial. There are papers like NLG-Eval in spade that showed really strong correlations between human judgments and LLM scores. And major model providers like OpenAI, Anthropic, are increasingly pursuing this direction for alignment as well.

Now, all of this sounds amazing,right? It's almost too good to be true. Hold that thought for a second, because this is where things really get very interesting. I think the most uncomfortable truth that we need to face is that LLM evaluators have 2 very major problems,right?

I think the first one is what I consider criteria drift. And it's a very sneaky one. If you're using a popular framework like, let's say, Fragas, PromptFu, LangChain, you're likely relying on their built-in evaluation criteria. And it seems pretty reasonable,right?

These tools are very established, they have hundreds of thousands, if not millions of downloads. But here's the thing, their evaluation criteria is designed for generalizability. It doesn't necessarily measure what's important to your unique use case. So here's a very real-world example that I have personally faced working with a customer.

We started with a working with a company, an AI startup building an LLM-based recommendation system for e-commerce websites. Their evaluator was checking things like all the standard boxes: context relevance on the retrieval side, answer relevance on the generation side of things.

And things really looked great in testing,right? But when they pushed this to production, that's where things started breaking. There were a lot of user complaints. The evaluator just completely missed the user's requirements for relevance. Their evaluator indexed too hard on keyword relevance, without really thinking about the larger context of what the product description means, how is it relevant to the user query, so on and so forth.

And as a result, their evals couldn't really catch any real relevance issues,right? And I've also seen, uh, the grading of these evaluators, where it might work fine on just a single test case, then it stops grading the user query on the same test case consistently because, you know, they were using maybe an LLM that the underlying model just changed.

They weren't using a stable version of OpenAI. So this is what we consider criteria drift. Essentially, when your evaluator's notion of what is good no longer aligns with the user's notion of good. Shreya Shankar and Tina Berkley, they published this paper called EvalGen, which really explores this concept in depth.

They found that evaluation criteria basically needs to evolve over time. And the main challenge is how you balance true positives with false positives, and basically maximize your F1 score when you're measuring alignment against human judgments. So this is a huge problem, but there's another one here.

This is the what I like to call dataset drift. This is essentially, your datasets basically lack test coverage,right? So picture this. You've spent weeks creating the perfect test cases: clear queries, obviousright answers, and wrong answers. Your test suite is golden,right?

And then you launch in beta. And then real-world users start using your system. And then suddenly they type way context-dependent, messy inputs, and your beautiful test cases that you spent so hard writing, they just don't hold up. They don't represent reality anymore.

This is a usage pattern that we see everywhere. Users are constantly asking about topics way broader than your actual test cases. Users sometimes ask data that requires you to use real-world user queries, like, let's say, SERP API results.

Sometimes users ask and combine multiple questions in ways you didn't really anticipate. So this is what really makes, you know, datasets particularly dangerous. Your metrics might still look good, your evaluator is happily scoring on these test cases. It's like practicing for a marathon,right?

And you're sort of running on the treadmill. You might think you're getting really good at it, but that's where now the race happens. You're ultimately not accounting for things like inclines, surface traction, and whatnot. Like, your test cases no longer just represent what the reality actually looks like.

And so how do we go about this,right? How do we fix these problems, and how do we actually make evals work for ourselves? The very simple insight that changed everything for us is this: evaluators and datasets, they need to be iteratively aligned.

Pretty much like how you align your actual LLM application. So here's a 3-step approach that I found that I really want to break down, and this has worked for a lot of our customers, and folks like Hamilton online have also written extensively about this.

But really, first off, you need to align your evaluators with domain experts. This is crucial. Have your experts regularly grade outputs. Not just once during setup, but continuously. Have them critique the evaluator results itself. What is it missing?

What is it overemphasizing? Use the critiques as few-shot examples in your evaluator prompt. And further ground your evaluator with a real-world notion of what's good, what's bad,right? There's a lot of massaging and iteration that just needs to be done on the evaluator prompt itself.

So don't just go ahead and rely on a templated library of metrics, but look at the underlying prompt, actually iterate upon it, see if you agree with the outputs yourself. And keep that process going until there's some level of agreement that feels satisfactory.

Second, really keep your datasets aligned with real-world user queries,right? It all starts by logging. Really, your test bank, it needs to be living, breathing thing,right? When you see underperforming queries in prod, automatically flow them back into your test suite.

You can do this manually, or you can set up automations using the various LLM ops tools that are out there. Third, this is where most teams really drop the ball. Measure and track alignment over time. Use concrete metrics like F1 score for binary judgments, or correlation coefficients for Likert scales.

Track how well your evaluator matches human judgment with every iteration. This will really inform you whether your evaluator is truly improving over time, or are you regressing over time, so on and so forth. And it's a lot of work.

It sounds like a lot of work, but here's the thing,right? It's far less work than dealing with the consequences of a meaningless eval that doesn't really tell you anything and doesn't measure anything meaningful. In practice, the most important step I've personally found is to actually customize the LLM evaluator prompt.

I think a lot of teams today are relying on these templated metrics that are rather meaningless. You want to carefully tune your evaluation criteria. Add few-shot examples of critiques provided by domain experts. Pay attention to whether you're actually using binary scales or Likert scales for ratings.

Highly recommend binary, by the way. And make sure you're measuring something which is actually meaningful. Instead of just relying on out-of-the-box metrics that, you know, don't really measure what's important to your use case, your application, your business context.

Next, you want to involve domain experts as early as possible and get them to evaluate the evaluator. Doing this in spreadsheets is a really good start. You can start with even, like, 20 examples, and it will give you a good sense of whether your evaluator judgments are actually in line with your domain experts or not.

This will also help inform what changes you should make next as you're improving your evaluator prompt. Putting this all together in practice, really, I'd recommend to start with logging. Like, as Hamilton really likes to say, just read your logs,right?

Every time your system underperforms in production, that's an opportunity to improve your test bank. These are real-world failures that are just golden, because these are the exact kinds of problems that your application needs to be improving upon. They show exactly where your evaluation system is falling short, and you should continuously add these test cases to your test bank.

And add the ground truth datasets as well, ground truth labels, so you can continuously improve your test bank over time. Next, iteration. Your LLM evaluator prompts, they aren't sacred texts,right? They need to evolve over time. Test new versions against your expanding test bank.

Make them more specific to your use case. Invest in an eval console sort of tool, build this internally if you have to, just to allow domain experts themselves to iterate on the evaluator prompt. And get a sense of whether they agree with the evaluator critiques and judgments or not.

Finally, measurement. You can't really improve what you don't measure. So tracking your alignment scores over time is extremely important. We recommend setting up a simple dashboard to track your alignment score. It could be F1 if you're using simple binary judgments.

It could be correlation metrics if you're using Likert scales. Really, this will just allow you to track in a more systematic manner as you're improving your evaluator template, whether it's improving over time or not. Very similar to how you might be testing your own original prompt for your LLM application as well,right?

And remember, the goal here isn't perfection. It's continuous improvement. And here's what I want you to take away from this talk, ultimately. Your LLM evals are really only as good as their alignment with real-world usage. So please don't fall into the trap of static evaluation.

Don't treat tests like static tests in traditional software. LLMs don't work that way. Don't just set it and forget it,right? Build these iterative feedback loops into your development process. The payoff is huge, especially when you're trying to improve your evals over time.

Yep, that's it. Thank you for all your time. Feel free to connect with me on LinkedIn if you want to discuss this more in depth. And if you're looking for tools to implement this workflow inside your team, check out our platform, HoneyHive, at honeyhive.ai.

Thank you.