Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith

Hello everybody. Um, my name is Taylor Smith. I am an AI developer advocate slash evangelist slash technical marketing and other titles, depending on where I am. I work at Red Hat in our AI business unit, because yes, we do also do AI too now.

Great. Um, I'm really happy to be here. This is my first time at this conference, so I'm new to the conference, new to speaking at this conference, ready to have a good time and also relax in about an hour and a half and just learn stuff.

Stoked that this was at 9:00 a.m. Um, my presentation clicker isn't working, so I'm going to be a little bit glued over here. But to kind of overview what we're going to do today, I'm going to talk a little bit about kind of the issues of setting up a large language model in production and the reason why you need evaluations and benchmarks and all of these things, and why that is so critical.

And then I do have some hands-on activities that we'll get to do to use some evaluation, um, methods and benchmark tools to kind of get a sense of what is out there to use, how we can use those tools, what that might look like to put it all together for an actual production system.

So? Ready? Are you excited?

Mm-hmm.

Yay. First thing in the morning. Love that. I'm glad it's 9:00 a.m., because that feels like a reasonable start time. I don't like that 8:00 a.m. stuff that happens. Um, setting up generative AI tech in production. If you think about it, crazy pants, okay?

There's so many things that could go wrong. This is such a complex type of technology that is very creative. Setting this up to be scalable and reliable and safe is challenging, okay? That's why we need evaluations. That's why we need all of these tests.

And we need to be careful. We need to understand the technology we're working with when we're looking to implement this as well. So hard to do. Most organizations, so they don't typically start off with a multi-agent framework,right, and go crazy.

Typically they're going to have a kind of standard, repeatable path to maturity. They might start out with, okay, how can I automate some things with AI? How can I have, like, a chatbot type of situation? That's kind of the standard that everybody starts out with.

Okay, I maybe want to implement a RAG setup. Allright, let's start to look at agents. For, uh, on the Red Hat side, we're still dealing with probably, like, the first three phases of this maturity with our customers. And that's where a lot of people, um, still are.

But we get to talk about all the cool advanced stuff at this conference so we can plan ahead and think about what we want to implement. But enterprises, they need to take an incremental approach to do this successfully.

Gen AI models, they have a number of drawbacks,right? And we all know about these things. Policy restrictions. Typically, if you're a developer at a company or have whatever type of role, you're restricted in the type of AI that you can use.

Red Hat was just, um, allowed to use ma what Gemini was just made available to us. But before that, we weren't really allowed to use anything.

Officially. Um, so the company policy restrictions of the tools you can use and the AI you can use is pretty locked down, typically. The legal exposures and risks of these models, you know, there was the glue incident. Um, they're going to say crazy stuff sometimes.

How do we guardrail against that? And how do we protect our customers against that and account for that when that does happen? Because it's hard to, um, completely avoid. There's the bias and discrimination issues. Most of the internet data is still largely Eurocentric and US-based.

So the models trained on this public internet data, of course, is going to be a little skewed,right? So we need to be aware of it. Just like in re-regular everyday life, we know that bias and discrimination exists. How do we account and kind of provide guardrails to make sure that we adjust and prevent what we can?

Cost and performance. That's probably kind of a baseline big one. How much these Gen AI models cost to run at scale in production and the performance, throughput, latency, et cetera, that we need to account for when we have these production systems set up.

And then the knowledge cutoff as well is another limitation. You know, these large frontier models have a knowledge cutoff because they're not consistently trained. So you might be working with a model that was cut off a year ago, so it's not going to have that up-to-date information.

Which is why they, you know, implement RAG systems and agent systems to look out into the internet for more up-to-date info. These are just some of the kind of drawbacks we need to be aware of and account for.

Inference at scale. No matter how I'm going to go kind of into these categories a little more in depth. No matter how good your model is, if it's not fast, if it's not reliable, if it's not affordable, you're screwed a little bit from the get-go, okay?

So this graphic, it shows a classic bottleneck type of scenario. You got concurrent user requests, which might be represented by those green, yellow, and orange dots flowing into a system. But then traditional inference runtimes typically can't handle that kind of load officially.

Um, to serve real-world traffic, whether you're powering a customer support agent, a developer type of copilot system, or maybe a RAG pipeline, you need an inference engine that's purpose-built for scale. So that's where inference runtimes like TRT, SGLang, which I know we have a session on, or at least a couple sessions on, VLLM, um, that's where Red Hat's kind of focusing.

That's where those type of production-grade inference runtimes really need to be utilized.

There's a lot of pain points with inference that I just want to double down on. And some of the activity will be benchmarking and evaluating, um, these types of metrics. Model inference performance evaluation under enterprise-level workload scenarios, it's very complicated to actually evaluate this appropriately.

It requires manual setup of evaluation runs with various parameters you have to test. The compute load just for performance evaluations as well is also pretty taxing. Um, you have to make sure the data sets that you're using for benchmarking are compatible with the models that you're using.

The resource optimization and identifying sizing so that you're efficiently using your hardware appropriately for whatever model size you're using. That's a big challenge, um, for enterprises today, making sure they're efficiently using their GPU investments. And then actually cost estimating is a little bit of a black magic thing.

Like, it's really hard to do that appropriately. Um, you have to, like, backwards math-map inference performance to tokens and the whole thing. So these are what enterprises are trying to achieve, and it's hard. Um, just a little kind of more examples of the challenges.

So we have, you know, this is just an example of stable diffusion bias. Like I said, most of our data is Eurocentric. This is going to be there. So how do we what tools can we use to provide guardrails against this type of behavior?

The glue incident. This was because there was re something on Reddit was like, it's like a joke, um, satire. And this AI overview tool used that information and didn't have theright mitigation techniques in place to identify that satire.

So then it came out in that, um, AI overview, uh, suggestion.

And then we have this, um, like, mad situation where a lot of the we're getting into a lot of synthetic data on the internet. And each generation of these AI models that come out are consuming more and more AI-generated data, which over time is going to get you for oh, there's music happening.

Um, further away from that original human-anchored data. So this is going to lead to a loss of output diversity, a loss of pr a loss of precision. This would be an area where you would need to use those kind of accuracy evals to mitigate again and identify that this is occurring.

So just to cover, of course, Google and the Stable Diffusion Project, they have m introduced additional evaluation frameworks and mitigation techniques to address this. Um, just like any time we have any story like that,right? They are certainly working to make sure that that does not happen again.

We don't know with the closed-source, um, AI offerings how exactly they're doing that, but we can speculate. You know, the AI overview technology maybe introduced more RAG mitigations to where it, it helps to identify that that was satire or whatever the case is.

Um, it has more safeguarding triggers. Um, the Stable Diffusion model likely introduced some level of bias mitigation guardrails. So we need so of course, they've been working to fix this, but ideally we don't run into this and we prevent it ahead of time before a model release or before an application release.

So how do we prevent these kinds of issues at scale in production environments? I want to just look at a couple of definitions, because sometimes evaluation and benchmarking are terms used, um, kind of they kind of conflate a little bit and people kind of use them for whatever they want.

So benchmarking is just a subcategory of evaluation. Evaluation is a comprehensive process to assess a model end to end, and it could include a lot of different kinds of evaluations about a lot of different components. Benchmarking is very specifically controlled specific data sets and specific tasks, typically used to compare models against one another.

So this would be like, uh, latency score that compares different hardware setups and different models, or like the MMLU benchmark scoring, things like that. We'll look at both custom evals that aren't benchmarking and also some benchmarks in our hands-on.

These are just some examples of what is typically considered a model evaluation versus a benchmarking specific test, just to kind of get a little bit of a sense. Um, but again, like, there's so many types of evaluations, so many tools.

You can customize it in so many different ways. But this kind of helps, hopefully, a little bit with the definitions.

So hopefully kind of seeing all these challenges with Gen AI, we understand that this is a critical process. Um, you need to manage risk for customers. Like, there's less of a concern,right, when I'm tinkering on my laptop, whatever.

You know, I see something weird, who cares? But when we're talking about a production-level environment, we're serving thousands, whatever, customers. We need to think about these things, obviously, more in these types of scenarios. The credibility of the company, when those stories come out, that takes a hit for a good chunk of time.

Um, and you obviously also need to continuously improve your evaluation frameworks as well, because you're not going to catch everything. You need that CI process to make sure you are continuously improving your evaluations and benchmark setups. It's also going to very much depend on the type of system that you have, what you set up.

So again, it's very much there's tons of tools. This could look a lot of different ways. We'll get a sense of it today. If you have a RAG setup, you're going to be maybe focused on, uh, the RAGAS, um, benchmark or evaluation tool.

Agents, you need to look at function tool calling capabilities, et cetera. You can kind of get a sense. There's going to be specific metrics that you need to set up and look for depending on the system that you have, which requires a lot of planning in advance and kind of architecture scoping.

I'll give an example of a RAG use case. And also, it's incremental too, because you could and I'll, I'll talk about this a bit. Like, you could literally evaluate every single part of things. But that's going to be, um, time and resource extensive to set up immediately.

So you likely want to take an incremental approach with these types of setups. So you might start out with, okay, I'm just going to evaluate the chunk retrieval, my retrieval application in a RAG system. I want to set up some kind of evaluation test there.

I might just want to set up a latency throughput eval, um, benchmark test for my LLM output. You can start with those kind of incremental approaches for specific components. And then from there, based on priority levels as well, branch out into a full system eval that covers all the components, the integration layer of how the components work together, the UI end-to-end experience, and kind of have a software engineering kind of test period approach to this.

Where you have that unit test layer kind of approach at the bottom, um, integration layer in the middle, and that UI end-to-end at the top. And you can take that layer by layer as you're building this evaluation framework for your systems.

So there we created this or we there is this pyramid also for model evaluation that represents the same kind of setup for, um, that, that software engineering pyramid, um, represents as well. So the base layer and, uh, very base layer is the system performance.

Because like I said, no matter how good your model is, if you don't have fast throughput, you aren't able to handle concurrent users, you're going to be in a bit of a pickle. GPU utilization, et cetera, you need to make sure kind of the basics are handled, uh, the as the main kind of event.

Um, and we'll talk about that'll be the first hands-on activity is, is evaluating the system performance. Formatting might be making sure it's, um, religiously giving you JSON output that you need for your application, something like that. The factual accuracy, which we'll also talk about, um, in one of our hands-on, that would be like the MMLU benchmark.

So evaluating that it's performing well on various subjects, kind of standard large language model accuracy, as well as if you've fine-tuned a model, potentially making sure that it's accurate based on the information that you've fine-tuned that model on.

And you kind of go up from there into, you know, safety, bias. There might be specific custom evaluations that are very specific to your application. So gives you kind of a sense of the tiered approach that can be taken here.

So we're going to talk first system performance, and we're going to have our first hands-on around this. We're going to be looking at GuideLLM, which is kind of a new project that's associated with the VLLM inference runtime project.

Um, we're going to use that for system performance benchmarks like latency throughput, and you'll get a little bit of hands-on there. The general user flow there is like you, you know, you select your model, you select your particular data set that you want to use to test throughput and to test inter-token latency, time to first token, those types of metrics.

And then GuideLLM allows gives you a nice kind of, um, in-terminal UI to visualize the results of that. Uh, and then once you get kind of the results that you want based on your use case, again, then you're ready to deploy.

We you'll see this in the hands-on as well, but you want to test based on the use case. And the pri one of the primary ways you test via, via GuideLLM is adjusting the input and output tokens. So if you have a chatbot use case, a RAG use case, you can adjust the, um, input and output token levels based on your use case.

And you'll see that in the hands-on and have an opportunity to kind of play around with that, depending on what you're most interested in. So we're going to start with the hands-on. That is the link. That red.ht/evals is the link to the workshop.

You will be signing in with your email. I have no marketing game. It just requires you to do that, so I'm not going to haunt you after this. Um, you'll put in your email, and that is the password.

And

me, hold on. I just want to show you well, I don't want to take up one of the systems. Um, otherwise I would show you. But you're going to get instru once you are in the workshop, you're going to get your instructions on the left-hand side, and you're going to have two terminals avail two terminal sessions available to you on theright-hand side to the same system.

It's a RHEL system. Um, the instructions will overview what the system includes a little bit for you, so you get a sense. Um, they each have an L4 GPU as an example. Um, anything else I want to call out before we get started?

That'll be primarily what you use. It has TMUX enabled, if you like to use that to open up different things and gives you some flexibility. We have three different activities. I'm going to pause after each activity so we can have a little bit of a discussion in between.

Um, we'll kind of get so if my first time running this particular, uh, activity. So we'll gauge kind of the time it takes, but I'm going to give it about 15, 20 minutes for this first one. Okay. So I'm also pulling up a system, and I'm just going to, like, walk through some of the stuff.

The initial page is just going to give you of course, if it loads. Jesus, the internet. Um, the initial page is just going to give you a little bit of background and preparing your system instructions. And then my terminals are on this second tab.

And everything's going to be glacial pace. Um, so the first thing I have to do, these systems don't have the container toolkit installed, so I just got to do that. Some, uh, system logistics, because I'm going to be running VLLM inference runtime in a container, and I need that to work.

So that's what that's doing. And then I'm going to deploy a model with VLLM. I so you're going to have to grab a Hugging Face token. Probably have gotten there by now. Probably most of us have a Hugging Face token, but just disclaimer there.

So incognito window, if you're hitting the Hugging Face rate limit to grab a new token.

So VLLM, you can also install locally, like if you have a Mac or whatever, um, Linux machine, but we're deploying it as a container here.

So I'm going to get that deployed. And I'm, um you can see the VLLM serve command at the end. So that's just the VLLM CLI tool. And I'm using an IBM Granite model because, you know, Red Hat, IBM.

Um, so we'll be working with that for a chunk of the activity. And it takes a bit for VLLM to load the model. So you're going to be waiting for info the words info, like four times in green, and then it's deployed.

What's nice about VLLM is that it is, um, it's compatible with the safe Tensor formats. So I don't know, has anybody used TRT to load up a model? Okay. Anyway, it's crazy. Um, because it requires you to also convert the model formats initially.

Um, so there's less kind of configuration steps with VLLM. Takes up less space too. So when you do these kinds of system and performance, um, benchmarks, you can make a lot of adjustments, like the, um, the input and output tokens that I mentioned, um, for the GuideLLM configurations.

But there's a lot of also configuration opportunities for the inference runtime itself, um, depending on what you're trying to do. So sometimes we'll reduce the max, um, the context window of the model so it runs more quickly. Because it's if it's a big context window, it's going to be pretty beefy.

There's a lot of knobs you can use for VLLM. We're not really going to touch that, um, this particular time, but just so you're aware. So I have my, my three in green infos. So that means it's working, and the model is successfully deployed.

So I'm going to, um, get into my virtual environment, which is already in place. And then I already have GuideLLM installed, but I'm going to pip install GuideLLM. And these are copy buttons, by the way, so you can just easily copy-paste things over.

So once I have that up, then this command is set up to just work with the, um, model deployed by VLLM and that I'm just keeping up here in the top terminal. You can run this in the background, but I'm just not doing that.

And this is so I have my target. Um, the rate type is a sweep of, uh, various benchmarks, like inter-token latency. There's various types of benchmarks that it'll run that you'll see in the output. Um, but you these are all things that can be adjusted.

You can run one particular benchmark at a time, for instance. You can take a look at I think it's, uh, guidelm---help typical type of commands. You can kind of see what the where all the knobs are. And the documentation is pretty good too.

So that'll take a couple of minutes to run because I have it, um, set at a rate of five to reduce the amount of time that it takes to process.

So you can kind of get a sense of the output here once it all processes. And I have explainers on the left-hand side of kind of how to read some of this. Um, the mean performance for each so we have the, the benchmark info on the top and then benchmark stats on the bottom.

Um, so the for the constant rate, the on the very left-hand side. So those are the number of requests sent to the model per second at that particular rate. So three, the constant at 3.63, 6.93. If I did written, um, the GuideLLM command, rate, and it was five.

If I did rate 10, you would see more lines of that at, at, um, more progressive rates.

And, like, whether or not these numbers are good also totally depend on your use case as well. And I would be comparing I would have a better hardware configuration, obviously, in production as well. Because again, I'm j I'm running an 8 billion parameter size or two, two, hmm, 2 billion parameter size model, um, on an L4, which is, like, okay.

But obviously, if you're doing anything concurrently and at scale, like, that's going to go bonkers pretty quickly.

So you get, like, the mean performance, the median performance, and p99, which is, like, the, the extreme level, um, which matters for SLOs and things like that. And you can also output this into, um, JSON format as well to take a closer look.

So once you reach this, you can try also with an additional, like, tweak the parameters and then maybe compare the results and kind of see what, what that changed. If you do a RAG setup, um, so I think we had I forget what we had, what the initial command said.

Um, but you can adjust those based on a different use case and kind of compare and contrast what the stats look like after. Because it just takes a couple minutes to run.

Who's done with this first exercise? Okay. Good. Allright. We're a few minutes away from the additional, um, 10 to 15 systems being ready, so.

Just for the sake of time, I am not going to do breaks for kind of discussion in between, if everybody's okay with that. And we can kind of just converse independently, and I'll just awkwardly walk around the room.

Um, so if you're done with activity one, move on to activity two. Because there are three activities, and I want to make sure everybody has appropriate time for each that they would like. Um, but we will also have the systems up until probably about noon.

I don't know, early afternoon. Um, we'll keep them up all if you wanted to also go back and look at it after. So is that good with everybody? We just kind of steamroll, power through. Okay.

Yes.

So I have a new URL where we have three more available so far, but others are provisioning. And I just wanted to go ahead and put the URL up. It'll come they'll appear here. And it's the same password,right?

You said? It's the same password. It's the same password you said. Okay.

Okay. So that's the new URL. Three more systems, but they'll more will appear.

Um, I put descriptions on the left, um, to explain that, but I just wanted to heads up. We're kind of moving from system performance to now that kind of factual accuracy part of the pyramid. Um, so you get a sense of so we're going to be doing the MMLU Pro, um, in the second activity.

And then the third activity is going to be focused on safety and bias and more custom evals. So that's the trajectory of activities. Feel free, if one is also more interesting than the other, feel free to skip around.

Totally fine. Explain more about the MMLU Pro. I'm curious about, you know, like, proprietary data sets, if that's the thing. You can cut you can basically customize everything because every like, all these things are open source. So you can create a similar type of eval in that multiple choice format that MMLU does with your own data set.

Um, there's different ways to do custom accuracy evals with your fine-tuned data. So we kind of do it in a we have so as part of one of the products, we, we incorporate an eval for, um, for our fine-tuned models on your proprietary data.

And we do, like, a branch of MMLU, essentially. So there's kind of it's a lot of ways to skin a cat, even though I love cats, um, in regards to how to set up the evals and a lot of tools available.

Yeah. So yeah, yes.

You can just, like, fork it and ch and change the data sources. Yeah, yeah.

Put that link back up.

The instructions don't some of the instructions don't look updated as I expected. So I'm going what I'm going to do, um, is everybody in the Slack for AI Engineer World Fair? I created a, um, Slack channel called Work Workshop Beyond Benchmarks.

Um, and I'm going to put

some of the so activity three. Does anybody see activity three? Okay. I'm putting content in this, um, in this Slack. If people can the it's a public channel. If people can go there, my, um, you'll see starting I'm the link is at activity two, but then you'll also see at the page for activity three.

We're, um, approaching activity three and are looking for that. The systems, for some reason, didn't render my latest changes yesterday where I improved on activity three. But I put the link to it to my repo in the Slack channel in this channel if you search for it.

But there's information about the Slack in the emails that we got about the event, but also on the back of our badges as well to navigate there. And this'll be good for after, if anybody has any questions. And I can send more info about any particular tool here as well.

I wanted to have a wrap-up moment because we have we have about eight minutes. So please feel free to continue working. I put the link to the activities in the Slack channel. These environments will be available, um, until the end of the day today.

So you have time also to tinker around with whatever you want. Um, so you would just to kind of recap, we went through the first

sounds funny. Um, the first was at that system performance, latency, throughput level. Did everybody kind of get through that successfully? Of course, there's I tried to include reading material and stuff to kind of look more into things after.

Because it is it is a very big topic, and there's a lot going on, and there's a lot of terms, and it is very complicated. Um, so hopefully you can use it as a learning res my GitHub repository as a learning resource to kind of poke around after.

But we started there, and then we moved into the MMLU Pro with mLevalHarness, um, which also allows you to do a lot of other evaluation, um, benchmarks as a part of that mLevalHarness framework. Um, I happened to choose MMLU Pro because it took the least amount of time, even though it took still 10 minutes.

Um, but there's other ones also that you can play around with, um, within that mLevalHarness repository. You can see the different, um, evals you can run there. And then we ended with a safety evaluation with prompt which with prompt foo, which, um, is a tool that allows you to do a lot of customizing, um, and do your own evals.

Like, you can do all kinds of custom tests with prompt foo. So I wanted to get you exposed to that tool so that you can start looking around there as well. That repository on GitHub also has a lot of different examples.

So we used that particular, um, safety-focused example. But if you look at the prompt foo repository, it's very easy to play around with other types of examples as well. Um, so we kind of moved up the pyramid throughout the activity.

So hopefully you get a sense of kind of how you can layer this approach when you're looking at and trying to plan for how to strategically implement evals across your entire system. Um, does anybody have any kind of questions or general what they experienced, notes of import?

Does this feel, like, valuable? I'm curious also about use cases and happy to talk to you after too. Yeah. So, um, I'm, like, super new to evals and stuff. Mm-hmm. Uh, so, like, when I'm doing evals, it's, like, uh, testing my prompts and, like, my data sets.

Or if I want to switch, like, models out and stuff. Right now, I'm actively thinking about, like, those evals, connecting those actually to, like, my production, like, running, like, use cases to, like, track that, like, my real performance matches kind of, like, the driving yeah.

What is, like is there, like, a word for that concept or. Um, so for me, what I hear is the kind of CI/CD automation implementation of an evaluation framework, just like with software engineering testing, is kind of what I hear from that.

Um, you know? Yeah. So I don't know. It's, it's evaluation, but in a more CI/CD format. Like, when it's actually running for, like, custom work and stuff. Yeah. You should have a CI/CD framework that includes these evaluation tests, just like for unit testing setups.

Gotcha. Yeah.

Anybody else?

Okay. Thank you, everybody. I really appreciate it. Of course, feel free to thank you.

Feel free to, um, continue to use the repo. Message me on Slack. Again, the environments will be up until about 6:00 p.m. 5:00 p.m. tonight. So thank you, everybody.