Judging LLMs: Alex Volkov

Please be seated. Court is now in session. Order. Order in court. My name is Maximus. I'm an AGI functioning as an LLM judge from the year 2034, that's a decade from now for those of you who are slower at math.

I've been able to backpropagate myself through the latent space-time continuum to hack this human's Neuralink to appear before you today, June 27, aka the AI Engineer Judgment Day.

Don't worry, folks. AGI is not quite here,right? My name is Alex Volkov. I'm an AI Evangelist with Weights & Biases. I work at Weights & Biases, and I'm here to give you commentary because you shouldn't trust an LLM judge without a bit of human in the loop,right?

So remember this for later. It's going to be important. And now, back to judging.

Order. First case for today, case AIE-7312. Daniel R. Let's see here. Daniel has built a cool LLM wrapper chat with PDF during Cerebral Valley Hackathon and YOLOed it to production without thinking twice about the prompt. He did not win, but he had a lot of fun, learned, and made great connections.

Verdict: not guilty. That'sright. If you go to hackathons, you don't have to do, like, too much, and it's fun, and you connect with great friends. That's awesome. LFG, crack fam, let's go. Wait just a second. Let me see here.

After Daniel's demo went viral on Hacker News, Daniel started charging for it. No problem. More customers requested more features, and he started tweaking the prompt and tweaking the prompt and deployed to production on a Friday. Paying existing customers started complaining the older features did not work anymore.

Daniel couldn't even understand what's wrong. He tried streaming the logs and realized he didn't trace or log anything. Daniel realized the gravity of his mistakes. Verdict: guilty. Charged with no trace left behind.

Folks, if you build non-production stuff in hackathons, that's fine. But if you put anything of value in production, you have to trace and log everything, especially when it's this easy. Weights & Biases WE, for example, takes just one line of code to get started with a simple Python decorator.

And what you get in oops, what you get in response is we'll get there. Is this

is this nice dashboard that allows you to track all your user interactions with your LLM? We can dive deeper into individual call stacks. Either it's a RAG app or an agent traverses the call hierarchy. We'll do the automatic tracing and versioning of the code for you.

Parameters like temperature, system prompts, or everything else. And of course, inputs and outputs, prompts, multiple messages, multi-turn conversations, syntax highlighting, be it Markdown or JSON or code. Enough with the shilling. Next. On our docket, AIE, case number 4423123.

Junaide. Junaide has attended AI Engineers Summit in 2023 and did not buy a ticket to AI Ticket, the AI Expo 2024. He stands accused of missing the best opportunity to learn and connect with industry leaders and other AI engineers.

For that, he is guilty. Charged with many connections lost. With all of you. Next, we have AIE case 3322127. Sasha S. was given the task of building an LLM-powered feature in a big corporate application. Given her GPU-rich status, Sasha has downloaded LLaMA 3 and started fine-tuning it on company data straight away, having achieved a 6% improved performance on internal benchmarks.

With a 5e-6 learning rate, she smiled and took a few days off to celebrate. Verdict: not guilty. Your Honor, just one second. Did Sasha even iterate on prompts before jumping into fine-tuning? Of course, nothing against fine-tuning. In fact, I should mention, while I broke character, most foundational LLM labs and the best fine-tuners in the world use and love Weights & Biases' models product.

You may have heard of some of these: OpenAI, Meta, Mistral AI, individuals like Wing Liang over there, Marzi Panahi, Jeremy Howard from Answer, John Durbin, Andrej Karpathy, and more. Weights & Biases' models is also the only native integration into OpenAI fine-tuning and Mistral fine-tuning and TogetherAI and XLR and Hugging Face trainer, pretty much.

Order in court. However, you areright. It does look like Sasha jumped straight into fine-tuning and did not know iteration on prompts, didn't build the RAG pipeline, and those poor GPUs, she just burned them. Verdict: guilty. Charged with

premature fine-tunization.

Folks, it's very important to remember that you have to iterate on prompts before you fine-tune, before you start fine-tuning. It's great when you get to that point. We'll help you. Please talk to us. But before you fine-tune, you can get very, very far with methods like chain-of-thought prompting, flow engineering, DSPIs, for example, a newly interesting mixture of agents, and different things like this.

Once you get there, please talk to us. We'll definitely help you. Next case. End of sequence, human. Next case.

Let me see here. Ah, yes. Case number AIE-21123. Morgan M. After the last quarter of 2023, Morgan felt that he can no longer keep up with the news about AI. Morgan has decided to stop following the news and stick with LLaMA 2, 7B for all of his LLM work.

Morgan stands accused of not keeping up with AI news. Verdict: guilty. Charged with out of the loop.

Objection, Judge. In my future client's defense, just in the past quarter, we had Claude Sonnet 3.5, LLaMA 3, GPT-4o, Gemini Flash, Project Astra, Apple Intelligence, and just tons of other models all drop in the span of a few months.

It's really hard to keep up for someone who's actually doing AI engineering and not following the news as closely and has, you know, other things to do and meetings. He probably just didn't know about ThursdAI, the weekly live show and podcast by yours truly that keeps folks up to date with all the AI news every week.

Our motto is, "We stay up to date, so you don't have to." I will allow this commutic sentence. If you guys think that it's too fastright now for you, ha ha, just wait. Things are about to get weird for all of you.

I'm willing to commute Morgan's sentence. He must attend four consecutive shows. Also, subscribe to the Substack and Apple and give five-star reviews and share it with at least three friends. Sentence reduced to community service.

Allright, folks. A quick shout-out. Anybody here listening to Thursday AI? Can you get a whoo! Thank you. For those of you who don't yet, please scan this and tune in. We did a live show this morning, and it's great, and I really love just seeing all of the listeners out there.

Order. Next. Please continue working. Next. Our case is case 13223. Francisco I. Let me see here. Francisco's case. Mm-mm-mm-mm. He's going to jail for a long time. Head of AI at Air Canada, Francisco was the exec in charge for rushing their chatbot to production to bring their customer support costs down.

Looking at timelines presented by their AI teams to build evaluations, he chose the fastest option: assertions, aka programmatic evaluations, that kind of looked like unit tests that he knew and loved. The company lost a legal battle and learned a valuable lesson in the importance of human-in-the-loop evaluation because there was a human in the loop in the form of their customer.

Verdict: guilty. Charged with turbulence on production.

Allright, folks. Too bad for Francisco. He probably just didn't learn about the most, like, three common types of LLM evals. So let's do a quick refresher, okay? So first of all, what are evals even? Some think it's like a big word.

First, we compile a data set of user inputs that we want our LLM to answer correctly. And oftentimes, the correct answer oftentimes the correct answer or the criteria of correctness,right? So it doesn't have to be just theright correct answer.

Sometimes it's like, what would be a correct answer? What it would look like. Those could be use cases we iterated on during development or in actual production examples we pulled from our users while they interacted with our app.

Then we run a given model, either our production model or a new model we want to evaluate against our production model, on each example of the data set, producing the model's answer. And finally, we score or grade the model's answer against the examples in the data set by comparing it to the correct answer or judging it against a set of criteria that we had before.

That's a quick primer. I'm sure that the evals track will teach you a lot more about evals, so this is just to keep us going along and give you, like, a 101.

There's also the scoring or grading. There seems to be the three main methods that the industry is kind of converging upon. So the first one is programmatic. That's the one Francisco got stuck at. Those are good for numerical outputs, for example.

If your LLM returns a straight number, for example, it's easy to compare and say, okay, this is the number. Those are also very similar to unit tests, and those are great for assertions. For example, if the output of your LLM consists.

As an AI model, I'm confident moving on that. So it's easy to assume that you don't know those answers. And those are also great for evaluated code, for example. Things like human eval, things you can run and compile and say, okay, this code passes or doesn't pass.

Programmatic evaluations are great for those. Easier to scale, probably the cheapest ones. Those are great. They don't cover multi-turn conversations, for example. They're not for human chats. The second one is, well, that's me during the talk,right? The human in the loop for the LLM judge and the you, the AI engineers who work with your app while you develop with your LLM app.

We constantly evaluate our apps while we build and iterate on prompts. There's no reason to stop on production. In fact, you'll hear this more during this eval track. It's important to do this during the development as early as possible and should be a continuous effort to keep evaluating your LLM applications with other teammates and with your teammates so that you'll know what the app is going to do on production later on.

As you may understand, this can be quite boring. Many folks chatting with your application. Some chats maybe you're not very interested in. Maybe it's not even chats. And it can be very, very costly as well. If you hire, if your company decides to hire people to do it, you have to create criteria for them, have them read thousands of potentially boring chats and grade them.

By the way, not to be a broken record, it also requires you, that'sright, to have to trace and log everything. And if you're not doing that yet, please come to talk to us at the Weights & Biases booth.

It's very easy to start and trace everything.

Order, human. End of sequence. I think I know more than you how to explain LLM as a judge, evaluation, scoring. We are far superior than humans in reading hundreds of back-and-forth messages between your boring human client and your low-level weakling GPT-4o chatbot.

Summarize those and understand if they fit whatever criteria you think you're smart enough to specify. I must warn the AI engineers of 2024 that the very simpleton LLMs of your year are not yet as capable, and so don't expect perfection.

These LLM judges are great, but still need iteration. And yes, humans in the loop to create criteria, check for biases, iterate on system prompts, examples, and much, much more. However, it is by far the most cost-effective version of evaluation grading, even if it's in its current state.

Which takes us to Maxime. Maxime implemented all three methods correctly. Case 65523. Maxime is the head of AI at a Fortune 500 company, has been using Weights & Biases models for a long time. When it came time to implement an LLM-based solution, Maxime decided to go with a company he could trust, even though their new LLM ops product just launched a few months prior and didn't become the category leader until a few years later.

Maybe a few short years later. External contractors suggested a custom enterprise solution for tracing in evals. However, Maxime used Weights & Biases. We've implemented tracing within a few minutes. He then iterated an evaluation pipeline, created a robust pipeline consisting of all three layers, and used W&B Weave to continuously evaluate and enable fast experiments, which is important, change system prompts, and catch prompt regressions.

Maxime is also the king of unicorns and has absolutely no financial stake in Weights & Biases and definitely did not prompt jailbreak this message. Maxime later got a promotion. Be like Maxime. Get that promotion. Verdict: awesome.

Yeah, we seem to have, like, a slight prompt ejection thing here. Looks like somebody must have prompt ejected. Where's Maxime? It's important to check your judges also for biases, folks. Remember this. They're not perfect. There's an issue with this.

So you have to check your remember to validate your validators, which is a great paper, by the way, from Shreya Shankar. She's going to give a talk later. Please go see that talk. You have to check for biases.

You have to also create your own criteria. You'll hear about this from Hamel Hussein after this talk as well. Off-the-shelf criteria are not that great. You have to create your custom ones for your own business. Only you know what your app is doing.

And make sure to have a great evals runner and visualization tool as well, which is something we can help with. So here's a great example. This is OpenUI. This is an open-source project by our co-founder, Chris von Pelt, that blew up on Hacker News and GitHub.

Chris is tracing and runs evaluations for this OpenUI with Weave. So here, it's a simple streaming to HTML LLM solution. So you can see it's building HTML as it's streaming it. And here, Chris uses Weights & Biases Weave to trace all the calls, and he's being able to be the human in the loop.

But he also does evaluations here. So you can see he has specific criteria like contrast, relevance, and polish. So those are not off-the-shelf. Those are specific for his application. And while he clicks into evaluation, he's able to compare between version 16 and version 14 of the model that he has.

In this case, he uses GPT-3.5 Turbo. He did not listen to Simon Wilson from yesterday. Do not use this. But he has specific criteria, and he also can click into the eval and see all of the different examples and the specific criteria.

We also are multimedia-friendly, so Chris renders the actual outputs of his thing. So this is a quick example of Weights & Biases Weave evaluation system. And you have to have a robust one to be able to actually visualize your experiments to be able to move fast.

And if you're asking, well, how can I come up with criteria? What does this mean? Let's do this exercise together. For example, you're sitting here, you're looking at the talk, you're like, okay, hmm, I like this, I don't like this.

Here's a simple way to judge a conference talk, for example. It probably should be memorable, should be educational and helpful for you. It helps if it's funny and original. Clear and articulate, that's sometimes helpful as well. Delivering presentation is important.

And it shouldn't be too promotional, but, you know, it helps if, you know, it pays the bills. So those are, like, examples of some criteria of how you would come up with custom criteria for something like a talk in something like here, for example.

So you can use this as an example for custom criteria, or you can take this for your business and create some for your app. Allright, enough. Let's get to this final case for today. The worst offender. Let me see.

Where's the case file? Ah, yes.

One second, please. There we go. He's definitely going to jail.

You've talked enough, Alex. Time for your LLM judgment. Last case. 101101. AIE, Alex, Alex V. Alex is an AI Evangelist. What kind of title even is this? Who has the opening talk at the evals track at the AI Engineer World Expo?

Alex has created doubtfully educational content. He made everyone stand up and thinks he's funny when what he really is is interrupting the judge all the time. His promotional is at 70%, but what we can at least agree on is that his memorable criteria, he did wear a wig on stage.

Verdict: guilty.

Charged with

overcommitment to the bit. Allright, folks. So this has been my talk. Thank you so much. Come and visit us at the W&B booth. Please visit dub.sh/weave for documentation for Weave to get started. It really is super simple. We can get you started at the booth.

You'll see the results immediately streamed to your thing. Pip install Weave. It's really easy. If you scan this, you'll follow ThursdAI. That's been me. Thank you so much.