Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen

Hi, this is Niklas. I'm the CTO and co-founder of Log10, and we want to talk about how you can scale the liability of LLM applications using a new tool that we've built. During this year, I think we all can agree that there's been this kind of craze in the industry, and we've been rolling out a ton of intelligence features based on GPT.

And we're now kind of finding ourselves in a "now what" moment, because without knowing what "good" means in a generative setting, it's usually hard and risky to evolve your applications, like changing your prompts, configurations, let alone considering going from one model provider to another to more advanced use cases like self-hosting or fine-tuning.

We want to introduce a new tool today called LLM eval that enables teams to ship reliable LLM products.

It is a command-line tool that you can run locally, and with these 4 lines of code, you should be good to go.

The initialization creates a folder structure and best practices for storing prompts and tests, and then this is based on a super configurable system from Meta called Hydra, so you can basically extend it to your heart's desire. And the metrics that we have wired up are in Python, so they could be any logic, could be called out to all the LLMs, whatever you want.

And after these evaluations have been run, you can generate some reports that basically give you a brief overview of how the entire app and all the tests are looking, but still support flexible test criteria. Because these models are very fuzzy, it's very hard to say with a guarantee that it's going to be one or the other, but it's fairly safe to say that the majority cases, or say 3 out of 5, should pass.

And we're going to jump into command line and taking a look. We're just going to create a directory for today

and go into this directory and create ourselves a virtual environment.

From here, we're going to install llm eval

and initialize the folder structure. What we should be able to see here is

a directory structure where we have our prompts. Let's say a simple case could be this, where we have this message template saying, "What is A plus B?" Only return the answer without any explanation. So in this case, we know that we have to prompt-engineer further in order to get an exact output, because let's take a look at how the test looks like.

In this case, we're taking the actual output from the LLM and comparing it with the expected, and this is a strict comparison. What we had taken the liberty to do is to strip any spaces that might come from the left, and that's because some models, in this case Claude, tends to preprint spaces, and so it's things like that that you have to watch out for.

Then we have the metric, which could be any metric that you want to surface in the report, and then the result, which is then a pass or fail. And in this case, we want to add 4 and 5, and we expect it to be 9.

And I'm just going to try to run this test here and try to revert some of the prompt engineering that we did earlier, so I'm going to remove

only return the answer without any explanation.

And the way you get started is the llm eval run, but if you want to override anything, if you just do llm eval run, it runs everything. But if you do prompts equals math, then it's only going to run the math example.

If you do n tries 1, then it's just going to do 1 sample. By default, we do 5 samples, so we get a better read on the stability of each test, but it might be too much for you.

But you can override anything. You can find these default settings here in the llm eval YAML, but let's try to run this and see what happens. And so this ran across Claude, GPT-4, and GPT-3.5 once, so we can go in and generate a report.

And see, actually something failed. What was it that failed? Let's take a look at the output here. And in this case, because we removed our prompt engineering, GPT-3.5 starts being a bit chatty and says like 4.5 equals 9.

Claude does something similar, it kind of writes out the equation. And now I'm going to try to revert and say, let's get this in,

and we try to run one more time.

Great. Now when we generate the report, it could say some tests failed, but the most recent tests that ran passed. So when you do the report, it's going to generate a summary. It's going to generate a report per run, but then also say overall, was there anything that failed out of these reports.

If you want to go a bit more advanced, let's say you want to use tools. We have an example here where we are generating some Python code, and again, we had to add a number of different clauses to make sure that it only outputs Python.

It tends to be very happy generating surrounding explanations. So in this case, we are going to see whether or not

it returns an actual Python program that could be parsed. So let's try to run that. If we go in and take a look at this report, you can see that these tests actually end up passing our tool use.

And to round up, we have model-based evaluation as well, where you can test using other models. And so in this case, say with grading, we can go in and define a full set of criteria. Here we're evaluating mermaid diagrams, giving a score between 1 and 5, and the reason, and that is also supported in llm eval.

One thing about the previous approach is that it takes quite an amount of work to set up these tests and gather your test cases. And one really compelling answer to evaluation has been model-based evaluation, and it's a setting where you have, typically, a larger model discriminate or kind of grade or be a judge over the output from another LLM.

And that makes it so you can get more nuanced output, like pass/fail, or a grade from 1 to 5, or preferences between different options, and it's reasoning behind it. There's a number of pitfalls, unfortunately, around this approach, around biases towards the output from the model itself.

If you're streaming different models, they tend to prefer their own output. They're not very good at giving point scores, saying, I think, between 0 and 1, or larger scores between 0 and 100. But there are different ways where you can start increasing the accuracy of the kind of feedback that's being generated.

And we've been working on this, where you basically start bridging between model-based and human feedback. So instead of removing the human completely from the feedback, you start taking in all feedback that might have been given prior and start modeling it and say, if you have all the feedback from John, then we create an auto-John that will start generating feedback for review for any incoming completions.

And so in this case here, we have two pieces of feedback that's been already given by a human. See here, it was overall just a score of 5, or here it was a bit more nuanced. But here we are kind of pending feedback.

And if we click this, we have AI suggested an answer to this.

And that's all I had today. If you want to get started on llm eval, we have our documentation at our usual documentation site, and you can find me at niklas.krafford on X, or formerly known as Twitter, or shoot me an email at nick@log10.io.

Thank you.