How to run Evals at Scale: Thinking beyond Accuracy or Similarity — Muktesh Mishra, Adobe

Hey everyone, um, hope you're having a great conference. So I'm going to talk about how to run Evals at scale, and thinking beyond accuracy or similarity. So in the last presentation we learned about, like, how to architect the AI applications, and then why Evals are important.

In this presentation I'm going to talk about, like, the importance of Evals as well as what type of Evals we have to choose when we're crafting an application. This is a bit about me, so I work as a lead engineer for Applied AI for developer platforms at Adobe.

I have also co-authored the CI/CD design patterns book, and also involved in a lot of open-source work across the communities. So let's get started. So how many of you have seen this or are active on the Twitterright now?

Like, have you seen these kind of patterns emerging? I think this morning there was a talk where this snapshot was, again, surfaced. So one of the most important trends in AI application development is Evals, because without Evals we can't craft any AI application.

Then, how many of you are developing an AI application, be it a RAG, chatbot, agents, anything? So if you're working on that, you often have come across these kind of questions, like how do I test applications when outputs are non-deterministic and require subjective judgment.

Because we all know in LLM world you can have the different output for the same set of input. LLMs are non-deterministic. Or how many times you wondering, like, if I'm changing a prompt, what is going to break, or how am I going to test that.

And then, most importantly, when you're developing an application, in order to measure the performance or accuracy, you need to find out what tools to use, what metrics to use, or what models are best, because models are getting capable day by day.

And the answer is Evals. So Evals is the fundamental approach where you are writing sort of test cases to measure your AI applications. And why do we why do they matter? Because without measuring something, it can have various impacts.

It can impact your business, you need to measure whatever system output is being produced, how do you align your application with system goals. Or one of the important aspect is how do you keep getting better, because applications are you are developing applications day by day, and you need to make sure it is getting better.

And then trust and accountability, this is one of the aspects which is very important because whenever you are developing something for a customer, you need to make sure they trust your application, whatever output is being generated. Now, when we talk about Evals, one of the important aspect to focus on is data.

So when we think about Evals, when we think about the tests, how do we start. So the very first step is starting with the data. Now, how do you get the data. So there are couple of approaches to get the data.

One is you start small, and you start with the synthetic data. Synthetic data means you can generate the you can generate the ancillary data, you can generate the artificial data, and start validating your applications output against that data.

Then it's Eval when you think about the data in Evals, it's a continuous improvement process. It means every time you generate some output, you need to observe the system, and then you need to keep on defining that data set, whatever data set you are procuring.

And another aspect is you need to label your data accordingly, so because data is fundamental to writing Evals, so you need to when generating the data you need to define your data set in a way where it is labeled into different aspects, it is covering multiple flows or application prospects, so things like that.

And then you need to continuously refine that. Another approach which I have learned from my experience is you one data set is never sufficient. So when you are thinking about Eval, you need to think about multiple data sets based on the flows, based on the applications, and whatever you are trying to achieve.

Now, when we think about evaluation, what do we think about. So what do you want to evaluate. The answer is everything, but what does that mean. So you need to define start by defining your goals and objectives, what do you want to evaluate in your system.

Then you need to design in a way where you have modules defined for each of the components. You need to optimize your data handling. And I noticed I'm mentioning data again and again, but the point is you need to have different data sets for different flows.

You need to test your flows, outputs, and paths. So if your application involves multiple flows, multiple path, you need to evaluate in all paths. Now, adaptive Evals. So one of the previous presentation talked about, like, there is no universal Eval.

And that is, again, most important thing because your Evals depend upon what you what type of application you want to evaluate. For example, evaluating a RAG application, a typical RAG application, is different from code generation. If you are dealing with a RAG, typical Q&A type of application, you can define your Evals such as accuracy, or similarity, or usefulness.

Versus when you are generating a code, you want to test the generated code against a actual code base. So that is where you need to measure your functional correctness of the code generated, or how robust that code is generated.

Then, when you are trying to evaluate agents. So one of the important aspect for evaluating agents is trajectory evaluation, because agents can take a different path, and oftentimes you need to define which path they are taking in order to execute a flow.

There is also multi-turn simulation where most of these agents are complex, and you need to check, like, when you are having a conversation, like, how do you evaluate that. Then if you are doing the tool call, then you also need to check the correctness, or test suite, or, like, how they are how the data is being generated.

Now, another aspect is how do you scale Eval. So one strategy is you can cache the intermediate results and regression. You need to focus on orchestration and parallelism, like how you are running your Evals, how you are orchestrating them, how you are parallelizing them.

You need to aggregate the results, and then you need the important aspect here is you need to run them frequently and then improve upon. So one of the term which is being used in industry is measure, monitor, analyze, and repeat.

So you need to often measure it, you need to analyze it, and iterate on that. Then you need to strategize what you want to measure. So again, depending upon the use case, there are different type of matrices or different type of methodologies you need to adapt to.

And then, again, use there is no fixed strategy to run your Eval, so use what fits best. In some cases you want your humans in the loop to be taking precedence, in some cases you have

automation test automation Evals running in. There is a fine balance or tradeoff between human in the loop versus automation, like whether you want the high speed versus high fidelity. So again, depending upon what you want to achieve, you want to take give a fine balance on that.

And rely on process over tools. Reason is because tools, again, you cannot automate everything. So you need to define and establish the process how do you want to run the Evals.

So these are some of the key takeaways we just talked about. So one is Evals are the most important aspect for AI application. There is a term being coined now, Eval-driven development, which is if you think about typical software like test-driven development, this is the Eval-driven development.

Define Evals based on the use cases. You need to focus on positive as well as negative cases. Then focus on the data. That is, I cannot emphasize enough on that. And then remember to measure, monitor, analyze, iterate in a loop continuously.

And always take a balanced approach in fidelity versus speed. If you have any questions, there is a barcode, you can come later and chat with me. Happy to chat more, and that's all from now.