Ensure AI Agents Work: Evaluation Frameworks for Scaling Success — Aparna Dhinkaran, CEO Arize

Well, thank you so much for being here. I'm going to start off by saying—apologize my voice a little bit, it's a little hoarse today—but you guys are going to hang in there with me. Today we're going to talk about a really important topic, which is, um, slideshow mode.

Awesome. We're going to talk about a really important topic, which is about evaluating AI agents and assistants. This will load. Just to set a little context before we jump in: a lot of you have probably heard today about different agents that are being built, how to build it, what are the cool tools out there to go build agents.

And today we're going to actually talk about when you put those agents into production, it's important to actually know how they're doing and evaluate them. It's super important to making sure that they actually work in the real world.

We're probably going to get a little technical in this talk, maybe a little bit more than some of the other talks, but hang in there. I think this is important even at the leadership level to understand how to make sure what you're putting out actually works in the real world.

So, a little bit about me. My name is Aparna. I'm one of the founders of Arize. Uh, fun update on us, actually: today we announced our series series. So, um, we have a lot of folks who are using us to evaluate agents.

So with that, let's jump in. Okay, well, everyone here has probably talked to you about text-based agents. So you have this chatbot, whatever, it's making an action, and it's figuring out all these things to do. The cool next frontier is actually voice.

AI is already taking over call centers. There are over 1 billion calls made in call centers all around the world with voice assistant aids, with voice APIs, and the real-time voice API. If any of you guys have played around with it, we're actually already seeing these types of, um, agents start to take over and revolutionize call centers.

This is actually a real production application of a travel agent. This is the priceline penny bot. You can go in and actually, hands-free, no text, book an entire vacation using priceline penny today. So we're not just talking about text-based agents anymore, we're talking about multimodal agents.

And it's important to address these because the way that you evaluate these types of agents, it's not just evaluate an agent, but also if it's on voice, there's specific types of evaluations you're going to need to do. If it's multimodal, there's additional types of evaluations you need to consider.

So we're going to break all that down, and hang in there with me for a fun one today. So before I jump in and talk about how to evaluate an agent, let's talk about what are the components of an agent.

You probably have heard different versions of this today, but I'll tell you the language we're going to use. One, um, there's something typically called a router, which is essentially what's deciding what the next step an agent will take.

There's skills, which is the actual logical chains that do the work. And then there's something that stores the memory. This is important because

there might be different architectures of how you're seeing people build these agents out there. It doesn't matter if you're using LangGraph or CrewAI or LlamaIndex workflows, there's all sorts of agent frameworks. They all have slightly different ways of building an agent.

You might not even use a framework. But what you're going to see is these common patterns of, okay, that's a router, that's a skill, and that's a memory. And these different components are going to have different ways of how you actually evaluate it.

So let's first talk about the first one. What the heck's a router? So you can think about a router almost like the boss. It's kind of deciding, hey, well, it's very common to have e-commerce agents in, you probably are all talking to e-commerce agents today to purchase things.

Amazon has one. All these e-commerce companies have one. When you type in a question, like, I want to make a return, give me an idea of what to go buy, are there any discounts on this, that user query funnels into something called a router.

And that router's goal is to, is really to determine, do I call this skill about hitting up a customer service agent, do I call this skill, um, to suggest all the discounts we have or suggest products. The router's really kind of the boss deciding who do I tap on to go actually execute the, the ask that the user made.

And the router might not always get itright, but you want it to get itright because then it goes down the pathway of a specific skill within an agent. So in this case, it will call a skill. Um, so if I asked, hey, uh, tell me the best, um, I don't know, leggings to go buy.

So it'll go in, it'll do a product search, and then this is actually the entire skill flow of execution that the agent needs to go through to execute, you know, whatever the user asked for. Some of these might be LLM calls, some of these might just be API calls.

It just really depends on how people actually implement them. And then lastly, this is an important piece, is there's always something storing the memory. Because these are usually not just single-turn conversations, they're multi-turn conversations, multi-turn interactions. And so you don't want to be talking to an agent that forgets what you previously said.

So there's really memory, which is storing what it previously asked for and keeping all of this in some sort of, um, in, in some, some sort of semblance of state. So with that, we're going to get a little fun here.

I'm going to show you, um, an actual example of what this could all look like. A router, skills, and memory. So this is an open-source project, um, that actually looks at the inner workings of an agent. These are called traces, for folks who may not be familiar.

If you're, you know, in leadership or you're, you're, this is really what your engineers are looking at when they're actually building and troubleshooting your agent. They're actually understanding what the heck went on under the scenes. So this is actually an example of a code-based agent.

Somebody asked a question like, what trends do you see in my trace latency, aka what's making my application slow. This is the router call that we were talking about earlier, where it actually decides, well, how do I then go ask, you know, how do I then go tackle that question.

So first, you can see here there's multiple router calls. There's not just one router call. This is pretty common as your application grows. You can have multiple times where it comes back and has to decide, what do I need to go do.

So the first time it calls the router, what it does is it actually, so the router then makes a tool call, um, which is essentially the skill that you need. The first time it actually makes a tool call to then go run a SQL query.

Go collect all of my traces of my application and go, go run a SQL query. Then it goes back up to the router, and then it calls the second skill, which is actually the data analyzer skill, which takes all of the traces and the application data, and then it passes it to something that actually analyzes that data.

So in this case, you can actually see there was a router, there was tool calls, we actually have memory that's actually storing everything that's happening under the scenes. And so really just shows all three of the different components that I actually just walked through.

So now that we have an example of a, of an agent with a router and skills and memory, let's talk about how to actually evaluate these agents. Every single step that I just walked through here actually is an area where the agent can go wrong.

For routers, typically what teams end up caring about is, did it call theright skill. Because if it didn't call theright skill, you know, user asks for, I asked for leggings, but then it sent me over to customer service.

Or it sent me over to, um, you know, something about discounts and deals. So you actually want to make sure that the router within an agent is correctly doing theright skill and calling theright skill. So that's the first piece that you'll want to make sure that your teams are evaluating.

So if your teams are building agents, you want to ask, well, hey, what's the ultimate control flow, what's the control flow, and are, do we have something like a router and are we evaluating it to make sure that it's correctly calling theright skill between A, B, C, and is it calling theright skill with theright parameters.

So not just, um, calling product search, but actually making sure that whatever way you've designed that skill, you're actually passing in the correct things like, um, you know, I want this type of material, I want this type of whatever cost range.

You're actually passing in all theright parameters into what the user actually is, is asking for. Can I get a raise of hands? Have any of you guys heard of, did any of you guys evaluate your agents today, actually?

Is that something you know your teams are doing? Okay, awesome. Are any of you guys evaluating this router level internally? Okay. Awesome. Wow, this is a great group. Okay. This is impressive. Um, okay, let's next go to the next one, which is actually evaluating a skill.

This is actually the part where it gets really interesting and tricky because there's many different components in a skill. There might be, in this case, I have a rag type of skill, so I want to look at things like evaluating the actual relevance of the chunks that were pulled.

I want to look at the actual correctness of the answer that was generated. But this skill itself can have many different LLM as a judge evals, or it can also have code-based evals that you might want to run to actually evaluate the skills, the skills of the agent.

And then lastly, this is kind of a really important one that we're seeing teams probably have the most trouble evaluating, which is actually the path that the agent took. Because, well, ideally you want it to converge. You call the same skill hundreds of times, and it always takes about five steps or six steps to actually query what the user asked for, put in theright parameters, call X, Y, Z components of the skill, and then ultimately, um, take theright, you know, generate theright answer.

But sometimes this can be a little longer. We've seen sometimes where the same skill, I, I don't know if you all have done this experiment, but you can put the same skill and build it with OpenAI, and you can also build it with Anthropic, and sometimes they have wildly different number of steps that the path actually takes.

And so the goal here is, how do you be succinct, and how do you also make sure there's reliability in the number of steps that your agent takes to actually consistently complete a task. So we call this convergence, um, but probably one of the hardest to actually evaluate.

Is anyone evaluating convergence today, or at least counting the number of steps? Awesome. Okay. You're awesome, dude.

Cool. Well, with that, I'm going to go maybe two more minutes, then I'll hop into one more demo here. So if any of you guys have watched the movie Her, this is from Her. Um, uh, this is where, you know, the, the main character asks like, who else are you talking to, and, you know, the Samantha says something like, 8,000 other people are in a conversation with meright now.

And so the future of voice applications is that these are probably some of the most complex type of applications that have ever been deployed, ever been built. It's going to require one more additional pieces to actually evaluate voice applications.

And the interesting part about these is that it's not just the text that needs to be evaluated, or the transcript, but it's also the audio chunk that needs to be evaluated. Um, in a lot of these voice assistant APIs, you have the generated transcript that happens actually after the audio chunk is really sent.

And so that's a whole nother dimension around, is the user, what's the user's sentiment. Is the speech-to-text transcription actually okay? Is the tone consistent throughout the entire conversation. And so you actually need to evaluate not just the audio piece and the flow of the conversation and everything else you're doing for all your other, you know, other parts of your agent, but also make sure that the audio chunks are getting their own evals defined on, um, you know, intent or speech quality or speech-to-text accuracy.

Um, so this is important for voice. So with that, um, I'm going to actually show you guys how we evaluate our own agent so that you can get a little bit of a example of, of what some agent in the wild actually does.

Um, this is our own agent. So let me actually show you what it looks like. Um, you can actually go on our product today, and there's a little Copilot. And our Copilot does something similar to what other Copilots do, where as people are spending time in our product, we actually help them do things like, hey, help me debug this, help me summarize this, help me look at this.

Um, can I search with natural language. So there's kind of this Copilot integrated throughout our entire product. But we're an evals company. So what do we do? We actually dogfood our own tool, and we decide to, what you're looking at here is actually the traces of our entire Copilot actually in the wild.

And every single step of this Copilot, we actually run evaluations of. So in this case, we have an eval at the very top, actually evaluating something around, was the overall response that was generated, this was actually a search question, is the overall search question actually correct or incorrect.

And then we also have one around, once it actually called the search router, did it pick theright router, and then did it pass in the correct arguments into the router. And then finally, ultimately, did it complete the task or the skill correctly in the execution of this, this entire skill.

And so evals aren't just at one layer of your entire trace. If you take anything away from this conversation, the goal here is really, how do you make sure that you have evals throughout your application so that when something goes wrong, I can debug if it actually happened at the router level, if it happened at the skill level, or if it happened somewhere else along the flow.

Um, and I think that's it from me. Any questions?