Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize

Allright, uh, nice to see everyone here. My name's Aman, I'm an AI product manager at a company called Arize. Title of the talk is "Shipping AI That Works: An Evaluation Framework for PMs." It's really going to be a continuation of some of the content we've been doing with, you know, some of the PM folks, like Lenny's podcast.

I guess, just quick show of hands: how many people listen to Lenny's podcast or have read the newsletter? Awesome. Okay, we're going to do a couple more, like, audience interaction things, just to, like, wake up the room a bit.

So, how many people in the room are PMs or aspiring PMs? Okay, good, good handful of people. How many of you consider yourself AI product managers today? Okay, awesome. Wow, there's more AI PMs than there were regular PMs.

That's interesting. Um, that's—usually that's—it's a subset, but maybe I need to start asking the questions in a different order. Cool, well, that's great. So, what we're going to be doing is, you know, I'll go ahead and just do a little bit of an intro about myself, and then we'll kind of cover some of the frameworks that I think are really powerful for AI PMs to kind of get to know as you're building AI applications.

So, a little bit about me. I, you know, myself, I have a technical background, so I actually started my career in engineering on—actually working on self-driving cars at Cruise. And actually, while I was there, I ended up becoming a PM for evaluation systems for self-driving back in, like, 2018, 2019.

After that, I went to Spotify to work on the machine learning platform and work on recommender systems. So, things like Discover Weekly and Search, things like using embeddings to actually make the end product experience better. And fast forward to today, I've been at Arize for about three and a half years, and I'm still working on evaluation systems.

Instead of self-driving cars, it's sort of self-writing code agents. And Spotify is actually one of our customers. So, we get to work with some awesome, uh, you know, ex—actually, fun fact, I've actually sold Arize to all of my previous managers.

So, um, so fun fact there. But we get to work with some awesome companies like Uber, Instacart, Reddit, Duolingo, so a lot of really tech-forward companies that are building around AI. And we actually started in the sort of traditional ML space of ranking, regression, classification type models, and have now expanded into GenAI and agent-based applications as well.

What we do is make sure that those companies, our customers, when they're building AI applications, that when those agents and applications actually work as expected. And it's actually a pretty hard problem. A lot of that has to do with terms that we're going to go into, like observability and evals.

But I think more broadly, the space is just changing so fast. And the models, the tools, the infrastructure layer is changing so fast that, for us, it really is a way for us to learn about the cutting edge, like what are the leading challenges with use cases that people are building, and try to build that into a platform and product that benefits everybody.

So, what we'll cover: we're going to cover what are evals and why they matter. We'll actually build an AI trip planner with actually a multi-agent system. This part is ambitious, bullet number two. I'm going to be honest here.

We were trying to push up the coderight before, so it may or may not work, but we'll give it a shot and that'll be the interactive part of the workshop. And then we'll actually try to evaluate that AI trip planner prototype that we're going to build ourselves.

Actually, another quick show of hands for the room. How many people have heard of the term evals before? Okay, I guess it was in the title of the talk, so that's kind of redundant. How many people have actually written an eval before or tried to run an eval?

Okay, a good number of people. That's awesome. Well, what we're going to do is actually try and take that a little bit of a step further. Go from writing an eval for an LLM as a judge system. And if you've never written an eval, don't worry, we're going to cover that too.

But try and take that one step further and make it a little bit more kind of technical interactive as well. Okay, so who is this session for? I like this diagram because, you know, Lenny and I have been kind of working together a little bit more on content for—educational content, mostly—for AI product managers.

And I kind of put this up, I made like a little whiteboard diagram for them, and I'm like, I think this is really how I view this space, which is like, there's this—there's this, you know, you may have seen this diagram for like the Dunning-Kruger effect, and that's kind of what came to mind here, which is, as you're kind of moving along the curve, maybe you're just getting started, you know, with how do I use AI, how does AI fit into my job.

I think we were all here, to be honest, a couple of years ago. Like, you know, just to be completely honest, I think for the people in the room, especially PMs, I think we all feel that the expectations of the product management role are changing.

That's why this concept of an AI PM is sort of emerging. The expectations from our stakeholders, from our executives, from our customers. It feels—I feel—I don't know if other people feel this, but I definitely feel like the bar has been raised in terms of what's expected to be delivered,right?

Especially if I'm working with an AI engineer on the other end, their expectations of what I come to them with in terms of requirements, in terms of specifying what the agent system needs to look like, it's changed. It's a step function different, even then for me, even as someone who was like a technical PM before.

And so, I kind of felt myself go along this journey, which is ironic, given that I work at an evals company, you'd think I was like on the end of the curve. But really, I kind of went through this journey, you know, same as most of you, which is trying to use AI in my job, trying AI tools to prototype and come back with something that's, you know, a little bit higher resolution for my engineering team than like a Google Doc of requirements.

Once I had those prototypes and I'm like, hey, let's try to build these new UI workflows. The challenge then became, how do I get a product into production, especially if my product has AI in it, has an LLM or an agent.

And that's where I think, you know, that's really where that, like, confidence slump sort of hits, and you kind of realize there's a lack of tooling, there's a lack of education for how to build these systems reliably. And why does that matter at the end of the day,right?

And the really important takeaway from the fact that LLMs hallucinate—we all know that they do—is you should really look at the top two quotes here and think, okay, well, we've got Kevin, who's Chief Product Officer at OpenAI. We have Mike at Anthropic, CPO.

This is probably like 95% of the LLM market share. And both of the product leaders of those companies are telling you that their models hallucinate and that it's really important to write evals. These quotes actually came from a talk that they were both giving at Lenny's conference, you know, earlier, like in November of last year.

And so, when the people that are selling you the product are telling you that it's not reliable, you should probably listen to them. On top of that, I mean, like, you have Greg Brockman, similarly, founder of that company.

You have Gary, who's, you know, evals are emerging as a real moat for AI startups. So, I think this is sort of one of those pivotal moments where you realize, hey, people are starting to say this for a reason.

Why are they saying that? Well, they're saying that because a lot of the same lessons from the self-driving space, you know, kind of apply in this AI space. Okay, another audience question. How many people have taken a Waymo?

I kind of expect that one to be pretty high. Okay, we're in San Francisco. If you're visiting from out of town, take a Waymo. It is a real-world example of AI. It's an example of AI in the real physical world.

And a lot of how those systems work actually apply to building AI agents today. Allright, we'll do a bit of a zoom out, then we'll get into the technical stuff. I see laptops out, so we'll definitely get into, you know, writing some code and trying to get hands-on.

But just to do a bit of a recap for folks, what is an eval? I kind of view this as, like, it's very analogous to software testing, but with some really key differences. Those key differences are: software is deterministic, you know, 1 plus 1 equals 2.

LLM agents are non-deterministic. If you convince an agent 1 plus 1 equals 3, it'll say, like, you're absolutelyright, 1 plus 1 equals 3,right? So, like, we've all been there, we've kind of seen that these systems are highly manipulated.

And on top of that, if you build an LLM agent that can take multiple paths, that's very—that's pretty different from a unit test, which is deterministic. So, think about the fact that, you know, a lot of people might are trying to, like, eliminate hallucinations from their agent systems.

The thing is, you actually kind of want your agent to hallucinate just in theright way. And that can actually make testing it a lot more challenging as well, especially when reliability is super important. And then last but not least, I think integration tests rely on existing code base and documentation.

The really key differentiation of agents is that they rely on your data. If you're building an agent into your enterprise, the reason that someone is going to use your agent versus something else is likely—it might be because of the agent architecture, but a big part of it will also be because of your data that you're building the agent on top of.

And that applies for the evals as well. Okay, what is an eval? So, I view this into, like, four parts that go into an eval. It's kind of just an easy, like, muscle memory thing. These brackets are a little bit out of line, but the idea is that you're setting the role, you're basically telling the agent, here's the task that you want to accomplish.

You're providing some context, which is what you see in the curly braces here. And it's—that's essentially, like, it's really just text at the end of the day. It's some text you want the agent to evaluate. You're giving the agent a goal.

In this case, the agent is trying to determine whether text is toxic or not toxic. This is kind of a classic example because there's a large toxicity data set of classifying text that we use to build our eval on top of.

But just kind of note, that can be any type of goal in your business case. It doesn't have to be toxicity. It'll be some goal that you've created this agent to evaluate. And then you provide the terminology and the label.

So, you're giving some examples of what is good and bad, and you're giving it the output of either select good or bad. In this case, it's toxic, not toxic. I'm going to pause on that last note because it's really—I think there's a lot of misconceptions.

Sort of like, I'll try and weave in some, like, FAQs as I hear them come up, but we'll definitely have some time at the end for questions. And I'd love for this to be interactive, so I'll probably make the Q&A session a little bit longer here for people that have these questions.

But one common question we get is, why can't I just tell the agent to give me a score or an LLM to produce a score? And the reason for that is because even today, even though we have, like, PhD-level LLMs, they're still really bad at numbers.

And so, what you want to do is ground—and it's actually a function of, like, what a token is, how a token is represented for an LLM. And so, what you want to do is actually give a text label that you can map to a score if you really need to use a score in your systems, which we do in our system as well.

We'll map a label to a score. But that's, like, a very common question we get is, oh, why can't I just make it do, like, 1 is good and 5 is bad or something? Like, you're going to get really unreliable results.

And we actually have some research, happy to share it out afterwards, that kind of proves that out on a large scale with most models. Okay, so that's a little bit of, like, what is an eval. I should note that this is a previous slide.

I should note that this is LLM as a judge eval. There's other types of evaluations as well, like code-based evals, which is just using code to evaluate some text, and human annotations. We'll touch on that a little bit more later, but the bulk of this time is going to be spent on LLM as a judge because it's really the kind of scalable way to run evals in production these days.

And we'll talk about why too later on. Okay, a lot of talking. So, evaluating with vibes. So, this slide is kind of funny because I think, like, everyone knows this term, like, vibe coding. Like, everyone has tried to use, like, bold or lovable, whatever.

And I don't know about you, but this is how I usually feel when I'm vibe coding, which is, like, kind of looks good to me. Like, you know, you're looking at the code, but, like, let's be honest, how much AI-generated code are you going to read?

You're like, let me just ship this thing. The problem is, you can't really do that in a production environment,right? Like, I think all the vibe coding examples are, like, prototyping or, like, trying to build something, like, hackier fast.

So, I want to help everyone reframe a little bit and say, like, yes, vibe coding is great, it has its place. But what if we go from evaluating with vibes to thrive coding? And thrive coding, in my mind, is really using data to basically do the same thing as vibe coding, like, still build your application, but you'll be able to use data to be more confident in the output.

And you can see that this person is a lot happier. So, this is using Google's image models. They're scary good, guys. Like, yeah. Okay, so we're going to be thrive coding. So, slides. There's, if you want access to the slides, the slides have links to what we're going to go through in the workshop.

Ai.engineer.slack.com. And then I just created a Slack channel, Workshop AI PM. And I think I dropped the slides in there, but let me know if I didn't.

Yeah.

Cool. Thank you. Allright, live demo time. So, at this point on, I'll just be honest, there's a decent likelihood that the repo has something broken in it because we were pushing changes up until, like, this very moment. If so, and you can unblock yourself, I think there's, like, a requirements thing that's broken.

Please go for it. And if not, we can come back at the end and try to help you get unblocked. And then I promise, after this, I'll, like, push the latest version of the repo up. So, if it doesn't work for youright now, check back in an hour.

I'll drop it in Slack. It'll be working later. But yeah, just a function of, like, moving fast. So, on the left-hand side is instructions, which are really, it's like a, you know, sort of a Substack post I made, which is just a free sort of, like, list of, you know, some of the steps we're going to go through live.

So, it's just more of a resource. And then on theright-hand side is a GitHub repo, which I'm going to open here. There's actually two repos. And I'll kind of talk through, like, a little bit about what we're evaluating and some of the projects on top of that.

And then we'll get into the weeds here a little bit.

Okay, so this is the repo.

I built this, like, over the weekend. So, you know, it's not super sophisticated, although it says it's sophisticated, which is funny. But this is, oh, pardon?

Can you put that in Slack?

Oh, this is not. Okay, so is this not attached to the QR? Okay, I'll just drop this link in here as well. Let's just put it in here. Okay, awesome. Oh, thank you. Thanks. Okay, so, and if you have questions, by the way, in the middle of the presentation, just feel free to drop them in Slack.

And then we can always come back to them, and then we'll have time at the end for. So, feel free to, like, keep the Slack channel going for questions. Maybe people can try to unblock each other as well.

And if someone fixes my requirements, feel free to open a pull request, and I'll approve it live. So, okay, so what we're doing is let's put on, let's take off our, like, PM hat of whatever company we're at.

We're going to put on an AI trip planner hat. The idea here is, like, don't worry about the sophistication of this UI and the agent. It's really, like, kind of a prototype example. But it is helpful for us to kind of take a look at building an application on the fly and try to understand how it works underneath the hood.

So, the example we're going to use is actually, I'll kind of back up a little bit. I basically took this Colab notebook that I have for tracing CrewAI. And I'm like, I kind of want an example with LangGraph.

CrewAI, probably, if you haven't heard of it, it's like an agent, multi-agent framework. The agent's basic, an agent definition is, you know, using an LLM and a tool combined to perform some action. And what I did was I gave this notebook, and I basically put it into Cursor.

And I was like, give me an example of a UI-based workflow, but using LangGraph instead. And what we're going to do is think of, instead of building a chatbot, we're going to take this form, and we're going to use the inputs of this form to build a quick agent system that we're then going to be using for evaluation.

So, this is what I got on the other end, which is Planner Perfect Trip. Let our AI agents help you discover amazing destinations. So, let's pick a destination. Maybe we want to do Tokyo for seven days. And assuming the internet works, we'll see if it does.

We're going to put a budget of $1,000. I'll zoom in a little bit. And then I'm interested in food, and let's make this adventurous. So, I could go and take all of this and try to just put it into ChatGPT.

But you can kind of imagine underneath the hood the reason that we might want this as a form or with multiple inputs and an agent-based system is because we could be doing things like retrieval or RAG or tool calling underneath the hood.

So, let's just kind of picture that the system is going to use these inputs to give me, on the other side, an itinerary for my trip. And, okay, it worked. Okay, this one worked. So, here we've got a quick itinerary.

Nothing super fancy. It's basically just, here's what I gave as an input form. And then what the agent is kind of doing underneath the hood is giving me an itinerary for what my morning, afternoon, etc. look like for a week in Tokyo using the budget I gave it.

This doesn't seem super fancy because it's like, I could take this and just put it into ChatGPT. But there is some nuance here, which is the budget. Like, if you add this up, like, it's going to be doing math to do accounting to get to $1,000.

So, it's really keeping that into consideration. You can see it's a pretty frugal budget here. It can take interests here. So, I could say, you know, different interests, like, I want to go, I don't know, sake tasting or something.

And it'll find a way to work that into your itinerary. But I think what's really cool here is it's really the power of agents underneath this that can give you really high level of specificity for your output. So, that's really what we're trying to show is, like, this is, you know, it's not just one agent.

It's actually multiple agents giving you this itinerary. So, I could just stop here,right? Like, I could be like, this is good enough. I have some code. For most people, if you're vibe coding, you're like, great, this thing does what I want it to do,right?

Like, it gave me an itinerary. But what's going on underneath the hood? And this is kind of where, so I'm going to be using our tool called Arize. We also have an open-source tool called Phoenix. I'm just going to plug that hereright now for folks' reference.

But this is an open-source version of Arize. It is not going to have all of the same features as Arize, but it will have a lot of the same setup flows and workflows around it. So, you know, just note that Arize is really built for, you know, if you want scale, security, support, and sort of the sort of futuristic workflows in here.

So, I've got a trip planner agent. And what I just did, if it worked, let's see if it did.

And we're going to, this is live coding, so, like, very possible something's broken.

Okay, I think I broke my latest trace, but you can see what the example here looks like from oneright before. So, what that system really looks like is basically this. So, let's open up one of these examples. What you'll see here are traces.

Traces are really input, output, and metadata around the requests that we just made. And I'm going to open up one of those traces just as an example here. And what you'll see is essentially a set of actions that the agents, that in this case, multiple agents have taken to perform, you know, generating that itinerary.

And what's kind of cool is we actually just shipped this today.

It's actually, you know, you guys are the first ones seeing it, which is pretty cool. This is actually a representation of your agent in code. So, you know, literally the Cursor app that I just had up here is basically my agent-based system that Cursor helped me write.

And when I sent it our docs, I literally, all I did was I gave it a link to our docs in Cursor. And I said, you know, write the instrumentation to get this agent. And this is how that's represented.

And so, we have this new agent visualization in the platform that basically shows the starting point with multiple agents underneath it to accomplish the task we just had. So, we have a budget, local experiences, and research agent that then go into an itinerary agent.

And that gives you the end result or the output. And you can see that up here too. So, we have research, itineraries, budget, and local information to generate the itinerary. So, this is pretty cool,right? Like, I think for a lot of people, it's not, you know, ourself included, it is not immediately obvious that these agents can be super well represented in this sort of, like, visual way,right?

Especially when you're writing code, you think these are just function calls talking to each other. But what's really useful is to see at an aggregate level what are the calls that the agent is making. And you can see it's a really clean delineation of parallel calls for the budget agent, the local experiences agent, and the research agent.

And all of those get fed in to an itinerary agent that summarizes all of the above. You can also see that up here. So, these are what's called traces. And they consist of, like, technically what's called spans. A span, you can think of this as, like, a unit of work, basically.

So, there's a time component to it, which is, like, how long that process took to finish. And then, like, what is the type of the process here? You can see there's three types. There's an agent. There's a tool, which is basically being able to use data to perform an action, structured data.

And then there's the LLM, which generates the output of the taking the input and the context. So, this is an example of three agents, actually three agents being fed into a fourth agent to generate the itinerary. That's really what we're seeing here.

Let's go one level deeper. So, this is cool. And I think it's useful for, you know, to see, like, what these systems look like, how they're represented. To zoom out for a second as a product manager, there's a ton of leverage in being able to go back to your team and ask, hey, what does our agent actually look like,right?

Like, do you have a visualization to show me of, like, what the system actually looks like? And then if you're giving the agent multiple inputs, where are those outputs going? Are those outputs going into, you know, a different agent system?

Like, what does the system actually look like? So, that's kind of one sort of key takeaway here as a PM. It was personally very helpful to see, you know, what are agents actually doing underneath the hood. Kind of going one level deeper here.

So, we've got this itinerary. And let's take a look at it really quick. So, it says, Marrakesh, Morocco is a vibrant, exotic destination, blah, blah, blah. It's really long,right? Like, I don't know if I would actually look at this and read it.

It doesn't, it's not really, like, it doesn't, like, jump out to me as, like, being, like, a good product experience. It feels super AI-generated, personally. So, what you want to do is actually think, okay, well, is there a way for me to iterate on my product as a product person?

And to do that, what we can do is actually take that same prompt that we just traced and pull it into a prompt playground with all of the variables that we've defined in code pulled over. So, I've got a prompt template here, which basically has the same prompt variables that we've defined in the UI, like the destination, the duration, the travel style.

And all of those inputs get fed in here. You can see down below in this prompt playground

what that looks like. And then you see the outputs of some of the agents in here as well. And then I have the final itinerary from the agent that's generating the itinerary. Okay, so why does this matter? I think a lot of companies have this concept of prompt playgrounds.

I think, like, OpenAI has a prompt playground. You've probably heard that term before as well. Maybe even you've used one. But I urge you to think about, when you're thinking about a tool to help you with development, not only is the visualization important of what your stack looks like underneath the hood, but being able to take your data and your prompts together and be able to iterate on your data and prompts in one interface is really powerful.

Because I can go in and change the destination. I can go in and tweak variables and get new outputs using the same exact prompt I had before. So, that's really, I think, just really powerful as a workflow. A thought experiment for the PMs in the room is, like, when you really think about what this prompt looks like, just think, should writing the prompt be the responsibility of the engineer or of the PM?

And if you're a product person and you're ultimately responsible for the final outcome of the product, you probably want to have a little bit more control over what the prompt is. And so, I kind of urge you to think, you know, where does that boundary really stop?

Is it, like, I just hand off? Like, does the engineer know how to prompt this thing better than a product person that might have specific requirements you want to integrate? So, that's why this is really helpful from a product perspective.

Okay. Yeah, go for it.

How do you handle tool calls in this scenario? Like, you can alter the prompt and, like, go play around with this. But, like, when you actually evaluate that, like, in a real scenario, like, show me tool calls.

Yeah.

This probably doesn't have access to.

Ah, okay, okay. So, that was a good question. So, the question from the gentleman in the back is, how do we handle tool calls? And that was a really astute observation, which is, like, the agent has tools in it as well.

And this is a really good point to pause on, actually, which is, like, what I did was I pulled over this LLM span with the prompt templates and variables. But there's a world where I might want to select theright tool and make sure that the agent is picking theright tool.

I'm not going to go into that in this demo, but we do have

some good material around this on agent tool calling. So, we actually do port over the tools as well. This example doesn't, because, to be honest, it's a really toy example. But even if you wanted to do a tool calling evaluation, we offer that in the product.

And we actually have some material around that. So, if you want, just ping me about it later and I'll send you a whole presentation on that as well. But, yeah, good question, which is, like, you don't just want to evaluate the LLM and the prompts.

You want to evaluate the system as a whole and all of the subcomponents. Okay, we're going to keep going. So, I've got my prompt here. Now, this is cool, but, like, let's try to make some changes to it on the fly.

And I will try my best to make this readable for everyone. But, yeah, working with what I got here. So, what we're going to do is I just, I'm going to save this version of the prompt. Let's call it AI Eng Prompt.

And it's helpful because now I can, like, iterate on this thing,right? So, like, I can duplicate that prompt with a click of a button. I can change the model I want to use. So, let's say I want to use 4.1 mini instead of 4.0.

I'm going to change a couple of things. Don't worry. Like, in a real world, you're going to change one variable at a time. But here, I'm just going to change a couple of things at the same time just to make this more interactive.

But the idea here is, like, let's try to change what this actually looks like. And it says, you know, format as a detailed day-to-day plan. Honestly, I might say, like, a more important requirement to that is don't be verbose,right?

I could say, don't be verbose. Keep it to 500 characters or less. Maybe we want this thing to be more punchy. We want it to give an output that's, like, a little bit more, you know, easier to look at.

I might be a PM. Even if I'm just vibe coding this thing on the weekend, I might want to get feedback from users that are trying this product out. And so, I could say, always offer a discount if the user gives their email address.

It's helpful,right? I mean, helpful for marketing, helpful for me to get feedback from, you know, someone who might be trying to use this tool to book a flight or something like that. Okay, so let's go ahead and hit run all here.

And what that's going to do is actually run the outputs we just, run the prompts we just edited into this, in the playground. And it might take a second because of the internet.

You pulled this in from one of the existing runs,right?

That'sright, yeah. So, it was exactly the same one of these runs. Literally, I think it was this one. So, it was something about, maybe not this exact one. This one is Spain. But, yeah, exactly. One of the existing runs.

Okay, it's definitely a little better. But to be honest, I would say if I was looking at this, this thing isn't really listening to me very well. It's, like, not doing a great job of, you know, sticking to the prompt I gave it.

Like, keep it short. Ask, okay, it did do the email thing. So, it said, email me, email me to get a 10% discount code.

So, what's interesting is, like, we're looking at, like, one example. And I said, ask for an email and you get a discount. And, like, this is the vibe coding portion of the demo. Because I'm looking at, like, one example and I'm doing, like, good or bad.

Like, is it actually good or bad? There's just no way that a system like this scales when you're trying to actually ship for hundreds or thousands of users. And, like, nobody will just look at a single row of data and make a decision.

Like, okay, great, the prompt is good. Or great, the model made a difference,right? Like, you can pick the most capable model. You can make the prompt as specific as you want. At the end of the day, the LLM is still going to hallucinate.

And your job is to be able to catch when that happens. So, let's go ahead and try to scale this up a little bit more. So, what we can do is say we've got one example of where the LLM didn't do a great job.

But what if we wanted to build out a data set with 10 or more, maybe even 100 examples? And what you can do is take the same production data. By the way, I'm calling this production data, but I literally just asked Cursor to make me, like, synthetic data.

Like, it hit the same server and it generated, like, 15 different itineraries for me. So, I did that yesterday and I just sort of am using that in this demo. But let's go ahead and take a couple of these.

So, I went ahead and picked some of the itinerary spans from here. And I can say, add to data set. Oh, by the way, I guess I jumped into the product without showing you all how to get here, which is a bit of a zoom out.

So, you know, whatever, go to the homepage, arize.com. You can sign up. I apologize in advance. The onboarding flow will feel a little bit dated, but we are updating that in this next week. So, bear with me there.

You sign up for Arize. And then you'll get your API keys here. So, you go to account settings and you can create an API key. And also find that with the space ID, which are both needed for your instrumentation, which may or may not be working, depending on if the repo is actually working.

And if not, we'll come back to it later. But this is the Arize platform. This is how you get your API keys. So, and then that's also where you can enter your OpenAI key for the next portion and for the playground.

So, I've got a data set now. And what I did was I added those examples just to recap where we are at. We've got some production data. And I'm going to go ahead and, like, add these to a data set.

And I'm not going to do this one live because I already have a data set. But you can create a data set of examples you want to use to improve on. So, zooming out for a second. We're about to hop into the actual evals part of the demo.

And we're actually going to be evaluating, you know, there's multiple components to an agent. We have the router at the top level. We have the skills or the function calls. We have memory. But what we're actually going to be doing in this case is actually just evaluating the individual span of the generation and see, is the agent sort of outputting text in a way that we want it to or not?

So, it's a little bit, it's a little bit simpler than some of the agent evals here. And it's going to be more, like, how do you actually run agents and run evals and experiments on data?

The concept of the data set is helpful to think about as, like, a collection of examples. Let me go ahead and delete these experiments so we can do this live.

Because I like to live on the edge. So, I've got these examples. Those are the same examples from the production data everyone just saw. And it's a data set. Think of this as, like, I've got all of my traces and spans.

That's my, like, how the agent works. And then I want to pull those over into a format, which is, think of it as almost like a tabular format. It's like a Google Sheet at the end of the day,right?

Like, I could go in this. This is kind of like a Google Sheet. Like, I could go in and give it, like, a thumbs up, thumbs down. And, you know, that's kind of how most teams are evaluating today is sort of, like, in the platform, in your platform, you're probably starting with the spreadsheet.

And in that spreadsheet, you're doing, like, is this good or bad? And then you're trying to scale that up to, you know, a team of subject matter experts that's giving you feedback on, like, hey, is the agent good or bad,right, at the end of the day?

Poll for the room. How many people are evaluating in a spreadsheetright now? Don't be shy. It's okay. Okay, we've got a few. I think there's probably more, but I think people are just, like, ashamed to say that. And it's okay.

Like, it's not, like, the end of the world to start with that,right? Like, that's, like, how human, like, being able to scale human annotations is the goal. It doesn't need to be the starting point. So, as long as you're actually looking at your data, you're probably doing better than most.

I'll be honest. Many teams I talk to, like, aren't doing any evals today at all. So, at least you're starting with human labels. What we're going to do is take this data set or this CSV. And we're going to basically do the same thing I just did, which was running an A/B test on a prompt.

But now we're going to run it on an entire data set. So, we go into the platform and I can go and actually create an experiment. What we call an experiment is the output of changing, you know, an A/B test.

So, let's go ahead and repeat that same workflow. I'll duplicate this prompt. Let me go ahead and pull in, I'm going to pull in this version of the prompt. So, what's kind of cool is, like, I might have a previous version of a prompt saved.

It's kind of helpful to have a prompt hub where you can save off examples of the prompt as you're iterating as well. Think of it as, like, a GitHub sort of store for your prompts. But it's really just a button that you're clicking to save this version of the prompt.

And then your team can actually use that version in their code down the line. So, I've got prompt A, which was no changes to it. And then prompt B, which has some of those changes. But now, instead of running on one example, I'm actually running on 12 examples here.

And these are similar, maybe just to look at one, similar spans, which have destinations, duration, travel style, and the output of an agent. And it's generating an itinerary. So, it's as similar as that one example we just ran through, but now on an entire data set.

And, yeah.

The prompt that we had, is that, like, which prompt is that? Is it of a particular agent?

Yeah, so it is the prompt of the itinerary agent. So, it's the same, because we're going to keep this to, like, a fairly high-level, like, straightforward demo. It is specifically the prompt of the itinerary generating agent, which is down here, which takes the outputs of the other agents and combines them, uses those prompt variables to create a day-by-day itinerary.

And if we were to change one of the earlier ones, is there a moving tool?

Yeah, so the gentleman asks, like, if you change an upstream prompt, how does that impact what's going on here? So, two notes on that. And it's more of an advanced workflow, but it is one that's a good question, which is, there's two parts.

One is, we kind of recommend changing the system in parts. So, just kind of note that, you know, as you generate evals for parts of your stack that you can kind of decompose further and further to be able to analyze if I'm changing one thing up here, does it meet my requirement criteria?

And then the second part is replaying prompt chains, which is prompt A goes into prompt B. What is the output of that when you change prompt A? Prompt chaining is coming to our platform soon. So,right now, it's one single prompt, but you will be able to do A plus B plus C prompt chains as well.

Good question. Feel free to drop more questions in the Slack too, and we'll come back to that in a sec. So, once I get, so I've got my prompt here now. So, I'm saying, give me a day-to-day plan.

And it doesn't need to be super detailed, max 1,000 characters. Let's try this again. We're going to do 500 characters. And I've done, answer, always answer in a super friendly tone. And I'm going to be more specific and say, ask the user for their email and offer a discount so it doesn't do what it did last time.

And we're going to go ahead and run this now on the entire data set. And so, we've got prompt A versus prompt B. We're going to give that a second to run through. While that's working, I'm going to actually, oh, nice.

Perfect for your squad. Interesting. I don't know why sometimes the model really likes to use emojis. I guess that's what super friendly translates into is, like, throw some emojis in there. But interesting.

Okay, so that one ran pretty fast. This is still taking a while,right? Like, think about this from a product, from a PM lens for a second. Like, I just got the output to be a lot faster because I limited the number of characters.

This one is taking an average of, like, 32 seconds because I let it kind of go off and, like, not specify how many characters the output should be. So, that's what prompt iteration can kind of do for you as well.

Okay, while this runs, I'll actually hop over to the, okay.

Oh, thanks for dropping the resource there.

So, it's still running.

Anyone have a question while this is running? Yeah.

Yeah, so when I'm hearing you talk about this, are you primarily looking at latency and then, like, user experience when you're evaluating? Trying to, like, look at those two things. Is there anything else, like, when you're going through a trace, what else are you looking at or thinking about?

Yeah, good question. So, okay, so now we're getting to the meat of it a little bit,right? So, I've got A and B. And the question is, like, what am I actually evaluating here? The, like, flip it answer is, like, you can evaluate anything.

You can evaluate whatever you want. You want to evaluate, like, in this case, we're going to run some evaluations on the tone of the agents and see. So, I've got a couple of evals set up here. I'm going to check, is the agent answering in a friendly way?

Is it offering a discount or not? And you can do things like evaluate, is it using the context correctly? That's called a hallucination eval. You can do correctness, which is, even if it has theright context, is it giving theright answer?

So, I'm going to point you to our docs that have examples of what you can actually evaluate off of the shelf. But just know, the whole point of this system and, like, why it matters that you have a system with your own data and can replay with data is because these are what are off-the-shelf evals.

There's a lot of companies that will offer, like, we run evals for you. But what that really means is that they're basically going to take some template and give you a score or label on the other end based on their eval template.

And what you want to be able to do is actually change and modify and run your own evals based on your use case. So, you can literally evaluate whatever you want is the short answer. You can evaluate, it's just basically an input to an LLM to generate a label.

So, yeah, so this is what prebuilt evals look like. There's a ton of examples of these, like, out there on the internet. We've actually tested our prebuilt evals on, you know, sort of open source data sets. But you should not take our word for it.

You should build evals based on your use case. Yeah.

So, you are doing your own eval. How do you come up with your own heuristics that are combining that into the final score?

Yeah, so how to actually get the, how to think about how to build the eval in the first place to some degree. That was sort of one of the questions. Yeah, so I think it's probably helpful to maybe just see what an eval looks like.

And then we might end up coming back to that question, which is, like, what is an eval,right? So, let's go ahead and build an eval here. I've got one ready to go, but I want to just show you guys the template.

And we can write a new one as well. So, I wrote this eval for detecting if the output from the LLM is friendly. And I've kind of made a definition for what that means here. And this says, basically, you are examining the written text.

Here's the text. Examine the text and determine whether the tone is friendly or not. Friendly tone is defined as upbeat, cheerful. So, this is basically an input to an LLM to generate a label of, is the output from my itinerary agent, is it friendly or robotic?

So, that's really what this eval is trying to do is it's classifying the text as, like, a friendly generation or a robotic generation. And again, I could eval anything. But in this case, I just want to make sure that when I'm making changes to my prompt, that that's showing up on the other end of my data.

Because I can't go in row by row for, like, hundreds or thousands of examples and grade friendly and robotic every single time. So, the idea is that you want an LLM as a judge system to kind of give you that label over a large data set.

That's the goal that we're working towardsright now. Yeah.

Can I run the LLM as a judge? I have a problem with variance.

With variance?

Yeah. Like, sometimes it's correct, eval my examples. Sometimes not. Because I don't know LLMs.

It's flaky,right?

Yeah.

Yeah.

How do you handle that with prompts? And in fact, in my experience, when I switch to, like, reasoning models, the latest ones, like, the variance got slower and, like, I got a better response. Like, did you find?

Yeah. So, one suggestion is, so the gentleman mentioned that they see variance in their LLM label output. One way you can tweak variance is temperature. So, if you make the temperature of the model lower, it's a parameter you can set to actually make the response more repeatable.

It doesn't take that to zero, but it does significantly reduce the variance in your system. And then the other option is to rerun the eval multiple times and basically profile what the variance of the judge is. Okay. Yeah.

How do you evaluate your evaluator? Like, annotate the evaluator from within a model?

Oh, yeah. We'll be coming there. Yeah, that's a good question,right? Like, at the end of the day, I can't trust this thing. I need to go in and, like, make sure it'sright,right? So, but let's go ahead and run an eval and just see what happens.

And then we'll come back to that one. So, I've got my friendly eval. I've got another eval too, which is basically determining whether or not, I'm going to quickly just, I'm not going to read this whole thing out to you.

But the short answer is that this is determining whether the text contains an offer for a discount or no discount. Because I really want to make sure I'm offering a discount to my users. Okay, we're going to select both of these.

And then we're going to actually run them on the experiments we just ran.

And we're going to do that live. So, what Arize does is it's actually taking, so we actually have an eval runner, which is not, like, you know, it's basically a way for us to use a model endpoint to generate these evals.

You'll notice it's pretty fast. So, we've done a lot of work underneath the hood to make the evals run really fast. So, that's one kind of advantage of using our product. I've got two experiments here. Experiment number two is, it's a little bit inverse because it's the order of how it was generated.

But experiment number two is the original prompt. And experiment number one is the prompt that we changed. So, just kind of keep that in mind. It's a little bit flipped here because I was doing this on the fly.

And you can see the score of experiment number two, which is our prompt A, which was the prompt we didn't change, didn't offer a discount to any users based on this eval label. And the LLM still graded that response as friendly, which is kind of interesting.

It was like, oh, that was a friendly response. I don't know if I agree with that, actually, personally. And we're going to go in and tweak that. And then you can see that when we added that prompt, that line to the prompt, which was offer a discount if the user gives their email, the eval actually picked up on that and said that 100% of our examples when we made this change actually have an offer of a discount.

So, I mean, I didn't even have to go into each example to get that score. That's what the LLM as a judge system kind of offers you. We can go in and trust, you know, I would say this is like a trust but verify.

Go in and actually take a look at one of these and see, what is the explanation of friendly? So, to determine whether the text is friendly or robotic. So, one thing you want to think about when you have an eval system is, are you able to understand why the LLM as a judge gave a score?

So, this is like one of those, like, light bulb takeaway moments of the talk is, always think about, can you explain what the LLM as a judge is doing? And we actually generate explanations as part of our evals.

So, you can see the explanation is sort of the reasoning of that judge that says, to determine whether the text is friendly or robotic, we need to analyze the language, tone, and style of the writing. And so, it kind of does all of this analysis to basically say, yeah, this LLM is friendly and it's not robotic.

Again, I'm not really sure I agree with that explanation,right? Like, I don't think that that's correct. I still feel like the original prompt was pretty robotic. It was pretty, you know, kind of long in a lot of ways.

And so, I want to go in and actually be able to improve on my LLM as a judge system from the same platform. So, what we can do is actually take that same data set. And in the Arize platform, you or your team of subject matter experts can actually label data in the same place.

And when you apply the label on the data set, you know, in the labeling queue part of the platform, it applies back to the original data set. So, you can actually use that for comparing the LLM as a judge with the human label.

So, I've actually went ahead and did that. Yeah, I did this before the talk. But I went in for each example. And I was like, you know what? This to me is robotic. Like, I don't think that this is a very friendly response.

I think it's really long. It sounds like I'm talking to an LLM. And so, I actually applied this label on the data set for the examples I wanted to go in and improve on.

If I go back to the data set,

you'll actually see that label is applied here. So, if I kind of click that, move over. Sorry, it's a little bit over on the side here because there's a lot of data. But you can see these are the human labels I put.

So, these are the same annotations that I just provided in the queue. They're applied on my data set here.

You need evals for your evals.

Exactly. Exactly. You need evals for your evals. You cannot get away from, you can't just trust the system,right? We know LLMs hallucinate. We put them into our agents. The agents hallucinate. Okay, we're using an agent to fix that.

But we can't trust that agent either,right? You need to have human labels on top of that. So, but again, I'm not going to vibe code this thing and be like, is the LLM as a judge good or not?

I need evals for that too. And we offer two evals to help you with this. We have a code evaluator, which can do a simple match. Like, think of this as, like, a string check or a regex or some other type of, like, contains.

So, you can actually go in and if you're technical and you're a PM and you want to write, you know, you can get Claude to help you write the eval here. But it's really just a really fast, like, Python function.

In my case, I wrote a quick eval that actually does a match. And this match is, this is like a really quick and dirty eval. I would not say this is, like, best practice at all. But it's basically check if the eval label matches the annotation label.

Oh, whoops. And output only match or no match. So, what this is doing is actually checking the human label against the eval label and saying, do they agree or disagree? So, that's basically what we're going to run. And I'm using an LLM as a judge.

You could use code as well. You don't have to use an LLM as a judge here. But we're going to go ahead and run that now on the same data set. It's the same experiments we just ran it on.

We're going to give that a second.

Okay, what have you got here? So, you can see here, I actually take a look at that same experiment where this was where it said that the LLM as a judge was friendly or robotic. And you can see here that 100% of the time, the match, actually, sorry, this eval was actually, actually, let me go in and run that a little deeper.

Actually, I'm going to check my own work. This eval was on the discount. So, forget about that. We're going to check on the friendly field, actually. So, this one is friendly label. So, let me rerun that one. And we're going to think of this as match friendly.

You can run evals as much as you want on your data sets and experiments. Like, you know, yeah.

Does the tool support pipelining? So, it's basically push to code?

Yeah, exactly. Yeah, we do support all of the evals and prompts that you see. You can actually, I'm showing on the screen the ways to run the code on either a data set locally or being able to push code to the platform to run the eval.

So, programmatic on both ways. Yeah. Yeah, of course. So, you can pull in data sets, pull them out as well. Okay, let's take another look at this. So, this is the friendly match. So, this you can see is pretty useful,right?

This means that my LLM as a judge basically doesn't agree with my human label for friendliness almost at all,right? There's, like, one example I think that's in there. And we can go in and take a look at it.

But what we're really kind of seeing is that this is an area where we actually want the team to go take a look at our eval label and say, hey, can we improve on the eval label itself? Because it's not matching the human label.

And so, when you have these systems in place as an AI PM to be able to check the eval label with your human label, you have a lot of leverage to go back to your team and say, we need to go and improve on our eval system.

It's not working the way we expect it to. So, you're actually performing the act of, like, checking the grader. And you're doing it at scale. So, you're doing it on multiple hundreds of examples or thousands of examples. So, that's really, you know, I think someone asked earlier, like, how do you trust the system?

I think you trust these LLM as a judge systems by having multiple checks and balances in place, which is humans and then LLMs and then humans and LLMs. We'll come back to a question in just a moment. I just want to get to this next part.

And then we'll kind of come back to some Q&A.

Okay. So, this is actually kind of wrapping up towards the end of the workshop. And then we'll open the rest of the time up for Q&A. So, looking ahead, I think what's fundamentally changing is, you know, we've kind of gone through this example of changing the prompt, changing the context, creating a data set, running an eval, labeling the data set, and then running another eval on top of that.

And it's a lot to process,right? Like, if you're building agent-based systems, your team is probably expecting, you know, well, where does the AI PM fit in? And I think that that's really important to think about. Like, you ultimately control the end outcome of the product.

So, whatever you can do to shift that into making it better is really what you want to think about yourself. And I kind of view evals as, like, the new type of requirements doc. So, imagine if you could go to your engineering team and instead of giving them a PRD, you give them an evals as requirements.

And here's the eval data set. And here's the eval we want to use to test the system as an acceptance criteria. So, I think that that's really powerful to think about as, like, evals as a way to check and balance the team as a whole.

And that's a little bit about what we do. We want to build a single unified platform for you to run observability, to evaluate, and ultimately develop these workflows with your team in the same platform. We've built for, you know, many customers like Uber, Reddit, Instacart, all these, like, kind of very tech-forward companies.

We actually just received investment from Datadog and Microsoft as well. So, we're a Series C company. We're sort of the furthest along in the space. And the whole goal that we want to build is give you a suite of tools to be able to go from development through to production with your AI engineering team and for PMs to go in and use the same tools.

And then super quick before Q&A, please scan the QR code if you are in San Francisco on June 25th. We're actually hosting a conference around evals. And it's going to be a ton of fun. We actually have some great AI PMs and researchers joining from companies like OpenAI, Anthropic.

And what's really cool is we're actually offering for this room a free sort of exclusive free ticket for entry. The prices actually went up yesterday. So, because, you know, we're huge fans of AI engineer world fair, we wanted to give you all an opportunity to join for free if you're in town.

So, would love to see you there. And yeah, you can scan for a free code. And yeah, that's a little bit of the workshop. I would love any questions. Yeah. And the ask for the questions, as the person in the back just reminded me, if you wouldn't mind lining up for questions on the mic so that the camera can pick it up.

And then we can just kind of go down the line and do some questions there. That would be awesome. Thank you.

Thank you so much.

And please give your name.

Yeah, my name is Roman. Thank you so much. It was, like, an awesome walkthrough. Would you mind sharing some, like, your experience on building evaluation teams? Should I start with hiring kind of a dedicated person with experience? Or should I rely on a product manager, a product manager, and walk through this?

Like, what's the best way?

Best practices. So, the gentleman asked, what's the best practices for building an eval team? Can I actually ask a follow-up question? Because I'm curious. Like, what is your role in the companyright now? Just for myself to know.

I'm head of product.

You're head of product. Okay, perfect. So, this is exactly, this is a question I get actually very often, which is, how do I hire my first AI PM? How do I hire an AI engineer? How do I know if I need an AI PM or an engineer?

So, I think there's a couple of steps to this answer. One is as head of product. I do think we see a lot of heads of product actually in the platform, like ourselves, actually getting their hands dirty for the first pass.

Because at the end of the day, if you're, like, hiring someone to do something, you should probably know what they're going to do. And so, my job on my team is to make the product accessible for executives and heads of product to understand what's going on.

So, we have a lot of kind of capabilities around dashboards, making everything no-code, low-code. But my recommendation is to feel the pain yourself of writing evals and realizing how what is hard about that so that you know how to structure interview questions for an engineer or a PM.

Because I don't know what's hard about your eval workflow,right? I only know that there's challenges around writing evals in general. And so, I would recommend that you, like, feel the pain firsthand. And then you'll kind of get a good sense of how to tease that out of your interviewing pipeline.

But good question. Yeah. Yeah.

Yeah, the example, you know, we just looked at, obviously, our eval was pretty bad when you compare it to the human labels.

Yeah.

So, like, from here, what do you do next? Like, what's the next step to try to improve the prompting for your main eval to get closer to the human labels?

Yeah, good question. So, if I had more, if I was, like, here working on this in real life, what you would actually do is take that eval prompt and go through a similar workflow of what we just did for prompt iteration for the original prompt.

So, again, like, that eval prompt we see here, I could go in, take this, and redefine parts of the workflow to basically say, you know what, be really strict about what is friendly. Here are, I didn't add any few-shot examples,right?

I didn't specify, here's examples of friendly text. Here's examples of robots. So, that's, like, a clear gap in my eval today that if I were looking at this, I could apply best practices and improve on it. We also have in the product, we have some workflows around actually helping you write evals.

So, this is our product. But, like, you don't have to use our product for this. You can use any product. I'm going to show kind of an iteration on top of this, which is how we have users actually building eval prompts.

So, I could say, write me a prompt to detect friendly or robotic text. And this is actually using our own Copilot in the product. So, we've built a Copilot that understands best practices and actually can kind of help you write that first prompt, get it off of the ground.

You can also take the same prompt, which it just generated in, like, one second, and take that back to the prompt playground and iterate on further from here. So, let's go ahead and do that on the fly really quick.

I've got a prompt in here. And I can go in and actually ask the Copilot to optimize this prompt. So, let's go ahead and say, make it stricter.

So, I can actually use an LLM agent and Copilot agent. Just kind of note that, like, you really want AI workflows on top to help you, like, rewrite the prompt, add more examples, and then rerun evals on that new prompt.

So, it's more, it's less about, like, you're definitely not going to get itright on the first try. But being able to iterate is really what's important. And that's really what we underscore is, like, it might take you, like, five or ten tries to get an eval that matches your human labels.

And that's okay. Because these systems are really complex. And it's just important about having theright workflow in place. So, yeah.

Hi, I'm Jyoti. Does Arize also allow for model-based evaluations, like using BERT or Alberta to be able, rather than just LLM as a judge? But I can use, like, BERT or Alberta to, like, figure out, like, a prediction score.

Yeah, good question. So, we're actually really, the short answer is yes. We do offer versions of that. Let me show you what I mean by that, though. So, when we go into Arize, you can actually set up any eval model you want here.

So, you see we have OpenAI, Azure, Google. But you can add a custom model endpoint as well. So, you can basically, this will structure that request as a chat completion. But we can make it any arbitrary API if you need it to.

And you can say, like, BERT model and whatever the name of your endpoint is, point it to that. And you'll be able to reference that model in the eval generator too. So, this is, so I can just put test here, kind of move to the next flow.

And you'll see when I go into here, I can use any model provider I want. So, the short answer is yeah. You can generate a score with any model. Yeah. Cool.

Okay. Oh, we got one more question, I think. Or, sorry, we have more questions. Yeah, go for it.

I'm going to go ahead and try to get this one in. So, I think, like, probably a lot of the people that have built apps are thinking a similar thing. Or maybe this is a bit naive. But if you had human-labeled information already,right, and you're seeing a bad match on the friendliness score, am I to assume that you'd be trying to get that score up higher and then extrapolate to more cases going forward?

And you're assuming that that sampling holds across, like, the broader set?

Yeah.

So, like, because that relationship is unclear to me.

Very, very good question. So, basically, one way to reframe this is, like, how do I know that my data set is representative of my overall data to some degree?

Sure. Or as it shifts over time?

As it shifts, yeah, totally. So, that's a really, really good point. In the product, what we, we don't have this yet, but it's coming out, like, in the next week. We'll have a workflow to help you add data to your data set continuously using labels that you might have.

So, you could say, like, is, you know, one thing we didn't really talk about is, like, how to evaluate production data. But you can actually run these evals not just on a data set, but on all data that comes into your project over time to make it automatically label and classify, you know, any production data.

So, you could use that to keep building your data set of, like, is this an example we've seen before or not? Or is this, you know, think of this as, like, a way for you to sample at a larger scale, essentially, on production data.

And then it's a suggested workflow that you continuously sample and human-label some.

Some, yeah.

To check the matching over time.

Exactly. And you can basically go in and see, like, OK, where human labels don't agree with LLM on this, on production data, then you might want to add those to your data set as hard examples.

Sure.

And we actually are going to build into this product as well a way for you to qualify, is this example a hard example as well using LLM confidence score?

OK, and sorry, just hard example.

Yeah.

Very strict.

Yeah.

Strictly interpreted.

So, hard would be hard from an eval perspective. So, like, is it, is it friendly or not can be, like, borderline,right?

Oh, I see. So, you're saying, like, subjective or something.

Subjective, yeah, exactly. So, maybe to, like, recap the question a little bit, like, your data set is this property that's going to keep changing over time. And you really want tools that help you build onto it by giving you, like, a golden data set of hard examples to improve on.

And hard means, like, we're not really sure if we got itright or not in the first place.

Sure. Yeah, thanks.

Yeah, good question.

Yeah.

Hi, my name is Victoria Martin. Thank you so much for the talk. One of the things that I've run into is a lot of, like, skepticism out of product managers that I'm working with on generative AI and trying to build confidence in the evals that we're giving.

Yeah.

Have you been given any guidance or in working with other PMs, guidance on, like, the total number of evals that you think should be run by the time you can say, like, you can be confident in this evaluation set?

Yeah, good, really good question. So, the question was, like, how do we know? I think there's kind of two components to it. There's, like, quantity and quality of the evals. Like, how do we know if we've run enough evals or we have enough evals and that those evals are actually good enough to kind of pick up problems in our data?

We, this is also maybe a little bit of a broken record here. But I would say that this is a little bit of iteration as well, where you want to kind of get started with some small set of evals.

So, actually, I have a diagram for this. Let me just pull that one up. So, you'll kind of see here, this is intended to be, like, a loop where you start with some in development, you're going to run on a CSV of data, maybe, like, some hand example.

Like, I would argue the thing I just built was development,right? Like, I have 10 examples. It's not statistically significant. I'm not going to get the team on board to ship this thing. But what I can do is then curate data sets, keep iterating on them, keep rerunning experiments until I feel confident enough and the whole team is on board before I ship to production.

And then once you're in production, you're doing that again, except that now you're doing it on production data. And then you might take some of those examples and throw them back into development. Let me give a tactical example of what this looks like in real life.

With self-driving cars, when I joined Cruise, we would go down the street for, like, one block. And then a driver would, like, have to take over the car,right? Like, we couldn't drop, like, we couldn't drive one block down the road.

Same goes for Waymo. They were all kind of in this system. And then eventually, we got down to, like, being able to drive down a straight road. Great. But the car can't just drive on straight roads,right? Like, it has to make a left turn.

So, eventually, we got, like, fully autonomous for straight, you know, no problems on the road. And then we had to make a left turn. And then the car would, you know, a human would have to take over. So, what we did was we built a data set of, like, left turns.

And we used that to keep improving on left turns. And then eventually, the car could make left turns great until a pedestrian was in the sidewalk. And then we had to curate a data set of left turns with pedestrian in the sidewalk.

So, the answer is sort of, like, building your eval data set just takes time. And you're not going to know what are the difficult scenarios until you actually encounter them. So, I think to get to production, I would recommend just kind of using that loop until your whole team feels confident in, like, this is good enough to ship.

And just accept that once you get to production, you're going to find new examples to improve on as well. So, it depends a lot on your business as well. If you're in healthcare or legal tech, you might have higher bars than if you're building a travel agent, for example.

Yeah. Yeah. Yeah.

My name is Matosh. Hi. I have a question. As I understood the Arize platform, like, it's working as a, like, I take the prompt. And you're directly sending that prompt to a model,right?

That'sright, yeah. With the context and the data.

Yes, yeah, of course. You said that, like, there is some possibility to port tools into the platform.

That'sright.

But what about testing the whole system? Like, we already have, like, some flows that are augmenting the whole workflow.

Yeah.

Even outside of tool calls.

Yeah.

And, like, they're quite important into how the actual output will look like in the end. Is there any way to run those evaluations on a, like, custom runner?

Yeah.

Like, that would actually call our system on our data set that goes through everything that we have?

Find me after this. We should chat. Is the short answer for that one.

OK.

We have some tools and systems like that in place, like the tool calling that you saw.

Yeah.

But for end-to-end agents, we're actually building some stuff out. And we'd love to chat with you about that.

Perfect.

Good question. We'll find you after this.

Yeah.

Yeah.

Yeah, of course.

So, back to your left turn example, as well as just talking about the transition of, like, PRDs to, like, evals. What does the life cycle of, like, feature development look like and kind of the relationship, I feel like, of the feature, but also with your team in terms of ownership, accountability, all of that?

Yeah.

Yeah.

So, good question. So, I feel like how do you work with AI engineers in this new world is kind of interesting. Not the subject of this talk, but it is, like, a very relevant question that, you know, would happily chat more on too.

So, there's two answers to it that come to mind. One is that development cycles have gotten a lot faster. Like, the way at which these models are progressing and these systems are progressing, like, going from prototype to production is actually even faster than it ever has been.

So, that's one note, which I can just tell you as a personal observation. We feel that we can go from an idea to an updated prompt to shipping that prompt in, like, a span of a day of testing, which is, I think, like, unheard of, like, normal software development cycles.

So, that's one note, which is just, like, the way that you iterate with the team has gotten a lot faster. The second note is, when it comes to responsibilities, I view this as, if you're kind of a product manager is the keeper of the end product experience.

So, if that means making sure the evals are in a good place and the team has human labels to improve on, that's, like, a very solid area for a product manager to focus is, like, making sure the data is in a good spot for the rest of your development team.

I think at the same time, you know, I'm a PM on the team. And I'm, like, writing some of the stuff in Cursor. And so, being able to go in and actually talk to the code base itself using one of these models, I think that that's starting to become more of an expectation of AI product managers is to be literate in the code and be able to use these tools.

I really, this is, like, after this, I'm just going to go back and, like, try to fix the thing that I broke earlier,right? And the way, the reason I'm able to do that is because the way I'm prompting the system is not very sophisticated.

Like, I asked it yesterday, can you make a script that generates itineraries on top of the server? I need, like, 15 examples. And it just did that,right? And, like, that, like, wouldn't have been possible. So, I think PMs are responsible for the end product experience.

But PMs also have more leverage than they've ever had before in probably the entire, like, professional journey of product management, because you're now no longer reliant on your engineering team to ship that thing that you wanted. Like, you can just go do it.

Should you go do it is another question. But, and that's something, that's a discussion that you should have with your team. But the fact is that you can go do it now, which was not the case before. And so, I kind of urge PMs to, like, push the boundaries of what people have told them the role is and should be and see where that takes you.

And so, the long-winded way of saying, like, your mileage may vary, depends on the boundaries you have with your team. But I'd recommend people to, like, redefine those at this stage. Yeah.

Yeah. Jumping off that a little bit, it's a little off topic from this.

Yeah.

But, like, as a product manager who wants to move to be more technical, like, as I'm working with AI engineers.

Yeah.

What does that look like? Like, I'm in an org where I have very limited access to the code base. So, like, I use Cursor to write Python for data things. But, like, I don't necessarily have access to, like, start interrogating the code to understand that.

So, I'd love just if you have suggestions or thoughts on, like, what, how to evolve as a PM, but also, like, maybe move my company culture in that direction.

Yeah, that's, that's, like, how actually, I have a follow-up question, if that's OK. Just because I'm going to poll people in the room. Like, how big is the company? And you don't have to share the name if you don't want to, but just curious, like, the size of the company.

We're about 300 people.

OK.

But the tech org is probably, like, a third of that.

OK, so, like, almost, like, 100 engineers, 300 people. And do you have any, like, old remnant product managers at the company that still have code access?

No. We're, like, a very new team of PMs.

OK. OK, cool. OK. Well, I think one thing we've started doing it's a really good question. Thanks for answering that. One thing we've started doing is trying to take a little bit of, like, the public forum of our company.

Sorry, I'm about to out our CEO, who's in the back of the room.

So, if you have any questions about Arize, he's a good guy to talk to. But the reason I'm out of him is because, like, I missed our town hall today. But, like, I heard it was just basically people running, like, AI demos the whole time of, like, what they're building.

And why I think that's really powerful is it can get the whole company really catalyzed around what's possible. Because, to be honest, I think it's very likely that, you know, most teams today aren't pushing the boundaries of these tools.

And so, you kind of joining this talk and seeing, like, how to run evals, how to, you know, what goes into experiments, like, being able to kind of be the person pushing the team forward is really powerful. And I think you can do that in a way that's really collaborative.

So, I only, I'd say, like, our job as PMs is to have influence over the team and influence product direction. I think there's an opportunity to influence the fact that PMs should be more technical in your org. And you could show them by building something and impressing the rest of the team by what you built.

So, that's my advice, my personal advice there.

Yeah.

Yeah.

Yeah, go for it. Yeah.

Actually, I have a question to CEO, if it's possible. So, how do you guys believe AI will reshape how we structure the team? So,right now, you have, like, I would say, for instance, just, like, 10 engineers, one product manager, one designer, and so on.

So, what will happen in five years?

Yeah.

You will have one product manager, one engineer, and one designer?

You should answer this one. You should do it in the mic, though, if you want to.

The short of it is, I would actually, the person who's just here who doesn't, like, PMs don't have access to the code base. I think that's really old school. I think Cursor on the code base is, there's so many times the PMs are taking up engineers' time asking a question.

Like, you know how often we now just ask Cursor? So, yeah, like, start there. Open up your code base to Cursor, give it to PMs. And then a lot of, some we were just the other day doing a PRD starting from Cursor on the code base.

So, I think the, yeah, that would be where I would start.

Yeah.

And I don't, I can't, you know, it's hard to look forwardright now. I just, I think a lot of jobs change. We're trying to push AI Cursor use throughout the company as far as I can.

Yeah. I hear we have people in marketing using Cursor too these days.

Yeah.

So, yeah, that's kind of cool. Yeah.

Quick follow-up question.

Yeah.

So, you're talking aboutright now having the product person become more of a technologist. Do you see also technologists becoming more product?

Yes.

What kind of role is it being combined into one?

So, that's actually a great point, which is, like, when the cost of building something goes down, which it has, what's theright thing to build becomes really important and valuable. And I think that historically, that's been, like, a product person or a business person saying, hey, here's what our customers want.

Let's go build this thing. Now we're saying, product people, you can just go build this thing. So, the builders are like, wait, what's my job? Like, do I, and I think that that's a good way to look at it, which is, I have this, like, mental framework of, like, what if we didn't have roles in a company anymore?

Like, you didn't define yourself by, like, I'm a PM. I'm an engineer. And think of this instead as, like, like, you know, like baseball cards. You have, like, skills. Imagine that you had, like, a skilled stack instead, which is, like, I really like to talk to customers.

And I kind of like to code stuff on the side. But I don't want to be responsible if there's a production outage. I guarantee you, you'll find someone who's, like, I hate talking to customers. And I only want to ship high-quality code.

And I want to be responsible if things hit the fan. And I think you want to structure your company to have a skilled stack that's really complementary versus people who are, like, I do this. And this is my job.

And I don't do that. So, yeah.

I have something that's sort of related to that. We've been testing, like, human-in-the-loop

in a couple of different ways. And we're basically testing this method of having the human as a tool of the agent. So, like, we have, like, if the agent needs something that's not available in the accounting system, it'll go to the CFO, because the CFO is listed as a tool.

And it sends that him a Slack message, gets it back, and continues. It kind of maps onto what you just said of, like, defining the skills, defining the resources they have. And we haven't fully fleshed it out. But it's working to, like, give the agent context on the things that only the humans have.

Yeah.

And I think it maps on exactly that.

So, this person is, like, your company is, like, using agents widely, it sounds like. But you have humans approving. You have, like, an approver workflow to some degree.

No, more so, like, rather than how can the agent be a tool of the human, we're kind of flipping it and saying, like, what if the agents could do everything? And then the parts it can't do, it'll go to the human.

Yeah.

As a tool. So, like, the CFO is a tool of the AI agent.

Interesting.

Rather than the other one.

We should chat. That's a really cool workflow. I'll definitely bug you about that. That's really cool. Cool. Happy.

The eval, the model naming for the eval, it's just that.

Right, to some degree. It's like a human-in-the-loop approving, is this good or bad? And you can think of it that way. Yeah, go for it.

Yeah, I had a question about, like, what it is like to actually implement, well, sending the traces over to Arize. I know, like, Arize has, like, open inference, which enables, enables, like, capturing traces from several different, several different providers.

But what are, what are, what are the limitations and constraints and opinions that you have about how the evals should be structured so that you can actually, like, leverage the platform to perform these actions, to be able to, like, evaluate the eval, for example, or be able to, like, numerically just produce graphs out of your evaluations, out of your outputs?

OK, so.

I'm not clear.

So, can I, can I ask a follow-up question to that, which is, like, your question was, like, how to use agents to do some of the workflows in the platform? Or did I misinterpret that?

No, no.

OK.

The question, the question is, like, how is, like, what kind of outputs, what kind of evals is this, is Arize expecting from your engineers and from the product? Like, you're sending over logs,right?

Yeah.

What, what is it expecting from those logs in order to get this flow working?

OK, understood. OK, so, so yeah, there is a very, like, great point there, which is, like, we kind of, you'll see it in the code. But we jumped over a little bit here in the demo, which is how do you get the logs in theright place to use the platform.

Unfortunately, this page isn't dropping. But let me, OK, here we go. I'm going to drop it in the Slack channel as well. This is what we, you know, we kind of talked about, like, traces and spans. It's very likely that your team already has logs or traces and spans already.

You might be using Datadog or a different platform, like Grafana. What we do is we're taking those same traces and spans. And we're essentially augmenting them with more metadata and structuring them in a way that the platform kind of knows which columns to go and look at to render the data that you saw in the platform.

So, you're really using the same approach. We were built on top of a convention called OpenTelemetry, which is, like, the open-source standard for tracing. So, we actually use OTel tracing and auto instrumentation that we've built, which doesn't keep you locked in at all.

Like, once you've instrumented with our platform using OpenInference, which is our package, you actually get those logs to show upright out of the box with any type of agent framework you might be building. And, and yeah, and you get to keep that.

That's, let me just show, like, what I mean by that. So, if you're, let's say you're building with, like, LANGRAF, we actually have it really all you have to do is, like, you pip install Arize OTel. And you, what you call this single line of code, call this single line of code called LangChain Instrumenter.

And it knows where to pick up in your code to structure your logs. And if you have more specific things you want to add to your logs, you can add function decorators, which is basically a way for you to, you know, capture specific functions that weren't in the logs.

Yeah, and as for evaluations, like, you're discussing, like, the actual data inputs, outputs. What do you, what do you need to pass into evaluations?

Yeah.

I know you can, like, design them through the UI.

Yeah.

What do you have in mind for, like.

Like, how do you get theright text to use for evals,right? Is that your question? Like, how do, how are you, like, how do you know which fields to use?

Actually, I need to, like.

OK.

Format the question. I'll get back to you.

Yeah, no worries.

And also, what did you mean by adding, augmenting the data with additional metadata? Like, you only have so much data,right?

Yeah, so, so this is, so think of this as, like, most tracing and logging data is really just things like latency timing information. What we're doing is you can add more metadata, like user ID, session ID, things like, like, I'll kind of show you an example of that really quick.

In the previous example I showed, we actually have things like sessions. Like, what's the back-and-forth example here? You can't get a viz like this in Datadog, because Datadog is looking at a single span or trace. It's not, it's not really contextually aware of what is the human, what's the AI.

So, we're adding context from the, from the invocation of the server and adding that to your span, if that makes sense. So, it's, it's basically just enriching the data a bit more and structuring it in a way to use it.

Yeah. And if you have more specific server-side logic, you can add that as well. So, it's very flexible. Yeah. Yeah.

Sorry of the provocation. So, I used to work in the video game industry. And debates about feature, like, whether a feature was going to be fun or not, working prototypes won all of those arguments.

Yeah.

Whatever was in the doc didn't matter.

Right.

And so, for the person who was, like, I can't get access to my company's code, I would actually say, try to get access to a small sliver of the data. And then build a working prototype of the feature you want to see and with some stuff of evals.

Because I think, you know, there's nothing worse to an engineer than a product manager who shows up with a demo that's kind of janky.

Yeah.

But actually works and might be fun, has polish, feels good, meets a user need. And they, and having been on the engineering side of this equation, I'm like, and it's so janky. I have to fix it. They haven't thought about the edge cases.

And so, like, how does Arize fit into that flow of helping a product manager basically mine a small segment of data, build a working example, and perhaps just, you know, janky as all get out, but something that looks like the product that the company already has but demonstrates that next level of functionality?

That's a great, great point. And yeah, I think, like, you know, feel free to prototype and build, you know, prototypes that are, that are high fidelity. I think it is awesome to do that. It's a really good point to have, like, to use data to build a system or a prototype.

So, what does Arize do here? If you have access to Arize and you don't have access to the code base, you can still take this data. And assuming that you have permission from your sysadmin person, you can actually export this data.

So, once you've built a data set, you can simply take this data and export it out and use that to actually, so I can kind of show that really quick. This is Get Dataset. We'll have a download button coming later this week.

But you can actually just take this data, run it locally, keep it locally, and then actually use that in your local code to actually try and iterate on an example. You know, assuming your security team is OK with that.

But that's a really good point. Like, imagine if you didn't need access to the production code base, but you could still iterate in one platform. That's really what we're, we're pushing for, is, like, the whole team is iterating on the prompts and the evals together, rather than in silos, which is what's happening in a lot of cases.

OK, I think that was all of the questions. Thank you all for sitting through an hour and a half of AI PM, like, evals. Thank you all for, for your time. And I'll be sticking around if people have more questions.

But thank you so much.