Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

Hey everyone, I'm going to get started here. Thanks so much for joining us today. I'm SallyAnn, I'm the director of product at Arize, and I'm going to be walking you through some of Proud Prompt Learning. We're actually going to be building an AI-driven optimization loop for the part of the workshop.

I come from a technical background, kind of started off in data science before I made my way over to product. I do like to still be touching code today. I think one of my favorite projects that I work on is building our own agent, Alex, into our platform, so I'm very familiar with all of the pain points and how important it is to optimize your prompts.

So I'm going to spend a little bit of time with the slides. I like to, like, just set the scene, make sure everybody here has context on what we're going to be doing, and then we'll jump into the code.

I have Fuad with me, so I'll let you do a little bit of an intro.

Yeah, thank you so much, SallyAnn. Great to meet all of you. Excited to be walking through Prompt Learning with you all. I don't know if you got a chance to see Aparna's talk yesterday, but hopefully that gave you some good background on how powerful meta prompting and prompt learning can be.

So my name is Fuad, I'm a product manager here at Arize as well, and like SallyAnn said, we like to stay in code. We'll be doing a few slides, then we'll walk through the code, and we'll be floating around helping you guys debug and things like that.

My background is also technical, so I was back in distributed systems engineering for a long time, so no stranger to how important observability infrastructure really is. And I think it's an appropriate setting in AWS for that. So yeah, excited to dive deep into Prompt Learning with you all.

Thank you.

Awesome. Allright, so we're going to get started. Just give you a little bit of an agenda of the things I'm going to be covering. So we're going to talk about why agents fail today, what is efficient prompt learning.

I want to go through a case study, kind of show y'all why this actually works. And we'll talk about prompt learning versus GEPA. I think everybody—I had a few people come up to me over the conference and be like, "What about GEPA?"—we have some benchmarking against that, and then we'll hop into our workshop.

But with this, I want to ask a question. How many people here are building agents today? Okay, that's what I expected. And how many people actually feel like the agents they're building are reliable? Yeah, that's what I also thought.

So let's talk a little bit about why agents fail today. So why do they fail? Well, there's a few things that we're seeing with a lot of our folks, and we're seeing even internally as we build with Alex for why agents are breaking.

So I think that a lot of times it's not because the models are weak, it's a lot of times the environment and the instructions are weak. So having no instructions from their learned environment, no planning, or very static planning.

I feel like a lot of agentsright now don't have planning. We do have some good examples of planning, like we have Claude Code, Cursor. Those are really great examples, but I'm not seeing it make its way into every agent that I come across.

Missing tools, big one. Sometimes you just don't have the tool sets that you need. And then missing kind of tool guidance on, like, which of the tools we should be picking. And then context engineering continues to be a big struggle for folks.

If I were to distill this out, I think it's like these three core issues. So adaptability and self-learning, so no system instructions, learned from the environment we touched on, determinism versus non-determinism balance. So having the planning or no planning versus doing, like, a very static planning.

You want to kind of have some flexibility there. And then context engineering. I think it's a term that just kind of emerged in the last, like, you know, six to eight months, but it's something that's really, really important that we're finding.

You know, missing tools, tool guidance, just not having context or pre-planning your data and not giving the LLM enough context. So these are kind of the core issues distilled, but I think there's one other pretty important thing. And that is kind of this distribution of who's responsible for what.

So there's these technical users, your AI engineers, your data scientists, developers, and they're really responsible for the code automation, pipelines, actually, you know, managing the performance and cost. But then we have our domain experts, subject matter experts, AI product managers.

These are the ones that actually know what the user experience should be. They probably are super familiar with the principles that we're actually kind of building to our AI applications. They're tracking our evals, and they're really trying to ensure that the product is a success.

So there's this split between responsibilities, but everybody is contributing. But then there's this difference in terms of, like, maybe technical abilities. And so with prompt learning, it's going to be a combination of all these things, where everybody's going to really need to be involved, and we can talk about that a little bit more.

So what even is prompt learning? I'm going to first kind of go through some of the

approaches that we kind of borrowed when we came up with prompt learning. So this is something that Arize has been really, really dedicated to doing some research. And so one of the first things we borrow from, which is reinforcement learning.

How many folks here are familiar with how reinforcement learning works? Allright, cool. So if I were to give like a really, like, silly kind of analogy, if we have a reinforcement model, pretend it's like a student brain that we're trying to kind of, you know, boost up.

And so they're going to take an action, which might be something like, you know, he's going to take a test, take an exam, and there's going to be a score. A teacher is going to come through and actually, you know, score the exam here.

That's going to produce this kind of, like, scalar reward. And, you know, pretend the student has an algorithm in their brain that can just kind of take those scores and update the weights in their brain and kind of like the learning behavior there, and then we kind of reprocess.

So, you know, in this kind of reinforcement learning, we're updating weights based off of some scalars. But it's really actually difficult to update the weights directly, especially in, like, the LLM world. So reinforcement learning isn't going to quite work that well when we're doing things like prompting.

So then there's meta prompting, which is very close to what we do with prompt learning, but still not quiteright. So here with meta prompting, we're asking an LLM to improve the prompt. So again, we use that kind of like student example.

We have an agent, which is our student, and that's going to produce some kind of output. Like, that's a user asking a question, getting an output. That's our test in this example. And then we're going to have a score.

Eval is pretty much what you can think of there, where it's going to output a score. And from there, we have, like, the meta prompting. So now the teacher is kind of like the meta prompt that's going to take the results from our scorer and update the prompt space off of that.

But still not quite what we want to do. And that's where we kind of introduce this idea of prompt learning. So prompt learning is going to take the exam, going to produce an output. We're going to have our LLM evals on there.

But there's also this really important piece, which is the English feedback. So which answers were wrong, why were the answers wrong, where the student needs to actually study, really pinpointing those issues. And then we still aren't using meta prompting.

We still are asking an LLM to improve the prompt. It's just the information that we are giving that LLM is quite different. And so we're going to update the prompt there with all of this kind of feedback. So from our evals, from a subject matter expert going in and labeling, and use that to kind of boost our prompt with better instructions and sometimes examples.

So this is kind of like the traditional prompt optimization, where it's like we have we're kind of treating it like an ML model, where we have our data and we have the prompt. We're saying, "Optimize this prompt and maximize our, like, prediction intervals."

But that doesn't quite work for LLMs. We're missing a lot of context. So what we've really found is that the human instructions of why it failed. So imagine you have your application data, your traces, a data set, whatever it is.

Your subject matter expert goes in, and they're not only annotating correct or incorrect. They're saying, "This is why this is wrong. It failed to adhere to this key instruction. It didn't adhere to the context. It's missing out," whatever it is.

And then you also have your eval explanations from LLM as a judge, which is the same kind of principle, where instead of just the label, it provides the reasoning behind the label. And then we're pointing it at the exact instructions to change.

We're changing the system prompt to help it improve so that we then get, you know, prediction labels, but we also get those evals and explanations of why it failed. So we're just kind of optimizing more than just our output here.

And I think the really key learning that we've had is the explanations and human instructions, or through your LLM as a judge, that text is really, really valuable. And I think that's what we see not being utilized in a lot of other prompt optimization approaches.

They're either kind of optimizing for a score, or they're just paying attention to the output. But you kind of think of it this way. It's like these LLMs are operating in the text domain. So we have all this rich text that tells us exactly what it needs to do to improve.

Why wouldn't we use that to actually improve our prompt there? So that's kind of the basics of prompt learning. But everybody always comes up to me and it's like, "This sounds great, Sally, but does it actually work?" It does.

And we have some examples of when we do this. We did a little bit of a case study. I think coding agents, everybody is pretty much using them at this point. There's quite a few that have been really, really successful.

I think Claude Code is a great example, Cursor. But there's also Kline, which is more of an open version of this. And so we decided to take a look and compare to see if we could, you know, do anything to improve.

So these are kind of the baseline of where we started here. You can see the difference between the different models. Claude Code is obviously using 2 and dropping kind of the state-of-the-art there. But we also had this opportunity where Kline was using, you know, 4.5, and it was working decently well at 30% versus 40.

And then there was kind of the conversation around 4.1. So this is where we started. And we took a pass at optimizing the system prompt here. So you can see this is what the old one was looking like.

It has, like, no rule section. So it was just very like, "You are a Claude agent. You're built on this model. You're here to do coding." But there were no rules. And so we took a pass at updating the system prompt so that there were all of these different rules associated.

So when dealing with errors or exceptions, handle them in a specific way. Make sure that the changes align with, you know, the system's design. Any changes should be accompanied by appropriate tests. So really just kind of building in, like, the rules that, like, a good engineer would have, which was completely missing before.

And so we found that Kline performs better with the updated system prompt. Pretty kind of simple. It's kind of the whole concept here. It's like you can see these different problems, and we're seeing, you know, things that were incorrect now being correctly done just by simply adding more instructions.

So it really demonstrates pretty well here how those system prompts can improve. And we benchmarked again with SWE BenchLite to get another just, like, kind of coding benchmark for these coding agents. And we were able to improve by 15% just through the addition of rules.

So I think that that's pretty powerful. So no fine-tuning, no tool changes, no architecture changes. I think those are the big things folks, like, reach for when they're trying to improve their agents. But sometimes it's just about your system prompt and just adding rules.

And I think we've really seen that. And that's why we're really passionate about prompt learning and prompt optimization in general, is it feels like the lowest liftway to get massive improvement gained in your agent. 4.1 achieved performance near Sally 4.5, which is pretty much consideredright now the state-of-the-art when it comes to coding questions.

And it's two-thirds the cost, which is always really beneficial. So these are some of the kind of the tables here. We'll definitely distribute this so you can kind of take a closer look. But I think the main point I want y'all to come away with is the fact that, like, you know, 15% is pretty, you know, powerful improvement in our performance.

Now, a question we get all the time is we're taking these examples for prompt learning. So how does this really work? So we're going to take a data set. A lot of time that data set is going to be a set of examples that didn't perform well.

Either a human went through and labeled them and found that they, you know, were incorrect, or you have your evals that are labeling them incorrect. And so you've gathered all these examples, and that's what we're going to use to optimize the prompt.

So I get a question all the time is like, "Well, are we going to overfit based off of these bad examples?" But there's this rule of generalization where meta prompting enforces high-level or usable coding standards rather than repro-specific fixes.

And we are doing this train-test split to ensure that the rules are generalized beyond just, like, these local quirks and whatever our training data set is. But if you kind of think of this as, like, you hire an engineer,right, to be an engineer at your company, you do kind of want them to overfit to the database that they're working on.

So we kind of feel that overfitting is maybe a better term for it is expertise. We are again not kind of training in the traditional world. We are trying to build expertise. And as we'll talk about, this is not something we feel that you do once.

You're actually going to kind of continuously be running this. So more problems are going to come up. We're going to kind of optimize our prompt for what the application is seeing now. And then we'll kind of continue. So we don't actually think it's a flaw.

We feel like it's expertise instead. We can kind of adapt as needed and kind of mirroring what humans would do if they were taking on the tasks themselves.

This is just another set of benchmarking, again, kind of proving here that this diverse evaluation suite that focuses on the tasks for those difficult or tasks that are difficult for large language models. And we're seeing, again, success with our improvements.

Now, GEBA just kind of came out recently, and I think that's something everybody's really excited about. I think the previous DSPY optimizers were a little bit more focused on optimizing a metric. And as we talked about, like, we really want to be using the text modality that these applications are working in that have a lot of the reasons or how we need to improve.

And so we definitely wanted to do some benchmarking here. So how many people are familiar with GEBA? Have you tried it? Read about it? Allright, cool. Well, just to give a, like, some high level, I just kind of noted that the main difference between GEBA and their other, like, NePro optimizers is that they are actually using this concept of reflection and evaluation while they are doing the optimization.

So it's this evolutionary optimization where there's this parental-based candidate selection and probabilistic merging of prompts. What this really does under the hood is we take candidate prompts, we evaluate them, then there's this reflection LLM that's reviewing the evaluations and then kind of making some mutations and some changes and kind of repeating until it feels like it has theright set of prompts.

So I think something that is important to note about GEBA is it doesn't really choose kind of just one. It does try to keep the top candidates and then, you know, do the merging from there. But we benchmarked it, and prompt learning actually does do a little bit of a better job.

And I think something that's really key is it does it in a lower number of loops. And I think something that we'll talk about in just a second here is that it does actually matter what your evals look like and how reliable those are.

I think that's something we really feel strongly about at Arize is you definitely want to be optimizing your agent prompts. But I think a lot of people forget about the fact that you should also be optimizing your eval prompts.

Because if you're using evals as a signal, you can't really rely on them if you don't feel confident in them. So it's just as important to invest there, making sure you're kind of applying the same principles that you are to your agent prompts.

As your eval prompts, you have a really reliable signal that you can trust and then feed that into your prompt optimization. But in both of these graphs, the pink line is prompt learning. And we did also benchmark it against NePro, their older optimization technique that I was mentioning, kind of functions off, like, optimizing around a score.

And evals really did make the difference. So kind of I highlighted on this slide here, like, the pull-through with eval engineering, we were able to do this. So we did have to make sure that the evals we were using as part of prompt learning were really high quality because, again, this only works if the eval itself is worth it.

So, yep, evals make all the difference. Kind of spent some time optimizing a prompt here. Again, it's all about making sure you have proper instructions. Same kind of rules apply. So I wanted to kind of walk through. I know there was a lot of content.

I think it's really important to have context. But before we jump into any of the workshop stuff, any questions that you can answer about what I discussed so far?

I have a question or just a general comment. So I think, you know, coding is the greatest example in terms of having the structure and evals. One thing I'm sort of curious about is if you have other examples, sort of general prompts for additional interactions with systems that are not as easily quantifiable.

I'm just curious about any experience you guys have there.

Yeah, is that for, like, evals or just, like, the prompts in general?

Well, I think it's just clearer how you would set up what the evals would look like. You know, I'm just wondering how you would do that for other types of.

Yes. Well, okay. So the question is, like, is there any kind of instruction for how you should set up your evals? Coding seems like a very straightforward example. You kind of want to make sure the code's correct,right? But where some of these other agent tasks, it's a little bit harder.

I think the advice that I usually give folks is we do have a set, like, out-of-the-box. You can always start with things like QA correctness or focus on the task. But what I always suggest is, like, getting all the stakeholders kind of in the room.

So getting those, you know, subject matter experts, PM, security, you know, leadership, and really defining what success would look like and then start kind of converting that to different evaluations. So I think an example is for, like, Alex.

I have some task-level evaluation. So, like, I really care. Did it find theright data that it should have? Did it create a filter using semantic search or structured, like, making theright tool call? And then I care. Did it call things in theright order?

Was the plan correct? So kind of thinking about, like, what each step was. And then, like, in security, it will be like, well, we care how often people are trying to jailbreak Alex. So it's just taking each of those success criteria, converting it to evals.

And we do have different tools that can help you. But that's usually the framework I give folks. It's like, start with just success and then worry about converting it to an eval after.

Yeah, and just to add on to that, maybe, like, more of, like, a subjective use case is, like, for example, like, Booking.com is one of our clients. And so when they do, like, what is a good posting for a property?

Like, what is a good picture? Defining that is really hard,right? Like, to you, you might think something is a very attractive posting for, like, a hotel or something,right? But to someone else, it might look really different. And sometimes, as kind of Sally was alluding to, it's sufficient to just gate it as a good/bad and then kind of iterate from there.

So, like, is this a good picture or bad picture? Let an LLM decide and then gate from there into specific factors. Like, oh, this was dimly lit. The layout of the room is different, et cetera, et cetera. Yeah.

Yeah, that you're actually building on the question I was going to ask, which is that if they end up with that binary outcome, which doesn't necessarily give you a gradient to advance upon, are you then effectively using those questions, like, digitally lit not to, like, get, like, a more continuous space?

Is thatright?

Right. Exactly. Right. And then from there, as you get more signal, you can refine your evaluator further and further and then use those dates. And you can actually put a lot of that in your prompting itself,right? So, yeah.

Yeah. Okay.

I have two questions, and I'm not sure if I should ask both of them or maybe your workshop will answer it. One is about rules and the rules section or, like, operating procedures. I'm curious how you do you just continuously refine that in the English language and maybe reduce the friction of any contradictory rules?

That's the first question. And then the other was, I would love to see the slide on evals and if you could just say a little bit more on how you approach that because my issue in doing this work is whether or not to have, like, a simulator of the product and then the simulator is evaluating or to do what I like to do, which is, like, an end-to-end evaluation that I build.

But I would love to see you talk about that if you could.

Yeah, absolutely. So I'll start with the first one about, like, how the instructions is it really something I think that, like, you iterate over time on them. So a lot of times, I think we take our best best, like, we write them by hand,right?

And I think what we're trying to do with prompt optimization is, like, leverage the data to dynamically change them. LLM is, I think, great at, like, removing redundant instructions, things like that. But the goal is we want to move away from static instructions.

We feel very confidently that, like, that is not going to really scale. It's not going to lead to, like, sustainable performance. So the idea exactly with prompt learning is something that you can kind of run over time. We see this even, like, a long-running task eventually where you're building up examples of incorrect things, maybe having a human annotate them, and then the task is kind of always running, producing optimized prompts that you can then pull in production.

And it kind of is, like, a cycle that repeats over time.

Sorry, just to intervene. So are you saying that when you're doing this over a long period of time and then you have examples, you're just running the shots back into your rules section?

Kind of. It's going to pass it. Like, when we get to the prompt optimization actual, like, loop we're going to build, you'll kind of see it as, like, you are feeding the data in that's going to build a new set of instructions that you would then, you know, push to production to use.

Okay.

And then I think your second question was around evals and, like, how to where to start, how to, like, write them and, like, how to optimize those. Is thatright?

Yes.

Yeah. So it's a very similar approach. I think it's like the data that you're reviewing is almost a little bit different. So I should have pulled up the loops. I don't know if you can find it.

It should be in the stack. Let me just check.

Something really quick to kind of show this. There we go. So this is kind of, like, how we see it is you have two co-evolving loops. I've been talking about the one on the left, the blue one, a lot about we're improving agent, we're collecting failures, kind of sending that to do kind of fine-tuning or prompt learning.

But you basically want to do the same thing with your evals where we're collecting a data set of failures, but instead of thinking about the failures being the output of your agent, we're actually talking about the eval output.

So having somebody go through and, you know, evaluate the evaluators or using things like log prompts as confidence scores or a jury as a judge to determine where things are not confident, we're kind of doing the same thing.

So figuring out where your eval is low confidence and then you're collecting that, annotating, maybe having somebody go through and say, okay, this is where the eval went wrong. And so it's the same pretty much process of optimizing your eval prompt.

It's just, you know, I think folks think they can just grab something off the shelf or write something once and then they can just forget about it. But this loop, I've said it a few times, but the left loop only works as well as your evals are working.

Sorry, I think my question is actually way more static and basic. It's like, do you are you talking about this orange circle as, like, are you building a system or simulator for the eval or are you just talking about, like, system prompt, user prompt, eval?

Yeah, I think it's moreright now what we're talking about is just, like, kind of the different prompts. You can definitely do simulation, but I think that's a whole different workshop.

Thank you.

Any one more maybe question before we get to the workshop? Any? Cool. Let me switch back.

Allright. So here is going to be a QR code for our prompt learning repo. So I'll give everyone a few minutes to get search. What should we do with that? Get it on your laptops. I know it's a little bit clunky.

The scan the QR code and, like, AirDrop it. I was not sure a better way. I can just show you also here if you want to find it. It is going to be in the Arize AI repo here and under prompt learning.

And you just want to kind of clone that. We are going to kind of be running it locally here.

Can you go back to the page with the URL?

Yes. Sorry about that.

No, the page with the URL.

Oh, sorry.

I'll give folks just a few minutes to get.

What's your process when you're building a new agent or workflow? Anything that could be evaluated? Do you guys start by just like, oh, chop something into a prototype and then see where it's bad and then do evals?

Yeah, I think there's different perspectives on this. Our perspective is evals should never block you. Like, you need to get started and you need to just build something really scrappy. We don't think, like, you should, you know, waste time doing evals.

I think it's helpful to pull something out of the box sometimes in those situations just because it's hard to go through your data. Like, it's something we've experienced with Alex of, like, when you're getting started, just running a test manually or reviewing, like, it's kind of painful.

So I think that having evals is helpful, but shouldn't be a blocker. Pull something off the shelf, maybe start with that. Then as you're iterating, you're understanding where your issues are, then you're starting to refine your evals as you're refining your agent.

Yeah.

One last question.

Yeah.

So it makes sense to, like, optimize the system. Do you use any kind of, like, buckets, sub-agents, or commands? Or how are you thinking about this, like, multi-agent?

Yeah. So the question is, is, like, are you just doing one single prompt or how do you think about this in a multi-agent? I think we're kind of thinking that thisright now is kind of independent tasks that can optimize your prompts kind of independently and then running tests when you get into, like, the agent simulation of running them all together.

Butright now, our approach is a little bit isolated. But I definitely see a future where we're going to kind of meet the standard of, like, sub-agents and everything else that's going onright now.

No, I think that's pretty accurate. And also, like, I mean, even in a single-agent use case versus, like, a multi-agent use case, like, ultimately, like, each of those agents may be specialized. They may have their own prompts that they need to learn from.

So I think doing this in isolation still has benefits for the multi-agent system as a whole that can pound over time, especially in scenarios like handoff, et cetera, and making something, like, really, really specialized. So kind of gets to, like, what we were talking about with, like, the overfitting as well, which is, again, like, the question we get all the time.

But really, you want to be overfitting on your own code base as an engineer. You don't want to be so generalized that you're no longer good at thinking of specific work scenarios based. But yeah.

Okay. Everybody kind of getting to the repo okay? Anybody need any help?

Allright. So we are going to be using OpenAI for this. So I think the next thing that I'll have everyone do is probably spend some time describing your API key. We'll get to it. And then I'll just kind of start walking through our notebook here.

So we are going to be doing the JSON webpage prompt examples. You're going to find that under notebooks here. And so we'll give everybody a second to pull that up. There's going to be just some slight adjustments we're going to add to this example just to make it run a little faster and work a little better.

So the first is what this is even doing. This is going to be a very simple example for just a JSON webpage prompt. If anybody has, like, a prompt or a use case that they want to kind of, like, code along, Fuad and I are absolutely help, like, glad to help kind of adapt what you're working on to the use case here.

This is something very simple just to kind of demonstrate the principles. And we are going to be using Quora. We can definitely experiment. If you want to swap out any other providers that you want to use, we can also definitely help you do that.

But the goal of this is essentially going to be to iterate through different versions of a prompt using a data set.

And we will optimize. So the first thing is, obviously, we need to do some installs. I am just going to have you all update. I think this says, like, greater than 2.00. But we're going to actually just use, I think, 2.2 today.

And then the next thing is just to make this run a little faster. We're going to run things in async, which is missing. So you can go ahead and add these lines in the cell as well. Allright. Everyone kind of following along?

And I never know. I don't ever want to move too fast, but it seems to have not. Cool. Let's talk about configuration. So I kind of talked about it a little bit when I was going through the slides.

We are going to be doing some looping. So the general idea is we start out with the data set with some feedback in it. And we'll look through the data set once we get there. But you're going to want to have either human evaluation, so, like, annotations, either free text, labels, or you're going to want to have some evaluation data.

But the feedback is really important. That's what makes this kind of work. We're going to then, you know, pass that to an LLM to do the optimization. And then it's going to basically have evals in the loop. So as it's optimizing, it's using that kind of data set to then run and assess whether or not it should, you know, kind of keep optimizing.

And then it also provides you data that you can kind of, like, use to gauge which of the prompts that it outputs in, you know, a production setting. So we're going to do some configuration. So I've kind of wrote out here kind of what each of these means.

So we have the number of samples. This controls how many rows of the sample data set. You can, you know, set to zero to use all data. You can, you know, use a positive number to limit for, you know, master experimentation.

So I think that sometimes folks use different approaches here. Sometimes you want to just move really quick. So you set a low sample. Sometimes you want to be a little bit more representative. So you up it. I have it here set as 100.

Feel free to adjust. And then the next thing is train split. So I think folks are probably pretty familiar with the concept here of, like, a train test split. But it's just how much of the data do we want to use into our training?

Again, that's what we're using to actually optimize. Then how much of it do we want to use when we're testing, when we're running the eval on the new prompt?

And there's a number of rules. It's basically the specific number of rules to use for evaluation. This just determines which prompts to use. And so this is, like, as we're running these loops, we're outputting, you know, a bunch of different prompts.

And this is just saying how many we should use for evaluation. And then key one here, number of optimization loops. So this sets how many optimization iterations run per experiment. And each loop basically generates those outputs, evaluates them, and refines the prompt.

And so these just control experiment scope, the data splitting, just pretty much the whole prompt learning loop and how much data we want to use. So you can kind of just run these as you are. Or if you want to adjust them, feel free.

And then the next step, pretty simple. We're just going to grab that OpenAI key if you haven't already set that up. So get passage is going to, like, pop up. I'll show you here quick. It's going to pop up there.

You can just paste in your API key. I'll say before we start looking at the data a little bit.

And just if anybody runs into any issues, just give this away. I think this part is the easiest part. So hopefully we get through this way.

How are we doing on API keys?

Pretty good. I'm doing good. But if you have a free one, you want to give me from the.

I wish.

Allright. Let's talk about the data. So we provided the data with you with queries. You can see here that we're doing the 80/20 split based off of kind of configuration we set above. I'm just going to pull this train set here.

And let's just.

Yeah. I run 80/20 because in the previous slide was, oh, sorry, 50/50.

Oh, yep. You'reright. That's a mistake on my part.

Yeah, it is the 50.

Let's take a look at this. Data set looks like no. Just so folks can kind of understand. So kind of starting here with some just basic input and output.

For the essential train set, we don't have any of the feedback in these rows that I printed out here. But you can imagine you can have different correctness labels here, explanations, any real validations. This data can be whatever it is that you'd like it to be.

Some folks use multiple evals. Some are just one. Sometimes it's human feedback. Sometimes it's a combination. But you really want to have, you know, the input and output that will use to evaluate.

Should my output of train set be the same as you?

Not necessarily. It depends on the.

I didn't know if head was sorted or not.

No, no, no. It all depends on kind of what the sample. We could look at, like, you know, if I did this, this should be the same for you, maybe. Just to make sure.

Yeah.

That's what you're saying? Okay. Yeah.

Quick question. Is it possible for the input to be, like, a chat history, not just.

Quick question. So I think it depends on, like, what it is you're trying to do. If you're doing just, like, a simple kind of system prompt with the input, you kind of want it to be one-to-one. You don't want to give it a ton of, like, conversation data that's not relevant to the prompt that you're optimizing.

We generally just use, like, the single input. But I think that there are applications that you could do, like, conversation-level inputs.

Yeah. Because quite often the failure is somewhere in the middle of the conversation,right? So if you put just the original task in, then the probability of you hitting, you know, a failure in the middle of the.

Totally. So in that case, what we generally see is, like, different rows of, like, having each of, like, the back and forths be, like, kind of independent rows. Because you're probably going to evaluate each of them. And honestly, probably, like, get the human feedback on each of them.

So we usually separate them out in that way. But it's a good point. Like, if you just always are focusing on the first turn, there's probably a lot of redundancy there. You definitely want to, like, sample parts of the conversation.

And how we can bifurcate, like, instruction versus we have some context also. So it should not touch the context. It should only whatever the manipulate the system instruction or the prompting. Context, it should be the static. It should not be, like, based on the answer, it will change my context.

Yeah. So what you're saying is, like, with the input, there might be, like, a dual volume of context you're kind of passing that in. You can absolutely include that in your data set so that the application kind of understands what other or not the application, but the prompt learning LLM can understand all of the data that's kind of, like, available.

So you can just have that pass in as extra column if you want. Most people start with just kind of input and the feedback. But you can absolutely add what other data you think is relevant.

And if you went for the rerunning when we're doing the experiment of testing, you'll definitely always want to have the data that would be required to answer.

Any agent, like, even very simple agent type, some tool call, some react call, or some context it is pulling, some API call. Whatever the prompt engineering, it should be based on the after getting the output,right? Whatever the context is contained in my prompt, plus whatever the tool call I have done, API call, all the context engineering.

And then last, final answer.

Totally. Yeah. So again, at this point, we're testing just, like, one prompt and not that kind of end-to-end. But you definitely want to have everything that, like, is flowing into the prompt that you're optimizing. So if your system prompt takes in the user input, for example, some data from an external API, you would definitely want to provide all of that data.

Does that make sense?

Because you're saying that, like, the, like, trajectories as you're going through the, like, tool calls and what the agent's going to do depending on what the tool call was is what you're trying to prompt learn to.

Yeah, exactly. You want to just, like, because we're kind of trying to replay and optimize one step of it. And we definitely don't want to do it completely in isolation. So if there's, like, data that flows into that prompt, that's context that's using that's producing the output,right?

So we want to make sure that we're including that. We don't want to exclude anything. But if it's data that comes, like, at a different step, probably not relevant, then you don't want to do it at that point.

It's just, like, think about what's relevant for the step that we're trying to optimize in this specific prompt.

Allright. Any other questions?

Allright. Cool. So we're going to set up our initial system prompt. You can see this is something very, very basic. We'll definitely, I think we can do a whole lot better than this. But just kind of want to illustrate something that we're going to test and optimize.

So we're just saying you're an expert in JSON webpage creation. Your task is input. And then so all these inputs that we're seeing here are going to be what we're actually generating outputs for and trying to optimize for.

Now, I already kind of touched on this. Evaluators are extremely important to make all of this work,right? So we're going to initialize two evaluators that use LLM as a judge to assess the quality of generated outputs. So we are using LLM as a judge.

If you have any other, like, code-based evaluations, whatever you need to do to evaluate, you can definitely swap those out. But we're going to do evaluate output. This is going to be a comprehensive evaluator that assesses the JSON webpage correctness against the input query and the evaluation rules.

It's going to provide an output label of correct or incorrect. So pretty simple binary. Again, you can use multi-label. And then it's going to have the detailed explanations as well. And then we have a rule checker. This is a more specialized evaluator that performs a granular rule-by-rule analysis.

And it examines if each rule was compliant. And then both of these are going to generate feedback that goes into our optimization loop to iteratively improve the system prompt, explanation, rule violations guide. And we'll get to this, the prompt learning optimizer and creating the more effective prompts.

So I have some imports here. Let's take a look at what the actual evaluate output has. So we do have some rules that are in

here. Wait. Sorry.

They're going to be in the repo. So we're going to open that as a file. We have this LLM provider. Again, we're using OpenAI here. And then we're going to do our classification evaluator. So we're just calling it evaluate output.

It's LLM. We have an evaluation template that we're reading from the file here. Then we just have choices, correct, incorrect. Now, we're mapping a label to a score. Sometimes it's helpful to be able to, like, average a score.

Sometimes a number is easier than just looking at a bunch of labels. It is optional how you want to map these. If you have, like, a multi-class use case, you can set the scores accordingly. But these are just going to be our choices, like the rails that we want our LLM as a judge to adhere to.

And then all we're doing here is getting a result. So yes, I have it doing some printing so you can kind of take a look. So this is going to be slightly different than what you're seeing in the notebook.

So I'm just going to pause here. If you want to make a code change to some of what you're seeing and probably your version, this is a good time for that.

Just kind of the setup of the evaluator makes sense to y'all, like kind of the keyboard. It's going to be the rails. It's going to be the output. And of course, our template.

Yeah, you will want to grab your own OpenAI key here to set.

And we can help you. If you want to use a different provider, we can help you swap this out quick if that is helpful to anybody.

Okay. I'm going to start walking you through the output generation. So this is just kind of, you know, you can imagine this as your own agent logic or the part that you're kind of testing. This is just going to provide a function that actually generates the JSON outputs.

We're using 401 here with JSON response format. So your temperature for consistent outputs. It's taking a data set, a system prompt, generates outputs for all rows, returns the results for evaluation. And it's called during each iteration to produce outputs.

So this is like our experimentation function that we're running. So as we're passing data, producing new prompts, we need a way to test it, evaluate, understand how we are kind of moving the needle here. So that's all this is.

So it's a pretty straightforward function just called generate output. We have that output model. Again, we're using OpenAI. If anybody wants help switching things around, happy to help. We are using response format because we are dealing with JSON here.

So we know that when you just prompt, I mean, some of the newer models are decent at it. But using response format is really helpful. And then we're also setting temperature to zero. And here is just kind of where we're passing all the data in.

So the data set, because again, we want to run this on all of the testing data, the system prompt that will be input. So as we get to the optimization loop, we're going to be passing in a new prompt to this with the data set and then evaluating.

We have our output model that we've already passed, concurrency, all that good stuff. And it's just returning all of the outputs there.

Would you, for the current generation of models, since this one's basically like in AI terms, ancient, would you, like, still recommend setting the temperature to zero? Or would you actually want to try to encourage some of the creativity in this?

I think it depends on the use case a little bit and what you're trying to do. You can definitely experiment that and kind of take it to the length of how important is consistency to use something like I feel like JSON webpage.

I feel like consistency is probably, like, temperature zero makes sense. But I definitely think not for every agent, every use case, do you want to use zero.

Any other questions? Keep it moving. Allright. Just a little note. So we kind of talked about before that we are kind of using some score mapping. This part is optional. You want to use the metrics that make sense to you.

We're not directly using this as, like, we are kind of, like, using it to know whether or not we optimize. But it's not like we're, you know, using this as our sole kind of indicator for the success. Here, we are just going to calculate some very basic metrics.

It's just you can, you know, choose something like accuracy, F1 precision recall, just some basic kind of classification metrics for us to understand. Again, because we are using binary mapping scores, we can do that. And so that's what you're seeing happen here.

We're mapping to binary. And then just based on the score, we calculate the metric. So very simple helper function here.

Allright. The good stuff. The optimization loop. We made it. Okay. So this cell implements the core prompt optimization algorithm. It's a three-part process. So we want to generate and evaluate. So generate outputs using the current prompt on the test data set and evaluate their correctness.

We want to train and optimize if results are unsatisfactory. Generate outputs on the training set, evaluate them, use the feedback to improve the prompt, and then iterate. So we kind of want to repeat until either the threshold is met or all the loops are kind of completed.

So if you remember above, we're kind of setting that to just, like, five loops. And then, you know, we can kind of repeat based off of that or if the threshold is met. It's going to track metrics across all the iterations, return to detailed results, including a train test accuracy scores, the optimized prompts, and the raw value.

So as I kind of mentioned at the beginning, as we're running these different loops on the experiments, we're going to be producing a lot of different prompts. And so we're kind of getting that information back that you can use.

And then these are our key parameters. I'll kind of go through them, you know, as we get to the code, but just to give you a heads up. This is the target accuracy score to stop optimizations. It could also be whatever other metric.

You'll see we have a score so you can kind of determine the number of loops of optimization iterations. We've set that score and then the number of rules. Again, these are some configurations we've already set.

Cool. So optimization loop. This is going to take in all of those, you know, parameters that I've mentioned there. It just kind of kicks off saying, "Hey, we're starting." It's going to do the initial evaluation so we understand how things are starting off.

Again, you can kind of pass in data too. You can kind of skip this initial evaluation. We're kind of running it at the start here. But if you were running a production setting, you might already have evals. You might already have feedback.

You can kind of adjust this for that. And then it's going to assess the threshold against kind of our initial evaluation. Again, this could kind of be skipped when we're coming from a production setting, but wanted to kind of start us off from scratch so that we can get a real feel for this.

And then it starts the loop. So we're generating output. It's setting that as the train output. So when I printed train, you kind of saw the outputs. I kind of stick the head there. And then it also will set, you know, correctness, explanation, any rule violations.

And then we'll actually use our prompt learning optimizer. So this comes with, like, the SDK, the prompt learning SDK that you can use with the RISE. So we're sending in that prompt optimization, the model choice, and then that API.

So under the hood, as we talked about in the slides, taking in that feedback, taking in the original prompt and trying to optimize to get better results, and then spitting out prompt. And then can also add an evaluator.

So again, those three kind of feedback columns we're looking to get back is correctness, explanation for that, if there are any rule violations. And then from there, we just kind of kicked off the optimizer and optimized with our train set, output those feedback columns again, and then, you know, any context size limitations you want to add.

Next step. So the optimizer again is going to take our data, produce a prompt we want to evaluate so we understand how we're doing, what this code block is doing here. So trying to get that new prompt running again with all those evals getting our results back.

And then we do that with our test set as well. And then we're getting back, like, our score and our metric value and then doing the checks. And then we repeat it all again till we either get above our threshold or we've hit the max number of loops and then returning our results.

So that's kind of what's going to be happening under the hood. I want to pause here. Any questions on that?

Just some results here and more. Helper functions here. So we do want to obviously save all these results. We don't want them just to be ephemeral that we can't ever access again. So just saving them all. You can also save off the single experimentation so you have all of that data towards the end.

We'll be able to kind of pull this and determine what the best prompt is. But these are just very basic helper functions. I don't spend too much time just saving them to CSV at the end of the day.

Now we get to the execution bit. So this cell runs the prompt optimization experiment, saves the results. We're getting the JSON format, the CSV format. It includes columns for the iteration number, the number of rules, test train accuracy scores, all the data that we're actually going to need to evaluate whether or not this thing is successful.

And then we're going to start getting results here. So this does take quite a while to run. So I think this will be a great point for discussion. But as you kind of are running it, you're going to start seeing the different loops, kind of outputs coming out as well.

And yeah, we'll just kind of, like, work through it as it runs. It's probably going to take, like, 20, 30 minutes for things to run. But happy to take any questions and help anybody out as they run into issues.

One small thing. Can you scroll back to the part of the code that we needed to change?

Oh, yeah.

Yeah.

It's going to be here.

Yeah.

So one reminder. Yeah. Looks are running into this and when it goes. So this line here, like when you're doing your pip install, you do want to be equals equals 2.2 because I think there's a little bit of a package issue.

So just make sure that you're hitting errors with the eval. That's probably why. If not, bite me down. I'll come try to fix it.

This is the reason why.

Uses, like, a generic evaluation promptright here.

Yes. And you can kind of see the evaluation prompt if you go to the. We've kind of just taken that part out of this, but we can definitely go through that. So if you look here on this line here, we're reading in

under prompts here. You can find the evaluation prompt if you're curious.

And this is the reason why everyone hates on Docker. This is why we use Docker.

Yes, absolutely.

It won't work a lot.

The notebooks.

The whole Docker is a screw.

So I would also recommend patching your code with Nestasync if you haven't already. Helps it run a lot faster. Also, for the purpose of the workshop, I switched our loops to one. That took me six minutes to run.

So I would recommend also doing that instead of adding five loops. Obviously, you wouldn't recommend doing that when you're actually trying to optimize your prompt. But for now, it'll help you get through the workshop.

Allright. I just want to kind of call out the last little bit here.

Of the last step

before folks head out. Let's see. Oh. So the last little bit of code here is just to extract the prompt that achieves the best test accuracy. So I mentioned how we're kind of, like, saving off all the results to use.

We just have a function that essentially gets the last or the best version of that, kind of showing you the original and then the best optimized version, which you can then use to pull and put into your code.

I did want to kind of just give one kind of call out. As you kind of saw today, doing this on the code can be a little bit difficult to manage. And so I want to call out for those of you who are kind of maybe looking for more of, like, an enterprise solution to this.

In a RISE, you do have these prompt optimization tasks. You can have your prompts living in our prompt hub data sets with all of your human annotations or evals that you can either create from traces or just by ingesting it into a RISE.

And then from there, all you really need to do is, like, give it a test name, choose what you want your training data set to be, where the output lives, where all your feedback columns are. You can adjust all of the parameters that you'd like.

And then from there, you can just, like, kick it off and it will produce an optimized prompt in the hub for you. So if I go over here, I think I have some. No, maybe not.

It will basically just create a new version here that says it's optimized prompts with all the results. And we are building on this. You can add all your evals to it, have that all running in the loop. I just wanted to call out that if you're not interested in maybe maintaining code loops and having to build, like, a task infrastructure yourself, it is something that we do offer in a RISE.

But yeah, hopefully, I know some folks are heading out. We'll be sticking around here for a little while as we can help you kind of work through issues. But thanks so much for joining us. Hopefully, you learned something useful.