Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

I'm Samuel, I'm probably best known as the creator of Pydantic, the open-source library. Now I run Pydantic the company. We do a bunch of stuff. We still maintain Pydantic validation, of course. We have Pydantic AI, the agent framework, and then we have Pydantic Logfire, our observability platform.

How many people coming into this know have heard the word Pydantic? Just to, OK, that's reassuring. And how many people have at some point played with Pydantic AI? Yeah, Pydantic AI? And Logfire? OK, so most people. Logfire is fundamentally under the hood: a general observability platform, open telemetry, logs, metrics, traces.

I don't really believe in AI observability. I think it's a feature not a category, and it will get eaten by either observability or AI at some point. But today we sell as AI observability because that is, understandably, the thing that people want.

So we go beyond the standard observability of logs, metrics, traces. We do stuff like evals, which I'll talk about a bit today, and managed variables is one of the newer features. And then the step beyond managed variables, which we're working on at the moment, is then optimizing your agent autonomously, effectively, from the platform.

That's not what I'm going to demonstrate today. I'm going to talk about GEPA and how you can use it with managed variables, kind of in a somewhat Heath Robinson way, until we have the full end-to-end all-single-dancing. You give us your money, and we magically optimize your agent.

Probably also more interesting than the "magically we make your agent better" argument anyway.

So yeah, there are two core subjects I'm going to talk about, and then I'm going to try and put them together at the end. So GEPA is a library. How many people here have heard of GEPA and kind of feel like they understand what it does?

OK, well, that probably includes me to an extent. So the name comes from genetic parado. So it's a genetic algorithm in the sense that it looks for the best

value of some kind, and then once it's found a good value, it kind of mixes it with some other values that appear to be good and produces a new candidate and tries to optimize that. So it's ultimately an optimization library.

It optimizes a string. Now that string can be a simple text prompt, or it can be some JSON data, which ultimately contains whatever you want. And the parado bit comes from it basically takes candidate from the kind of parado frontier of the best examples it has.

So it is effectively, if you imagine you're breeding racehorses, you take the best racehorses and you breed them each time. You don't, for the most part, go and take some really slow horse and add it into the mix to see what happens.

You take all of the best racehorses and breed them and hope to get to better racehorses. And it does a similar thing. And then the second is managed variables. So lots of platforms, including ours, have prompt management. So basically the idea that you can have your text prompt in the platform and you can edit it from there.

We take managed variables one step further. Sorry, we take prompt management one step further, and we have managed variables. So they don't have to be just text. They can be effectively any object that you can define with a Pydantic model, can be managed inside Logfire.

And so we're going to kind of try and combine those two things today. I will say at the beginning of this, due to some family situations, I wrote most of this talk overnight. So maybe in keeping with the rest of this conference, it may be slightly chaotic.

If what I'm saying doesn't make sense or I go to sleep at any point in this talk, please throw something at me and I will endeavor to make more sense.

So I'm going to talk quickly about the subject that we're going to go and optimize. It has to do with politics. Don't worry, it's not particularly controversial politics relative to what's going on at the moment. But how many people, actually, I won't do the how many people.

There's a podcast called The Rest Is Politics, which I'm a big, big fan of and a big listener to. They were having a conversation back in April last year, almost exactly a year ago, in fact, about how many politicians basically come from political dynasties, families of politicians.

And there was a lot of bluster about what the answer might be. And I used Pydantic AI to go and analyze the Wikipedia article for each MP to basically look for references to their relations who were also politicians, and therefore was able to come up with some percentage.

I think it might have been 24% of MPs have some kind of ancestor who is a politician. But at the time, I basically just ran it with whatever model I could get, got an answer, submitted my question to The Rest Is Politics.

They read it out. That was very nice. But I didn't actually go and check how well it had done. And so here we're going to take that challenge and try to optimize it to improve the prompt that we use.

The subtlety, let me go into some code here and talk through the task. So hopefully this is a scale you can read.

There's a bunch of fluff here. In particular, there's an archive file here that will be automatically downloaded if you run the script, so you don't have to go and do all the scraping from Wikipedia, which just contains basically the raw HTML of the Wikipedia pages.

We then have some schema for the MPs' data, and then we have schema for their relations. So in particular, so this is obviously the name, the role they might have had, in particular their relation. So whether they're a sibling or a spouse or a grandparent.

And in particular, with regard to the question that we're trying to answer about kind of political dynasties, it isn't really relevant if someone has a child or a spouse or a sibling who is also a politician. It's to do with their parents' generation or parents' parents, basically ancestors.

And this is something that I found really hard to get models

to respect. Once you said relations, it just couldn't help but add spouse or children or sister-in-law into the mix. And so the way I actually went and solved this last year was I basically said, include the relation, whatever it is.

And then afterwards I went through and scanned and removed the ones which were obviously the same generation or beneath, as it were. So here, one of the things we're trying to optimize is finding a prompt which will get the agent to efficiently discount, yeah, non-ancestors.

So this is a schema we have. We've got a task input, which is relevant. We have some very basic initial instructions for the agent. We have some slightly more advanced instructions so we can kind of see how the two perform.

And then ultimately we're going to try to optimize to find an even better prompt.

So the core of this is this Pydantic AI agent. It's actually just using a hard-coded model. It's a variable because in theory you can substitute it with a command line, but I don't think we need to today. The most interesting bit is that the way that we're doing this is effectively a structured output question.

So this could be, this is equivalent to any other structured output: find the address in this email, find the invoice lines in this PDF, whatever it might be. But we're using the same idea of structured outputs to go and find this political relation.

So we're just substituting in, yeah, so we're just setting the output type to be list of political relations. We set the instructions from above. And then ultimately when we go and run the agent, all we're doing is taking the HTML, doing a little bit of processing with Beautiful Soup to get out the text so we don't have to take all of the other crap going on in the HTML page, and then we're passing it to the agent.

And this works surprisingly well. If you go and read through the cases, you will see that it is pretty damn accurate at finding all political relations. Even the relatively dumb models do a pretty good job. The main place where they get confused is, are their relations political versus otherwise public figures?

And this idea of a relation being an ancestor seemed to really confuse the models.

You will see if you look in the rest of the files here, I'm sorry if the file explorer is small, but once you run it once, you'll have this MPs directory, which contains a JSON file with the MPs and the URLs and the raw data, and most importantly of all the pages, which is the HTML pages.

As I say, that will just be downloaded and put into that directory the first time you run it.

Yeah, and then there's also this golden relations. So this is supposedly the exactly correct answer for each politician. So

Stephen Kinnock has, I mean, for anyone who's British will know a reasonably famous political name. He has a bunch of political relations. You see here that, again, it's included his wife, who was the prime minister of Denmark, even though I'm seeing some questions there.

I believe that's true. Even though we said don't include relations, that's why I've used this technique of basically including those relations and then stripping them out based on what the relation is,

et cetera, et cetera. And I believe this is pretty accurate. Truth is we ran it, we just ran a similar script with Opus 4.6 to get that data, but I've checked it quite a lot and it appears to be pretty much correct.

And so in terms of the first step here is to run some evals against this and

demonstrate the performance relative to the golden dataset. But before we do that, I'm just going to run it, run a single test case in the terminal and show what's going on, just so that you have some feeling for what we're doing here.

So I think if I look in task here,

yeah, if I go uv run task two is the idea of the first politician for some reason, Stephen Kinnock. We'll go and run the model and successfully find the two relations. And in this case, we've prompted it and said

don't include, only include ancestors. So it's done that correctly, and it's excluded his wife.

I mean, I guess people would prefer to run along with this and try this themselves. Is that true? OK, so to do that, you will need, and

ideally you'd have a Logfire account that will allow you to do the managed variables stuff. You can use a free Logfire account. The free tier is extremely generous, so everything you're doing there is free. The other thing you need is obviously an API key to connect to the models.

You have two choices there. Either in the code, we'll basically be using Pydantic AI Gateway as a simple way of being able to access all of the models. So you just see Gateway prefix in front of the model name.

If you have your own OpenAI or Anthropic key and you want to use that, just remove the Gateway prefix and it will work. If you want to use Pydantic AI Gateway so that everything just works, I'm just going to generate an API key in Logfire and share it with you all, hence that channel, and put a limit on it and hope no one goes and uses it all up before we finish the lesson, finish the workshop.

So I'm in Logfire here. I'm going to go into Gateway. If I had set this up more formally, we could basically invite you all and you could all go and generate your own keys and you'd all have your own limits.

I have not prepped that, and so I'm just going to generate one key with a $1,000 limit and hope that does the job today, and then delete it after the workshop.

So I'm just going to copy that into

this channel here. So you should just be able to export that environment variable. And then when you run your code, for example, this task here, you've run task two, you should get an output there.

Should I wait a couple of minutes for everyone to, the other thing you'll need to do is set up Logfire for when you run the evals. And the simplest way of doing that is once you've run uv sync in this, so you'll be in this directory, you'll run uv sync.

So note we're in the sub-directory for this talk. We'll run uv sync. And then to connect it to your project, you'll see here my project is called demo. So I'm going to run uv run Logfire projects

use demo, demo being the name of the project that I want to connect it to. And that will go and connect Logfire in this directory to the demo. So when I go and run stuff, it will output, record traces to Logfire and use the managed variables from there.

I'll wait a couple of minutes because I'm sure people want to get a bit set up and get that kind of very simple hello world thing working, and then we can move on. Anyone got any questions while we're just getting that set up?

Yeah, what does gateway mean in the model?

So AI gateways, we have one, but by no means the only ones. The idea is you can basically make requests to most models through our gateway with one API key. And in particular, we have observability. So if you're using the production version, you can see requests that your team are making or that your agents are making.

We have things like caching and fallbacks. So if one model fails, we can fall back to another one, et cetera, et cetera. But in this context, it just means you can have one API key and connect to Anthropic or OpenAI or Grok with a queue or Gemini models.

So gateway is your own product?

Yes. But as I say, you don't have to use it here, but it might just make things easier to get all of the models working.

I'll wait a couple of minutes, but any questions or should I go on?

What was the directory name?

Sorry?

The directory name.

So it's the repo itself

is

GitHub Pydantic slash talks, and then the directory is 202604 AI engineer.

It wants me to authenticate through a browser even though.

Have you logged in with Logfire before?

No.

OK, so you'll need to do that so that you have the machine key.

So I think it's uv run Logfire auth. Sorry, I've done that.

So I'm going to keep going just to keep things moving, but unless anyone has any questions. So we now have the very basic case working. What we want to do is run an eval to see how

our model with our prompt is performing relative to our golden dataset. So you will see if we go into evals here, we have,

where's the dataset defined? Sorry.

You'll see at the top, we have the load dataset, which is going to basically create

our dataset, which is the object that we can then call eval on. This is going to load

the cases and basically create one case for each of the MPs that we're going to run. And then it's going to create, yeah, go through and generate these cases and ultimately return this dataset object. And we're going to register one custom evaluator here, which in turn actually generates a whole bunch of metrics or assertions for how accurate

this run is.

I don't think, I mean, you can go and read through the logic. I'm not going to go through each individual bit. I'm not claiming here that these are the perfect set of evals in terms of the actual evaluators, but they're a useful start.

The point is, you can look through the docs, but we have a bunch of prebuilt evaluators like LLM as a judge, and then we allow you to define your own. And generally, defining your own is far better than the LLM as a judge because the LLM as a judge is effectively the kind of, I don't know if it's politically acceptable, but the kind of lunatics running the asylum, and it can lead to slight issues.

So if you can have a deterministic eval like this where we're comparing what the result is versus a golden dataset, that's much better.

And so if you look in main where we're going to go and run this, ultimately main is really just a CLI. So we set up a bunch of CLI cases, and then in the case of eval, we run the run evaluation function.

If you look in here, ultimately what we're doing, print out better stuff, and then we're just going to call this evaluate function. We're using Pydantic AI's override functionality to override the prompt that is set by default

with the custom prompt that we're using for that particular that we're evaluating and a model that we're evaluating. We're not actually going to change model in this case.

And so we're going to, and we'll then go and call, so ultimately we're calling dataset evaluate, which takes a function, which is the actual function we're going to go and perform the evaluation on. And that will in parallel with a max concurrency of five, go and run all of our cases, which you will see in a minute.

And then we're going to give it some name so that we can look at it in Logfire. Now, you will also see here we've set up Logfire at the top here. If token present just means it would run without causing an error if you don't have the token set.

We set an environment, service name. We're not going to print to the console because in these complex cases, it's actually quite tricky to see what's going on in the terminal. It doesn't really help to have Logfire kind of printing to the terminal.

And then we've switched off scrubbing because occasionally something like the word password or auth or something will appear in an output, and we're happy for that to go to Logfire. And then we're going to instrument Pydantic AI. We don't actually need it now, so I won't instrument prints.

But when we get to using GEPA, instrument prints just as a nice way of getting the output from GEPA into Logfire as well. And so if I look, and as you may have realized, the help file has help.md has some useful examples of what we're going to run.

So we're going to go and run this script here. So we're going to uv run main.py. We're going to call with the arguments eval. We'll split. So we're splitting it based on just the test set that we're running.

And then the prompt is the initial one, so the very simple prompt, kind of one-liner description of what the agent should do rather than the more complex prompt. And if we go and run this eval, you won't see that much in the terminal because we switched off terminal, but you will see it running through the cases.

And if we go and look in Logfire, you can see those cases running and their costs, et cetera. We can dive into what an individual case looks like. So if you look here, we're running

the agent, and I guess that's probably quite small, but I will try and zoom in to let you see what's happening here. We have the system input. Again, that's the simple instructions that we talked about. And then we have the input, which is the contents of the Wikipedia page as text.

Then we have the final output, and in this particular case, it found no relations.

That has now finished. And so as well as this view of the individual cases here, we can go into evals in Logfire, and you will see this case here. And you will, sorry, this dataset, and then within the dataset, you will see that most recent case from one minute ago.

And if we go in and look at that, we should get a view of what's going on. So again, this is pretty small on the big screen, I guess, but it'll look much clearer on your screen. We can see where the cases have, the most relevant metric is accuracy where you'll see there are ones where it's correct and zeros where it's incorrect.

And you'll see a few values where it's lower, where the output from the golden dataset is slightly different to the output

from the model that we're running here. So you see here this guy has a father who is something to do with UKIP, and it's got it correctly, his name, but there's a slight difference in the description, hence it's like got a 0.9 score.

And you can see the overall performance of 85%. So 85% of the time, it got all of the stuffright. That's not particularly good, but it's also not particularly interesting

until we go and compare it to running with a better prompt. So how many people are up to kind of being able to run that eval? A few people. OK, I'll wait a couple of minutes because I think it's probably slightly more entertaining for us to, yeah, we're 20 minutes in.

We've got quite a lot of time. I'm going to wait a couple of minutes.

Can you show the command again?

Yeah. So it's this command. If you look in help.md, we're basically running the eval command.

Can you look?

It's taking some time.

Yeah.

Does it take about 10 minutes?

Running eval?

Yeah.

It took.

Relations 9% seven minutes ago.

It ran really quickly for me.

Me too. It took 30 seconds.

I think it should take about 30 seconds. So I don't know whether you're on a different internet or

changed a model.

Do you think it's on that?

Ah, yeah. That will be much slower than I think we're using. Well, A, Anthropic was down this morning when I was trying to use it a lot. So it's being really, really slow. And GPT-4.1 is flying because not many people are using it, and it's doing a reasonable job.

That's why I'm using one of those models. It's partly to show the eval performance, but also bluntly to get the answers more quickly.

Yeah.

While people are getting there, I'm just going to run the next one, which is compare. If you go and look at what this is doing, it's basically just running the eval twice with the bad prompt and the good prompt, and then we'll be able to compare those outputs in Logfire and kind of see where the differences are coming from.

And then we can go on to the full GEPA optimization step. I'll just run the compare, which should take a little bit more time, but still hopefully relatively quick here with these models. You see we're just running the 65 test cases here rather than the full 650 MPs as we'll use when we get to the full optimization.

If you accidentally run the same command twice, you'll just create a new.

You'll create a new, obviously you'll have more spans in Logfire, and you'll get another run in the Logfire evals view. You can archive them when you've got confused by how many different ones you've got. So if you come over here to evals, come into that most recent case, you'll see I've now got a bunch.

I could go and archive, so yeah, archive the older ones. Do it the other way around just to keep things clear. But if I take those, so I've now got two new runs here, two new experiments, and I'm going to select them and

compare. And now we have a bunch of different metrics

in particular. And you will see that ultimately accuracy being kind of the most important one here, slightly higher, 92% versus 87% for the expert prompt versus the initial prompt. And we have the possibly slightly useless graph of different metrics and how they've performed.

With fewer metrics or assertions, this can be more useful. And again, what's probably most interesting though is to look into individual cases where the performance differs. So you see here with this MP, Joe White, that the initial prompt found one match

relation spouse, which obviously it should not have found because we're not looking for spouses, whereas the expert prompt correctly ignored that one. And then if you look in this case, Laura Kyle, you will see that the expected output is no matches at all.

Both of these most recent runs, they found a match because they had some four times great grandfather who was the governor of Hudson Bay, which is not a politician, but is a public figure. So you can kind of start to see where the differences are coming from and the kinds of things you might want to do in the prompt to improve the performance.

That's ultimately the kind of data that GEPA is going to use to

inform it, generating the next Pareto frontier prompt for it to go and try.

I didn't manage to see the graphs.

You will see the graphs only if you're comparing two cases. So if you open a single case, it doesn't show the graphs.

Two cases.

So I've run compare. Once I had run eval, I then ran compare with this command here. And then

here I've taken two cases.

Oh, you select compare. Sorry.

And then you hit compare, and then you see the graphs comparing the performance.

Oh, cool.

It wasn't quite clear what you compared it to. Is that the expert prompt?

So if you look in the task.py, there are two prompts as the kind of starting point. So one is a very simple one-liner, and then the second is a bit more detailed description of the kinds of things you should be looking for.

Still not kind of optimized, but what you might get if you, as a human, wrote out a decent prompt or asked a model like, "Write me a better prompt." So you can see we've improved the performance from 87 to 92% with this model through doing that approximately.

And we should, with GEPA, be able to get better. I think I got 96% when I was running it earlier. By no means perfect, but this is, you know, but there are examples of using GEPA to, I suppose what people generally do is they either hold one variable constant, like we need the following quality, and now we want to reduce time or reduce cost, or we need to drive up the quality.

So there was an example from Shopify using GEPA. They were basically looking at

Shopify sites and looking for things like whether they were fraudulent or which tax category they fell into. And they switched from basically just giving the entire website to GPT-5 and saying, "What is this?" to using an agent and using a Qwen model and GEPA to optimize the prompt.

And they got the price down from $5 million a year to $73,000 a year and improved performance over time. So that's obviously also, that's not just optimizing the prompt. That's like some mixture of optimizing the prompt and moving to Agentic and yada yada.

But in this case, we're just trying to kind of improve performance whilst using the same relatively fast model for the sake of the demo.

So now we've done the comparison. I guess it's time to look into, well, anyone want me to wait a bit longer, or should I keep going? I'll take the no one asking me to wait as a signal to keep going.

So we now get to the kind of GEPA phase of this. So I'm going to re-enable that while I remember. And then ultimately, if you look here where we are going to run the optimize case, you'll see we have this run optimization function, run optimization here.

It's going to load in the training and the validation datasets. Error if it's not there. We're then going to go and create this GEPA adapter. So the GEPA adapter is effectively, GEPA is not, it's an amazing, I guess the state of the art now in agent optimization.

Lakshay, who created it, is a first-year PhD student at Berkeley. He is not perhaps the most experienced kind of engineering Python developer. And so GEPA doesn't do async that nicely. It's not as type-safe as I would like it to be.

I'm like itching to go and either fix it for him or fork it bluntly, but I have so far not done so. But yeah, so GEPA has this concept of adapters, which are basically how you define the agent that is going to propose the next case.

And so we have this function here to create the adapter, which in turn generates one of this adapter here, which we've defined, which subclass is

the adapter type. It's a data class. It takes a bunch of arguments. But ultimately, the bit that matters is build propose agent, which is obviously their method name. And what we're going to do here is return a Pydantic AI agent in turn.

So we're using a Pydantic AI agent to propose new cases to optimize a Pydantic AI agent. And we're going to use again GPT-4.1 for the sake of speed in this case. There is a slightly annoying characteristic where GEPA is sync, not async.

And so if you try and run, each time it runs the proposer agent, it runs it in a new

async context. And so we have to go and make sure we create a new HTTP connection, HTTPX connection. Otherwise, we get a bunch of errors, hence this slightly weird formulation here. We have again a prompt. Obviously, you can get a bit going around in circles on optimizing the prompt of optimizing the prompt, but we're basically saying, you're an expert prompt engineer, improve the system prompt for this agent, yada yada yada.

Here are the things to consider. And then we're going to call the ultimate evaluate function, which is going to

generate this bunch of cases, generate a Pydantic AI eval, run that eval, and that is what we're going to use to basically test the performance of our new proposed prompt. This will probably make some more sense when I actually go and run it.

Yeah, and then we

apply skills for the failures, and then we return this evaluation batch, which kind of is a summary of how it's performed, which is what GEPA will then use as to whether or not this is a new Pareto frontier or whether to propose a new prompt.

So I think the simplest thing to do is just go and run this. It will take a little bit longer, and it will cost a little bit more, but that's fine. I'm going to run it. If you run it with 50, it basically dies before it gets anywhere.

But if I

run this with 400 calls,

you will see

GEPA starts to have a similar kind of progress bar. And as I said, I've switched off Logfire printing so that you don't see what's going on in the background here because it basically just becomes impossible to view. But if you look inside Logfire, you will see already this optimization is going on.

So you can see that we

evaluate the agent, and then we call the proposer agent here. So the proposer agent, if you look at this step here, this is what we just had, the system prompt I said for the proposer agent. And then the input, which is basically GEPA's description of the context for the proposer agent to enable it to come up with a new prompt.

And then from there, the proposer agent is going to propose a new system prompt. And we're going to then evaluate that, see how it performs, and iterate towards finding a better solution. So that's what's going on here. You see why I did the instrument print is so that we can get the print output from GEPA nicely viewable inside Logfire.

There may be a neater way of doing this, but this worked for me this morning. And it's going to work through. You can see it's a little bit more expensive. We've already spent $2.

So I guess if everyone runs it, we will spend a few hundred dollars. OpenAI will no doubt appreciate it. And that's going to work through. But you see we're already like 50% of the way through the optimization, and it is proposing these new prompts and then

evaluating them and using the feedback from the evaluation to decide where to go next. And I would encourage you, if you have some time, to go and have a read through how these traces because they're quite interesting. They're quite illuminating in terms of how the optimization is working.

I think one of the things to note is people love to say that anything to do with AI is incredibly sophisticated and complicated and advanced. This optimization technique, whilst the state of the art, is not actually that groundbreaking.

Relative to the complexity of the model itself, it's a relatively crude sense of ask an agent to generate a new prompt. If it does better, take bits of that, put it into a new prompt, keep doing that until you either run out of time or get to some prescribed score.

Yep.

I have a question. How big the batches are? And do you have an agent skill for doing that?

How big the batches are? So we have.

Does the agent process

the prompt that is in the user prompt of the agent? How many evals does it run? How many prompts?

So I think you can see it here where it's running the evals. So it's running, I think it is running all of the cases in the test dataset.

All of them.

But I think I've seen it run fewer. So I think GEPA is clever enough to alter the number of cases based on how they performed. But I'm honestly not that sure.

Not 100%.

Yep. So in terms of skill, what was the question?

Just thinking how to reproduce that through optimizing some prompts in my agent workflow. So I was thinking whether you had a synthesized, a premade skill that could just.

I mean, I think we will make this much easier with open source. We will also make it something where you just plug in Logfire and it just happens in the background. But we also want to make it super easy to do it locally or open source as well.

It's like we don't just want to say, oh, you must plug it into Logfire to get optimization.

Thank you.

Yep.

Now you just said the max calls,right, which is, I think, max calls for OpenAI?

Yep.

Is there a smarter way to do this? Like if you don't improve for 10 iterations, just stop.

I'm sure there is. I haven't looked into it. Like I say, I did this in a bit of a hurry. Totally. And presumably, for the most part, you're saying reach this threshold, or as you say, run out of optimization or whatever it might be.

I don't quite understand how you see the prompt optimization.

So if you look here in the proposer agent run, you will see effectively

what input, this is the input to the agent that is

proposing a new prompt. And it is being told, based on this information, propose a new prompt. And then if you look, the principle of the Pareto frontier is this contains components of previous prompts that have done well. And in fact, it's a bit like the.

It's not like a diff. It's a whole new.

Yes, it is proposing a whole new diff.

Thank you.

So I think if you have multiple, so I think GEPA operates on the idea of a kind of object or dict of known keys. And so here we only have basically one, a dict with one key value pair, e.g., like system prompt value.

If you have multiple different keys, then you can optimize by basically combining the best values from multiple different keys. So you could have, you could imagine a model and system prompt or model and some set of tools or even different lines from a system prompt that you're basically

allowing it to, the proposer would just be choosing different values from your set of different lines. That's the kind of standard way of doing this in DSPY is like basically take all of my different examples and choose which ones to include in the prompt.

What is DSPY?

So DSPY is another agent framework, somewhat similar to Pydantic AI. It comes from a very machine learning background. I find it hideous to use because it's not type-safe, but the caliber of people using it is generally very high.

And it has had this concept of optimization for a long time. For a long time, it was basically the only agent framework that had this idea. Now GEPA has come along and is available within DSPY, but is not only available within DSPY.

Yeah. What's your personal take on GEPA and those types of algorithms? From your personal experience, have they worked well in practice? Have you all used more home-built optimizers? Yeah, curious on this informal take.

So I think that in the case where you're trying to basically get a dumber model or a faster model or a cheaper model to be able to do some task, they make a lot of sense. If you go and take the state-of-the-art models are like Opus 4.6 and you ask it most questions, it will just, if it has all the information it needs, it will just go and get itright for the most part.

And so the optimization is slightly less relevant. I think one of the subtleties here is that the statistic is that 98% of data is private. I don't know if that number is correct or not, but let's say even if it's off by miles and it's 50-50 data.

When you have a private set of data where the models have not been trained on it, you have some massive internal spec for how you're supposed to operate as a bank, let's say, adding theright bits of context into the system prompt or into the instructions is incredibly valuable, and the optimization really matters.

The problem we have when we're trying to give demos like this is by definition, there isn't private data that we can use. Because if someone lets us use their private data, they're not like, oh, yeah, can we go and talk about it at AI engineering?

So I've got this slightly weird example of MPs, but actually, you'll see in cases, stuff I'm going to show you in the middle in a minute, the model actually just goes and figures out who the MP for some borough is, even if it hasn't looked at the data, because actually, this is all public data that it knows.

So I think that one of the subtleties is that optimization matters way more where you have a large amount of private data. And honestly, we're still trying to figure this out as a company. One option for us is to use stuff that is public, but new enough that it's not in the training data.

So one option would be like news, but then that could get slightly politically dicey if we were using current news as our source of stuff to optimize on. So another option is to use what's going on in sports, because whether you find it interesting or not, at least it's not controversial.

And we would ask it which footballer has done well in this Premier League season over the last five matches. And that might be public data, but equivalent to private in the sense that it does not exist within the model, but it's not a really good example.

It's something we're trying to figure out.

Yes. I have a question regarding GEPA. Does it.

I think it's not working your mic, but if you just speak up, I think we can keep going.

Yeah. I'm just trying to understand if there is a way where it prioritizes kind of optimizing and adjusting the system prompt rather than kind of appending to it. Because I feel like that's a recurrent issue. You find new edge cases and you're appending and adding more information.

Yep.

So does it strike that balance of, OK, this we append and this we adjust?

So I was using GPT-5 Mini this morning, and one of the problems, A, it was slower in general, but also it was just producing this enormous system prompt, which then meant that the agent was slower. I don't know if it did better or not.

I think it did roughly similarly well. So that definitely is a problem. I think the solution to that is the thing I was talking about earlier where ultimately GEPA wants a key value store, a dict of keys and values.

And then instead of basically saying, instead of the proposer agent being like, go just generate me a new system prompt, you're basically saying, choose a new subset of our list of different inputs available for like, you would go and basically split your system prompt up into like 200 sentences and say which is the best 20 sentences to use.

And now by definition, if it can only choose 20 sentences, it's not going to get for both. In this case, we're using a Pydantic AI agent to basically summon up a whole new system prompt, which is probably better at finding the edge cases, but does result in a verbose prompt.

So in fact, on that very point, it has now finished. It has given us this optimal prompt. It has achieved a performance score of 96.7%, so significantly above the 92 we got from our best case before, and at the expense of a relatively verbose prompt.

So you can see where it's gone through in relative detail. And in particular, it's banged on about what relation to use, when to include a relation, what sort of things count as a relation.

Yeah.

OK, so that is GEPA working. The other bit I was going to talk about is the managed variables. So I will now switch onto the managed variables. But before we go onto that, any other questions on this bit of the talk or on GEPA?

Yep.

This is just probably me hard of hearing, but you got the validation scores compared to a golden set,right?

Do you want to take that?

The validation scores comparing to a golden set,right?

Yep.

The golden set in this repository, it is here,right?

Yep.

Oh, OK, OK.

Yeah. So if you look in the cases directory, you will see golden relations. And there's a JSON file with the supposedly golden set. Now, I'm sure it's not 100% perfect. As I say, we generated it with Opus 4.6 using a similar script.

But.

I just had a silly thought. If you didn't have that golden set, then things get quite difficult, doesn't it?

Yeah. Yeah. So evals are much easier if you have some golden reference against what it's supposed to be doing. In general, what people end up doing is they have some subset of data that's been human annotated that works.

But you can also have cases where, for example, if you have an agent that's writing code, you go run the code and check whether or not the code fails, for example. Or if you can have a feedback, if you can have a full loop of

basically returning that data to the original source and seeing how accurate it is, something like that. Working out what your judge is is the hard bit of evals.

But just for your example,

if you were running the code, that would be like it either runs or doesn't,right? It's not like a how do you do a percentage then? I'm a little bit confused there.

Well, you could check, for example, whether or not it was going to generate invalid code or use libraries that weren't available. That would be a first step of how performance is going. But yes, I mean, this is the hard bit of evals is like what isright.

And as models get more intelligent, figuring out whatright looks like is harder and harder. I mean, the ultimate case,right, is like you have an agent whose job it is to persuade people to stop smoking. The ultimate eval is wait 40 years and see when they died.

But obviously, you can't have an eval where you wait 40 years to see if they died. So your eval is probably like, does it include the correct words? Does it not include some bad words that you definitely don't want it to suggest?

Like take up a cigar instead,right? So text does not include cigar would be one of your evals, perhaps. And that sounds dumb, but that is what an awful lot of evals end up being.

Can I, yeah, sorry.

Two questions. One is a very short one. With Pydantic AI, you kind of break up the structure a lot of the time in the prompt. Does it just jam everything together and you just get one prompt at the end?

So you always have one set of instructions.

Yeah.

So it is building one set of instructions.

Right. OK, that's fine. So the real question was, variance is huge with this. I mean, you showed your results. I ran it. I got results. Is there a way of having the variance be part of the actual optimization?

I wouldn't say that. So when you run evals, I'm just looking for where we call run, you can basically set the number of times you run each case. And the best way of reducing variance is to run them lots and lots of times.

I know a bunch of hedge funds who are spending about $20,000 a night running their whole set of evals to see if they can improve over time. And if you have your own GPUs, you're like, well, it's sort of free to go and run them, so we should run them more and more times.

We should run everything 100 times and look in the morning. It's definitely one of the challenges.

And then do you create a cost function, or do you have like, you can point it out, like I want you to optimize both accuracy and minimize variance. Do you write your own cost function then within?

Yeah. So if you look in the adapter code here, you will see, I mean, have a look through it, but where am I looking? Evaluate is returning this evaluation batch type, where you can basically include whatever scores is, a list of floats,right?

So ultimately, you can return whatever floats you want in there.

OK.

And just in case people are wondering, trajectories is effectively similar to traces. It is the sequence of different steps that agents went through to get to a particular point.

So how do you handle systematic errors? I ran the optimization and it got a high value of accuracy, 96%, but it got wrong once and uncles. So the final prompt excludes explicitly once and uncles, even though they are in the golden relation.

So do you suggest running and Cloud Code spotted this immediately?

Yeah.

So maybe one could have a harness where there is another check at the end.

I think what it's doing is it's using a small set of test cases and then a bigger set of evaluation cases. And so the subset of cases that the optimizer agent gets exposed to maybe doesn't include uncle and aunt, and so it just made it simpler to exclude it.

So one of the other things you end up needing is like a kind of at least 2x the amount of data that covers all of the different space so that you can have one full set of data that you train on and one set of full data that you evaluate on.

This is like classic problems of machine learning of how do we go about we need an awful lot of data to build something that's really reliably, yeah, really reliable. Because ultimately, you're already getting to the point with the agent where you're kind of encouraging it to overfit.

And I suspect the uncle and aunt thing is it overfitting.

OK, thanks.

There's a question behind you.

This is probably a very silly question, but short of the prompts, can I assume that the prompt only works with the particular model that you would.

That is people's take is that you would need to run this all over again if you changed your model.

That's going to happen all the time, isn't it?

Yep. And that's one of the reasons that evals are hard. And that's one of the reasons that most people don't run evals and don't run optimization. They write out a decent prompt. They ask their coding agent of choice, does this prompt look good?

If it says yes, they kind of eyeball it, and then they put it into prod and they worry about other things, because probably the model that comes out in a month's time is going to go and supersede whatever optimization you did in some cases.

Now, if you are a private equity firm who have 200 million invoices from across your portfolio that you want to go and analyze, it's worth going and optimizing for a particular version of Qwen 3.5 Lite Mini to go and do that job instead of just throwing GPT-5 at it.

Because the difference is like $10 million and it costs you you have your one analyst who earns half a million dollars and you put them on it for three weeks and you solved it. So it depends on the question that you're trying to solve.

But the ultimate example of this is the coding agents, where the trajectories are extremely sparse, as in there's an extremely wide range of things that they're doing. And for the most part, we are told that those companies don't have very many evals or don't have any evals at all.

If you go and ask Boris Cheney, how does he work on Cloud Code, he says, mostly vibes. Mostly we just see what works and tweak it a bit. So this is not a I'm not claiming this is a panacea in all cases.

But there are definitely situations where being able to optimize your prompt or your agent choice is valuable. The other thing that we don't have a demo of here, but there's an awful lot more than just the prompt you can go and optimize.

So there's the model. There's things like the compaction strategy. There is how you register tools. There are things like including code mode in this. And the next step of what we want to allow is basically for you to go and optimize across the full range of things you can do within your agent to find the optimal choice, where perhaps the particular model choice doesn't matter so much.

And it just makes me think that you evaluate the prompt here, but could we just switch a row a bit and start evaluating different models, or that would be silly?

Yeah. So that would be something you could totally go and do is try different models. I mean, given that there are maybe you have 10 models you want to try, you probably just run all those 10 models and see which one performs best and pick that one, or which one optimizes for your particular subset of performance, price, time, et cetera.

I think there was another question behind you.

Yeah. Thanks. I already got the mic. So what you've shown is a task which is relatively narrow in what should be achieved. And I'm wondering how you would approach it if the task is more open-ended.

I think that my suspicion is that

that is a case where this optimization technique is harder to do. Because like I say, the coding agent case, you could imagine you end up with 5,000 distinct tests for a coding agent, and you go and optimize as much as possible in those 5,000 cases.

And it turns out that they are a drop in the ocean of different things people do, and now your agent is over-optimized for your 1,000 cases or 5,000 cases. I think the bigger the breadth of the task your agent is performing, the harder it is to do this, and the more you end up just having a cleverer agent.

Thanks.

The other thing to say is that my point earlier about where you have large amounts of private data, and so you have to put masses of context into the system prompt for understanding a specific domain, then choosing theright examples from that to put in, which basically cover the full data set, is more and more relevant.

Did you have another question?

So

if I understand correctly, if you're a company and you would use this GEPA optimization technique, and if you had more money and maybe more expertise, you could decide to fine-tune a model. So they are kind of competing strategies.

Correct. Yeah. And in answer to the previous question about, oh, but isn't it made obsolete by the next model, the main reason that the big model labs say, don't bother fine-tuning, is that there you really are spending tens of thousands of dollars to go and fine-tune a model and actually, for the most part, improve the harness, wait for the next model, show a nicer loading icon to your users is probably better than fine-tuning.

But that misses the cases, in particular in places like finance, where they have enormous numbers of runs where they really do care about that optimization, and fine-tuning applies.

Thanks.

So I'm going to go on to talk about managed variables. As I said earlier, this is a somewhat

it's not going to be a particularly sophisticated answer I've got here, but I'm going to kind of show the example just to explain how it works. So we have here a very simple FastAPI web server. You will see that it has ultimately two endpoints.

One returns some HTML, and the other one is a form submission endpoint. And if I go and run this,

where have I had it running before? Here we are. It was, in fact, running. So if you want to run it, you'll probably want to use this command here to run it. I'll put that into Slack as well in case anyone wants to run it.

And if you run this web server, you will

see Logfire, which is not so interesting yet. But you will also see the server running locally. And this is just a very simple, basically, form where you can ask questions about the same bunch of NP data. And I can go and ask a question like, we'll see how well this performs.

Hammersmith, where I live.

It will run for quite a long time because what it's got in the background is basically one tool which lets it grep through the HTML pages. And it has replied relatively quickly on this occasion. And if I just load this here, you will see the HTTP request coming in in Logfire.

And then you will see the agent run. And it ran a rather

good, sorry, good regex, got back the data, and was able to correctly identify Andy Slaughter as the MP for Hammersmith, where I live, all very well. But if you look in how this code is defined, we have here this Logfire variable.

And so this allows us to basically go and change the behavior of this agent without redeploying. Now, obviously, redeploying in this particular case where I'm running locally is very, very simple. But where I've got some big production CI stack to go and run and deployment takes hours, being able to go and change variables in production or in staging very quickly is extremely valuable.

So the way that Logfire variables work is that we have a type here, a Pydantic model, which has in turn three strings, the instructions, the model, and the max tokens, which is an integer. And then we've given it a default value here, some very simple prompt.

And then we've chosen the model and we've chosen the temperature. And what I have run here is I've run uv run web. And then basically, there's a CLI command for this, which just pushes the variable. Ultimately, we're just calling Logfire push variables in this case.

When I run that, I'll get asked, do I want to overwrite this variable, which I don't. But if you're running it for the first time, you won't be asked and you'll create a new variable. Now, it's worth saying to do this, we need to make the experience of using these variables slightly easier in Logfire.

At the moment, they're not configured with the Logfire projects use command that you did before. So you need to go and set a separate API key to use variables. That will be solved soon. But if you want to do it today, the way to do that is to go into settings, API keys within Logfire.

You see I've created I could show you the form. I can create a new API key, and I just need to give it the three variables permissions, well, or whichever ones you want. And now you can go and we're ultimately using that API key to push the variables from local development, from the code definition, through into Logfire, and then pulling them to update the values.

And so because I have pushed that, if I go into managed variables here, you will see I have two variables set up. This NP search config is the one we were looking at. You see we have those three fields that we've set.

And we have some history of some updates. But in particular, what we can do in targeting is we can basically define what percentage of calls are using which of our different values. So this is very much like A/B testing.

In fact, under the hood, our managed variables use the open feature, open standard for doing this. So in theory, you can connect to Logfire with anything that speaks open feature.

But yeah, so at the moment, we're using the code default. So you saw when I ran this now, I got back this answer of Andy Slaughter. But now let's go and update the value here. I want to go and edit.

And I'm going to leave those values the same. And I'm going to say, reply in French to make it very clear what's happened. Save that. And I'm going to go into targeting, and I'm going to dial up the use of latest to 100%.

So it should now use

that latest value. And now if I come in here and I ask, for example, the same question again,

and I stop the server, which was silly.

But you could imagine the server is still running and it's going to pull the new variable each time that that function is called. And if the demo gods are with me and everything is hooked up correctly,

it has indeed gone and replied in French. So we've updated the server without redeploying. Obviously, I did, in fact, restart the server, but I could change it to not. So just to prove that that works, let me come back over here, edit this again, and say, reply in German.

And oh, that's very annoying.

And now if I ask the same question again,

that variable that defines the system prompt should be updated.

It's taking its sweet time, but it is indeed replying in German. But the nice thing here is, coming back to your point about, is it just a system prompt, we can set up our variables to not just be simple text values, but be, for example, a Pydantic model, which allows us to edit multiple different fields.

So I can switch here from Gateway Anthropic Sonnet 4.5 to OpenAI GPT 4.1. I'll get rid of that because that's going to confuse me.

Save that, 4.1. And the point is that we can update multiple different values. So if I instead ask it

hopefully, it won't bother going off and running. It said it's now ChatGPT rather than Anthropic. So we can use this to change variables and ultimately go and experiment with what new prompts or what new models or temperature or whatever it might be, how they behave in production.

Now, of course, the ultimate aim here is to be able to take this whole system, wire it up, and instead of us having to basically tweak the managed variables manually, we basically get self-driving for managed variables. So given some set of evals, it can basically go and perform, optimize towards hill climb towards some peak without us having to do anything.

And that's the feature that we're working on. But I can kind of prototype what that might look like here. So let me just reload this.

You will see in managed variables, I actually have two variables here. And the other one is NP relations instructions, which is instructions for the task agent we were running before. And you will see we don't have any values in here yet.

But what we have in our code, in the web server here, is we have registered so we register with this main agent two different tools. One was the NP search, which it was using before to find out who the MP for Hammersmith was.

But it's also got this extract political relations, which is basically running the same extraction logic. And then

it should be I'm just looking through it. Yeah, ultimately, it's calling that same extract relations function that we were optimizing earlier. So we can now use this agent. This agent can be asked, who are the political relations of a politician?

And with a bit of luck, it will go and use this tool to go and do the calculation. At the moment, it's going to use the simple dumb prompt. But the point is we can go and alter that prompt based on the optimized value that we got and thereby basically apply our new state-of-the-art prompt without having to redeploy.

So if I come over here, the first thing I'm going to do, actually, is

dial that previous variable back down to zero because we want it to be told correctly what to do. So I'm going to put that to zero.

And just to prove that something here is working, I'm going to say, who are the political relations of what was he called? Stephen Kinnick, because I know he does have some. I'm going to go run this case. And we can go and look in Logfire in a minute and see what it's done.

But hopefully, it has used that tool as the correct way of identifying who the relations are. So yeah, so it's found them correctly. But more importantly, if we look in sorry, if we look in Logfire here,

we'll see that most recent call here. And you can see that the search agent run

called the extract political relations tool with the input Stephen Kinnick, which is correct. And so it's then gone and correctly output the value. But if you look inside the tool call, we then ran the other agent, nested. So it ran and it correctly extracted theright data.

And so we're now using our agent, and we can now basically click the value and improve the prompt now. And so I can come over here. And to be fair, I'm not going to have a particularly good way of showing you that it's working better because it seems to be working pretty well, unless I can find an example where it wasn't.

So let me actually try. I've got a bit of time. Let me try and see if I can find an eval case where the dumb prompt was doing badly. Oh, we've got loads of runs now.

Sorry, let me go and want this case here. And I want to compare these two. And so if you look so, for example, this person here, Paul Maskey, it was getting it wrong. The dumb prompt was

no, wrong way around.

I want to find a case where it's done badly, where I can get it to do better. OK, let's use Andrew Gwynn, where the correct answer is no political relations. But the two prompts both return suggested that they did have relations.

And I'm sure that's because if we look, it'll be because, yeah, they're a sports commentator. They're their father, which the dumb prompt wrongly suggested was a political relation. And so if I go and run

this now, with a bit of luck, this is using the dumb or the poor prompt. And so it should find those relations. And then conversely, if I update it to the correct prompt, it should find yeah, so it's identified the sports commentator as a political relation.

And now, without redeploying, I can come over to managed variables, update the prompt that we're using. I think I may have a mistake here. You didn't see me just go and tweak the code.

I was going to tweak the code to use this version of the task where we are using the instructions managed variable.

Let me restart the server.

And now, without redeploying, I promise you, I'm going to go and create a new case here. And instead of the simple prompt, I'm going to put in the best one we got back from

running GEPA.

Save that.

And without further ado, without redeploying, I promise, we go and run that again. And it'll almost certainly go wrong because that's what things do. But we can hope. And it's gone wrong. It's again identified it, unfortunately. But hopefully, you get the principle here.

Yes, there's a question.

-

This web interface, where you search and get a response from the model what do you call that? The form that you submitted to, is that an answer?

Well, the form and then the agent that I'm submitting to.

Yeah. Do you have any thoughts about collecting feedback from users themselves, like MPS and whether it's good or bad?

Yeah. My number one piece of feedback is no one clicks them. And so there's a well, maybe there's a mic, but.

You ready? Sure.

Or

you could be nefarious like myself and just click bad.

So my experience is that if you're going to do it, you do only thumbs up, thumbs down. We have an annotation system that I'm not demoing today where you can basically record annotations against prompts. That is another place to build your golden data set from, or at least a starting point for your golden data set.

The best thing you can possibly do is something which is implicit in the user's interaction. So if you have something more advanced than this simple text prompt, but you have a chat, the single best way of evaluating the performance of a given question response pair is to look at what the user did next.

Because if the user says, you idiot, try again, you can get pretty strong evidence that the agent behaved badly. And if the user was like, thanks, that's great, or goes away, you assume it'sright. And if they it's probably not as extreme as you idiot, try again, but it's probably like, no, I mean X.

And so I mean, that's how Google works,right? The way they identify how good a website is, back in the old days when we used Google, was to see how long you spent on a site. If you came straight back and clicked on another link, they assumed the page was bad.

And this form here, is it a chat or is it.

This is just I honestly just asked Claude, go build me a single-page chat. We have an integration with the Cell AI protocol so that you can basically build a full chat interface on a Pydantic AI agent in a few lines of code, supposedly, that I would try and demo if I had a bit more time.

But this is just me trying to have the simplest example to run. In fact, gone. I'll have one try at doing it now since that one didn't work.

And a chat would mean that it kept its context somewhat,right?

Yeah. Yeah, so if I

get rid of this single run and instead I do

what's the agent called?

relationsAgent.toWeb. And then I go and run let me kill that web server to avoid confusion.

Why is that not running?

Maybe I do need to call .run.

Maybe I'm forgetting exactly how to call it. But the principle is that we basically have one line of code here. If I remember what it is, you can basically turn that into a proper chat interface.

Let me just see if the docs will tell me what it is quickly.

So I define the app, which gives you a starlet app. And then I run

UVRunUVicornTaskApp. And there's a typo.

And that should give me a nice interface now to go and ask questions where I can say

maybe

I don't know if that's going to work, but we will try it. But the point is now

you would get back a final result. It would go and you could use it like an agent, and then you could ask follow-up questions. I mean, this agent is designed to always reply with structured data. So it's not very chatty, but you get the idea.

I'm going to stop there. Thank you very much. Happy to answer any questions afterwards.

Sorry, did you have a question?

Yeah.

Go on.

Thank you. Can you hear me?

Yeah.

Do you have cool use cases internally where you use your stack, but agents that you've built? Because I hear a lot of, look, how many tokens are we burning? And look, my evals. But have you seen some like you mentioned the private equity firms earlier.

These are classic old problems classifiers. Do you have use cases that internally you're really impressed by agents or what the team is using internally to ship faster or create more value for either you or your clients?

So we have there's an agent inside Logfire to basically do free text search. And it converts to SQL. And we optimize that agent a fair bit. We don't care particularly about token count in that case because it's not high enough volume.

We do care about generating good SQL. The reason we can't use it is that it very often ends up with private data within the SQL. As in, someone will say, find me results for invoice 1, 2, 3, 4, 5.

And we don't want to go and show people that. It's one of the reasons we can't use that example in demos. I think there was another question. Does that answer the question?

Great. So in Logfire, is it possible to kind of obfuscate or encrypt the input and the output and just evaluate against a golden set? Let's say if you set up an application to work in a field with sensitive information like medical or.

So you can choose not to send the system prompt or anything to the agent and just basically record the output. Could be, for example, the performance, like good or bad. And then you have the metrics. You can choose not to send the raw data.

It's obviously less valuable. My somewhat biased answer would be you can obviously enterprise self-host Logfire, and then you can have it inside your VPC. That's probably what people do. But yes, you can do it without recording the actual input and output.

Great.

What people do in those cases so, for example, the big like Legora and Harvey, they're like the legal tech companies where they are very strictly not allowed to exfiltrate the data because it is their client's client's very private data is they record categorical performance.

So like good, bad, whatever. You could have like 50 grades of how it performed. And you can exfiltrate that data without actually exfiltrating any

generated content where you might include private data.

Great.

Any other questions? Yeah, there's one here.

Thank you. This is a tiny bit of an internal question. I noticed that you're using data classes throughout.

Yep.

Is that for speed reasons or any other reasons?

I think that is the canonical thing in Python. And so they're like if you're not trying to perform validation, I think that they're generally the kind of yeah, the canonical choice over Pydantic models. It doesn't make any difference.

The performance of any of these cases, the Pydantic validation or the data class construction time will be shrinking 0.0001% of the performance.

Thank you.

Thanks for the presentation. I had a quick question. So with the prompt optimization, obviously, that's like a type of context engineering,right? So I was wondering, have you done evals where you've combined that with essentially some because obviously, when you there's context degradation.

Even though the context windows are so big, obviously, after like five or so, it'll still be less than or after 20 calls, it'll be less than very quickly. So combining prompt optimization with just spinning up new LLM calls and new context restarts, as opposed to, say, compaction, which still reduces as well?

The compaction strategy is definitely one of the things you want to go and optimize in a more complex case. I think here we optimize for something that everyone can grok in a session rather than the most complex use cases.

But yes, definitely, that's relevant. So for example, the Shopify example of how they were analyzing these sites, with GPT-5, you could just give it the entire content of the website and be like, hey, go figure out whether this is fraudulent.

Using a Qwen model, you couldn't do that. You needed to do something more agentic where it performed a bunch of queries to look for certain terms. And so that is a variation on context engineering to basically allow a smaller model to perform the same task and be more deterministic.

Great. Thank you.

Stop. Apparently, I have to stop. Thank you, everyone.