From Stateless Nightmares to Durable Agents — Samuel Colvin, Pydantic

Hi, I'm Samuel from Pydantic, and today I'm going to give a demo of Pydantic AI, Temporal, and Pydantic Logfire. I'll also cover Pydantic Evals. So we have, in Pydantic AI, support for Temporal and DBOSS, two durable execution frameworks.

We're actually adding a bunch more; I think we've had something like 5 pull requests to add other durable execution or, like, workflow orchestration backends. But at the moment it's these two, and I think it's fair to say Temporal are like the big incumbent in this space, and they're kind of, I guess, leaders—leaders in how you do this.

To demonstrate this, a simple example of, like, I go ask an LLM a question, it replies, mostly just works, and we don't need the durable execution component. So, but once we get into longer-running workflows, that's where it really becomes a problem.

In particular where we've done enough compute that we don't want to lose it, or we've spent enough time on that compute that we really don't want to have to start again for the user. That's what, for example, I think OpenAI—it's, I think it's public that OpenAI used Temporal for their Deep Research, and I think some of the other LLM Deep Research do the same thing.

So I'll, I'll start with a kind of toy example, and then I'll move on to a more Deep Research type example, in fact a Deep Research example. But before we get into that, let me, let me run this example without.

So this is a, uh, two agents that play 20 questions. So instead of just a yes/no answer, they get to give a little bit more detail, like yes, kind of, not really, no, completely wrong. That was because I was getting bored of waiting for them to take ages to succeed on the, with the just the yes/no.

So we have an answer agent, which is, runs a relatively small model, Haiku 3.5, because I didn't know 4.5 came out until about an hour ago. And this basically takes a question and answers yes or well, with one of these answers, and it gets added into its context, the, like, secret object that you're looking for, which in this example is a potato.

So, and then we have the questioner agent, or the player agent, has a bit more context on what it's going to go and do. You don't need to read through all of this stuff; this code is public now, it's a pull request, but I'll merge that afterwards, but you should get the idea.

And the way that the questioner agent gets to ask its questions is by calling a tool, ask a question. Inside that tool, we run the other agent, the answer agent, to basically decide the answer to this question, and then we respond.

And it takes a little bit of time to run. You can see in this case it succeeded pretty quickly. Sometimes it's amazing how even these very simple questions, even very intelligent LLMs, get themselves completely confused and go down some weird track and, like, get very confused.

But you can see in the last run that it asked a bunch of questions, got down to, like, is this a fruit or vegetable, is it a fruit, no, so it, you know, it was a vegetable, is it orange, and it worked out the answer was, was potato.

But obviously, and here it is running again, I don't know how many steps it's going to take. Sometimes it can take up to, like, 50 steps to get this questionright. And obviously the problem is, if this dies, either because we have some unreliable endpoint within our system, or because we're running in the cloud and Kubernetes decides it wants to scale or whatever it might be, if we run this again, we obviously have to start from scratch.

That is problematic in this case, but you can imagine as the tasks get longer and longer, just restarting it gets more and more problematic. So, and I think the other thing to say about this, this, like, 20 questions example is, although it's pretty simple to understand and it feels like a toy, it is actually directly equivalent to a Deep Research case where effectively the agent is, is like, going off on a quest to go and find an answer to a question where it needs to ask the, like, you know, troll at the bottom of the garden, the, the, like, question to the next riddle to get to the next endpoint,right?

Like, Deep Research is effectively this 20 questions just with, like, web, like, web search or RAG or whatever it might be is your intermediate steps. So let's turn that 20 questions into a durable agent.

So this is mostly the same code. For simplicity, I've actually copied it here. But you see we have our answer agent. We need to wrap it in this temporal agent, which gives us another thing that behaves like an agent, like a Pydantic AI agent.

So it's also a subclass of abstract agent. We do the same with our questioner agent. To keep things simple, we aren't doing the same stuff about passing around the answer in context. We've just hard-coded the answer into the system prompt here for the, for the answer agent.

But apart from these, like, adding the temporal wrappers, you can, as I will show later, just apply durable execution later. But here's where the, the temporal bit comes in. And I am not a salesperson for Temporal, and although the underlying stuff they do is amazing, I do think some of their Python abstractions are kind of ugly.

But anyway, the, the principle of Temporal is that you have workflows and activities. And workflows need to be entirely deterministic, and activities then need to do anything that is non-deterministic, like I/O in particular. So you can basically do anything inside a workflow other than I/O and calling random.

And if you're, if you, if that's the case, then you have a deterministic system. And what you can think of what Temporal is doing in the background as, it's running that workflow and it's basically recording the, every activity that runs and both the, the inputs to that and the outputs.

And so if you want to rerun it, like, from the beginning to a certain point, it can basically plug in those answers. And I'll show what that looks like. So this is how we define our workflow. The activities here are implicit.

The point is that this temporal agent takes care of turning all of the I/O that you need to do to call an LLM into activities in the background, including tool calls. So OpenAI claim to have temporal support, but they don't support tool calls as activities, which to me makes it slightly a chocolate teapot.

Like, there's actually no point in having any of these things without, without tool calling. Or little, little point. But we define our workflow like this. I think you can, for the most part, just copy-paste their, their definitions of how to do it.

Here we have our play mechanism. The point here is we're going to, we're going to connect to the temporal server, which is what's going to record the state of our task or our agent as it's executing and be able to resume and stuff.

I have Temporal running locally here. This is just the open source version of Temporal, which runs as a separate process. And I can restart that to kind of kill the state. And that's why we're connecting to local host.

In production, you use Temporal's cloud. That's why they make so much money. And here we, we run the worker. This is where we're actually going to kick off our workflow. In general, we just kick off our workflow with execute workflow.

We pass in the workflow that we want to run. We pass the inputs. There aren't any inputs in this case because we just start, so there aren't any here. And it will then go and run. And so if I run this, we will, you will see it start to, to execute.

You'll see it running. The only couple of things to note, there's a couple of log messages. Ah, and we immediately have this exception broken. And that was because, to simulate some system that's unreliable inside the tool, I added 20% of the time it's going to, it's going to break.

What you will see is that Temporal has immediately taken care of continuing after that. So even though this broke, it will continue to run. And I may have set 20% to be too high because it's now failing all the time, but it's actually going to continue and deal with those runtime errors and just continue to operate absolutely fine.

Let me, however, I think I dialed up 20% too high just before. So I'm going to, I'm going to actually see if this is going to continue to operate. Obviously, when you give a demo, everything suddenly grinds to a halt.

Someone recently said they hate demos where everything goes to plan, and I said, you never need to worry about that with me. I don't know why that has actually ground to a complete halt. I don't know whether that's it just repeatedly failing.

Let me, I'm going to kill Temporal server here and restart it so that we don't have the state stored. And I will clear this and run it again. And you should now see it succeeding most of the time and failing 10% of the time.

So yeah, you see it now asking questions and occasionally breaking. Good timing, but continuing. So that's one of the things Temporal does. It just does the, like, retry logic that is, like, you could implement without Temporal, but they do it very nicely.

But there are more powerful things. So let's say this process that's in the middle of running gets killed by Kubernetes. Now, so we go across here and we, we just, like, kill it. Process gets killed. Now, what I didn't show you is I also instrumented this with Logfire.

So if we look at our workflow, we can see exactly what was going on here. And we, we can see what's going on here. We have the different calls to Claude, and then inside that we have, we're running the activity, which is then running the other agent.

But in particular, if we come to the top level start for the workflow, I can take the workflow ID. And if we come back over to code here, you'll see I had some, some code in here to basically allow me to continue with a given resume ID and to continue a workflow.

Now, for the most part, you wouldn't have to do this. This is just for the sake of the demo. If I just reran the script again, it would kick off this workflow again, and it would run the two in parallel.

That would look really confusing. So instead of doing that, I'm, I'm specifically hanging on a particular workflow to finish so you can see what's going on. So if I, if I run my script again, but I give it the workflow that was ongoing, now you see it's already whizzed forward to question 6 and it's continuing to operate.

So we've got it to basically resume without having to add any resume code anywhere in our actual agent code. We just set up Temporal and it works. And you can see exactly what's happened if you look at Logfire.

What you will see is that that whole, that first bunch of calls to the LLM responded in, like, 5 milliseconds. So these were not actually sent to the LLM. Temporal just returned the result, the kind of cached result that it already had for each of these cases.

So we're able to effectively zoom forward to the point where it then continues to call the LLM. It's, it's as if you've gone through everywhere that you're doing I/O and you've set up caching on each individual call so that you can run your code.

I see some people nodding, which is making me feel a bit better about explaining this. But we don't have to do the inference. We don't have to wait the time. We can basically run our workflow code. It's generally very fast because it's no I/O.

It's just procedural. And it will just keep getting results instantly until it gets to the point where it needs to, needs to continue. And you see in this case, it's got itself completely confused and it's off wondering about whether this thing is a salad bowl.

So you see how sometimes the LLM does well, sometimes it does, does terribly. I'm going to, actually, I'll leave that running to see whether it, you see it's got, it knows it's related to food, but it's got itself really confused.

I will just say, as, as it might interest you, just before this, I was wondering how the different models would perform. And so I, I ran some evals with Pydantic Evals on these different cases. And you can see here we have, it's a bit hard to read on this screen, but GPT-4.1, Gemini, and Claude Sonnet 4.5.

And you can see the different assertions for each case, whether they passed or failed here. And you can see the, the average cost. You can see Gemini was way, way cheaper, way, way faster. And somewhere we should have, if we look at an individual case, we have a, a metric for how many steps it took to succeed.

I think maybe we have to scroll over. Yeah, question count. You can see here, Gemini was way quicker each time. I discovered subsequently having, having checked the results that actually the reason Gemini is way faster and answers much more quickly is it just invents an answer that's wrong and I wasn't checking it.

So this is not perfect yet, but like, it's a, the, the, there's definitely an interesting case for evals and seeing and like working out which model is actually better because they're definitely not particularly good at it by default.

But yeah, in my naive case, Gemini did way better, but that's not representative. Anyway, I'm going to leave that because it's got 46 steps in and it's still failing to work out that that thing's a potato. I, I can show the evals case if you want, but I think it might be more interesting to look at a Deep Research case, which is a kind of more meaningful example of where you would run durable execution and also doing stuff in parallel, which is also one of the things that like just works out of the box with Temporal without you having to write any, any code.

So this is my very quick last night hours attempt at building Deep Research. I honestly think it's as good as lots of the actual Deep Research systems. So we have, we define our plan for Deep Research. And this is, this is effectively our Deep Research plan.

So it has an executive summary, what you would effectively pump out to the user about what I'm going to go and do. Then we have a list of web search steps. We maximum, we have a maximum of five here so it doesn't take forever.

And then we have analysis instructions. And the point is that like, I mean, I think this is one of the big change in AI this year, answering a bit the question I had on the other Zoom. I think we've moved from thinking that like agents in the sense of, so there were three definitions of agents.

There is the like AI definition, which is LLMs calling tools in a loop. There is the tech definition, which is a microservice. And then there is the business definition, which is something that can replace a human. Ignoring the business definition for a minute, if you think about the AI and the like engineering definitions, we thought at the beginning of this year you would have one AI agent, one LLM calling tools in the loop within each microservice.

I think we've moved more and more to think that the agents are actually the kind of quantum of development. They are the, the micro tasks that are doing that you build up to, to form a like what most people would think of as an agent, something that actually goes and autonomously completes a task.

And so our Deep Research agent is actually made up of multiple agents. So we have this plan agent, which goes off with a prompt and it returns an instance structured data extraction, gives you an instance of this Pydantic model, which is your plan to run it.

Then you have the search agent, which has access in this case to search tool, or I'll show using Tavilli in the other case, which is using a faster model, Gemini Flash in this case. And then you're using, in this case, I'm using Claude Sonnet 4.5 for the final analysis stage.

So I suppose this is a bit what people talk about when they're leaning towards graphs. I haven't built this in a graph, although I could, because it doesn't need a graph. It's not complex enough to need a graph.

And durable execution is a way better way of getting snapshotting, but like much more granular support for, for durable execution. We added a tool that allowed the analysis agent to do a bit more web search if it really wanted to.

I don't think it uses it, but this is the actual Deep Research code. So you can see how concise it is. We run the plan agent, we get back our plan, we run in parallel all of the search agents.

So we're just using a task group from Python to run all of these. We get those results, which will all be the text results of the different bits of search. We use format as XML to basically smash all of that into a massive lump of reasonably readable data for the analysis agent.

And then we go off and run the agent. And we run the kind of question that I'm asking AI relatively regularly for sales, which is, find me a list of hedge funds that write Python in London. And if I go and run this, uh, you've run Deep Research, we'll see it starting to churn away.

We can see it in the terminal with Logfire, but we can also

come over here to Logfire. Let me clear that. Go to the bottom here. And we can see this, this run here as it's going on. It's, it's run the plan step in 9 seconds. And you can see all of the search steps going on in parallel.

Once they've finished, it will start. You can see the analysis agent has just started. We can look at the individual searches. So you get a pretty good idea of what happened, the question it got asked, the queries it decided to run, a bunch of data from Medium, different sites, structured data, and then like the agent also bangs in quite a lot of context,right?

So this is each individual one of our like 10 parallel searches. And now the analysis is going to go and run with all of that input. You can see so far we've spent 8 cents on this particular run.

We'll see what it gets to by the time it finishes. But obviously the problem with this is if I kill this now, it's just going to die and I'd have to restart from the beginning if I wanted to run it again.

So while that churns away, let me start introducing you to the durable execution example. Spoiler, it's going to be pretty similar. I discovered last night that there's a bug with the Vertex SDK. That means that you can't use it with Temporalright now.

So I swapped out, I think we should fix that, or at least I'll be whinging at DeepMind today to go and fix that. So I've switched it out to OpenAI responses here and I'm using Tavilli instead of the built-in search.

Yeah, but other than that, this is all pretty similar code. I could have probably imported the code from the other module. I just decided to duplicate it just to keep things easy. But you see again, we do the same thing.

We wrap our agents in Temporal agent. This analysis one can take more than I think whatever the default activity duration is because it's a long basically build up a summary. And so I give it, I think it was taking longer than 2 minutes or whatever the default is.

So I just gave it an hour so it's not going to fail. And then the, the, for me the most powerful bit is everything here inside my workflow looks exactly the same. I don't have to do any crazy stuff to do parallelism.

I just use a task group exactly the same. I could use asyncio.gather. It's all just imperative Python code as you would be used to. I could have a, if I wanted to run this periodically, I could sleep for 7 days in here.

Temporal would take care of pausing everything. Again, I'm not here to be a Temporal salesperson. I don't love everything about what they do, but it's a pretty powerful way of thinking about code. We don't have to do all the infra stuff.

And then ultimately, again, smash all of my context into the last agent and run it. And again, there's a bit of plugin stuff. I have to plug in log, add some plugins, add the agents as plugins. But again, my code to actually go and kick it off is just execute workflow.

Simple as that. And I asked it here slightly more controversial question of what's the best Python agent framework to use for durable execution and type safety. And we will pray to God it gives theright answer when we run it in front of everyone.

If I go and kick that off and run this again, we should see it. If we come over here, we should see it running in Logfire. You can see we have the stuff related to kicking off the agent.

It's kicking off the workflow, excuse me, here. And we have the searches beginning to happen, happen. But the, the powerful bit here is again, imagine that we're halfway through running all these searches. We're about to start the final step and something comes along and kills the process.

And by in general, you'd have to go and completely restart this process and run your Deep Research all over again with Temporal. It will just go and rerun that workflow automatically. In this case, I'm restarting it and just running that one workflow.

But in general, it would just automatically go and be restarted. And on the, the next time that Kubernetes comes up, the workflow will run as it would have done before, but it will get answers to each individual question basically instantly.

And so

if it's not going to fail for me, which it seems to be, there we are. It started again. You see our plan took 24 milliseconds. Search all took no time at all in the grand scheme of things because it just got the result back from Temporal immediately.

And then the analysis that was the, the, the task we needed to, that we hadn't run yet. Obviously that needs to go and start again because that's an activity and you can't, activities obviously have to run again from scratch.

And so once that finishes, I think it does take quite a long time. Maybe I can show the previous output. Or did we not get to displaying the previous output? Did it ironically actually fail a time before? But hopefully once this finishes, we should be able to see its analysis, which, you know, I think is on a par with what I see from the other Deep Research things.

Obviously there will be some, there's some UI work to do to display this in a nice Deep Research interface. There we are. It's completed and it has primary recommendation is Pydantic AI with Temporal. So it, it, it did what I hoped it would do.

And you see it's given a reasonable report here of like the relative trade-offs of the other inferior agent frameworks. And it should have done an executive summary at the beginning with, with links. Yeah. So it said Pydantic AI, LangGraph.

Obviously if you love snapshotting or writing unsafe code, type unsafe code, Temporal on its own, which makes sense. Yeah. So there's a, there's the summary. That is the main stuff I had to show. I will merge the, the durable execution stuff in here.

So go here. I just, other thing I want to just say quickly while I have, I can't work out how to post a comment, but like you'll find it on Pydantic if you, if you, if you look for it.

Oh, I have, I can do that. If anyone wants to take a picture of that QR code. The other thing I just wanted to mention, we're about to announce Pydantic AI Gateway. So if anyone wants to try it early, let us know.

But yeah, that platform will be landing soon. You'll be able to use Pydantic AI Gateway directly to buy inference from any of the big models or most of the open source models and self-hosting for enterprise, all the observability stuff.

But I'll, I'll save you the full spiel, but that's coming soon. I think some of you will find it interesting. That's it. Thanks so much for watching. If you want to learn more about Pydantic AI, Pydantic AI Gateway, or Pydantic Logfire, please scan these QR codes.

If you have any feedback, please come and talk to us. Thanks so much for listening.