Two Roads to Durable Agents: Replay vs. Snapshot — Eric Allam, CEO, Trigger.dev

How's everyone doing?

Good.

It's a full room, look at this thing. Okay, let's get started. Okay, so here is our, you know, agent. You know, it's got the turn loop, it's got the LLM loop. You know, this little example sort of works well enough running on your own machine, but what if we want to sort of deploy these to production backends and, you know, run them on our servers?

So what do we want them to do,right, when they run on our servers? We want them to do, you know, long-running, meaningful work. It should be durable across turns and versions of our code, and it should be able to, you know, recover from errors.

So I'm Eric, I'm one of the founders of Trigger.dev, and we've been sort of trying to make it easy to deploy these types of agents to production for the last few years. And what I like about this little meme here is, which one is the agent and which one is the human?

I like to think.

Yeah. So this talk is sort of about, like, the fundamental shift that agents are, like, posing to backend infrastructure and some of the ideas for sort of how to achieve these durable agents. So before we go into that, I want to do a little history lesson here.

Let's take a step back and see sort of how we got here. So the very first dynamic web backend was CGI back in 1993. Anyone here ever done CGI stuff? Cool, thanks.

So the model was really simple. HTTP request comes in, the server forks a whole new process, request data goes in, the process does some stuff, and then it writes the response to standard out, and then the process goes away.

So it's completely stateless. Shortly after that, PHP came out, which sort of turned into the LAMP stack. And oops. So sort of the LAMP stack sort of reused the PHP process,right? But it kept sort of the principle that, like, all you needed to do to create a response was the request, some state from the database, and then it would do the request.

So the second request would come in, and it would do all the same work again, and it would produce the response. So this is sort of request plus DB equals the response, is sort of became known as the shared nothing architecture,right?

So looking at another way, shared nothing sort of means that the compute layer is stateless,right? There's nothing, there's no meaningful state, like, in the compute. The state is in the database,right? So this became the dominant backend infrastructure for the last 30 years,right?

Everything that followed from this, like Ruby on Rails, Node.js, serverless, it all follows the same paradigm,right? As web applications became more, you know, complicated and sophisticated, they started performing these, like, sort of side effects outside of the request and DB lifecycle.

These side effects are async tasks. So they start out simple, send an email, charge a credit card, you know, resize an image. But soon they became sort of these, like, multi-step side effects,right? Like this process order example here, where you sort of do things in sequence,right?

You'd quickly run into a problem how to handle failures in something like this,right? So if send receipt fails, you can't just retry the whole process order thing again without charging the credit card again twice, which is bad. So about 10 to 15 years ago, workflow and durable execution engines were sort of adopted to solve this problem,right?

So you'd write your code like this now, where you'd sort of wrap every single side effect in, like, a step that becomes cached as it's executed. So now that, you know, solves the problem nicely, when you call process order for the second time, you skip the things you've already done, and then you do the thing that you want to do originally,right?

And you don't charge a credit card twice. So this is, I call this model sort of the replay model. So it builds durable execution on top of existing, like, stateless compute architecture, which I covered was the, you know, that's how everything works,right?

So, you know, you get this nice side effect of you get this execution history, this audit trail of everything that happened. And also by being able to sort of resume to a specific point in time, you can, yeah, you can recover from a failure for that, but you can also, like, wait for something else to happen,right?

So you can wait for, like, a human to do something, and then you can resume execution.

Some of the downsides of this replay system is, like, because now you sort of have to wrap everything in these steps, and everything outside of steps has to be deterministic, you kind of get this, like, rigid structure. You have to write your code in a certain way or things break.

And also, like, replay journaling, the replay journal versioning is kind of tricky if you deploy a new version. So this is sort of the very simple and truncated history of, like, sort of the state of the world in 2023 when LLMs came out.

At first, they really fit neatly into this paradigm,right? They would just become another step in a workflow,right? They would classify some text or something, but it was still in this old workflow era,right? Not long after that, we sort of got tool calling, and tool calling got good, and we were sort of introduced to the agent loop,right?

And the big difference there is code is sort of no longer orchestrating the LLM. LLM sort of orchestrates the code,right? So we're back at our agent loop,right? And, like, what happens if we, yeah, you can see that. If you can, if basically we want to, you know, make this agent loop durable, and can we do it with this replay model,right?

What does that look like?

So what does that look like? Every LLM call,right, becomes a step in the replay journal. Every tool call becomes a step. On resume, you know, the function re-executes on top and sort of replays all that stuff,right? So after a single turn of the LLM not doing too much, you know, this is sort of what the replay log looks like,right?

And as you sort of keep interacting with the agent, the log grows and grows and grows. At a certain point, you might hit into some sort of, like, fundamental limit of your replay system. That could be there, like, too many actual entries, or it could be, like, the entries grow too large.

But yeah, this sort of kind of falls over once you hit that limit. And sort of there's this measure of, like, how long agents can actually do meaningful work. And apparently, it's doubling every four to seven months. Soright now, we're on about, like, a few hours, but, like, not too long from now, we'll be on, like, multiple days of length as these agents build to actually do meaningful work.

So, you know, replay gave us these, like, sort of durable transactions, but, you know, an agent isn't, like, a transaction. It's like a session,right? And it lasts, like, as long as the user wants it to last. Multi-step workflows are sort of start and end, and sessions keep going for as long as possible.

So if we sort of take a step back and think about what an agent needs to be durable, like, from first principles, I think of it as, like, an agent sort of has these two halves,right?

The first half is the context. So this is all your system messages, user messages, tool calls, tool results, assistant responses,right? So this is all, like, the actual context, everything that went in and out of the LLM.

So this is extremely valuable, obviously. You want to make that durable,right? But you also have this sort of execution layer. And as agents are, like, more complicated, doing more things, they kind of want a machine,right? They want to be able to do stuff like they could do on your laptop,right?

They want to be able to write files, use memory, like, create sub-processes. And so I think of both of these as super valuable pieces of state, but they can be treated separately. So the context is first and the most important.

And it's just an append-only log of sort of everything that happened,right, like I said. And you can make this log durable using any sort of, like, primitive that already exists, like a database, object storage, like distributed file system.

You know, there's a ton of, like, technologies that are coming out that are specialized in making this sort of thing durable,right? And when that is durable, when that context log is saved somewhere, now you can have durability across versions of your code,right?

So you upgrade your harness, and you can still use that same context,right? Maybe the machine crashes, and you can still, that is saved somewhere, so you can pick up where you left off,right? And append-only logs scale really well.

But what about making this sort of execution side durable,right? For these, you know, saying the types of agentsright now that are doing meaningful work, there's a lot of state that happens in the compute layer that we might want to save.

Maybe you've cloned a GitHub repo, you know, you've installed some packages, you've got some data sets in memory, you're running a dev server,right? You sandbox in a sub-process, whatever it is,right? You can't really make that durable using a log.

And how do we get this to work,right? So you have to wait for some amount of time for the next user message,right? And we can't just keep the machine running. It'd be nice, but we can't. It'd be too expensive.

So instead of recreating the execution state from a log, we should use snapshot and restore. So this allows us to snapshot the machine, shut it down, save it to disk, and then when the user message comes in, we restore it,right?

So this gives us durability across turns. So when the user goes to lunch,right, we don't have to run the machine the whole time. It allows us to preserve everything that the agent was doing. And, you know, effectively, compared to running the machine live, it's pretty cheap.

So I think if you combine these two things, then you sort of get a durable agent,right? You've got the context, so you're sort of, yeah, you're context durability and execution durability,right? And this also allows you to recover from errors.

So one of the whole points of having these, like, durability guarantees is to recover,right? And so it depends on what happened, what went wrong, you can recover in different ways. So say the LLM isn't working for some reason.

That never happens, but you never know, it could happen. And it takes a long time to, like, retry. Maybe it says, like, wait, you know, 15 minutes so you retry your next message. Well, you don't want to wait in memory, so you snapshot, and then you restore when you can retry.

But if there's something wrong with the machine, maybe you've, like, shipped a bug, or maybe there's just an issue with the machine,right? It crashes. You have the context log, and you can recover that. So I think, you know, for 30 years, we sort of had this stateless compute as the sort of core of backend infrastructure.

And I think agents are sort of forcing this move to become stateful compute. So, and sort of at the heart of that, I think, is going to have to be this snapshot and restore capability. But sort of, you know, this isn't actually new.

This is an IBM mainframe from 1966, and it actually has checkpoint and restore. Because they would run these super expensive jobs for hours, and, you know, something went wrong, and they couldn't afford to run it all again, so they would add these, like, checkpoints into their code,right?

Fast forward to 2011. This thing called Crew was developed. It was a way to, like, suspend and restore a process, like, from user space. So it would basically, like, inject a process with this, like, a parasite, basically. And then they would force the process to, like, dump everything to memory, and then it would remove all the traces of the parasite.

And it actually worked. In 2024, we actually shipped this, and we've done millions of snapshot restores since. You know, it's transparent for the process, so the process doesn't have to, like, participate in it. And it's compatible with container runtimes, which is good.

So the downsides are you sort of can only checkpoint, like, a process. So if you're doing stuff with, like, FFmpeg, or, like, you've got a Chrome instance running, or anything else,right? It sort of doesn't work. It only captures open files, so if you're working with the file system, it has to be open at the time of snapshot, or you won't get snapshots.

And then also, if you, yeah, it's nice that it's compatible with containers, but once you are compatible with containers, you have to work with registries and push and pull, and it gets very slow. So last year, we moved to Firecracker microVMs.

And this allows us to sort of snapshot, like, the entire machine,right? So everything that's on a machine, on a VM, we can snapshot it, and then we can restore it, and it picked upright where it left off, no matter what was happening in the machine,right?

But if you do that sort of in a naive way, it can be quite expensive. So say you have a default machine size of 512 megabytes, you know. If you do a snapshot, it's 512 megabytes on disk. So that's not great.

So obviously, you've got, like, network transfer costs, you've got storage costs, and there's a lot of memory there that's not actually being used. So we actually solved this with compressing it. We actually use a sequable compression. So when we restore, we actually don't restore all the memory pages at once.

We actually, like, capture when it needs to be restored and just, like, decompress, like, that little bit that needs to be restored at a time. We also have a couple other techniques for layering the snapshot. And we can get the snapshot down to, like, 14 megabytes compressed.

And that's sort of, like, a knob you can tweak, depending on how, what kind of performance you want. You can compress more or less.

So that's pretty much all we had to do other than all that.

And once we did that, we got super fast snapshot and restore times. So this is sort of a stupid graph comparing Crew and Firecracker, but it's basically the moral of the story is that snapshots are, like, slightly under a second, and restores are a couple hundred milliseconds.

We've actually bundled all of this into a tool that's going to be open source here soon. It's called FCRun, or FCRun, depending on who you ask. So this allows you, it's like a Docker-like CLI, so you can drop in replacement for, like, the Docker command for running containers in Firecracker VMs and snapshotting and restoring them.

So, for example, you can run Alpine, and it's super fast. And you can snapshot running VM, and it's super fast. You can, like, fork a VM, also very fast. This is a little benchmark for TTI, so basically how long it takes the VM to become interactable with the internet.

So this is, we're doing, like, 15,000 VM starts per minute. You can almost render, like, a video. The FPS would be about 30 FPS. So it's extremely, extremely fast. So this is going to be powering sort of our future, like, compute layer.

But it's open source. Not yet, but very soon.

So kind of back to where we started with our little agent loop here. And we've sort of made it durable now by doing two different things, context log and execution snapshots. So we get durability across versions, durability across turns, across failures, and this will lead to a future of, you know, stateful compute.

That's it. Yeah.