Breaking the Chain: Agent Continuations for Resumable AI Workflows - Greg Benson

So you built an agents, and they're working. Actually, they may be working really well, doing amazing things, and now you want to put these agents into a production setting. What are the other things you have to worry about?

For example, do your agents need some form of human approval during their processing steps? Will your agents be running for long periods of time in the presence of failure? How do you address these issues? Today we're going to be learning about a new mechanism that addresses both these issues, and I'm really excited to talk about it, so let's get started.

Hi, I'm Greg Benson. I'm a professor of computer science at the University of San Francisco, and I'm chief scientist at SnapLogic, and today I'm excited to talk about a new mechanism that we've developed at SnapLogic called Agent Continuations.

Agent Continuations allow you to capture the full state of complex agents that can be used both to do arbitrary human-in-the-loop processing and also be the basis for reliable agent continuation through taking snapshots. This work came out of the Agent Creator research group at SnapLogic.

First, let's talk about the general challenges that we face with agents today. First of all, there's a lot of concern about where the human sits relative to agent processing. That is, for people to get comfortable with agent automation, key aspects of agent execution require some amount of human oversight, and so there's been lots of work and lots of techniques for including human-in-the-loop.

I'm not going to go over all of those, but essentially the agent will get to a point where there's a designated tool or task that might be a high-value, high-risk task, like transferring a lot of money or deleting an account.

And we need a way to ensure that when the agent gets to that point, that in some manner it gets to the human, the human can provide the final determination or decision. Another problem with agents is that many of them will be long-running.

There's lots of steps in agentic processing, and the longer that any process runs, the greater chance that there may be a failure. And we'd like some way to not lose all the work that an agent has done and be able to potentially, uh, checkpoint our agent state to be able to resume from a point that is not the beginning.

And finally, increasingly, agents are going to be operating in a distributed environment and not just on your desktop, for example. And so what are the considerations for running agents in a scalable distributed environment? Let's talk about basic agent execution just so we can level set for the rest of the talk today.

An agent is essentially a loop that involves calls to an LLM that specify potential tools for that LLM to use, and if the LLM determines that it needs to use those tools, it will come back to the agent loop, which will then call the tools on behalf of the LLM request and collect the tool results and send them back to the LLM.

And it does this in a progressive loop: LLM tool call, LLM tool call, and so on. Now, it's also possible that a tool can be an agent itself, and that's illustrated in the picture here. We have a subagent.

So this is the basic agent processing setup, pretty standard across agent frameworks today. Clearly, there's a lot more going on, there's a lot of interaction, and this gives us a detailed view of the interactions I just mentioned. The main point of this is to indicate that even simple agents have quite a bit of interaction with the LLM and with the tools.

So what are some of the agent execution scenarios that we're concerned with addressing? I've mentioned tool calls that require human approval. This is a common idiom that is, uh, a tool call is an access to something in the environment, something that may require, uh, human approval for the access itself, or it may be an action in the environment, like, like deleting an account, transferring money, um, doing something of consequence.

And in those situations, it's desirable to allow the human an opportunity to approve that request. Another situation that can happen, it happens to me all the time with coding assistants, is the landscape of LLMs and LLM APIs is changing rapidly, and their use is, is dynamic.

And often new models come out, they get used heavily, and you might encounter rate limiting. And there's other reasons you might encounter rate limiting, uh, but there are different ways to handle this situation, and we should seek mechanisms in the presence of this.

And finally, I mentioned long-running agents, uh, especially agents that have to do very complicated tasks or extensive research to, to multiple systems, either external to your organization or internal to your organization. Um, the longer that the agent runs, the more susceptible it is to some potential failure, either some sort of network failure or hardware failure.

Um, and as these sophisticated agents run longer, we would like a way to not lose all of the work that the agent has done. So, so we want to also tolerate failures in the presence of long-running agents.

Another thing I want to elaborate on is

the fact that increasingly agents are becoming more sophisticated in the sense that not only will you have, say, a single-level agent that has its access to an LLM and its tool calls, but you may have an agent configuration in which you have a main agent or an orchestrator agent and then several subagents.

In fact, those subagents could have their own subagents. This is, this pattern is, is becoming quite common. So then the question is, in the presence of these more sophisticated multi-level agents, how are we going to deal with those two primary, uh, primary tasks that I mentioned earlier: human approval and state saving in the presence of failure?

The last concept I want to talk about is what I'm calling agent loop persistence, and this is the idea of if you have an agent running, there is this loop. That code that actually is doing the loop has to run somewhere.

It has to be on a physical machine running. It might be in the cloud, it might be in your desktop, it could be in, in many locations. That agent loop is going to be interacting with the user in some manner.

Maybe you're doing it from the command line, maybe you have a web app, um, maybe it's interacting through some sort of third-party communication channel like Slack. Most of the frameworks today, to support things like human-in-the-loop, require that that agent loop be running continuously.

Even if it's waiting for something, like waiting for a human response from Slack, it requires, they require this, this agent loop to, to persist. And this is something that we wanted to address in our agent continuations work. That is, we wanted to develop a mechanism that would, would allow us to fully shut down the agent loop or multiple agent loops in some cases, and then restart them at a later point, say when the human has provided their approval.

So what are agent continuations? So first off, agent continuations are inspired from the programming language theory concept of continuations. This is a long-standing concept in programming languages in which you can, at any point in time during program execution, you can say, "Hey, I want to stop the program execution at this particular point and sort of bundle that up so that I could resume or continue the execution from that point going forward at some later time."

In effect, what that lets you do is it lets you take a snapshot of the program's execution so far so that later it could be continued. Um, it's a powerful mechanism that doesn't exist in, in all programming languages, um, because it requires fairly sophisticated runtime support.

But this mechanism, this language continuations mechanism, was the inspiration for what we're calling agent continuations, in which we want to do the same thing. We want to be able to say at some point during agent execution, which again might be through multiple tool calls, multiple LLM calls, and maybe even calls to subagents and their tool calls and LLM calls, at any point in time we want to be able to say, "Let me pause what the agent is doing.

Let me be able to save that state and then return it back, say, to the application layer so it can, for example, process a human approval or perhaps persist the state of the agent at that moment in time to be resumed later."

So now let's go into the details of agent continuations. First, let's make some observations about why this may even be possible. The way we work with LLMs and agents today is we maintain the messages array. The messages array is really almost a log of all the interactions that you've had with the agent or with the LLMs.

The, and if you, if you look at itright, it records a history which is actually replayed back to the LLM to make its next inference. And so in a sense, uh, agent interactions with LLMs are already doing a lot of the bookkeeping to keep track of everything that's been done so far so that that past history can be sent back to the LLM for the next action, say, for the next tool call response or the next decision that the LLM has made.

So one of our, one of the key insights is that we're building on top of this or this messages array, which already is saving quite a bit of the state of the agent. Now, it's not quite enough, but it's pretty close.

Before we get into the details of how agent continuations are implemented, let's look at how you use the prototype framework. First, I'm going to show you how to use the framework like other agent frameworks, that is without continuation support, and this should look pretty familiar.

We have tools, and we use a decorator to indicate that these Python functions are tools. Uh, and then when we go to create the agent,right, we just send the tool list to the agent so the agent knows to use those tools in processing a user request.

Now, one thing I'm not showing you here for brevity is you can also include instructions or a system prompt to the agent. Now, once you've instantiated your agent, then you can make a request and give a user prompt, and the agent will go off and do its LLM request and its tool calling and then eventually come back with a response.

Let's look at our continuation extension to this standard approach. There's only a couple of differences. So now let's look at using the framework with continuation support, and you'll notice there's only a couple of differences. Let me point those out.

First off, now with continuations, we can designate a tool as needing approval. So here we've said that this tool will need approval. In addition, instead of creating an agent from the agent class, we're going to use the continuation agent class.

And other than that, it's pretty identical to the standard agent usage. Now, in terms of execution, it's a little bit different. The response may be the complete response of the agent if the agent was able to complete its task.

But if the agent for some reason needs to suspend, then the response will be a continuation object. This continuation continuation object has metadata that can be inspected so that you can understand the reason for suspension. In this example here, we're assuming that because we know we built the agent with the tool call that needed approval, that that continuation object is going to need a human approval response, which could be true or false depending on what the human ultimately decides.

But let's say in this case that we decide to approve this tool call request. So we update the continuation object appropriately with the positive human approval, and then you can send the continuation back to the agent using the same request method.

And the agent will know, because this is a continuation object, that it needs to resume from the place that it suspended in the first place, that is, in this case, just before the tool call so that we can get the human approval.

Then

the agent can continue to completion and get the final response. So let's go into a little more detail about the implementation.

So with continuations, the forward processing of the agent remains the same. You're going to do LLM calls. The LLMs are going to come back with a tool call request. We'll pull, form the tool call request as long as they don't necessarily need human approval.

Uh, but if the human approval isn't needed, it's standard agent processing. If we get to a point where we determine that we need to suspend, like for human approval, or if we've reached some other suspension condition, that's when we create the continuation object.

The continuation object embeds the standard messages array with some additional metadata. That metadata provides the continuation enough information of how to resume when we want to send the continuation back to the agent. Ultimately, this continuation object will be sent back to the top-level agent, which we do some final processing of to make the application layer, to make it easier for the application layer to understand why the continuation was suspended and also to allow it to provide human approval updates if required.

So essentially what we do is we take this fairly potentially complicated nested continuation object and extract the core information that the application layer needs to act on to make it easy for that application layer to make those approvals.

Then once the application layer has updated the continuation object, we send it back to the agent. The agent knows that it's a continuation and has the logic in order to reconstruct the agent back to its state so that we can continue the execution.

Now, one really important point of all this is that

once you've suspended the agent and you've created the continuation object, you actually don't need to keep any of the agent loops running in your system. That is, they can be shut down because we've captured enough information to restart everything back to where it was.

And this is a really powerful aspect of agent continuations. So going into a little bit of the detail just to get a sense of what this continuation object actually looks like, I've sh, I'm showing two versions. First is a single-level continuation.

Uh, you can see that we've wrapped it. We have the top-level continuation, the messages array, which is the normal messages array, and the two other pieces of metadata are this resume request, which indicates exactly which tool call, for example, we need to resume at.

And also processed is going to be populated with whether we approved or, say, disapproved in the case of human approval. And I mentioned that before the continuation is returned is that we extract to the top level of this object the metadata that allows the application layer to easily update the continuation, um, so that it can proceed after we send it back to the agent.

On theright, a slightly more complicated continuation object in which we have an agent that calls a subagent. And the main point here is that our format is recursive and it can handle arbitrary layers of nesting. And you can see that here in the fact that we have a resume request that now has its own continuations object with its own messages, and that's the messages of the subagent and its own resume request and processed.

So let's look at a full example. Here we have a multi-level HR agent. And so the top-level agent has an email tool and it has access to a subagent, the account agent tool, which is responsible for creating an account and setting account privileges.

I have an abbreviated system prompt up here, and we construct our continuation agent for the HR agent here. Now, let's take a look at the account agent for a moment. It's its own agent with its own tools, create account, and authorize account.

In this case here, authorize account needs approval. But you can see it uses the same basic setup. Uh, we give it its system prompt up here and its tools. And this subagent is going to be used by the top-level HR agent.

Here I have an example prompt to create a new account, and that prompt will be sent to the agent to be processed. Now, in this example here, it's set upright such that eventually the account agent subagent will need human approval for the authorize account tool, which will cause the agent to suspend, create a continuation, return that back to the application layer for processing.

Here is an illustrated diagram of the steps that I mentioned. In this diagram, we have the application layer that's interacting with the user. We have our HR agent and its tool call send mail, and then it's using the account agent as a tool.

And that account agent has its create account tool and its authorize account tool. Now, in the scenario I just described, we'll get down to the point where we know we need approval for authorize account. So that's the point at which we need to suspend and start creating or building up our continuation object.

And that's what happens here. So the approval propagates a continuation object that gets expanded at each level as it's propagating back to the application layer until finally we have the full continuation object for the application to inspect and then act on.

Now, once the application layer has done that, then that continuation object can be sent back into the HR agent and the framework will know that we need to restore the agent state and the subagent state back to where it was when we needed the approval in the first place.

And that will be handled all by the framework. So let's do a quick demo. So let's look at our prompt. So we have the prompt here as a JSON, and

I'm going to

run

this prompt

here. So I'm just giving that prompt as the first prompt to the agent starting off. And I'm going to send this out to

a response file. And the whole point of this is to show that these things are fully decoupled and that these objects contain all of the state that we need. So let's look at the

response object. And this has the things that you can imagine. There's the continuation, there's the top-level messages. And if you go down,right, here is the subagent messages with its system prompt. And finally, if we go down to the bottom of this object, this is that top-level approval object that the application layer can then present to the user and then update.

So here I'm just doing it manually. Obviously, you would do this in application code. So I'm going to save that. And now what I'm going to do is I'm going to send that edited response back to the agent.

And the point of this is that now the agent will reconstruct its state, but with the information that the tool has been approved so that the agent can continue processing.

And there you go. If you look here,right, we agent is indicated that the account has been successfully created, uh, and that a security level has been granted, and that we now have the end reason of completed. The other thing I'll note in this output here is that

everything above now is normal, a normal messages array. That is, uh, there's no sign of continuation in the final result object. The continuation is just introduced when we need to suspend. Okay. So let me go over just a few details.

Um, our prototype is built on top of the OpenAI Python API. There's no other dependencies. We have the prototype implementation. Here's the link that you can go to on GitHub, and it should be in the channel notes as well.

Where do we want to take this? We've already implemented a version with more general agent suspension that is beyond just human approval, is that setting up arbitrary suspension points like after a certain amount of time, a certain number of turns, even asynchronous requests for suspension.

And what we're looking at is not just not necessarily developing, uh, this prototype into a full separate agent framework, um, because there's lots of good frameworks out there, but rather looking at ways to extend existing frameworks, something like Strands or Pydantic AI.

Now, I should also mention that, um, we're not the first framework to consider state management. Some of the other frameworks do have forms of state management. However, uh,

in our investigation of the state management, they either sort of lack the human approval element or the, um, the sophistication of having arbitrary depths of nesting into subagents. And so we think our approach is novel in the sense that it combines both those, both a human approval mechanism and also this arbitrary nesting, uh, of complex agent.

This work came out of the Agent Creator team at SnapLogic. Agent Creator is SnapLogic's visual agent building interface and platform. Not only did we build it out in Python, but we built it out in Agent Creator itself. And you can see here that we allow our users to create sophisticated agents using a more visual approach.

And you can also visualize in the platform agent execution. We prototyped continuations in the SnapLogic Agent Creator environment as well. Uh, so we, we did it both at the Python layer and we also did it at our higher level, uh, layer in, in the SnapLogic platform.

In conclusion, Agent Continuations are a new mechanism for managing agent state and human-in-the-loop processing. We have a prototype implementation that you can access on GitHub from the link on this page. And if you want to learn more about building agents digitally with SnapLogic Agent Creator, go to agentcreator.com.