Building Reliable Agentic Systems: Eno Reyes

So for this talk, basically what we thought was it'd be cool to give kind of practical examples and lessons of problems and solutions that we identified while building the Droids. And so, for context about Factory, our mission is to bring autonomy to software engineering, and what that means concretely: we build these products that we call Droids.

They are autonomous systems that are applied to different stages of the software development lifecycle. Think code review, documentation, testing, all the way to end-to-end coding tasks like a refactor, a migration, feature work. And so each of the Droids has separate cognitive architectures which are mapped to the tasks at hand.

And in particular, the Droids like Review Droid or the Documentation Droid, which handle processes which we think are kind of more on guardrails, are pretty different from something like the Code Droid, which is able to take nearly any natural language task and make an attempt or complete a coding task that is associated with that request.

And so the idea for this talk is let's just start describing what we think of as an agentic system. Agentic systems have a lot of different interpretations, a lot of different definitions. We think that there's three characteristics which are kind of most representative of an agentic system.

The first is planning. I think you've probably seen a lot of this in this track. The idea that the agentic system can make decisions about one or future or many future actions that it's going to take. Decision making.

Some people call this reasoning. I think that in a lot of these systems, just the ability to make a decision, have some criteria and some algorithm that's associated with making that decision, is kind of critical in order for agentic systems to take on more general or broad tasks where the decision space is very wide.

And then you have environmental grounding. And so a lot of systems have planning, they have decision making, but I think the existence of an agent within an external environment is very critical to understanding some of the unique properties of agentic systems when you're actually implementing them.

So being able to read and write to these environments is a critical part of this process. And so

let's talk about planning first. So the first idea that we kind of encountered is inspired by control systems and robotics. The idea of the pseudo-Colman filter is: as you are working through a plan, what happens is you tend to notice that your agentic systems will be led astray.

Their reasoning will change rapidly, and as you kind of iterate through especially long plans, you can imagine that in order to migrate even a small section of a code base, you might have literally hundreds of steps in a process.

And so inspired by a lot of the folks that on our team come from backgrounds in self-driving and robotics, the pseudo-Colman filter is basically passing intermediate reasoning. And you can get pretty complex about how you modify or share the intermediate reasoning through the different plan steps, but as you pass that intermediate reasoning through the execution of the plan steps, it allows the individual decisions that happen on the plan to slowly converge towards at least consistent reasoning.

The kind of core issue with this is it also facilitates error propagation. A simple mistake, especially early on in the plan, can lead to strong downstream effects. Subtask decomposition. This is pretty well known. Even Reacts, like some of the earlier agentic action systems, have some form of subtask decomposition.

But what we found is that experimenting with different forms or structures of subtask decomposition in our planning process has led to a lot of pretty interesting downstream positive effects. In particular, when you basically increase the resolution or the fidelity of your subtasks in a given plan, it gives you more fine-grained control.

It allows you to define kind of the action space a lot more clearly. However, the risk is you're introducing a lot of decisions for the LLM to make. The more small, tiny tasks you introduce, the harder it is for the system or the agentic system to decide what'sright.

Then you have model predictive control. This is, again, not a new idea, not invented by anyone here, but the idea of evaluating outcomes of your subtasks in your current state and enabling kind of adaptive replanning based on real-time feedback that occurs during the execution of the plan.

So if you have rapidly changing environment or information, this can occur especially in situations for us where there are other humans actively engaged in either a development process or some other workflow while your agent is executing perhaps a long-running workflow.

And so if you have, you can honestly find a lot of information about these techniques and kind of reason around. You don't have to necessarily jump into the true Bayesian statistics, predictive modeling stuff. I think a lot of the general ideas of replanning, taking in the trajectory information, and making sure that you keep that up to date is honestly enough to see some pretty solid quality improvements on your agents.

And then finally, explicit plan criteria. So I think you actually kind of see this pretty often for very simple agents, and then people start to remove explicit plan criteria in order to increase generalizability. But what we've found is that clearly defining kind of successful structures or at least successful initial states for plans can lead to very strong kind of downstream effects.

And this can be done in a lot of different ways: instruction tuning, few-shot prompt examples. You can validate your plans with different kind of hard-coded logic. Ultimately, all of this is about error reduction and keeping your trajectories as successful as possible as long as possible.

But it's difficult to build and scale these because basically what I'm recommending is that you hard-code a lot of logic into your system. I think this is a general idea that people maybe, depending on your domain, this may seem appealing or not appealing.

But ultimately, we're probably not building AGI tomorrow or next week or in the next six months. And so I think taking some of the lessons from the kind of symbolic era of AI is useful if you have a domain challenge that you want to solve in the next three months.

For us, we think about delivering real value to customers today, and that means that a lot of what we do is thinking about things like explicit plan criteria, hard-coded logic. And I think that a lot of folks may not be open to admitting that, but I think it's a very important part of the process of building these kind of agentic systems.

So now talking about decision making. And I know this is also meant to be kind of like a word vomit, honestly, because I think hopefully you guys are building agents and you care a lot about this stuff, and it can spark or inspire some ideas in your own systems when you're working on them.

So first, for decision making, consensus mechanisms. There's a lot of these. Self-consistency is a very popular one. Prompt ensembles, cluster sampling. Basically, the more inference at runtime that you run and the more that you can build a clever way of selecting ideal or optimal samples from those many number of inferences, the higher the accuracy.

It's just going to cost more. It may introduce longer inference wait times if you're not parallelizing. But we found this very important to getting high-quality decisions that are consistent. Next, you have explicit and analogical reasoning. These also have a lot of different names: chain of thought, checklists, chain of density, analogical prompting.

Basically, you want the system to explicitly outline its reasoning or decision-making criteria to reduce the complexity of the decision-making process. So when you have, for example, a checklist of things that, let's say, you want to make a decision about left orright, if you just create a checklist of what constitutes a reasonable left and what constitutes a reasonableright, your system will do better at choosing left orright.

Obviously, though, this introduces challenge if you have very broad domain stuff and/or you have a decision which has a huge action space, in which case techniques like chain of thought reasoning, chain of density, honestly, there's a galactic tree of mega brain thought that there's a lot of these techniques that exist, and I think exploring these is super worth it for improving performance.

Fine-tuning. This is kind of a cop-out answer, but I actually think it is pretty valuable once you really get into having data for specific decisions that you want to make. It may just be true that the best thing to do is to spend a weekend pulling the latest open-source model, generate a bunch of training data with a high-quality model, validate it with a bunch of your team members, and just train or fine-tune a model.

This is expensive, and it locks in the quality of your system. A lot of the benefit of relying on basically being able to sample from different models is every time a new state-of-the-art model comes out, your system gets better.

I think that's a huge benefit. However, for certain decisions, especially those that are really out of distribution, fine-tuning is a pretty effective way to make a good decision. And then simulation. So simulation of decision making is super tricky.

This is definitely going to be very domain specific. If you're working with software development, simulation is luckily kind of built into the thought process here. The ability to execute code, the ability to reason through code trajectories is super doable.

And so for us, simulation makes up a huge amount of how we think about decision-making processes. So sampling multiple decision paths, simulating them both with real and kind of like LLM-imagined execution paths, techniques like language agent tree search, which basically says, amongst this simulation of decision nodes, let's implement a fancy Monte Carlo tree search algorithm.

And it's kind of like Monte Carlo tree search. It's not exactly, but let's do fancy algorithms to decide where we want to move based on the simulation results. And so you can do all these things, and ultimately, the kind of core goal here is you want to have some evaluation of whether or not your system makes good decisions when you want it to make good decisions.

You can take all these techniques, apply them, and then see what's working, what's not working, look at the pros and cons, and then adapt. So that's decision making.

Now we have environmental grounding. So

the first, this is oftentimes called tool use. I think that that's an equivalently valid way to describe AI computer interfaces. But I think the idea of building these is kind of the interesting challenge. So there's dedicated tools that are very common.

You can pull down LangChain and start using a calculator, a sandbox, even Python script execution. I'm pretty sure you can clone an open-source repo today that implements Claude's artifacts in your own environment. It's awesome how great the open-source community has pushed tool use.

But I think a lot of the kind of edge of tool use and where you start to move towards building custom AI computer interfaces is when you need workflows or trajectories which don't exist with the kind of known tool set that you have today.

So if you have a calculator, you can definitely run a calculation. But what if you know that very consistently you are going to use a calculator, take the output of that calculator, and maybe pass it to another system, and then you're going to read and parse the logs, and then take the output of those parsed logs and do some additional transformation?

If you're going to consistently do that and you want your agent to just come up with that every single time, then that's probably not theright mental model for what the agent should be thinking about or making a decision about.

Instead, what you want to do is say, how can we build this tool and then build the interface to the LLM for this tool so that you can kind of streamline those types of actions that you want to take?

And so this is honestly really effective, especially in domains like code where there's tons of dev tools. I mean, there's infinite numbers of tools that people have made to be really good at developing software that are honestly really effective, and they just have kind of weird interfaces.

Maybe it's a CLI. Maybe it's command shift clicking to go to definition on your VS Code. All these things that exist in the world need a way for an LLM-based system to invoke and reason around the outputs. So we spend a ton of our time building these AI computer interfaces.

And then I kind of alluded to this, but designing explicit feedback processing is, I think, a very critical step of grounding your agent in an external environment. In particular, if you have, for example, logs, I think, is a great example of this.

Your CI/CD probably outputs an enormous amount of data about all the tests that ran, all the debug statements, all this kind of garbage that you don't care about. If you know that you're going to need to process that data and use it at some point, it's definitely good to build into the system explicit kind of paths or decisions or tools that take all of this feedback, process it, and then maybe even do a step which is solely about LLM reasoning.

I guess the example here is you have your logs, you parse through it, and then you say, well, I know the LLM is going to want the failing tests. So let me get the failing tests and also provide a brief explainer of what the rest of the logs were doing.

That type of feedback processing is pretty critical into making these systems work really well. It also applies to I'm talking a lot about external tools and external mechanisms, like the LLM or agentic system interacting with something else. There's also the LLM or agentic system interacting with itself.

And so as the agent kind of reflects and reasons about error trajectories, processing that feedback in a meaningful way and passing it back to the LLM is super important because a lot of the time, the LLM is not great at actually criticizing its own action.

It's just good at listening to you tell it to criticize itself. And so that we found to be very important.

Bounded exploration. So you definitely want your agents to be able to gather as much context as they can about the problem space. As you introduce additional ways of gathering this context, probably at the beginning of the problem, not always, but probably, there is a huge benefit for models that can handle this, like very long context models.

If you use Gemini Pro 1.5 or Sonnet 3.5, honestly, there's a lot of models now that have very long context windows. You can continue to include information, but at a certain point, you need to kind of say, let's jump into the problem.

And what we found is that finding theright balance of bounded exploration time is very difficult to actually know in advance and honestly requires a lot of evaluation. And so if you can allow your agent to gather this context, have longer exploration phases, collect data, decide which data is or isn't relevant, and kind of begin the problem with the maximum likelihood of success, then that is one of the single most important things to having a successful trajectory in the kind of extent.

And then human guidance. So this is also a little bit of a cop-out answer, but I think it's important to describe. As you have your agentic system interact with humans, you want to decide when in those interaction patterns do we want to ask the human to interfere or when do we want them to provide guidance.

So

with careful UX design and interaction design, I think that this can be extremely effective at allowing your systems to go from 30 or 40% reliability to 90 or 100% reliability. But ultimately, this is balancing autonomy with human oversight, and so it's a trade-off that you have to make.

Anyway, so those are all the things that we learned at Factory. There's many more. And if you'd like to join us, we are hiring AI engineers, software engineers, go-to-market, everybody. So please give me an email.