Automating Large Scale Refactors with Parallel Agents - Robert Brennan, OpenHands

Allright, thank you all for joining for our "Automating Massive Refactors with Parallel Agents." I'm super excited to talk to you all today about, you know, what we're doing with OpenHands to really automate large-scale chunks of software engineering work—lots of, uh, lots of toil related to tech debt, code maintenance, code modernization.

These are tasks that are super automatable; you can throw agents at them, but they tend to be way too big for, like, you know, a single agent, just one-shot. So it involves a lot of what we call agent orchestration.

We're going to talk a little bit about how we do that with OpenHands, and also just more generically.

A little bit about me: my name is Robert Brennan. I'm the co-founder and CEO at OpenHands. My background is in dev tooling; I've been working in open-source dev tools for over a decade now. I've also been working in natural language processing for about the same amount of time.

I've been really excited over the last few years to see those two fields suddenly converge as LLMs get really good at writing code, and I'm super excited to be working in this space. Then, OpenHands is an MIT-licensed coding agent.

OpenHands started as OpenDevin about a year and a half ago, when Devin first launched their demo video of a fully autonomous software engineering agent. My co-founders and I saw that and got super excited about, you know, what was possible, what the future of software engineering might look like, but realized that that shouldn't happen in a black box,right?

If our jobs are going to change, we want that change to be driven by the software development community. We want to have a say in that change. And so we started OpenHands, then OpenDevin, as a way to give the community a way to help drive what the future of software engineering might look like in an AI-powered world.

So hopefully not controversial for me to say that software development is changing. I know my workflow has changed a great deal in the last year and a half. I would say now, like, you know, pretty much every line of code that I write goes through an agent.

Rather than me opening up my IDE and typing out lines of code, I'm now asking an agent to do the work for me. I'm still, you know, doing a lot of critical thinking. You know, a lot of the mentality of the job hasn't changed, but what the actual work looks like has changed quite a bit.

But what I want to convince you all of is that it's still changing. We're still just in the first innings of this change. We still haven't realized all the, um, all the impact that large language models have already brought to the job and are going to continue to bring to the job as they improve.

I would say even if you froze large language models today and they didn't get any better, you would still see the job of software engineering changing very drastically over the next 2 to 3 years as we figure out ways to operationalize the technology.

I think there's still a lot of, sort of, psychological and organizational hurdles to adopting large language models within software engineering, and we're seeing a lot of those hurdles disappear as time goes on.

A brief history of kind of how we got here: everything started, I would say, with what I call context-unaware code snippets. Some of the first large language models, it turned out, were very good at writing chunks of code, especially things that they'd seen over and over again.

So you could ask it to write BubbleStory, you could ask it for, you know, small algorithms, you know, how to access a SQL database, things like that. And it was able to generate little bits of code. It was able to, you know, it seemed to understand the logic a bit.

But this was totally context-unaware,right? It was just dropping code into a chat window that you would ask for. It had no idea what project you were working on, what the context was. Shortly thereafter, we got these context-aware code generation tools.

So, like, GitHub Copilot as output was probably like the best example here,right? So it was in your IDE, it could see, you know, where you're typing, you know, what the code you're working on in. And it could generate code that was specific to your codebase that referenced, you know, local variable names, that referenced, you know, local table names in your database.

Huge, huge improvement for, you know, our productivity. So instead of copy-pasting back and forth between the ChatGPT window and your IDE, now all of a sudden you could see the little robot that has eyes, it could see inside your codebase, and it can actually generate relevant code for your codebase.

And then I think the giant leap happened in early 2024 with the launch of Devin, and then the next day the launch of OpenDevin, now OpenHands. This is where we first started to see autonomous coding agents. So this is when AI started not just writing code, but it could run the code that it wrote.

And it could Google an error message that came out, find a Stack Overflow article, apply that to the code, add some debug statements into the code, and run it and see what happens. Basically automating the entire inner loop of development.

This was a huge step function forward. You can see the little robot gets arms in this picture. This was a huge jump, at least in my own productivity, being able to, like, just write a couple sentences of English, give it to an agent, and let it churn through the task until it's got something that's actually working, running, tested, passing.

And then now what we're seeing is parallel agents, what we're calling agent orchestration. Folks are figuring out how to get multiple agents working in parallel, sometimes talking to each other, sometimes spinning up new agents under the hood. You have agents creating agents.

This is, I would say, kind of bleeding edge of what's possible. People are just starting to experiment with this, just starting to see success with this at scale. But there are some really good tasks that are

very amenable to this sort of workflow, and it has the potential to really automate away a huge amount of tech debt that sits under, you know, every contemporary software company.

A little bit about kind of like the market landscape here. Again, you can kind of see that same evolution from left toright. We really started with, you know, plugins like GitHub Copilot inside of our existing IDEs, then we got these, like, AI-empowered IDEs, IDEs with, like, AI tacked onto them.

I would say your median developer is kind of adopting local agents now and maybe running Claude Code locally for one or two things, maybe some ad-hoc tasks. Your early adopters, though, are starting to look at cloud-based agents, agents that get their own sandbox running in the cloud.

This allows those early adopters to run as many agents as they want in parallel. It allows them to run those agents much more autonomously than if they were running on their local laptop. But if it's running on your local laptop, there's nothing stopping the agent from doing rmrf/, trying to delete everything in your home directory, whatever it might do, installing some weird software.

Whereas if it's got its own, like, containerized environment somewhere in the cloud, you can run a little bit more safely knowing that, you know, the worst it can do is ruin its own environment. And you don't have to, like, sit there babysitting it hitting the Y key every time it wants to run a command.

So those cloud-based environments are much more scalable, a bit more secure. And then I would say at the farright here, what we're really just seeing the top, like, 1% of early adopters start to experiment with is orchestration. This idea that you don't only have these agents running in the cloud, but you have them talking to each other, you're coordinating those agents, you know, on a larger task.

Maybe those agents are spinning up sub-agents within the cloud that have their own sandbox environments. Some really cool stuff happening there. I would say, you know, with OpenHands, we generally started with cloud agents. We've leaned back a little bit and built a local CLI similar to Claude Code in order to meet developers where they are today.

You know, these types of experiences are much more comfortable for developers. You know, we've been using Autocomplete for decades, just got a million times better with GitHub Copilot. I would say these experiences on theright side are very foreign to developers.

They feel very strange to, like, give off the task to an agent or a fleet of agents and let them do the work for you. It feels kind of like, for me at least, the jump that I made when I went from being an IC to being a manager is what it feels like going from writing code myself to giving that code to agents.

So very, very different way of working. I think one that developers have been very slow to adopt. But again, the top 1% or so of engineers that we've seen adopt the stuff on theright side of this landscape, they've been able to get, you know, massive, massive lifts in productivity and tackle huge backlogs of tech debt that their teams just weren't getting to.

Some examples of where you would want to use orchestration rather than a single agent: typically these are tasks that are going to be very repeatable and very automatable. So some examples are things like, you know, basic code maintenance tasks,right?

Every codebase has to, you know, there's a certain amount of work to do to just keep the lights on,right? To keep dependencies up to date, to make sure that any vulnerabilities get solved. We have one client, for instance, that is using OpenHands to remediate CVEs throughout their entire codebase.

They have tens of thousands of developers, thousands and thousands of repositories, and basically every time a new vulnerability gets announced in an open-source project, they have to go through their entire codebase, figure out which of the repos are vulnerable, submit a pull request to that codebase to actually, you know, resolve the CVE, update whatever dependency, fix breaking API changes.

And they have seen a 30x improvement on time-to-resolution for these CVEs by doing orchestration at scale. They basically have a setup now where every time a new CVE gets announced, new vulnerability comes in, they kick off an OpenHands session to scan a repo for that vulnerability, make any code changes that are necessary, and open up a pull request.

And all the downstream team has to do is click merge validated changes. You can also do this for, like, automating documentation and release notes. There's a bunch of modernization challenges that companies face. For instance, you might want to add pipe annotations to your Python codebase if you're working in Python 3.

You might want to split your Java, you know, like a monolith into microservices. These are the sorts of tasks that are still going to take a lot of thought for an engineer. You know, you can't just, like, again, one-shot it with Claude Code and say, like, you know, refactor my monolith into microservices.

But it is still very rote work,right? You're still just kind of like copying and pasting a lot of code around. So if you thoughtfully orchestrate agents together, they can do this. A lot of migration stuff. So migrating from, like, old versions of Java to new versions of Java.

We're working with one client to migrate a bunch of Spark 2 jobs to Spark 3. We've used OpenHands to migrate our entire front end from React from Redux to Zustand. So you can do these very large migrations. Again, lots of very rote work.

It still takes a lot of thinking from a human about how to orchestrate these agents. And then a lot of tech debt: detecting unused code, getting rid of that. You know, we have one client who is using our SDK to basically scan their data dot logs every time there's a new error pattern, go into the codebase, and add their error handling to fix whatever problem is cropping up.

So lots of things that, you know, are a little too big for a single agent to just one-shot, but are super automatable, are good tasks to handle with an agent as long as you're thoughtful about orchestrating them.

A bit about why these aren't one-shottable tasks: some of them are technological problems, some of them are more like human psychological problems. On the technology side, you have a limited amount of context that you can give to the agent.

So extremely long-running tasks are tasks that span, like, a very large codebase. Usually, you don't really have enough there. You're going to have to compact that context window to the point where the agent might get lost. We've all seen the laziness problem.

I've tried to one-shot some of these types of tasks, and the agent will say, "Okay, I've migrated three of your hundred services. I need to hire a team of six people to do the rest."

The agents often lack domain knowledge within your codebase,right? They don't have the same intuition that you do for the problem. And errors compound when you go on these really long trajectories with an agent. A tiny error in the beginning is going to, you know, compound over time.

The agent is going to basically repeat that error over and over and over again for every single step that it takes in its task. And then on the human side, you know, we do have this intuition for the problem that we can't convey.

You know, say you want to break your monolith into microservices, you probably have a mental model of how that's going to work. If you just tell the agent, "Break the monolith into microservices," it's just going to take a shot in the dark based on patterns it's seen in the past without any real understanding of your codebase.

We have some difficulty decomposing tasks for agents and understanding, like, what an agent can actually get done in one shot.

We also, like, you do need this intermediate review, intermediate check-in from the human as the agent's doing its work. We'll talk a little bit about what that loop looks like later. But it's, again, not something you can just, like, tell an agent to do and expect the final result to come in.

You have to kind of approve things as the agent goes along. And then not having a clear definition of that. I think if you don't really know what finished looks like for this project, it's hard to tell the agent.

On these types of orchestration tasks, I want to make it super clear that we don't expect every developer to be doing agent orchestration. We think most developers are going to use a single agent locally for, you know, sort of ad-hoc tasks that aren't common for engineers: building new features, fixing a bug, things like that.

I think running Claude Code locally in a familiar environment alongside an IDE is probably going to be a common workflow, at least for the next couple years. What we're seeing is that a small percentage of engineers who are early adopters of agents or really excited about agents are finding ways to orchestrate agents to tackle, like, huge mountains of tech debt at scale and get a much bigger lift in productivity for that smaller select set of tasks,right?

You're not going to see a 3,000% lift in productivity for all software engineering. Probably going to get more of that, you know, 20% lift that everybody's been reporting. But for some select tasks, like CVE remediation or codebase modernization, you can get a massive, massive lift.

You can do, you know, engineer years of work in a couple weeks.

I want to talk a little bit about what these workflows look like in practice. So this loop probably looks pretty familiar if you're used to working with local agents. This is a very typical loop that looks a lot like the inner loop of development for, you know, non-AI coding as well.

But basically, you know, you give the agent some prompt, it does some work in the background, maybe you babysit it and watch, you know, everything it's doing and hit the Y key every time it wants to run a command.

Then the agent finishes, you look at the output, you see the tests are passing, you see if this actually satisfies what you've asked for, and then maybe you prompt the agent again to get it to get a little closer to the answer.

Or maybe you're satisfied with the result, you, you know, you commit the results and push.

For bigger orchestrated tasks, this becomes a little bit more complicated. Basically, what you need to do is you or maybe hand in hand with Claude, you want to decompose your task into a series of tasks that can be executed individually by agents.

Then you'll send off an agent for each one of those individual tasks, and you'll do one of those agents for each of the individual tasks. And then finally, at the end, you, maybe with the help of an agent, are going to need to collate all the output together from all those individual agents into a single change and merge that into your codebase.

Very importantly, there's still a lot of human-in-the-loop here. You need to review not just the final output of the collated result, but the intermediate outputs for each agent. I like to tell folks the goal is not to automate this process 100%.

It's something like 90% automation. That's still, you know, an order of magnitude productivity lift. I think this is really tricky to getright. This is where a lot of, like, thought comes into the process of, like, how am I going to break the task down so that I can verify each individual step and so that I can actually automate this whole process without just ending up with a viable mess.

This is a typical Git workflow that I like to use for tasks like this. Typically, we'll start a new branch on our repository. We might add some high-level context to that branch using, like, an agent.md or an OpenHands view, the concept of a micro-agent.

But adding just a markdown explaining, you know, here's what we're doing here, just so the agent knows, "Okay, we're migrating from Redux to Zustand," or, "We're going to migrate these Spark 2 jobs to Spark 3." You might want to put some kind of scaffolding in place.

I'll talk a little bit more about examples of scaffolding later. You're going to create a bunch of agents based on that first branch. The idea is that they're going to be submitting their work into that branch, and it's basically going to accumulate our work as we go along.

And then eventually, once we get to the end, we can rip out our scaffolding and merge that branch into main. Now, for if you're kind of getting started with this, I would suggest limiting yourself to about three to five concurrent agents.

If I need more than that, your brain starts to break. But for folks that have really adopted orchestration at scale, we see them running hundreds, even thousands of agents concurrently. Usually, a human is not in the loop for, you know, one human is not on the hook to review every single one, but maybe those agents are sending out pull requests to individual teams, things like that.

So you can scale up very aggressively once you start to get a feel for how all this works and you feel like you have a very good way of getting that human input into the loop.

I'm going to kick it off to my coworker Calvin here. He's going to talk about a very, very large-scale migration, basically eliminating code smells from the OpenHands database that he did using our refactor SDK. We'll shift off here.

Hi, Calvin.

OpenHands excels at solving topic-loaded tasks. Give it a focused problem, something like, "Fix my failing CI, add and debug this endpoint," and it delivers. But like all agents, it can stumble when the scope grows too large. Let's say I want to refactor an entire codebase.

Maybe enforce your type checking, update a major dependency, or even migrate from one framework to another. These are not atomic tasks. They're sprawling, interconnected changes that can touch hundreds of files. To tackle problems at this scale, we're using the OpenHands Agent SDK to build tools designed specifically to orchestrate collaboration between humans and multiple agents.

As an example, let's work to eliminate code smells from the OpenHands repository. Here's the repository structure. Just the core agent definition has about 380 files spanning 60,000 lines of code. This is a lot about the volume of the code, but not much about the structure.

So let's use our new tools to visualize the dependency graph of this chunk of the repository. Here, each node represents a file. The edges show dependencies, who imports who. And as we keep zooming out, it becomes clear this tangled web is why refactoring at scale is hard.

To make this manageable, we need to break this graph up into human-sized chunks. Think PR-sized batches that an agent can handle and a human can understand. There are many ways to batch based on what's important to you. Graph theoretic algorithms give strong guarantees about the structure of edges in between induced batches.

But for our purposes, we can simply use the existing directory structure to make sure that semantically related files appear inside the same batch. Navigating back to the dependency graph, we can see that the codes of the nodes are no longer randomly distributed.

Instead, they correspond to the batch that each of those associated files exist. Zooming out and zooming back in, we easily find a cluster of adjacent nodes that are all the same color, which indicates that an agent is going to access all of those files simultaneously.

Of course, this graph is still large and incredibly tangible. If you construct a simpler view, we'll build a new graph where nodes are batches and the edges between those nodes are dependencies that are inherited from the files within each of those batches.

This view is much simpler. We can see the entire structure on our screen at the same time. But this is not the only view we have of batches. Using a graph, we can identify batches that have node indices and then extract the files that go into them.

This batch, for example, has 16 files. Looks like it's an empty file. It's probably empty. Let's check. Now, this is the tool intended for human and AI collaboration. So once we know that this file is empty, we might determine that it's better to move it elsewhere.

Or maybe we're okay keeping it inside this batch, and all that we want to do is add a note to ourselves or rename the batch so we know the contents. Of course, when refactoring code, it's important to consider the complexity of what it is you're really looking for.

This batch is trivial. Let's find one that's a little bit more complex. Here's a batch that has four files. They all seem to be based in the original key, and the complexity measures reflect this. These are useful indicators to achieve that we should be more careful when manipulating this batch than, for example, the first one we examined.

So how are we actually going to get rid of code smells? Well, it's a two-step process to get rid of these bugsright here. Before we can fix the code by removing code smells, we need to identify what's wrong in the first place.

Enter the verifier. There are several different ways of defining the verifier based on what you care about. You can set it to be programmatic, so it calls a batch command. This is useful if your verification is checking unit tests or running a linter or a type check.

Instead, though, because I'm interested in code smells, I'm going to be using a language model that's going to be looking at the code and trying to identify any problematic patterns based on a set of rules that I've provided.

Now, let's go back to our first batch and actually put this verifier to use. Remember, this batch is trivial, and fortunately, the verifier recognizes it as such. It comes back with an excel report indicating which code smells it identified and which it didn't.

And the status of this batch is turned to completed, green, good. And this change in status is also reflected in the batch graph. Navigating back and toggling the color display, we can see that we have exactly one node out of many completed, and the rest are still yet to be handled.

But this already gives us a really good sense of the work that we've done and how it fits into the bigger picture. So now our strategy for ensuring that there are no code smells in the entirety of our repository is straightforward.

We just have to ensure that every single node on this batch graph turns green. So let's go back to our batches and continue verifying until we run across a failure. We'll keep going in dependency, making sure that we pick nodes that don't have any dependencies on other batches that we have yet to analyze.

This next batch is about as simple as the first, but because the init file is a little bit more complex, the report that gets generated is a little bit more verbose. Continuing down the list, we come across the batch we identified earlier with some chunky files and relatively high code complexity.

And this batch happens to give us our first very trivial failure. Notice that the status turns red instead of green. Now, this batch has more files than what we've seen in the past, so the verification report is proportionally longer.

Looking through, we see that it is listing file by file the code smells that it's identified and which branches. I see one file is particularly egregious with its violations. We'll have to come back to that in a moment.

And if we zoom all the way back out to the batch graph and look at the status indicators, we'll see the two green nodes representing the batches we've already successfully verified. We'll also see the red node representing the batch that we just saw fail verification.

Now, our stated goal is to turn this entire graph green. This red node presents a little bit of an issue. To convert this red node into a green node, we need to address the problems that the verifier found using the next step of the pipeline, the fixer.

Just like the verifier, the fixer can be defined in a number of different ways. The programmatic fixer can run a batch command, or you can feed and have a batch into a language model and hope that it addresses the issues in a single step.

But by far the most powerful fixer that we have uses the OpenHands Agent SDK to make a clean copy of the code and spin up an agent that has access to all sorts of tools to run tests, examine the code, look at documentation on the internet, and do whatever it needs to do to address these issues.

So let's go back to the spin-up batch and run the fixer and see what happens. Now, this part of the demo is sped up considerably, but because we're exploring these batches in dependency order, while we're waiting, we can continue to drill down the list, run our verifiers, and spinning up new instances of the OpenHands Agent using the SDK until we come across a node that's blocked because one of its stream dependencies is still complete.

When the fixer is done, the status of the batch is reset. We'll need to rerun verification in the future to make sure that we associate it with OpenHands again. Looking at the report that the fixer has returned, there's not much information, just the title of a PR.

We've set this up so that every fixer produces a nice, tidy forward request ready for human approval. Just because the refactor is automated doesn't mean it needs to be a human. And here's the generated PR. The agent has done an excellent job of summarizing the code smells it identified, the changes it made to address those, as well as any changes it decided not to make.

It's also left some help notes for the reviewer and some notes for anybody working on this part of the code in future agent development labs.

And when we look at the contents of this PR, we see it's very recent. All of the changes are tightly focused on addressing the code smells that we provided earlier, and we've only modified a couple hundred lines of code, the bulk of which is simply refactoring a deeply nested block into its own function call.

Not all PRs are built to be this small, but our batching strategy and narrow instructions ensure that the scope of the changes are well considered. This helps to improve agent performance, but it also alleviates the reviewer burden down the line.

From here, the full process for removing code smells from the entirety of the OpenHands codebase becomes clear. Use the verifier to identify problems. Use the fixer to spin up PRs that address those problems. Review and merge those PRs, unblock new fixes, and repeat until that entire graph turns green.

We've already used this tool to make some pretty significant changes to the OpenHands codebase, including strict typing and improving test coverage. And we could have done it without the OpenHands Agent SDK powering everything under the hood.

Allright. So that's the OpenHands refactor SDK powered by our OpenHands Agent SDK. We're going to walk through a little bit later on the workshop how to build something a little simpler, but very similar, where we have parallel agents working together to fix tasks that were discovered by the initial agent.

I want to talk a little bit about strategy for both decomposing tasks and sharing context between these agents. Those are both really big, important parts of agent orchestration. So to start with task decomposition, you're really looking to break down your very big problem into tasks that a single agent can solve, a single agent can one-shot.

Something that can fit in a single commit, single pull request. Super, super important because you don't want to be constantly iterating with each of the sub-agents. You want each one you want a pretty good guarantee that each one is just going to one-shot the thing.

You'll be able to rubber stamp it and get it merged into your ongoing branch. You want to look for things that can be parallelized. This is going to be a huge way to increase the speed of the task.

If you're just executing a bunch of different agents serially, you might as well just have a single agent moving through the task serially. The more you can parallelize, the more you can get many agents working at once, the faster you're going to be able to move through the task and iterate.

You want things that you can verify as correct very easily and quickly. Ideally, you'll have something where you can just look at the CI/CD status and have good confidence that if everything's green, you're good. Maybe you'll need to click through the application itself, something like that, run a command yourself to verify that things look good to you.

But you want to be able to very quickly understand whether an agent has done the work you asked it to or not. And you want to have clear dependencies in order between tasks. You'll notice these criteria are pretty similar to how you might break down work for an engineering team,right?

You need to make sure that you have tasks that are neatly separable, tasks that different people on your team can execute in parallel, and then collect the results together. You want to know once I get task A done, then I'll launch task B, C, and D, and then once those are done, we can do E.

So very similar to breaking down work for a team of engineers.

There are a few different strategies for breaking down a very large refactor like the one we talked about just now. The simplest, like most, that you want is to just go piece by piece. You might iterate through every file in your repository, every directory, maybe every function or class.

This is a fairly straightforward way to do things. It works well if those dependencies can be kind of executed without depending on one another too much. So good examples might be adding type annotations throughout your Python codebase.

And then at the very end, once you've migrated every single file, say, you can collate all those results into a single PR.

A slightly more sophisticated thing would be to create a dependency tree. And the idea here is to add some ordering to that piece-by-piece approach where you start, as we saw Calvin do, you start with the leaf notes in your dependency graph,right?

You start with maybe your utility files, get those migrated over. And then anything that depends on those, it's going to have those initial fixes in place, and the dependencies can start working through their set of the process. You can basically back your way up to whatever the entry point of the application is.

This is often a better way to proceed. It's more kind of like a principled approach for how you're going to order through these tasks.

Another example is to create some kind of scaffolding that allows you to live in both the pre-migrated and post-migrated worlds. We did this, for example, when migrating our React state management system. We basically had an agent set up some scaffolding that would allow us to work with both Redux and Zustand at the same time.

Pretty ugly, not something you would actually really want to do, but it allowed us to test the application as each individual component got migrated from the old state management system to the new state management system. And then we set up parallel agents for each of the components, got each component done, and then at the very end, once everything was using Zustand, we were able to rip out all of the scaffolding so there was no more mention of Redux and everything was working.

But having that scaffolding in place allowed us to validate, as each agent finished its work for just that one component, we could validate the application was still working, that component still works. We didn't have to do everything all at once before we got some kind of human feedback for the agents.

Next, I want to talk a bit about context sharing. As you go through a big large-scale project like this, you're going to learn things,right? You're going to figure out, okay, my original mental model wasn't actually complete. I didn't actually understand the problem correctly.

Your agents might run into you might have a fleet of agents, you've got 10 agents running, they're all hitting the exact same problem. You kind of want to share the solution to that problem so they're not all getting stuck,right?

There's a bunch of different strategies for doing this context sharing between agents.

One strategy, the most naive thing you can do, is share everything. Basically, every agent sees every other agent's context. This is not great. It's basically the same thing as just having a single agent working iteratively through the task.

You're going to bleed your context window really quickly if you do something like this. So this is not going to help.

A better naive approach would be to have the human being just sort of manually enter information into the agents. If you have a chat message, a chat window with each agent, you can just paste in, like, hey, use library 1.2.3 instead of 1.2.2.

The human can also modify like an agent.md or an OpenHands microagent to pass messages to these agents. But this doesn't involve manual human effort. It involves a lot more like babysitting of the agents, so it's not super scalable.

You can also have the agents basically share context with each other through a file, like agent.md. You can allow the agents to actually modify this file themselves. Maybe they send a pull request into the file as they learn new things.

Downside here is that sometimes agents will try and learn unimportant things. They can get kind of aggressive about pushing information to this file. So doing some kind of human review seems to help.

And then last, this is probably the most bleeding edge idea here, but you can basically give each agent a tool that allows it to send messages to other agents. It could be like a broadcast message that goes out to all the other agents, or it could be point-to-point conversation.

This is super fun to experiment with. We're doing a lot to experiment with this now with our SDK, but it's tricky to getright. Once you get agents talking to each other, you're increasing the level of nondeterministicity in the system.

Things can get a little hairy. I have an example here on theright. This is from another project's report. They had two agents just talk to each other. They just entered into a loop of wishing each other Zen perfection.

Cool. Now I want to work through an exercise. I would love it if you all want to follow along. You can access this presentation for copy-pasting purposes at dub.sh/openhands-workshop. We'll work through some coding exercises with the OpenHands SDK, specifically to do CDE remediation at scale.

We're going to write a script that will take in a GitHub repository, scan it for open source vulnerabilities, for CVEs, and then set up a parallel agent for every single vulnerability we find to solve that and open up a pull request.

So dub.sh/openhands-workshop. Let me know if anybody can't access it.

It's going to be the slideshow.

So it should be the slideshow. If you want to, there will be

copy-pasteable prompts and links and stuff like that around slide 29.

Got it.

I'll get there.

So in terms of how this process is going to work, basically we're going to start with one agent that runs a CVE scan on this repository. It's going to scan for vulnerabilities. What's nice about using an agent for this is it can look at

the repository and decide, well, how am I going to scan for vulnerabilities,right? Am I going to use Trivy to scan a Docker image? Am I going to run npm audit on a package.json? So it can basically detect the programming language and figure out, how am I going to scan for CVEs here?

Then once we have our list of vulnerabilities, we're going to run a separate agent for each individual vulnerability. Each of these agents is going to research whether or not it's solvable. It's going to update the relevant dependency, fix any breaking API changes throughout the codebase, and then open up a pull request.

What's nice about this is that we can merge those individual PRs once they're ready.

Can you show the link again?

Yeah.

What's nice about running the solving in parallel is that we get a bunch of different PRs, so we can merge them as they're ready. If one agent gets stuck, one of the vulnerabilities is unsolvable, all the other ones are still going to work.

Maybe we get to 90% or 95% solved. We don't have to get to 100% in order to have any value here. Just some quick pseudocode of what this is going to look like. So this is an example using the OpenHands SDK of how to create an agent.

You can see we create a large language model. We then pass that large language model to an agent object along with some tools, like a terminal, a file editor, a task tracker for planning. We give it a workspace, and then we just tell it what we want it to do and hit run.

This is a pretty naive hello world example. We'll see how it gets a little bit more complicated as we progress through this particular task. But then once that first agent is done, we're going to iterate through all of the vulnerabilities to get back out.

And then for each one, we'll send out a new agent asking it to solve that particular CVE.

Allright. So to get started here, I would say create a new GitHub repository. We can start to save our work there. You're also going to need both a GitHub token and an LLM token.

If you sign up for openhands@app.allhands.dev, you can get a $10 free credit of LLM credits there. If you're already an existing user, let me know and I can bump up your existing credits for the purpose of this exercise.

Then we're going to start an agent server. This is

basically like a Docker container that's going to house all the work that our agents are doing. This is a great way, again, to run agents securely and more scalably. So instead of running the agents on our local machine to solve all these CVEs, we're going to run them inside of a container.

Hypothetically, if we were doing thousands of CVEs, we could run this in a Kubernetes cluster so that we have as many workstations as we want for our agents. But for the purposes of this exercise, we'll just run one Docker container as a setup for our agents.

Then we can create an agent.md or an OpenHands microagent to start working through this task. I'm going to be using the OpenHands CLI as we go here. You're welcome to check out the OpenHands CLI. You can also use Cursor or Claude Code or whatever you're used to using as we kind of bifurcate our way through a CVE remediation process with OpenHands.

I'm going to give it a couple of minutes. I'm going to walk through creating my GitHub repo, getting my GitHub token, etc.

If you all have any trouble, feel free to raise your hand. I can come around and help getting an LLM token, etc.

You said app.allhands.dev?

App.allhands.dev, yeah.

So I've got my new GitHub repo here.

So I'm going to add a quick OpenHands microagent here.

I'm just going to tell OpenHands building a new

process for remediating CVEs with agents.

Relevant docs for the OpenHands SDK are at https://openhands.dev/openhands-sdk/ref. So this will give OpenHands a little bit of context. Similar to agents.md, we know how to just read code out of me.

To get a token, I'm not actually going to do it here so that it won't close my token, but you can go to GitHub settings, under your profile,

then developer settings,

personal access tokens. I like to do classic tokens.

Generate a new classic token, give it a name, and then the repo scope is really what you'll need. That way, we can open up pull requests to solve each of the CVEs involved.

Are you doing a classic token, not the new thing?

I haven't gotten

used to this exercise. You're welcome to do. I guess you could create a new repository.

I haven't gotten used to them either, so I'm with you.

Stuck in the old days.

So what permissions do we need to.

Just the repo permission.

I'm also just going to show you sign up for app.allhands.dev.

If you go to the app keys under your profile here, you can get your OpenHands API key, your LLM key here. I won't show it, but this will allow you to use our LLM proxy.

Last, I'm going to start up some agent server here. You'll probably want to copy-paste this out of the presentation.

Got my repo closed.

Run the agent server container.

Back here. If you do want to work with the OpenHands CLI,

you can use the tool install OpenHands.

I'm going to start up the OpenHands CLI. Again, you can use Claude Code, Cursor, whatever else that you want. I think folks need a little more time with the setup. Get an LLM key, get a token set up, start checking through.

So I'm going to start with this first prompt. Basically, what we're going to do is we're going to point our agent at the OpenHands SDK, point it at the documentation, and just ask it to basically check that our LLM API key is working, that it can actually do an LLM definition.

This will be like a very basic hello world, so it can just kind of get started here. I'm going to tell it I'm using the OpenHands key that I generated at app.allhands.dev. So I'm telling it to use this openhands/claude-song4 model.

You can replace this with Anthropic if you want to use just like a regular Anthropic API key. You may need to set this model a bit differently depending on if you're using OpenAI. It's using my LLMs here in the pitch.

You can look at the LLM docs to figure out if you have an OpenAI key or an OpenAI key. You can look at the LLM docs to figure out which model you might want to plug in for this string.

But I'm just going to copy-paste this as is.

Sorry, what was the step for agents.md or the one for OpenHands?

So I would say just create a file, either the agents.md if you're working with a tool that's compatible with that, or for OpenHands, we have what's called a microagent. I can get to it. So openhands/.openhands/microagent/byconventionrepo.md is the description of the repository you're in.

And I just gave it a couple of links to the SDK documentation and the repository for the SDK so that it has access to basically the API docs there.

This is kind of an optional step. It'll make things a little easier, though.

See how the agent's doing. Allright. It thinks it's got something good. So let's see what's going on. Python CVE solver.

We need environment variables.

I'm using Vera to set my environment variables here. Make sure I don't type those in.

One more time.

Got a small error. It's like the agent didn't quite get the API docsright. Let's paste the error back to what happens.

Allright. Try again.

Of course, live demos never go.

Got an auth error.

See if it's working.

Version security.

Let's use Claude Code.

Do you have any tools on OpenHands that breaks?

Yeah.

You know what version of UV you're on?

I'm on 0.9.6. 0.9.6.

What error are you getting?

0.9.11.

I don't know why.

Going to build the same as very. No, it's a table

provided by package OpenHands. Removing tool error failed to install entry points.

I'm new-ish to the Python world, so I assumed I was good.

Silly.

You could try updating to 0.9.11, which is what I'm on, but.

Okay. Yeah, I'll try.

Another question.

Yeah?

So I was able to, after you ran through the CLI, I was able to run this on the allhands.dev.

Yeah, cool.

And it submitted a PR and created it. Looks good.

Awesome.

Why are you doing it through the CLI?

Really just for

normally, I actually prefer to work through the web UI here.

I think being able to run and show that the script is working locally is a little bit better to hand off. I actually like to work through the web UI normally and then have the agent push and I pull locally if I really want to work locally.

But I figured that was just extra steps for presenting purposes. But yeah, feel free to use the web or the tool.

Looks like I got myright API key set up here. Just got to fix the output.

What's that? Should we get 200?

Yeah, you should get something like this. I think I just got finally where the LLM says hello OpenHands.

I'm just going to.

Others. Lens action.

Anybody manage to get an LLM connection working?

I think so. I created the file.

Nice.

Just a quick review of what this looks like.

I have it in the.

First, let's start the agent server.

Here. Basically, you can see we create an LLM, tell what model we want to use, tell the API key we want it to use, and then just send a quick message to the LLM to make sure it's actually working.

Allright. For the second time, I'm going to move towards prompt 2. So here, we're going to actually start to do some work with agents. So we're going to tell the agent we're working with, we want to use the SDK to create a new agent that's going to take in a GitHub repository.

It's going to connect to a remote workspace running at mobile host 8000. Again, that's the Docker start command from before. If you haven't already run that, now's a good time to get Docker running. Docker run this agent server.

It's going to clone our repository into that Docker container. We're going to create an agent that's going to work inside that Docker container. And we're going to tell that agent to scan this repository for enable or replace.

With the OpenHands CLI, is there a way to interrupt and get it to stop?

I think Ctrl+P will pause. Yeah.

And then can I insert corrections?

Yeah, then you can type a new message or just type continue.

So I got the CLI to install, but I had to add dash AI.

Seems on PyPy that there's a dash AI version, but then it says in the docs that it's I don't know. I think the AI one is duplicated, but it is a usable CLI if you want to use that.

I'm going to service that one up with our team.

Did you get the dash AI one to work? Because as soon as I tried to run it, it crashed.

Oh. Oops. It installed. I was so happy.

It installed and then it didn't work.

It does a deprecation warning when I go to version, so yeah, fair enough.

There is if you want to download an executable binary on our release page. That might be straightforward. You can also run it in a Docker container

if you do our CLI docs. I think there's a UV run as well.

Try UV run.

You got the version. You got the version? Yeah.

I think it was the wrong one.

Oh, interesting.

So you can try adding a patch version. That's for the not OpenHands AI.

The regular OpenHands?

The regular OpenHands.

Okay. Thank you.

Allright. Supposedly have an agent working here. Let's see. I'm going to run it with the virus repo. It should have a few CVEs in it. Let's see if we find any vulnerable. By default, OpenHands will

visualize the output here so we can see the agent working even with the SDK. Very similar to how we saw the CLI looking. You can see it's updating its task list. It's

going to the repository.

It doesn't have Trivy installed, so it's installing Trivy. It's basically doing what we would expect an agent to do given the tasks we've added to it.

So we're running Trivy now.

Show a bit about what this generated code looks like. You can see, so we instantiated our LLM in the first step. Now we're actually passing this LLM to an agent. We're also giving it a terminal tool and a file editor tool.

We're creating this remote workspace that's connecting to our Docker container so that the agent can start working in its own environment. Then we create what's called a conversation, which is basically one chunk of context that the agent's going to manage as it goes about its work.

We pass it a task with some clear instructions for what it's supposed to do. Then send that task to the conversation.

Looks like that initial scanner agent is almost done.

Looks like that agent ran just fine. Get these results.

At the time, I'll keep plugging along here. We've got an agent that's scanning for vulnerabilities.

So the next thing I'm going to ask this to do is basically we're going to breach into the agent's environment and get the vulnerability list out from it. The idea is we're going to have it save status to vulnerabilities to a JSON file.

Then we can, on that workspace object inside of the Docker container, we can run an execute command in order to get those vulnerabilities back out. We also have some functions for manipulating files within the workspace. And then for now, we're just going to iterate over the vulnerabilities in that JSON file and print them just so we can see if we were able to breach into this workspace and get some information back out.

Allright. Supposedly good to go. See what happens. Should it run it?

Actually, we've got some vulnerability results.

Agent's finished. Let's see if our script can get

results back.

Here, parse in JSON.

We have a hard time. The cause I was going to GPT-5 codecs moving. Agent messages.

What is the observation event?

So for every agent that opens the interface, there's an action and then an observation. So it might be run this command, and then an observation goes back with the output of that command.

Is it a hook?

It's more core than a hook. It's

basically the entire trajectory the agent takes is made up of events. And then there's two kinds of events, actions and observations. So OpenHands, whenever we make calls to the LLM, it comes back with an action to take from basically a tool call.

And then the observation is like a tool call.

If anyone's stuck on anything, I'm happy to come around too. Feel free to raise a hand.

What's that?

Number four?

I just did number three.

Three?

Nice. Yeah. It looks like it's printing the CVE list. Perfect. Yeah, that looks good. Good job. That's good. That's good. That's good. That's good.

I'm just worried that you have a specific scheme or sub-agent for each script you're running. Why are you authorizing or manipulating the same file again and again?

So the process we're going through here with the five prompts is really

to demonstrate what it would feel like to actually build with our SDK,right? This is not the way that I would this is the way I would maybe work if I was actively working on a problem. I could have just given you this full fully packaged code base pre-built,right, that had all this built.

But

is that what you're asking? Why are we pasting these prompts one by one?

Eventually, we get a very large script that we should break it into separate files or sections.

Sure.

Tools.

Yeah, yeah, yeah. No, I think there's definitely better ways to organize this code than to have one single script. Just easier for demo purposes. Yes. I do have a demo repo. I think it's openhands/cve-demo that uses special classes.

There's a single CVE agent subclass that uses the same LLM. That's a little bit more better organized than just this one script.

CVEs. We're still having trouble parsing JSON.