The Missing Primitive for Agent Swarms — Lou Bichard, Ona

Cool.

Allright. Hello, everyone. We're getting to the back end of the conference; I don't know if that's a good thing or not for you. Uh, start the weekend, or maybe sad that the conference is over. So, I'm going to be talking about the missing primitive for agent swarm, so talking a bit about sub-agents and swarms within the context of coding agents and the infrastructure underneath them.

So, just a quick introduction: my name is Lou, I'm the field CTO at a company called Owner. Uh, previous life I was principal engineer and sort of platform engineer. Joined Owner doing product management, and now I work a little bit more with our sort of customers on the field side.

So, in my world at least, everyone is trying to build a form of a Software Factory. I've seen a few different talks at this conference of similar sort of ideas, trying to take coding agents and then apply them across the software development lifecycle.

I'm wondering, actually, if this statement is entirely correct, because I've definitely chatted to a lot of people over the course of this week that are not yet at this point of thinking about this, but it's been very much occupying my headspace for the last few months as well.

So, I added a definition in this slide as well, just to quickly define what I believe to be a Software Factory, which is sort of the commitment to incrementally moving the human out of the loop within the SDLC, such that the human is not proactively interacting with the computer.

I say that because I've seen a few talks and some people talking about Software Factory with these, like, parallel agents, like one individual IC running lots of coding agents at the same time. It's not my personal definition. My personal definition is that you're slowly bringing the human out of it, and then work is flowing from sort of development into production, theoretically, in an automated fashion.

But we're extremely early in terms of where we're at with Software Factories. So, obviously there's a bunch of funky little visualizations that I've been showing, and a lot of these are actually taken from this website, backgroundagents.com, that I created, which seem to resonate quite a lot.

So please do have a look at it, it encapsulates some of these ideas as well. But we have, like, different patterns, really, for running these agents at scale, coding agents at scale. So one of them really is sort of this swarm pattern, which, from my definition, is starting off with an individual intent, firing that out to a number of different agents, then funneling that back in, let's say, to an individual PR or task.

And that's almost like your typical kind of sub-agent process that we've seen quite a few times. Fleets is something you can't see this very well in the room, unfortunately, but on the topright, the sort of fleets is where you're fanning out agents across, let's say, a number of different repositories inside of an organization, which is a capability we've had in Owner for quite some time now.

And you see it a little bit popping up here or there, but not so much, but I think it will be something that organizations will start to take advantage of a lot more in the future. And then at the bottom you've kind of got events.

So if you think about how do we take the human out of the loop and build this sort of Software Factory, you need to know how and when are you going to trigger those agents, when do they come online.

And a lot of this already exists with existing sort of webhook infrastructure and things like that, PR is raised, linear ticket is created, etc.

One thing that's been useful for us over the last couple of months is a lot of these large companies have also come forward and shared some of their implementations of this infrastructure. So I'll mention a few notable ones, but Stripe has one, they're what they call Minions, which is built on top of their existing infrastructure where they've then plugged in these coding agents and they're able to then drive thousands of pull requests inside of Stripe.

Another very noteworthy one, and RAMP has been so loud on social media these last couple of weeks, it's been quite insane, but they also built one that they internally call Inspect, which again is their sort of infrastructure for running these background agents.

And it's something that we've been doing now for quite some time. So Owner as a platform has been infrastructure for development environments for about six years, but over the last year or two, obviously integrating further with the agent side of that as well.

What Owner effectively does is allow you to spin up any number of different development environments, but one of the additional features that we have that I mentioned is this fleet feature. So you can automate on schedules or triggers agents that spin up to resolve issues across a number of different repositories.

Like use cases for this is things like CVE remediation, or bumping test coverage, or enforcing something at scale. As it stands today with current LLMs, that seems to be often tasks that are somehow simple, but the hard part is that you're doing this across a number of different teams, thousands of teams, or thousands of repositories.

Which is what basically I'm showing you here. So it works kind of like a workflow kind of creation, you're adding prompts, scripts, and things like that, and then using that to drive this change across your organization. I did want to give a notable shout-out also to the Harness Engineering blog that OpenAI and Ryan created, because this encapsulates much of this mindset of trying to then take and encode as much of your process into your repository, into your context files, to agents.md, in order to build, effectively, that Software Factory.

Over the last couple of days, I had a few questions with people talking about Harness Engineering and what is it. For me, it's really another extension on Context Engineering, whereby everything in your repository, from skills to agents.md to unit tests, everything that you could possibly use to give feedback to your agent, is for me Harness Engineering.

So I include this little visualization in the top corner, because as well, for me, Harness Engineering is very much about doing things, letting the agent run through, figuring out where the agent gets lost, and then encoding that knowledge back into your repository or context, again, to try and get the agent flowing through the Software Factory as much as it can.

So if we take a step back and then think about, from an infrastructure level, what you need, effectively, to build this form of sort of Software Factory, the first piece of that puzzle is a runtime, or you need somewhere for the agent to run.

And I believe this mostly is pretty much a solved problem now. You then need a way to orchestrate these, so you need to run them at scale, so you need a way to run these agents, you know, scale up, scale down, horizontally.

You need some way to trigger them, but for me, one of the biggest difficulties if you try and build this today is, effectively, agent coordination. So how do you get the agents to sort of interact with each other, pick up tasks from each other, how do they collaborate.

For runtimes, lots of, you know, different approaches here, but you can run agents in separate threads, you can isolate them more in work trees, you can then go one step of abstraction further and put them in containers and VMs, or micro-VMs.

Or what Owner does, basically, we really call these dev environments, so like the sandbox conversation has made this very blurry, but we at least believe that for running sort of proper development tasks, it has to be inside of a virtual machine.

The reason for that is, for the isolation from a security standpoint, a container is not a bulletproof isolation boundary. So if you have an agent running in there and you want to secure it, there's challenges for a container.

They're also bursty; if you run them on Kubernetes or in pods, you have noisy neighbor problems, you're going to have compute contention across different containers. And only with having sort of full isolation of a VM will you be able to, effectively, do this properly.

Let me run back through my presentation, sorry.

Cool. So let me actually just quickly, it's a good point, actually, at this point, I will show you a quick demo of how this actually looks inside of the Owner interface, because a lot of this is a little bit theoretical.

I did a quick recording of this yesterday in my hotel room, just in case the Wi-Fi was terrible in here. But let me just run you through this. So in here you see the Owner interface. On the left-hand side you have all the different tasks that I have running.

So I kicked off two different tasks here. The first one I asked Owner to implement me Symfony, so Symfony has a spec in the repository that talks about how to implement Symfony. It's a very detailed spec, so I wanted to use it as an example.

I gave one of the agents and asked it to spin this up using process-based agents, so sub-agents running within the environment itself. So take the VM, run the agent inside, and that agent will spin up sub-agents within that VM.

The other one I asked it to is, effectively, to run me a fleet with a number of different VMs. So the agent is actually then empowered to create other VMs inside of the platform, and it can spin up technically infinite of these.

So wherever you're running this, you're only really inhibited by as much as you're willing to pay and as much as your cloud provider can scale to. So if I run this through, what we see, and I might have to skip through here a little bit, is we see, I jumped too fast.

This bottom agent, the VM one, then it will spawn these three different sub-agents. So it spins up the different VMs, which we see coming in on the left-hand side. The parent is the controlling agent, and the sub-agents obviously then are given small bits of context, individual tasks to complete, and then we'll do message passing back to that parent agent to control and govern the overall task itself.

One challenge for sure we have is that how do we build the UX for this? Like, as you build more and more complicated tasks, how do you think about sort of managing and controlling these sub-agents, and how do you think about the UX on top of them?

So as this progresses, you'll see also on the left-hand side, eventually you start to see, as the agents come online, they start their environments and they start to work through tasks. Both of these have, for some reason, seven sort of sub-items that they're working through, so you can see those.

When that gets to the end, it's then obviously going to terminate those VMs, that's all going to collapse down, and your task is complete. The second form of UX for this that we have with the sub-agents is this one here with the process level, which I'll pause so that it's not jumping around.

When you launch a process-level sub-agent, it happens all within the single agent window. So you actually see at the bottom here a stack of a number of different sub-agents that are starting. And then when you click on those, you can then open up a new chat window, which is almost your new context, and use that.

So you've got this, like, two different forms of this, one entirely isolated VMs to scale out these swarms, and another one at the sort of process level where you can run it within the individual VM itself.

So lots of stuff going on there, but I wanted to show you conceptually. People say, what does this swarm look like in reality? How does this actually look? And this is how it looks within Owner.

So give me one second to fly back through.

So, coming back to the Software Factory idea. So how do I know about some of the challenges about Software Factories, and that is because I've also tried to build one. Obviously I implement many of these ideas in our own projects, but I also, similarly to what OpenAI did building out their sort of Symphony projects and their Harness Engineering blog, is, okay, what can we build a project without touching any lines of code, and can we automate as much of this process as possible such that the agent can actually then develop everything as autonomously and self-driven as possible?

And it turns out, yes, the technology is there today, but there are some different challenges that you have. What's missing? One thing that you find out is that the SDLC is not this. You know, this is the conceptual SDLC that we present and talk about with each other, very coarse-grained, five steps of the SDLC.

Agents don't respect this and they don't understand, I mean, these boxes contain a ton of complexity. So if we take something like plan, or a plan stage, actually within that, our SDLC has a ton of different sort of micro-steps almost to it.

And if we're wanting then to train agents to then step through the SDLC, we need to find ways to actually break down the SDLC into some of these micro-steps. And that happens all the way through the SDLC. So if you want to build some form of Software Factory, we then now need to start figuring out how to solve these sort of micro-steps, how do we get agents to sufficiently follow those steps and do them in sort of deterministic ways as well.

And the hard part of this is context. Probably not a, you know, huge innovation to everyone in this room, context windows are some of the hardest parts about working with LLMs, because as the last speaker said here as well, Context Rot, context, you know, once the context window becomes consumed, the agent starts to lose track of where it's going and things like that, it gets less effective.

They also skip steps, typically. You know, they want to please us, they're quite sycophantic, so they might, you know, ask them to write some tests, and then they'll go into skip some tests in order to complete the task.

So a lot of the tools that we have today as well for coordinating these agents, as you start to spin them up, once you've got this infrastructure, you have the runtimes, you have the orchestration, the missing piece then you also need is coordination.

So if you use something like GitHub, GitHub is not a coordination layer for agents, it gets incredibly overwhelming. You know, you can have your agents raise a pull request, you can review it, you can then solve the merge conflicts, you can fix a CI build, etc.

But this gets incredibly noisy for you as a human to make sense of where you should step in to intervene with that agent itself. And GitHub is a poor solution for this. Symphony is built on top of Linear, but suffers the same problem, that we're reusing existing sort of human tools in very weird ways for agents.

But I think we are now at the cusp of now effectively solving this, and I think it might be solved in a few different ways. One is through sort of state machines, you know, by building out workflows and effectively state machines that we've had for a very long time, and then building these as our versions of our SDLC.

I do believe to a degree some of the ideas from durable executions comes into this as well. You know, lots of companies have pioneered this over time to be able to run a process in a durable fashion. But we do need to solve also the gates and compliance section of this.

And I do believe, actually, there is definitely a gapright now also for packaging this in some form of like CLI construct, whereby you can run this locally in a sort of development environment, but also then remotely in, you know, in a CI or some other fashion like this.

But that's effectively it. If anyone wants to chat and go a little bit deeper, I think we've got a couple of minutes, we can also take some questions if we need to. Yes, basically my definition of Software Factory is moving this human on the loop, so they're not necessarily driving each of these individual changes.

Context and context management is by far the hardest part of building this Software Factory. Out of these primitives, I do believe we've effectively solved the runtime. There are many options for this now, sandboxes and containers and other solutions.

The orchestration is effectively solved, the triggers are solved, but the thing that's missing for me is coordination. Also, one of our folks here is here from our security team, and security is another piece of the puzzle that we really need to solve as well to drive more automation.

But it's this coordination layer that I think is largely what's missing. Obviously a lot of this is a little bit high-level. On the 6th of May, we'll run a virtual summit to go into these topics about Software Factories and background agents.

So if anyone here would like to attend, you can do, it's on backgroundagents.com, and we have a CFP up for it as well. If anyone would like to speak, that would also be great. There's not many people in the world doing this, so if you're at least experimenting with it, it would be great to also talk to you as well.

The other thing is a few people that I've been speaking to at the conference, especially about Software Factory, complain that obviously a lot of these talks are a little bit theoretical, which is true. So some of you might want to be really getting into the weeds of this.

So actually next week, with Zach from Owner, we're actually just going to build one of these in public, starting from scratch, build a Software Factory. We'll do this over the course of two weeks in public, just to show what that looks like with today's technology.

Just to really, like, show all the ins and outs of the workflows, if anyone wants to see what that actually looks like as well. But there you go, thank you very much. We do have two and a half minutes if anyone has questions, but if not, I'll also be outside and downstairs.

Yes?

Thank you for the talk, super helpful. Could you maybe go back to the slide where you were talking about the problems with the coordination layer?

Yep.

Memory, and then also if you could just elaborate a bit more on what you propose as a solution and more concrete, that would be pretty interesting to hear your thoughts.

Yeah, sure. So I have been, so I need to repeat the question actually for the stream. So the question was, problems with the coordination layer and sort of more concreteness about the solution, what could that potentially look like?

So I have, we have a number of different sort of prototypes for this internally, and I've also been trying to think about how we, what we do with this. So I think one form factor that works is the CLI.

So I've seen some other solutions, graph-based ones, there are some open-source thingsright now where defining the workflow effectively as a graph, you know, like, kind of like what you drive sort of mermaid diagrams out of, like NA10 type of workflows, where you can define sort of these prompts in that form.

I think that ultimately needs to be packaged in some form of CLI though. What I mean by that is, I think that if you have a local running agent, Claude Code, whatever, any local running CLI, I think that now needs to have something that integrates with its tools so it can invoke this to say, hey, have I achieved this part of my SDLC and can I now proceed to the next part of it as a CLI?

If you want as well, I can, I have a full spec for this, I can show you after the talk as well, we can go a bit deeper into it. But I'm seeing this solved in a number of different ways.

People are solving this in a few different ways. There's the NA10 sort of workflow type of diagram way, the CLI sort of gateway is the prototype that I have as well. And then there's a range of different sort of hacky interim solutions.

There's one even the OpenClaw folks have as well, which the name is eluding me, but I will, I think it's ACPX, which they've built on top of ACP, which builds out some of this workflow. There's a GitHub one, Fabro, some folks are playing around with, but it's very nascentright now.

Yeah. Any other questions?

Are you using some protocol for the CLI? Because, you know, there's ACP, there's A2A, there's a bunch of, I don't know if people have agreed on a standard.

Currently not. So the implementation that I have, I'm trying to understand whether or not to release it as an implementation or as a standard, to be honest, because I largely kind of don't really care about the implementation. I almost care about the standard for it so we can collaborate on the standard.

But it's not built on top of ACP, at least not yet, but it might be.

Not A2A either?

Not A2A either, no. It's solving, I feel, like a slightly different problem space, so.

Cool. Allright. Thank you very much. Enjoy the conference, everyone. Thank you.