Scaling Agents on Kubernetes with acpx and ACP — Onur Solmaz, OpenClaw

Welcome. So the talk is building an ACP at OpenClaw. It's also about other things, like how to put open source agents on open source agent frameworks on Kubernetes and stuff. So I hope I'll today share, in a nice way, what I've been working on in the last two months.

A little bit about me, very brief. A little bit about me. I've been building harnesses since a few months before ChatGPT came out. I built a JupyterLab extension over OG Codex model, like DaVinci Code 2, back in the days that eventually so I'm currently working for a startup.

And that initial coding harness turned into this current harness over time, like a ship of theses. It got ripped apart and put back together so many times. And yeah, I'm a founding engineer there. I've been in the industry since three and a half, four years.

I'm using OpenClaw since ClawdBot first dropped in Discord. I went in there. I've been following Peter since he wrote "ClawdCode is My Computer." I was like, this guy is crazy. Because Clawd4 saw it, I wouldn't give my machine to it.

But he ventured forth, and he's paid the way for us. And when I saw ClawdBot in Discord, my mind was blown. The next day, I installed it at the company, in our cluster. And it was basically talking to people on Discord, and also everybody else's minds were blown.

And we were used to selling to enterprise for a few years now. So that's why I started by adding an MS Teams integration in case it might be useful at some point. And I became a maintainer along the way.

I was there when it was renamed two times. And today, my focus is on agent interoperability and orchestration.

One of my goals is to accelerate enterprise adoption of OpenClaw and adjacent software, and also address OpenClaw is not secure. Well, it will be secure. Or Peter talked a lot about that earlier, so I don't have anything else to add on top of it.

It's a work in progress.

Sorry.

I'll use this. So I started on developer workflowsright away. So I created a PR and then called it Discord-driven development. Well, Telegram-driven development suits better because it's TDD.

And thenright after, I set it up personally, and I realized Opus is not so reliable for complex it wasn't back in the days. It's now better. Agents are improving. But Codex was my main harness, and I wanted to use Codex in Discord.

But it was basic I was playing a telephone game. I was telling Opus to tell Codex to do something. God knows what it's saying, because wording matters when prompting. But it was working somehow. And I would go and look at the Codex session, and then it paraphrased kind of what I was saying.

But eventually, I got some stuff done. But I knew this could be done much easier.

And today, I'm running a full IDE on Discord. We have parallel workloads. We have five channels. At any point, I'm working with one of five agents. So you see Codex 1 to 5, and then Clawd is for testing the OpenClaw's ACP feature.

And yeah, basically, that's how some of my channels look like. And it's very good for coding on the go, because I'm a guy who's addicted to side projects. And I like to just

AI is making things a lot easier to do these sort of things. Like in parallel, you have an inspiration. You execute on a weekend, and you just get it ready. You ship it. ACPX, which I'm going to talk about, is similar.

I built it through Discord. And what you do is you bind a Discord channel to Codex through ACP. You can also use Codex App Server Protocol. Harold and other maintainers developed it. And here, it is me using it before I'm flying to London to create a PDF about converting the docs of ACP into a PDF.

And then I have to say, put it in temp, because

Codex doesn't know about the harness, and it cannot send me in Discord. So then I go to another channel and tell it to send it to me on that channel. So we are developers, and our tools, we don't have time to polish them very well.

But we know the

advantages and disadvantages. What is ACP? So it's not most people say, is it like MCP? MCP is for giving tools to the model. ACP is for

standardizing agent-to-client interaction. Shout out to Zed. I actually forgot to put Zed's logo.

Zed is building a new editor in Rust, more efficient, lower memory usage, not Electron. So I'm using Zed's also since last fall. And if you use Codex on VS Code or ClawdCode, they are all building different plugins. It's not exactly it's like so much wasted work.

If only you could just standardize them under one interface, and you just build it once, and then you ship it. And that's what the idea that's the idea they had. It's also more it's much less duplicated work.

There are competing standards. So

Agent Protocol, that's for agent talking to agent. ACP, Agent Client Protocol, is for human talking to agent. But agent can use the human one to talk to other agents as well. In the long run, as these protocols get adopted, we will support all of them.

We will weigh the

advantages, and then we will use them somehow. But when I need so I chose ACP because when I needed it the most, I needed adapters for Codex and ClawdCode. And only Zed had built them. And Google didn't have them back at the time.

So that's why I chose it. So when you're adding in functionality to OpenClaw, you do it through a CLI.

So I said, OK, let's create a CLI for ACP. Let an agent call any other agent over the command line. So that's how it started. And it's slowly turning into a Swiss army now for ACP, as I will show in a bit.

So at OpenClaw, we have fire hoses. We have 60 KPRs, over 60 KPRs total, 300 to 500 per day on average are open. And basically, overnight, people woke up and decided they want they like OpenClaw. So we have tens of thousands of stakeholders who want to add features to OpenClaw.

And the biggest challenge of the project currently is, how do you absorb all the needs and wants of these people? And how do you balance them? How do you you can't please everyone. How do you create an elegant system without creating AI slop that can cater everyone's needs?

And Peter's workflow gave me an idea. So this is also something I do. You go and you ask your clanker, what is this? So PR comes your way. Most of the time, it's AI-generated description. Like, what is this?

If the human put thought into it, great. But yeah, you need to ask what it's doing. You ask, is this the best possible fix? Most of the time, it's no. And then you either continue he wrote it. You do some back and forth with the agent.

The reason is people just run into an issue with their OpenClaw. And it can use GitHub. And then they just say, please fix. And then they just send some slop your way. You can't merge it, but you can also fully discard.

You need to take this data point. It's like crucial feedback from the user. So you need to put it, categorize it, put it in a bin. It tells you when some part of the code is broken. And Vincent also talked about that a bit.

And here is

his Codex session doing that. On one side, there's the one that says it's a good fix. And on the other side, it's just saying it's not bad. And it's just so mechanical. And once you do this over and over, you realize you're repeating something.

If only you could program something to automate it. So you are automating the automator. So I started to create workflows in an abstract way. So item comes, PR, and then you find the intent. You're judging implementation. You're looking into if that's conflicts.

If reviews give you some issues that need to be addressed. So you need to make the CI pass. If it's not passing already, most people just don't care about that. All this mechanical work, by the time the PR ends up in front of you, should be resolved, ideally, and can be resolved.

And that's what I'm working on.

Ah, in the workflow, we have the shameful Ralph review refactor loops. I'm a believer that when people say,

just running an agent in a loop does not necessarily have to be

something that will create slop. As long as you're not making it design something, but you're making it uncover shallow bugs that can be easily fixed, it should be fine. So in the abstract workflow that I created, which is actually called turn into a program, you can tell it to do superficial refactors.

And then you can tell if it needs a fundamental refactor related to the human. Resolving conflicts also doesn't need it was hard back in the days. I don't think anyone is resolving conflicts by hand by now. So we are basically creating standard operating procedures for agents.

That's a fancy word for workflow. So that's what I built into ACPX. It's an A10 workflow engine, but it's driving a Codex session. You can see on theright. Let me show you in action.

So this was one PR.

It's loading. Let's speed it up a bit. So there are some programmatic parts. So this is just replaying what it is doing. It's reproducing the bug. It's judging refactor. It's reviewing. Now I'm doing a review loop. And then review didn't bring anything.

It's what I do. But I make it output JSON structured data, so I can put it in an A10-like workflow. And I will talk this is a general workflow engine.

You can use it on other things as well.

You need to apply agents generously on problems. So I see it's like an ointment that you apply generously. On any problem that can be solved with agents, you need to take yourself out of the loop and solve it with agents.

And personal agents, I think, are on a there's a spectrum: enterprise and personal agent. And normally, you see the work you use at the computer and then the PC you use at work and the PC you use at home used to be relatively similar.

But that will not be the case with agents, because at work, you will be using consuming a lot more inference. And that means in enterprise, there will be a lot more money to be made. So that's why I'm a bit also excited about enterprise agents and OpenClaw's potential.

That's why I believe in on-demand disposable agents. If you use OpenClaw on Slack or Teams or Discord, this is one instance. You create an app. The problem is, you can't really talk to multiple instances of this.

You create a connection. You create a Slack app. To connect another agent and another name and another profile picture, you need to create another app. And you have to create an app manifest.

And it's something that shouldn't be managed manually by clicking. And the platforms, like chat apps, don't have this standard yet where you can do multi-agent provisioning, like cosmetically create different agents. This is what this is on the screen you see.

I asked ChatGPT to generate the idea. You've got agents, and then the name can be generated by an underlying app. And you can talk to them separately. This is not supported, and this must be supported for this vision to work.

Until it's supported, I'm using it on another UI. Because we are all going to start one agent per task, and they will work on these tasks. And they will be creating files, editing files, and it will all be synchronized.

It will be a tad bit different than what you're used to with your personal agent. And to have that, you need to have a few key components. You need to have Kubernetes. You need to have an agent harness.

OpenClaw could be one of them. Codex, ClawedCode could be one of them. It could be ACP. It could be GitHub. You give read/write access, and you do state data synchronization. Maybe you do something that's used Rsync, something like whatever algorithm Dropbox is using.

And there are some projects that are taking this on. I've been working on this. This is outside of OpenClaw. This is my day job. And it's an open source orchestrator. It's a Go operator that basically handles the complicated parts.

There's a user experience. You want to create a concierge on Slack, like concierge agent, and you're talking to it. But you get bottleneck because you have 100 employees on Slack. So for some other task, you may need to create a new one.

And then it creates and gives you a website link. I'm going to skip the shout outs to Cognition and Devin, because they invented the category, but I'm running low on time. So the repo is at textcortex/spritz. I'm just going to demo it for our use case.

We use it on error reporting currently. So if you're on Slack, you can ask it to dispatch an agent to debug it. I'm asking any new bugs after prod release, and it's saying something. And then you ask it to create an agent.

If I could put that agent into Slack, I would. But I can't do that. So I have to put it in another UI. And this is an open source project. You can take this UI is also like a React app hosted in the cluster that you're deploying these Helm charts to.

And yeah, it starts the conversation there. It starts working on the problem. This is like Codex web or Devin or anything. But it's actually using a full Kubernetes pod, wasteful, but I think it's a better abstraction. Because OpenClaw showed the power.

When you give a full computer to an agent, it's a lot more powerful. And I believe that as well.

I think OpenHands uses Firecracker. So I'm also not so well-versed on all the different virtualization frameworks. So I'm also learning along the way. But I have a working product that's running on Kubernetes. And you can use this product if you're interested in deploying internally and using Codex on the web.

And then just spin things off. If you're like, I don't know, a backend if you have an open source project, first of all, I can help you set this up. If you have a system like inlet of just hundreds of issues per day, I can help you process it.

This

does all the wiring around Slack and keeping those agents on and the user experience. And then the intro you're not locked in any agent. You can switch. It's all abstract the way with ACP. Yeah, that was my talk.

Thank you for listening. Some social links in case you want to get in contact with me. I just want to make clear of my OpenClaw side and textcortex side. So the last part was about the work I do at textcortex, just to give a disclaimer.

Thank you for listening. I guess.