Infra that fixes itself, thanks to coding agents — Mahmoud Abdelwahab, Railway

Your app's infrastructure should fix itself. Let me show you. So,right now I'm on the Railway dashboard, and I have a bunch of services that are deployed, and all of these services have one thing in common: they all have bugs and problems.

So, for example, this service has a memory leak. If I click on it, go to metrics, we can just see memory utilization keeps growing high and very quickly. This is just a sign of a memory leak, and I'm pretty sure the service would eventually crash.

If I look over at the amount of requests, we have a high number of 500s, so the server is failing to respond. We have a high request error rate of 94%, and we also have an extremely high response time of, like, multiple seconds for the service to respond, which is not ideal.

Like, if this was a service running in production, everything would be on fire, you'd be getting paged, and you'd just try to bring back the service up back quickly. But the thing is, not all problems are this obvious.

For example, this service, all it does is just query the Postgres database, and if we go to metrics, we'll just see that, well, CPU utilization seems fine, memory usage is also fine, sure it's a bit spiky, but okay, whatever.

We have some fails, but okay, nothing too alarming. Request error rate is somewhat high, so that also should make us kind of, like, want to investigate, but the response time is extremely high. The thing is, this is because the service makes queries that are super slow.

And the thing is, if you're an end user that's trying to use this experience, you would just suffer. You would need, like, 30 seconds for, like, a page to load, which would be a nightmare. So, the thing is, when you deploy your app to production, maybe some, you know, bugs or issues make their way to production.

Things happen. And kind of like the typical way of dealing with these things is, maybe you set up a bunch of thresholds, and when these thresholds are met for, let's say, CPU or memory utilization, maybe you want to have a threshold for the request error rate that shouldn't exceed a certain amount.

Well, what will happen is, you're going to get alerted, and you'll be aware that there's an issue, but you still have to do the investigation yourself. You have to dig through logs, metrics, and traces to try to paint a picture in your head and try to piece things together so that you can ship a fix.

Now, what I'm proposing is, you should have a coding agent that monitors the state of your project and your application's infrastructure, and if any issue is detected, so, you know, any of the thresholds we define are met, we should just have a fix shipped,right?

So, like, instead of, you know, getting the alert and investigating, you just review a pull request and you're like, oh, looks good to me, you ship it, and then everything is good and crisis averted. So, today I'm going to show you what I have in terms of a demo that kind of paints a picture of how this could be achieved.

So, at a high level, I want to have a series of workflows that will kick in, that will help me go from issue detected in on Railway, my deployment provider, to a pull request being opened in my GitHub repo.

And this is what I have in mind. The first workflow that I want to have is a workflow that runs on a schedule. So, let's say it runs every 10 minutes, 15 minutes, 30 minutes. And what this workflow will do is, one, it should fetch the application's architecture.

We should have an understanding of what services are deployed, which, you know, like front ends, back ends, crons, queues are live in my project. And I then want to fetch each service's resource metrics, so CPU and memory utilization.

And I also want to fetch each service's HTTP metrics. So I want to see the request error rate, the number of failed requests for, you know, 500, 400 errors. And once that's done, I will want to then see which services have exceeded which thresholds, and then I just want to return a list of the affected services.

So, this would be essentially the goal. Now, you might be wondering, well, why not make this an alert-based system? So maybe we configure something like webhooks for alerts, and then that would kick off, essentially, this workflow instead. I would argue that it's probably better to be able to analyze a slice of time rather than just having a threshold being met, because it can get pretty noisy.

Like, imagine you have a spiky workload, and, you know, you reach the 80% resource utilization for, like, your CPU, but things are still fine. And that's not, like, in my mind, this is enough to investigate, but it might not, like, it might mean that there just aren't issues when we try to look at, like, the bigger picture and all the details.

Now, once we have this list of impacted impact services, we essentially want to pull in even more context for them. So, like, at a high level, we want to see project health, all the services, is everything operating as expected?

Oh, we have this thing that we're suspicious about, let's actually pull all of, you know, additional context for this service. Because imagine again, you have, like, high resource utilization, maybe you're just successful, you have high usage, but then when you pull the logs, it's like, oh, everything seems fine, there aren't any errors, well, you're good.

And you can imagine that we can even pull even more context. Like, imagine maybe we scan the code in the repo, and based on that, we infer the upstream providers that the repo relies on, and then we can automatically check the status pages of these services.

Imagine, like, a payment processor goes down, well, that's kind of how you can know, and then the coding agent will be able to maybe tell you, like, hey, you should just, like, wait out this issue. And once we have all this information, we can just write a detailed plan.

So, like, we can look at, oh, we have a high number of 500 requests, we see that we have very high resource utilization for memory, and we see that we have, you know, just errors specifying that a specific endpoint is failing.

Well, this is enough information that we can write a detailed plan of, hey, this is my application's architecture, these are the affected services, we just then give this plan to an agent, and then the agent will just follow the process of, hey, let me clone this repo, I'll just create a to-do list based on the plan you gave me, I'll implement all the fixes, and I'll just create a pull request.

And this is kind of how we go from issue detected to an open pull request. So, let's actually see this in practice. So, because we have the idea of workflows, what I want to do is actually use what is known as durable execution.

So, the idea of durable workflows has been around for a while, and it's really one of my favorite abstractions because it can help you simplify complex logic while making it more reliable. So, for example, here, we have this workflow.

So, this actually is Inngest, but there are lots of solutions out there that pretty much do the same thing. And we have this function that, you know, called processVideoUpload, it listens on an event of video uploaded, and we essentially want to do three things.

We first want to generate a transcript, and we do this by making an API call to a third-party API. Once we get that transcript, we want to generate a summary by also making a request to an LLM provider.

And once we have the transcript and the summary, we want to store them in the database. The thing is, all of these steps, they are not 100% guaranteed to work. They are prone to failure. And what's neat about this pattern is, by default, these steps will be automatically retried.

You don't even have to think about it. But if you, let's say, want to modify this behavior, maybe you want the retry to happen, like, on a certain schedule, like, you know, exponential backoff, maybe you want to define another thing that should happen in the case of failure, you'll be able to do it.

But what's neat is, each step, when it succeeds, the result is cached. So, if, for example, we are able to transcribe the video correctly, we summarize the transcript correctly, but we fail to write to the database, if we were to retry this workflow, we'd just continue where we left off.

We don't, we won't really repeat any work, which is, one, awesome because it's faster, but also it's more cost-effective. So, at a high level, this is the thing that I'll be relying on in my code, because I'll be making API calls to the Railway API to be able to fetch the project architecture, all the resource metrics, as well as, you know, the HTTP metrics and whatnot.

So, yeah, this is kind of like the first thing that we need to talk about. The second thing is the coding agent. And for the coding agent, I'll be using OpenCode. OpenCode is an AI agent that's built for the terminal.

You can think of it as an alternative to something like Claude Code. But the main difference is OpenCode is fully open source, and you can choose any LLM provider or, you know, model that you like, which is pretty nice.

You have this nice terminal UI, but honestly, what's so cool about the project is how it's architected. So, if you go to their docs, they actually have a server implementation. You can have a headless server that runs, that exposes an API for you to essentially interact with an agent.

So, the way it works is, when you run the command OpenCode, which is what starts up the agent in your terminal, it doesn't just run a single app. It actually starts a terminal UI and a server. And because the terminal UI here is the client, we can essentially bring our own client and talk to the server, which is awesome.

Because now we can run OpenCode on a server, in this case it would be on Railway, and we can just have this server have all the tools that the agent would need. So, we'd install all of the necessary, you know, tools, we can configure Git, and then the agent will be able to open pull requests and, you know, go through the file system and do everything.

Let me show you what, how easy it is to essentially have this deployed on Railway. So, if we go to the code hereright now, this is my project, it's called Railway Autofix, I know, great name. I have essentially two directories.

One is for my API, the other one is for OpenCode. And OpenCode, really, we just have a single server running using Bun, and all we're doing is we're just calling a function that is called createOpenCodeServer. So, if I actually stop this here, you can see it runs on port

4096, and this is pretty much all we need. And I have a Docker file, and in this Docker file, we're essentially defining the environment. So, we're installing a bunch of tools. You can see we're installing curl, jq-bash, all the other tools, even Git.

We're installing the GitHub CLI, which is what will allow us to open pull requests against a given repo. We're then installing OpenCode in the environment, we're configuring Git, and at the end, we're just exposing the port, and we're just authenticating the GitHub CLI, which is pretty neat.

By the way, the code will be linked somewhere down below. But that's really it for OpenCode. And when it comes to the actual API, let me actually run it. So, now the, this is the OpenCode server that's running, and if I go here, I have my actual API running on localhost:3000, and I have a UI that is provided by Inngest, which is very useful for debugging.

So, if I go here, and I go to functions, essentially each function here is a workflow, and it has a bunch of steps. So, let's actually try to run it to see what happens. Now, in production, when this is live, this monitorProjectHealth workflow should run on a schedule, and if an issue is detected, we will call the pullServiceContext, and then pullServiceContext will call the workflow for generating a fix.

So, if we actually just kick things off, this is how the flow of things will happen. So, if I actually have now, I have this function run, we called monitorProjectHealth, then we called pullServiceContext, and now we're actually calling generateFix, because we detected an issue.

And we're just setting, like, the Railway-specific variables as environment variables, and all of these are actually available on Railway, they're just set automatically, which is pretty neat. So, if I actually go to monitorProjectHealth, you'll see we have a bunch of steps.

The first one is getting the project architecture, and this stepright here, this is, we can actually see its output. So, we can see all of the databases that I have in my project, I just have one. We can see also a list of all the services, as well as their configuration.

We can see which, like, where is the repo for them. And we just now have a high-level overview of our application's infrastructure. We also see that we have any kind of, like, volumes that are there, which is cool.

And then we have a series of steps that are actually running in parallel, so, like, you know, things are efficient. So, we're getting the database resources. We can see on average, well, what's the max CPU? And it's like 0.9 CPU, okay.

Same thing for memory. And we actually have a summary. And this summary essentially is us formatting these results so that we can then pass it to the coding agent. So, you can see CPU usage average 0.93 BCPU, and, you know, this is the max.

And memory usage as well. Now, this is actually high, and we'll be able to kind of understand that, because it's like, oh, memory usage here is 31.96 gigabytes out of a max, which is 32 gigs. And then we just pull even more, like, resources.

So, like, because we have multiple services, we will call each step for it. So, like, we will pull the HTTP metrics for each of the three services that we have deployed, for example. But also, for this one, for the HTTP metrics, we can see the error rate percentage for 400s, for 500s.

We see, like, the latency, and we just have, like, a status count. So, we can also have a summary, and then we can say, hey, these, this is the rate of, like, request error rates, these are the latencies.

And this way, when we actually, at the end of, like, this workflow, so if I go to runs, go here again, towards the end, we will actually give this pullServiceContext function just all of this information in a nicely formatted way.

So, if I actually go now to this function run, we will see here that we're fetching the HTTP logs, the build logs, the deployment logs for, like, all the services that are affected. And we can see here, like, this is the function payload.

So, this is the stuff that we passed from the other function. And we can see we just have all this info. We also have an architecture summary. So, this, actually, we can expand this. The architecture summary is just a nicely formatted text saying, like, this is the project architecture.

We have three services. We are running in the production environment. We have one database. We have all these volumes. And we just have all of this information. It's just harder to read because it's, like, in one line. But for the coding agent, we'll just give it to it as, like, markdown.

So, now that we have that, I'm going to go to runs again. Now that we have that, we are just going to make a call to another workflow, which is generateFix. And for this one, what it does is, one, it will analyze with AI.

So, this is the actual output. In terms of, like, the input, it's a bit large to render here. But we analyze it with AI. So, like, you can imagine we give a large language model saying, like, hey, this is my project architecture.

This is the data. This is how things are performing. And then we take all of this information, and now we actually come up with a plan. So, you can see here, debugging steps. We want to see reproduce locally with the same load.

Maybe we want to run it. We want to see what will happen if we see that the agent is like, oh, I ran into an error. Then it's going to fix it. And then we have, like, recommendations. So, like, this is the plan that we'll then just pass to our coding agent.

And then we have a step to create a session. So, on the coding agent, you can imagine each session being its own chat. So, this will run, like, imagine you have multiple repos. Each repo will have its own session.

The coding agent will work. And then at the end, it should, you know, if, as expected, it should open a pull request. So, yeah, that's pretty much it. This is how it works. Now, if everything works as expected, we should see a pull request on the project.

And here we go. We have a pull request that is open with all of our changes. If we go to the conversation, we'll actually be able to see that we have a summary of all the changes, an analysis summary, the root causes, what was fixed.

So, we should be able to just review this. If everything looks good, we merge and we're good to go. And that's it. I hope you enjoyed this talk as much as I enjoyed making it. If you have any questions, feel free to reach out to me on X or Twitter.

This is where I mostly hang out. Also, the repo for this project will be available somewhere down below. So, make sure to check it out. And with that, thank you so much for watching, and I'll see you in the next one.