Self Coding Agents — Colin Flaherty, Augment Code

Hi everyone, thanks for coming today. So I want to talk to you about something that sounds like science fiction but very much is reality: an AI coding agent that helped build itself. My name's Colin, I'm an AI researcher at Augment Code, a company building AI-powered dev tools for software engineering orgs, and I want to share with you a little bit about my—our journey working on AI coding agents.

So, zooming out, AI dev tools is a fast-changing space. Everyone remembers in 2023 we were all talking about auto-complete models, GitHub Copilot being the one that probably really comes to mind. In 2024, chat models really started to penetrate software engineering orgs.

In 2025, though, we think AI agents are going to dominate the conversation about how software engineering is changing.

So, naturally, a few months ago we started building our own agent at Augment. I want to show you a sneak peek of what we built and share some hard-learned lessons about how this tech works. And I just want to, you know, reiterate: I've been really amazed to see the extent to which this agent has helped build itself.

I'll note a kind of fun statistic: we have about 20,000 lines of code in our agent codebase, and over 90% of that was written by our agent, with human supervision. So what does it mean for the agent to write itself?

Implementing core features. So one of the first things we had to add was third-party integration. So our agent, you know, if it's going to work like a software engineer, it needs to interact with Slack, Linear, Jira, Notion, search Google, muck around in your codebase.

And so we wanted to have the agent help us build these features. We had it—we found after we added the first few ourselves, when we asked it to, you know, give it an instruction like "add a Google Search integration," it was able to go look in our codebase for theright file to add it in, figure out theright interface to use, and go add it.

One kind of fun anecdote is when we were adding the Linear integration, it didn't know the Linear API docs, the foundation model we're using didn't have those memorized. And so it used the Google Search integration, which it had written previously, to go look up the Linear API docs and then was able to add that.

We used it to write tests. So we found if we asked it something like "add unit tests for the Google Search integration," it was able to go add those. In order to do this, we just had to give it some basic process management tools: things like running a sub-process, interacting with it, not hanging if there's an infinite loop in some test it wrote, and reading output.

I think this is super interesting to everyone seeing the Twitter demos of these agents writing features and writing tests, but I haven't yet seen a compelling example of them performing some kind of optimization. Well, over the course of our project, we noticed the agent was pretty slow and we weren't sure why.

So we asked it to profile itself, and what it ended up doing, using all these tools we'd given it, was add some print statements to its own codebase, run essentially sub-copies of itself, look through these print statements, and it figured out there was a part of our codebase where we were loading up all the files in the user's repository synchronously and hashing them synchronously.

And then it added a process pool for these to speed it up and stress test to confirm it was all working. And by the end of this, we reached about 20,000 lines of code, and again, over 90% of that was written by the agent with our help and supervision.

So let's walk through a quick example, a couple quick examples to see how the agent works. I focus on simple examples where it's reliable so you can follow along easily. So here I ask the agent, "Are you able to search Google?"

And then it notes that it found a tool called Google Search. For those who aren't familiar with the notion of tools, I'm sure most of you are, but I'll just kind of quickly reiterate: the idea is we have this kind of master-level agent that's doing all the planning and it has access to certain tools that it can use to interact with its environment, whether that's the third-party integrations I talked about like Google or it's editing a file in the user's repository.

And then it wants to confirm that this Google Search tool is working, so it sends a query to it of tests, and the agent responds to us, "Yes, I can search Google, and I see the first 10 results."

Let's try something a little bit more complicated. I ask it, "Instrument agent's Google Search tool with logs," and then generate an example. Then it uses our retrieval tool, which is, you know, allows it to search the local codebase, and it's looking for a file related to Google Search integrations.

It finds this file deep in our directory hierarchy at services/integrations/thirdparty/googlesearchtool.py, and then it calls its file editing tool to quickly and performantly edit that file to add those print statements.

This is a continuation of the last example. So it added those print statements, and now it wants to run a sub-copy of itself so it can look at the output of those print statements, because we asked it for example logs.

But in doing so, it finds that we don't have Google credentials authorized, so it uses its clarify tool to ask for clarification from the user. It asks, "I don't see Google credentials. Would you like me to, one, add stub for Google API, or two, guide you through setting up credentials?"

I note that the credentials are actually stored in .augment/googleapi.json; it had just missed this. And then here's a really cool extra feature we have, which is we want the agent to continuously learn as it interacts with humans. And so here it thought, "Well, it's probably a good idea to remember where the Google credentials are stored."

So it called this memory tool to create a memory of where the Google credentials are stored to save that for later. This is another example: if you have that really good context engine, it's really critical to getting the agent to work well.

And so now we get our output. So it prints out these logs, and it's searched with an example string, Python programming language, and it gives some example URLs that were returned by Google, python.org, and wikipedia.org. So we had the agent add logs to itself, run itself, learn from user feedback, and it used all kinds of tools: Google Search, codebase retrieval, file editing, clarification from the user, and memorizing useful learnings.

So let's fast forward and talk through some of our lessons building this. I just want to note, you know, we've been working on AI coding tools for a couple years now, and we didn't set out to build agents.

We've worked on things like completion models and chat and so forth, but our focus the whole time was around building a super powerful, scalable, enterprise-ready context engine. Because we knew no matter what, no matter how good these LLMs get, you're going to need that context.

And we also thought a lot about how do you build great UI/UX so AI can seamlessly interoperate with humans. It turns out this context engine and all these thoughts around design provided a great foundation for us to quickly build this agent in just a couple months.

The three most important things were that access to context, that context engine with all those different types of context sources, whether it's Slack or the codebase, the reasoning capabilities from a best-in-class foundation model, and that code execution environment so you can safely run commands in a customer's environment.

So let's talk through a couple assumptions that we frequently fall into, we've have frequently fallen into and remedied, and some of you might encounter as well. So the first one is that, you know, L5 agents are here. The senior soft agents are at senior software engineering level.

If you look at the Twitter demos, it oftentimes can seem like this: you have an agent write an entire website all on its own. In reality, professional software engineering is rarely zero to one, and the environments that we're coding in are a lot messier than what those demos show you.

As a result, these tools, you know, aren't quite there yet, but they're still super useful.

The way one framework I've seen people think through when they're trying to figure out, you know, how to use these agents and how to build them is they think agents will take over entire categories of tasks. So first you build an agent that will solve back-end programming, and then you build an agent focused on front-end and maybe one focused on testing.

In reality, this technology is very general-purpose. And so instead of thinking about categories of tasks, we found it more helpful to think through levels of complexity. So our agents, you know, kind of good, decently good at tasks across front-end, back-end, security, and so forth, and we're improving the capability level along all those fronts at once.

Because, you know, again, it's a very general-purpose technology.

We've also seen people anthropomorphize agents. So they think they're just like human software engineers, and they map the characteristics of a weak software engineer to what they think a weak agent would look like, and vice versa for strengths as well.

In reality, agents have different strengths and weaknesses than humans. And so you may have an agent that can't do math, but it can implement a whole front-end feature way faster than any human could. And it's important that we keep this in mind.

Let's talk through a couple reflections and lessons. So here I ask Augment, "Can you create a stack of 2 PRs for the new reasoning module using Graphite?" Unfortunately, so Graphite is a version control tool for working with Git.

You can like stack PRs; it makes it a lot easier to review. Unfortunately, foundation models have not memorized how Graphite works. So our general agent responds, "I don't know what Graphite is, so I'll use Git." And then it calls our terminal tool to run a command, running git checkout.

Well, what do we do here? We wanted it to use Graphite. We can't necessarily go tell OpenAI or Anthropic to retrain the model to understand Graphite overnight. So we came up with this notion of a knowledge base, which is essentially a set of, a sort, a set of information that we want the agent to understand that it currently doesn't.

We can kind of patch holes. One thing we wanted to add to it was this Graphite knowledge. So we created this Markdown file describing Graphite, how to, you know, run common commands, things like how to create a PR, use GT create, some things not to do.

We created other files in our knowledge base for things like details in our tool stack, how to run tests, the style guide. And then we added this into the context for the agent so it can dynamically go search in this knowledge base when it doesn't understand something.

And once we added this, then, you know, we go ask it, "Can you create a stack of 2 PRs for the new reasoning module using Graphite?" And it calls that knowledge graph, reads about Graphite, and then can run the GT create command.

So what's the learning here? Well, onboarding the agent to your organization is crucial. The analogy I like to think about is if you just hired a new, a new hire software engineer, you wouldn't go tell them to just stare at the codebase for three days to figure out how your tech stack works.

You'd let them ask you questions, maybe there's some things they didn't understand, and you add some additional documents to your Notion. We should think similarly about agents.

Recall I was talking about how we had all these third-party integrations we added, whether it's Linear tools or Slack tools and so forth. When we were working on these, we weren't really sure of which ones to prioritize and start with on our product roadmap.

In a normal world, we'd make some educated guesses, we'd implement a couple of them, and go from there. But with the agents, we were able to iterate them, build them all at once. And so this starts to change the calculus around how product management works.

If you can build everything at once, well, then maybe, maybe engineering hours aren't the bottleneck on what we build, and it starts to, we start to be bottlenecked a little bit more on good product insights and good design.

So when code is cheap, you know, you can explore more ideas.

Also recall earlier, we were talking through this example of, you know, instrumenting the agent's Google Search tool with logs, and it was able to go find the file to edit. Notice here how we didn't have to give a very precise instruction to the model.

We just told it in natural language, like how we'd talk to another engineer to instrument the agent's Google Search tool, and it was able to go figure out the file to edit. This only worked because we had that really good codebase awareness.

We can also use the agent for tasks outside of writing code, but still within the software development life cycle. So here we asked it to look at the latest PRs in our codebase and generate an announcement on them, and then we posted it to Slack.

And so it was titled "New tools for this CLI agent," and we talked about some things around Slack notifications and Linear integrations. This only worked because we had that Slack integration and understood our codebase well. This figure may look familiar from the beginning of the talk.

We actually had the agent make this as well. So we asked it, "Make me a plot of the interactive agent's line of code as a function of the date."

And so good context is critical. In all three of these tasks, we needed to pull in some different context from some different sources, and it's not just the codebase. Context comes in many forms. And also note that it's multiplicative.

So having access to the codebase and having access to Slack is 4X as useful as just having access to one of those. Finally, I want to switch over and talk about testing. So here's a really hard-to-test edge case in our code.

The agent actually wrote this, and we only caught it because of some unexpected runtime behavior. So we have these caches that the agents store relevant information for their runs in. We can run multiple agents in parallel, and they all write to the same cache location.

And the agent wrote this save function to save to that location. And I had this lock around the JSON dump, so there were no race conditions that would explicitly fail if you had multiple agents all writing to this cache at the same time.

But notice here how there's no read before writing to the cache. And as a result, you could hit a race condition where if multiple agents are running in parallel, they're all overwriting each other's caches. And when the agent wrote this save function, why did it miss this issue?

Well, these agents sometimes make mistakes, and this is a hard-to-test situation. There's some parallel programming, there's a cache involved, and so we didn't have a test. And because we didn't have a test, the agent messed up. My learning here is we need to be very careful about having sufficient tests.

We have this pretty incredible statistic. So we have an internal bug-fixing benchmark. We found when we upgraded our foundation model by about six months, our scores from this benchmark improved by 4%. But when we added the ability to run tests, so the agent could suggest a fix for the bugs, run tests, look at the feedback, suggest another fix, run tests, and do that four times, that led to a 20% gain on this benchmark.

So what's the lesson? Well, better tests enable more autonomy. You can trust these agents more, and it just makes them smarter.

What does software engineering look like in a world of agents? Well, agents didn't work last year, but now we're pretty good. If you'd asked me two years ago if we'd be working on this tech, I frankly wouldn't have guessed it.

There's a compounding effect where these agents are starting to help build themselves, and that's only going to accelerate the pace at which they improve. Code isn't going away because it's a spec of our systems, but our relationships to it are changing.

Good test harnesses are becoming more important than ever, and we need to be especially careful about those parts of our codebase that tend to be less well-tested. And the calculus of product development is changing. If code becomes super cheap to write, then the focus, our focus is more on good product work, gathering customer feedback quickly, building insights.

We're really excited for how this tech's going to positively transform our industry, and we'll be releasing our agent soon, so I'm really excited to share that with you. Find me after the talk if you want to discuss any more.

Thanks.