Making Codebases Agent Ready – Eno Reyes, Factory AI

Hey everybody, my name's Eno. Really pumped to talk today about something that at Factory we care a lot about. When we started 2 and a half years ago, we said that our mission is to bring autonomy to software engineering.

And that is, like, got a ton of loaded words in it that sounds a little buzzwordyright now, but I think that the my goal is that you guys leave this, like, roughly 20 minutes with a bunch of insights that will apply to your organization and the teams that you build, the companies you advise, and if you're building products in the space.

Insight into, like, sort of maybe how to think about building autonomous systems and also making your engineering org one that's able to use agents really successfully. A sort of, like, plus of this is that ideally this applies to any tools you're using that involve AI, so it won't be specific to, like, our product or any of the other amazing tools out there.

I'd like to start with a little bit about, you know, Andrej Karpathy had a very well-timed tweet, so of course I'm going to mention it. You know, he kind of talked about this idea of Software 2.0 coming from the ability to verify things,right?

This is something that's in sort of, like, the mind of Silicon Valleyright now as the most frontier models are built with post-training that involve lots of, like, verifiable tasks. And really I think the most interesting thing here is the sort of frontier and boundary of what can be solved by AI systems is really just a sort of an input function of whether or not you can specify an objective and search through the space of possible solutions,right?

And so we are used to building software purely via specification. We say, like, the algorithm does this and, like, input is X, output is Y. But if you sort of shift your mindset to thinking about automation via verification, it is a little bit of a difference in what is possible to build.

And there is another great blog post by Jason where he talks about the asymmetry of verification. This is, like, pretty intuitive to most people who know about, like, P versus NP. It's like a thing that a lot of people have talked about throughout the, like, history of computing and software, but there are a ton of tasks that are much easier to verify than they are to solve.

And vice versa. But the most interesting sorts of easy-to-verify problems are ones where there's an objective truth, they're pretty quick to validate whether or not they're true, they're scalable, so validating a bunch of these things maybe in parallel is easy.

It's low noise, so your chance of validating it is, like, really, really high. And they have continuous sort of signals. It's not just, like, a binary yes/no, but, like, maybe you're 30%, 70%, 100% accurate or correct. And, you know, the reason I bring both these things up is software development is highly verifiable,right?

This is, like, the frontier. It's why software development agents are the most advanced agents in the worldright now. And there are so much there's so much work that has been put in over the last, you know, 20 to 30 years around the automated validation and verification of software that you build.

Testing,right? Unit tests, end-to-end tests, QA tests,right? The frontier of this is expanding. There's tons of cool companies like BrowserBase and, you know, Computer Use Agents and all these things that are making it easier to validate really complex visual or front-end changes.

Docs,right? Having, like, an open API spec for your codebase is something that can be automated, it's validated. I can go through and enumerate a bunch of these, but I actually think it is sort of a nice checklist for yourself,right?

Do you have some automated validation for the format of your code? Do you have linters? These things for professional software engineers are sort of like, oh, of course we do. But I think you can go a step further,right?

This is where that continuous validation component comes in. Do you have linters that are so opinionated that a coding agent will always make code that is exactly at the level of what your senior engineers will produce? How do you do that?

What does that even mean,right? Do you have tests that will fail when AI slop has been introduced? And when high-quality AI code is introduced, those tests pass,right? These additional layers of validators are things that most codebases actually lack because humans are pretty good at handling most of this stuff without the automated validation,right?

Your company may be at some test coverage rate that's, like, 50% or 60%, and that's good enough because humans will test manually. You may have a flaky build that every third build it sort of fails and everyone at your company secretly hates it, but no one says anything,right?

These are the sorts of things that we know are true about large codebases. And as you scale out to extremely large codebases, organizations with 44,000-plus engineers,right? This starts to become a very accepted norm that the bar is sort of maybe at 50% or 60%.

And the reality is, is most software orgs can actually scale like that. It's sort of fine to be at that lower barrier. But when you start introducing AI agents into your software development lifecycle, and I don't just mean in interactive coding, but really across the board,right?

Review, documentation, testing, all this stuff, this breaks their capabilities. Most of you have probably only seen an AI agent that operates in a codebase that has a decent amount of validation. I think a lot of the best companies in the worldright now actually have introduced very rigorous validation criteria.

And it means that their ability to use agents is significantly greater than your, like, average developer. You know, and if you think about it, this, like, traditional loop of understanding a problem, designing a solution to the problem, coding it out, and then testing it sort of shifts if you have really rigorous validation.

It becomes a process of, when you're using agents, specifying the constraints by which you would like to be validated and what should be built. Generating solutions to that outcome. Verifying both with your automated validation as well as with your own intuition.

And then iteration, where you continue to iterate on that loop. This move from sort of, like, traditional development to specification-driven development is one that we're starting to see sort of bleed into all of the different tools. Different tools have spec mode.

Droids have, like or Droid is our coding agent, have, like, specification mode, plan mode. There are entire IDEs that orient you around this, like, specification-driven flow. And if you combine these two things together, this is really how you build reliable and high-quality solutions.

So if you think about it, what is, like, the best decision for you to make as an organization? Is it spending 45 days comparing every single possible coding tool in the space and then determining that one tool is slightly better because it's 10% more accurate at Swebench?

Or is it making changes to your organizational practices that enable all of these coding agents to succeed and then picking one that your, you know, developers like or honestly letting people choose from the tons of amazing tools out there?

And when you have these validation criteria, you can actually introduce way more complex AI workflows to your organization,right? If you cannot automatically validate whether or not a PR is, like, reasonably successful or has code that won't definitely break prod, you are not going to be parallelizing several, like, agents at once,right?

You are not going to be decomposing a large-scale modernization project into a bunch of different subtasks. Like, that is a very frontier-style task to use AI for. And if the single task execution,right? The simple "I would like to get this done, here's exactly how I'd like it to be done, and here's how you should validate," if that does not work nearly 100% of the time, you can sort of forget successfully using these other things at scale in your company.

When you get into other tools like code review,right? If you want a really high-quality AI-generated code review, you need documentation for your AI systems. And yes, agents will get better at, you know, picking out, you know, whether or not to run lint or test.

They will get better at finding solutions when you don't have explicit pointers. They'll get better at search. But they won't get better at just randomly creating this validation criteria out of thin air,right? This is why we believe software developers, by the way, are going to continue to be heavily involved in the process of building software.

Because your role starts to shift to curating the sort of environment and garden that your software is built from. You're setting the constraints, you're building these automations and introducing continued opinionatedness into these automations. And, you know, if your company doesn't have at least all of these,right?

Then that means that there's a lot of work that you can do totally absent of a procurement cycle or buying one tool or trying out another one. And so plug is that we help organizations do this,right? I think that it's great to have tools that allow you to go in and assess this stuff.

They have ROI analytics that let you interact. But I think that for most organizations, there is actually, like, a very clear way to do this,right? You can go and analyze where are you across those eight different pillars of, like, automated validation.

Do you have a linter? How good is the linter? Do you have agents.md files, an open standard that almost every single coding agent supports? You can improve and systematically enhance these different validation criteria. And you can go through and say, "Well, we're seeing that coding agents are reliable enough for a senior developer to use, but our junior developers, if you have the tooling to tell, by the way, like, which developer is using what tools, you can ask questions like, 'Maybe our junior developers are actually totally unable to use these coding agents.'"

And you'll learn that the reason why is not because they're, like, more incompetent or they don't know how to use the tool, but because there's these niche practices that you don't have automated validation for,right? And if you think about what is the difference between a, like, Google or a Meta and

a still large, but, like, 2,000-person engineering org? The difference is that a new grad with effectively zero context can go and ship a change to make YouTube's, like, boundary, like, slightly more round. And it won't, with some degree of confidence, take down YouTube for, like, a billion users,right?

And the reason that's possible is because of the insane amounts of validation that have to happen on that code for it to be shipped. The big difference that we now have is we have coding agents that can go and identify exactly where these gaps are, and they can actually remediate those fixes,right?

So you can ask a coding agent, "Could you figure out where we're not being opinionated enough about our linters?" You can ask a coding agent to generate tests. We have an engineer named Alvin who I love this quote.

He said, "A slop test is better than no test." And I think that that's slightly controversial. But the thing that I would argue here is that just having something there,right? That it passes when changes are correct and somewhat accurately matches to the spec of what you want built, people will enhance it, they'll upgrade it, and other agents will actually notice these tests.

They will follow the patterns. So the more opinionated you get, the faster the cycle continues. So I think that what you guys should be thinking about is what are the feedback loops in our organization that we are catering towards?

If you have better agents, they will make the environment better, which will make the agents better, which will mean you have more time to make the environment better. And this is sort of the new DevX loop as well that organizations can invest in that will enhance all of the tools that you're procuring,right?

So no matter whether it's a code review tool, a coding agent, et cetera, they will all benefit. And I would argue that it sort of shifts your mental model about what you're, as a leader, investing in when you're investing in your software org.

Right now, the idea of, you know, OPEX as, like, the input to engineering projects. Like, we are investing in we want more people. In order to solve this problem, we need 10 more people. I would argue that the other thing that you can now start investing in is this environment feedback loop that enables these additional people to be significantly more successful,right?

And I think that that's the feedback loop that can actually take quite a lot of value because coding agents can just scale this out. So, you know, all of this is to say there's a lot that can be done outside of the, like, product itself to enable these systems.

And the best coding agents will actually take advantage of these validation loops,right? So if your coding agent isn't proactively seeking linters, tests, et cetera, then, you know, at the end of the day, it's not going to be as good as one that will seek those validation criteria.

And in addition to that, when organizations think about these sorts of things, if you're the person who's able to say, "Here's my opinion, here's how I want software to be built," it scales your capabilities out greater than ever before.

Like, one opinionated engineer can actually meaningfully change the velocity of the entire business if you take this to heart and you have a way to measure and systematically improve. So that's, you know, the majority of what I came here to say.

I think that the only thing that I'd leave you with is that when you think about where AI is going and, like, where we're at today, we are still really early in our journey of using software development agents.

If you want a world where the moment a customer issue comes in, a bug is filed, that ticket is picked up, a coding agent executes on that, that feedback is presented to a developer, they click approve, that code is merged and deployed to production in a feedback loop that takes maybe an hour, two hours, that will be possible,right?

We all are sort of skeptical about that fully autonomous flow. That is technically feasible today. The limiter is not the capability of the coding agent. The limit is your organization's validation criteria. So this is, like, an investment that made today will make your organization not 1.5x, not 2x, but that is where the real, like, 5x, 6x, 7x comes from.

And it's sort of an easy thing to say, and it's an unfortunate story, because what that means is you have to invest in this. It's not something that, like, AI will just magically give to you. It's a choice that you, as an organization, have.

And if you make it now, I can guarantee you that you will be in the top 1, 5% of organizations in terms of eng velocity, and you will outcompete everybody else in the field. So I highly recommend investing in this sort of stuff.

And hopefully you found this helpful and have some lessons to take home. Thanks.