Code Generation and Maintenance at Scale: Morgante Pell

So, I'm Morgante, the founder of Grit. I'm going to be talking about code generation and maintenance at scale, uh, or CPUs still matter, and what it takes to actually make one of these agentic workflows work in production. Quinn was talking about how most people have not merged them; that's 100% true.

Grit has probably merged more PRs than any other company in the world at this point. Because we focused very narrowly, and have done a lot of work above the model layer, and we're going to talk about how we did that.

Uh, it's helpful to know why I started Grit. My background is all developer tools. I've been working at Google Cloud for 5 years, and built a lot of stuff on the DevOps layer,right? Thinking about Kubernetes and how do you orchestrate very large-scale systems, working on tools like Customize or Terraform templates.

And one of the biggest things I learned from this was how rare it was for a customer to come to us and ask for a brand-new application,right? People didn't come and say, "I want to build a new app on Google Cloud."

It sounds cool. Ninety percent of the time, customers came and said, "I have this line-of-business application that is doing $100 million in revenue. How do I run that on Kubernetes?" Right? And that's what all of our templates did.

That's what everything we built, uh, in the sort of pre-AI era of automation was all on how you ran existing applications. And that's why we started Grit, because every demo that you usually see the typed one of these ones on Twitter, it's usually type a prompt, get a new application from scratch,right?

Build something brand-new. It's exciting. It goes really well on Twitter. That's not what devs do day in and day out. Developers spend most of their time modifying huge applications so that flights run on time. And this goes into three sort of categories of developer tools,right?

There's ID developer assistants. This clearly has the most product-market fit today. It's really easy,right? Like Quinn was saying, you just do auto-complete,right? That's a very simple thing. Then there's AI agents that are focused on lowering the floor,right? They're allowing people to do tasks that they otherwise don't have the skills for,right?

Allowing a product manager or other non-technical user to build an application that they don't have the skill set for. This is powerful, but I actually am pretty skeptical that that's how most software is going to be built in the future.

It requires a real thinking of how do you actually spec things out, how do you think about edge cases, basically how we train as engineers that's required to build great software. Which is why with Grit, we focus on raising the ceiling of what great engineers can do,right?

Principal engineers, the most high-level engineers that you work with, they're primarily limited by time,right? They can't be in ten places at once. But AI agents can be in ten places at once if there's theright engineer controlling them. And that's what we focus on, is supercharging the productivity of the top 1% of engineers.

It also helpfully gets around the problem of 95% of engineers not using AI. Uh, the great thing about Grit is 95% of our customers, 95% of their engineers don't touch it,right? There'll be 100 engineers on the team. They're not using Grit.

There's one engineer who is deeply embedded with Grit and is generating hundreds of PRs with their agents.

But to do this, tools need to change,right? The IDE that you have today, it's a scalpel,right? It's focused on editing individual lines of code, and it's great for that, but it's not focused on editing hundreds of repos at once.

It's not focused on how do you open 1,000 files and make those changes in them, and that's why we built Grit. It's basically we want to have bulldozers for code,right? If you're generating huge quantities of code, how do you push that around in an effective way, uh, when you're not editing individual lines, when you're working on a higher level of abstraction?

And this is super necessary,right? We've seen an explosion in how much code is being generated. A lot of our customers are seeing, uh, 20 to 30 percent more code, uh, that's coming out of their teams now, just because there's more PRs, there's more CI, there's everything that's running because you have code genetics.

And this is going to accelerate,right? Once we go from 5% to maybe 50% of people actually using AI, there's going to be way, way more code in the world, and we need better tools for managing that code once it's in production.

So, just to give an example of what this looks like in practice, uh, this is a real customer that we had. Uh, they have been around. They've got thousands of repos. They've got thousands of developers. And they wanted to use open telemetry instead of logging,right?

This is traditionally a massive effort,right? You have to coordinate across hundreds of teams to get them to understand open telemetry, to understand how to instrument their code. You have to get them to do actual code changes to swap out their logging library.

You have to do a bunch of education efforts. And it's actually very much a people and process problem, usually,right? Something where you have a program manager who has a massive Excel spreadsheet. That's what I say, like, what Grit we compete with is actually Excel, not any other AI dev tool.

It's that you have these spreadsheets where you go and manage these changes,right? And tens of thousands of developer hours go into a change like that. So a lot of companies just say, "You know what? That's not worth it,right?

I'm not going to migrate to open telemetry. I'm not going to go into the cloud. I'm going to stay in my older ways." You know, people still have millions of lines of cobalt because it's just so much work to do this kind of coordinated change.

Uh, with Grit, you don't have to do that coordination effort,right? Because you can have one engineer who is actually coordinating that change, is driving the individual AI developer agents to do the changes. You don't have to have a bunch of meetings because it's just one person telling their little agents what to do,right?

And you can do it with under 100 developer hours because they're just doing high-level orchestration and then thousands of, uh, compute hours that the AI is doing as it's healing these changes. And we've seen change. This is literally a project that they had postponed for years because it just was not feasible.

They couldn't get enough on the roadmaps. They got it done with Grit in a week,right? It just opened up 1,000 PRs across all the repos, fixed them, uh, iterated on the changes, merged, and migrated over. So how do we actually do a change like that?

Uh, it's sort of a three-level process. Um, planning is a big part of it,right? So we index the entire code base. Uh, we do both semantic indexing, so understanding, uh, embeddings and understanding the intent of each file, but we also do a lot of traditional, more static analysis indexing.

So we understand what's the structure of the code, what's being imported from where, what's the dependency graph,right? This is all the sort of thing you need to know to actually do really high-quality agentic flows. Then once we have the plan of how we're going to make changes, we execute the plan,right?

So we use large language models that are going to take that change, delegate it to a sub-agent. The sub-agent is going to make a modification in one file. Uh, it uses something called GritQL, as well as diff generation from language models.

Uh, GritQL is our custom query engine that's able to actually index, uh, millions of lines of code and find theright places to modify things, and then finally push it up for PR review, uh, and be able to, uh, both have developers who are the director of it,right?

So a typical scenario is that there will be the principal engineer who's driving the change. They'll review the PRs. Then individual developers, their primary interaction with Grit is just seeing a PR land in their repo, and they might leave like one line comment, uh, that Grit will learn from, but they don't actually open up the Grit UI ever because they're just responding to the changes that come from Grit.

Cool. So a little bit more about how we find code,right? So our goal here is to find all the error logs in the code base because we want to migrate those over to open telemetry. Uh, the naive approach, if you go to many of the workshops yesterday, would be, "Allright, just chunk it, put it into RAG.

Uh, you have a bunch of embeddings." Uh, and, you know, that, you know, theoretically could work for maybe some document use cases. I'll tell you that absolutely will not work for this problem,right? If you just go to try to find stuff that looks like a logging error, uh, it's going to find a lot of irrelevant context,right?

It's going to find anything that looks like log. Like, an LLM has a hard time differentiating between a user-facing log, like an alert in a UI, and an actual log that you want to be putting into open telemetry,right?

There's also unpredictable thresholds,right? You don't actually know how much code you're looking for. You can't do, you know, retrieve the top 10, uh, closest matches. In some cases, you want to retrieve 10,000. In a lot of cases, developers don't even actually know how big of a change it is until Grit starts to propose it for them,right?

So that's where we built the GritQL query engine. Uh, it's our own custom query system that combines the best of static analysis with the best of AI. So you've got this query here that's looking for, uh, logger, uh, with some set of arcs,right?

So we're just going to look for a function call, basically. Uh, and that's a syntactic query. So we're just looking for all of our function calls across our entire code base. Uh, and then we're going to say that our arcs should be like an error occurred,right?

And that's just we're giving an example of, like, what's an error message that we might be trying to look like. Like, there is a magical word that converts it into a semantic representation. So we want to say, "What's some code that, you know, embedding search, the cosine similarity, is sufficiently above a threshold that this is an actual log message versus some other function call that we wouldn't be wanting to modify?"

Uh, and then we can finally do an imported from. We've got a built-in library to be able to understand the whole dependency graph. So we can do things like make sure this is actually imported from log for j.

In this example, we wanted to make sure that we're only substituting our log for j logs, and that will go and traverse, uh, the import graph earlier in the program.

So that brings us to finding the code that we want to change. But once we've actually found the code, how do we make reliable changes? Uh, and unfortunately, really smart models, uh, still have a hard time doing this completely autonomously.

Uh, just to give an example, uh, I just used Claude Sonnet today. Uh, 3.5, it's a really good model. Uh, and it uses, uh, we put the entire files, uh, so a bunch put a bunch of context into the context window, 100,000 tokens from Grit's VS Code extension, uh, and some of our linter outputs.

Uh, and we wanted to just write a function that's going to convert from our linter JSON output, uh, and puts it into diagnostics for the Grit VS Code extension,right? Pretty simple task. I promise that everything that's required was in the context window,right?

It's not something where I had to go retrieve additional information. Uh, it was all there. Uh, came back with a pretty reasonable completion. Converts ESLint to LSP diagnostics. Uh, this looks reasonable to me. Like, I imagine if you look at this code, you wouldn't be able to tell anything that's wrong with it.

I certainly couldn't tell anything that's wrong with it from eyeballing it,right? Uh, but this is wrong,right? And this is one of the main things to understand is that, uh, humans also can't look at this code and understand what's wrong with it,right?

This is why we have systems that allow us to, uh, type check, lint things,right? It allows us to understand code that even looks kind of correct is, in fact, incorrect and will fail in production. But I went back and just asked, uh, Claude to fix the code for me.

I said, "This is broken in production. I tried to put it into my VS Code extension. It broke." And I just asked it why. Uh, as you can kind of imagine, it doesn't do any better than I do of just looking at it, looking and eyeballing the code and understanding why it's wrong,right?

Uh, so it just comes in, specs some totally irrelevant answer of how to fix it because, again, it's not grounded in what the actual errors are. Uh, fortunately, uh, we have a great tool for TypeScript called TSC,right? We can compile this, and it's going to go and tell me, "Actually, uh, Grit positions and Grit ranges have a slightly different type signature, uh, than LSP ranges,"right?

And this is, you know, why compilers are great, is we can actually get that information really close in the dev loop. With this information fed back into an LLM, it's able to correct that mistake, uh, no problem,right? That's a pretty easy change.

It uses the convert LSP range to Grit range, which, by the way, was in the context window. It could have used that before. It just didn't realize that it needed to use that until it had the compiler error forcing it to,right?

So this is already how I think I it's important to see that IDEs are already making us superhuman, and we need to make sure that all of our AI agents have access to the same tools that make them, uh, super AI.

Uh, so compilers rock,right? This basic flow of prompt, get some code, uh, build it, type check it, and then fix that output based on the LLM. This is actually really powerful. This is probably, uh, half of what you need to do to build a really good agent is make sure that you have this flow working reliably.

Uh, but they're really slow when you're talking about enterprise code bases. Uh, so this is real numbers from, like, one of our customers. Uh, it takes them 10 minutes to build their application from scratch, and that's just for type checking.

It's not even pushing a production build,right? And this is actually pretty typical if you look at very large-scale enterprise code bases. Uh, that's why large companies have had to build a lot of caching, because it's hard to build a large code base from scratch, which this is completely different than what people usually expect for AI.

People usually think inference takes a long time,right? You're waiting for the AI. And this is actually a pretty long prompt, 30 seconds,right? We're using a huge model to generate this code. It takes 30 seconds. But that's dwarfed by the 10 minutes to build the application,right?

This basically destroys our entire agentic flow if we're waiting 10 minutes for every single change to validate that it's correct. But this is even more compounded if we're trying to do that in a loop,right? If we're trying to do a single change, uh, it might take a day if you're just doing this naively.

Uh, there's some agent projects that, in fact, do take a day to make very basic changes because you don't have haven't done this optimization level. But then you might ask, like, how are you able to make changes in your IDE at a fast rate,right?

You're not waiting 10 minutes every time you make a single keystroke to get a compiler error. Uh, it's much faster than that. It's because there's been a lot of work with language servers, uh, to solve this so that you can do a bunch of upfront prep so you can build the index in memory, have that in-memory queryable index, and then only rewrite the parts or only recheck the parts that you've modified,right?

On every keystroke, uh, most tools, like TS Server, for example, and TypeScript, uh, is doing live reconciliation of figuring out that specific file,right? And this is much, much faster. You can do the 30-second prompt, then one-second recompute from the TS Server, then 30 seconds to fix it.

Uh, and this is a much more reasonable flow,right? So you obviously want to be using the same kind of language server tools that you'd be using as a human, uh, not CLI-based tools, which often don't have the same heuristics in place to be able to optimize.

Uh, and then, you know, ideally, you do this in a nice loop. You eventually get to the point where you can commit and get a fresh PR to do that migration to open telemetry. This is what it looks like in theory.

In practice, at some point, it hits an error that it can't fix,right? It hits an error that gets into a loop, and it's continuously trying to fix the same error. It uses five different techniques, then goes back, and your context window is completely polluted with the wrong errors,right?

Everyone often says, like, "Agents don't work." Uh, this is probably half of the reason that agents don't work, is that you just have compounding failures,right? We found anytime we actually have more than 10 prompts in a row, uh, our chance of having a successful PR is dramatically lower,right?

Uh, so the way we work around that is instead of trying to repeatedly fix an error, uh, we should actually just save our original state, revert back to that, uh, and then continue to edit from there,right? So if we went down a path that was just a bad path,right, we got stuck in a row, we want to go back to a known good checkpoint, uh, and then build from there.

And this is actually how we're able to do this quickly. You don't want to spend 10 minutes recomputing each time. Uh, we want to actually build our in-memory graph that we were talking about with TS Server. We want to save that.

We want to take a snapshot of memory. Uh, so we use Firecracker. It's a VM manager, uh, that's used for AWS Lambda, but we can actually use it for dev environments too. Uh, and we can actually take the in-memory state, snapshot that, and then fork it into 10 different isolated environments that all have everything precomputed.

You can try 10 different changes in them and then figure out the correct change that is most likely to yield good results from there. In fact, this becomes massively parallel. You can end up with an AI system that looks more like a distributed database than it does a traditional agent or something that you're running on your laptop,right?

We actually have flows where we often have, uh, 6 up to 10 different agents working in parallel, all working from a known good state. They're supposed to report back once they're done. And then we'll actually look at the different evaluations.

We look at, uh, both some LLM-based evals, but also heuristics like how many errors are there currently in the code base, uh, how many unit tests are currently passing, uh, and then actually compute, like, what of these, which is the quorum,right?

It's actually similar to, uh, again, a database system where you would have a voting of, like, what's the new master. Uh, here it's like, what's the new good state that we want to fork from? Uh, there's four here that have similar states that we want to use that as our new known good state, uh, save that as our known good state, and then fork from there going forward,right?

And this ends up being much, much more reliable because we can have an entire PR that, yes, we've done 30 or 40, uh, different generations on it, but in the final chain, there were only four different generations,right? Because we had one that we got went back to known good state, then a second one that's all operating from that quorum at each checkpoint.

But these edits get pretty expensive,right? If you're doing, uh, 40 different edits to make a single PR across very large files, uh, that's a lot of money that you're spending on inference. Uh, this is a common problem with making good edits.

Everyone naively just asks for, "Generate the whole file again,"right? It's the simplest approach. You definitely should start with that if you're building your own AI tool. Uh, but then you run into the classic problem of laziness. So this is actually still from, uh, Sonnet.

It still said, you know, uh, the start of the function remains the same,right? It left this comment in because it didn't want to output that code. And it's just because output tokens are fundamentally more expensive. If you look at GPT-4.0, it's, uh, $5 to $15 ratio of input tokens to output tokens.

Uh, Claude 3.5 Sonnet is $3 to $15. Uh, this is pretty consistent across the board. And then response limits are not growing at the same level with context size,right? We've got models out there that have 1.5 million tokens, 2 million tokens in their context window, and still only it's outputting 4,000 tokens at a time,right?

Because it's auto-aggressive, it gets much more expensive. So you really don't want to output entire large files as you're making edits. You want to find a good edit format. Uh, so whole edit format works well. Uh, it's very expensive, though.

Uh, you can do diffs,right? You can say, like, "Generate a unified diff for this. Try to apply that." Um, there's some problems with this. One is, like, line numbers. Uh, LLMs are still not very good at knowing what theright line number is.

Even if you give it them, they're just not that good at the math part. Uh, and there's also off-distribution,right? Real-world code that's trained on is largely not trained on diffs,right? It's trained on actual full files. Uh, you can do simple search and replace with function calls.

Uh, the problem with this is function calls are underneath, for the most part, JSON. Uh, escaping code in JSON format is terrible. You end up using a lot of tokens just for, uh, just escape characters,right? It's just not a very good format to use.

Uh, so that's why we actually developed a GritQL loose search and replace. So we could actually do something that's similar to what you would have on the model of being just a before snippet. And this is something you might have, like, in a tutorial, which is, like, "Replace this with that,"right?

This is what we and this is actually the exact same output that comes from the LLM. We'll do a match,right? We'll do a loose match to try to find what's the code that looks like that, and replace it with the code that looks most similar to that afterwards,right?

And this works really, really well because we don't have to you can allow irrelevant details, like what's currently inside the make match function, and just give enough detail to make the replacement. Cool. Uh, and just want to leave you with, like, where we're going next.

This is our current UI. It still is very traditional,right? It still is, uh, building, you know, what's an AI workflow looks kind of like your CI, even though it's thousands of agents executing. Uh, I'm really excited about where we go next with this.

Uh, figure out, like, what does it look like to manage an entire code base? I think of, like, SimCity is, like, the ultimate where you can zoom in and out and understand, uh, different levels of granularity and edit things there.

Cool. Uh, thanks so much. I'm Mugante, and we are hiring, so, uh, scan the QR code.