Can you prove AI ROI in Software Eng? (Stanford 120k Devs Study) – Yegor Denisov-Blanch, Stanford

So companies spend millions on AI tools for software engineering, but do we actually know how well these tools work in the enterprise, or are these tools just all hype? To answer this, and for the past 2 years, we've been researching the impact of AI on software engineering productivity, and our research is time series because we look at get historical data, meaning we can go back in time, and it's also cross-sectional because we cut across companies.

And the way we used to measure most of the impact is by a machine learning model that replicates a panel of human experts. The way this works is that imagine you have a software engineer who writes a code commit, and this code commit would be evaluated by multiple panels of 10 and 15 independent experts who would evaluate that code commit across implementation time, maintainability, and complexity, and then produce an output evaluation.

So we took the labels of these panels across, you know, millions of kind of evaluations, and then trained a model to replicate this panel of experts, meaning that we can deploy this at scale, and if there's ever any doubts around the model's output, you can always kind of assemble your own panel and see that it correlates pretty well with reality.

Today we'll talk about 4 things. We'll start off with looking at some of the things that are driving AI productivity gains in software. Then we'll look at an AI practices benchmark that we developed. We'll then look at how we propose to measure AI return on investment in software engineering, and lastly we'll finish things off with a case study.

So here we took 46 teams that were using AI, and we matched them with 46 similar teams that were not using AI, and we measured their net productivity gains from AI quarterly. And the shaded area is the middle 50% of the data, and the dark blue line is the median, which as of July of this year stands at about 10% for this cohort.

I'd like to direct your attention to the fact that the discrepancy between the top performers and the bottom ones is increasing. There's a widening gap. And so if we very unscientifically and very illustratively project this forward, we might get something like this,right, where you can have these top performers being part of this the rich gets richer effect, where the these successful early AI adopters might compound their gains while these strugglers could fall further behind.

At some point this is going to converge, and this is very directional, but my point here is that if you're a leader in a company, you definitely need to know in which cohort you areright now so that you can course correct.

And without measuring the impact of AI on your engineers, you're not going to be able to do this.

So we started investigating. What are some of the factors that drive these top teams to perform better? And the first thing we looked at is AI usage, or basically token spend. In this graph you have the same kind of on the vertical axis the productivity increase, and then on the horizontal one you have the token usage per engineer per month on a logarithmic scale.

And what you can see is that the correlation is quite loose, 0.20 or so linearly, and there is a bit of a death valley effect around the 10 million token mark, whereby teams that were using that amount of tokens seem to be doing worse than teams that were using a bit less tokens.

It's very directional but interesting nevertheless. The conclusion here might be that AI usage quality matters more than AI usage volume. We dug deeper and we said, well, does the environment in which the engineers work impact the productivity from AI?

And we came up with an Environment Cleanliness Index. It's quite experimental. It's a composite score that looks at tests, looks at types and documentation, and at modularity and at code quality. And that index is on the bottom axis here from 0 to 1, and then on the vertical axis once again you have the kind of productivity lift relative to teams not using AI.

And so what you can see is that there is a 0.40 R-squared, meaning a pretty decent correlation around environment cleanliness and gains from AI, or productivity gains from using AI. And so the takeaway here is to invest in codebase hygiene to unlock these AI productivity gains.

We dug deeper to illustrate this concept, and here we have on this graph on the vertical axis the percentage of tasks that might be able to be completed by AI based on 3 colors. And so green means that AI can do most of the work for that task in that sprint.

Yellow means that AI can help someone, and red means that AI is not very useful. And this is quite illustrative, but it it conveys the point. And so then any codebase at any point in time sits on a vertical line across this graphic.

And what you can see is that clean code amplifies AI gains. Secondly is that you need to manage your codebase entropy,right, your codebase tech debt, because if you just use AI unchecked, this is going to accelerate this entropy, which is going to push and degrade your cleanliness to the left, kind ofright?

And then you as as a human need to push on the other side to kind of improve or maintain that cleanliness to keep reaping the benefits from AI. Thirdly is that it's important that engineers need to know when to use AI and when not to use AI.

And what happens when they don't is this kind of line on the left whereby you have AI AI outputs that are rejected or need heavy rewriting, which then leads to engineers losing trust in AI, saying, okay, this just doesn't work, I'm not going to use it, which then further collapses your AI gains.

Now we said, can we find out whether we can look not only at usage but at how are these companies and these engineers using AI? And we came up with an AI Engineering Practices Benchmark. The way this works is that we can scan your codebase and detect these AI fingerprints or artifacts, basically traces of how your team is using AI.

It's quite directional at this point but evolving. And we can quantify this based on the percentage of your active engineering work that uses each AI pattern, and then we kind of repeat this monthly using get history. And the way this works is more or less you have kind of a few levels, and level 0 might be how humans are just not using AI and write all of the code.

Level 1 is kind of like personal use where engineers are not sharing prompts across the team or not versioning them. Level 2 is team use whereby teams are are sharing these kind of prompts and rules. And then level 3 is even more sophisticated.

It's where AI autonomously does specific tasks, maybe not the entire workflow. And level 4 is, you know, agentic orchestration, which is where AI just runs the entire process. And so this is going to be an open source tool which you can leverage if you sign up on the Sweeper Research Portal.

We applied this benchmark to one of the companies in our research data set, and we saw this. This company had 2 business units with equal access to AI tools,right? Same licenses, same spend, same tools, same everything. But the adoption rate and the usage rate was very different by business unit.

On the left, the first business unit, as you can see in the area in the blue, seemed to be using AI a lot more for almost 40% of their work, whereas on the on theright, the second business unit seemed to struggle behind a bit more.

And so the takeaway here is that access to AI and even AI usage doesn't mean or doesn't guarantee that that AI is going to be used in the same way across a company. As a leader, you really want to be understanding not just whether they're using, but also how your engineers are using AI.

Great. Now let's dive into how do we actually measure AI return on investment in software engineering.

Oh, there we go. Okay. So here ideally we would be measuring this based on business outcomes,right? I give my AI engineer my engineer's AI, and then I make more money, more revenue, net revenue retention, whatever business KPI you want to track.

The problem is that there's too much noise between the treatment,right, giving AI, and the result, which is the business outcome. And on top of this, there's confounding variables such as your sales execution in the macro environment, your product strategy.

And therefore, although that would be ideal, unfortunately I think we need to find alternative paths, and the most logical one is to simply look at the engineering outcomes because there is a clear signal,right? But here we need to go beyond measuring AI usage into measuring engineering outcomes.

There's a few caveats, and this topic is quite heavily discussed, and so I want to mention some of them. The first one is that this is assuming that our product function can properly direct that increased capacity into something that generates value.

And if they aren't directing that, then it's a product problem, which although sits quite close to engineering, it's slightly different,right? The second caveat is that this assumes that engineering is a meaningful bottleneck for value, which frankly it typically is, and that you can guard against Goodhart's law by using a balanced set of metrics and also by having a good company culture that doesn't weaponize these metrics.

And thirdly is that AI is still very new, and measuring proxy metrics is still better than not measuring. There's going to be winners and losers in this AI race, and progress is better than perfection here. And so metrics don't need to be flawless to be useful is what I want to illustrate.

So then here we have 2 parts which you need to do to get the ROI from AI,right? You kind of need to measure usage, and then you need to measure engineering outcomes. And so let's start with usage. There's really 2 buckets for enterprises.

There's kind of more in a research environment, but to make it simple, there's access based and there's usage based. Access based is basically looking at when did people get access to the tool. And here we have you can kind of do a pilot group, give that group AI, and then compare it to a similar group without AI, or you can measure the same team across time.

The problem is that access based is noisy, and the gold standard is really usage based, which uses telemetry from APIs from these coding assistants,right, to give you theright data to know who's using AI and and where. And the caveat here is that the vendor API is different.

Unfortunately, tools like GitHub Copilot aggregate the data, and other tools like Cursor give you more granular data. The big takeaway is that you can measure impact of retroactively by using get history. And so you don't need to set up an experiment now and wait 6 months.

You can actually, if you've already adopted AI, you can go back in time and and and do this. It's quite easy. Now we've seen usage. Let's look into how do we actually measure engineering outcomes. What are some of the metrics we propose?

Here we have our framework which we propose, which is using a primary metric and a guardrail metric. And so here the primary metric is engineering output. It's not lines of code, it's not PR counts, and it's not DORA.

And it's basically based on this machine learning model that replicates the panel of experts,right? And the second set of metrics are the guardrail ones, which you want to maintain at a healthy level, but you don't want to maximize.

It doesn't make sense to maximize them, truly. And so then there's 3 categories within the guardrail ones. We work on refactoring, quality tech debt and risk, and then people and DevOps. The third bucket is important to highlight that these are not productivity metrics.

They're useful, but you cannot just kind of use them like maximize them to maximize developer productivity. They kind of fall off at some point. And so the goal here might be to keep your guardrail metrics healthy while increasing the primary metric to whatever degree possible.

Now let's dive into a case study. Here we worked with

a company that large enterprise. We took a team of 350 people under a vice president, and we measured pool requests. The reason we did this is to illustrate that you cannot measure pool requests to understand whether AI is helping you.

And so here this team adopted AI in May of this year, and we measured the 4 months before, 4 months after. We saw a 14% increase. Great. That's fantastic. But what about reviewer burden? What about code quality? So we measured code quality.

And here what we saw is, I mean, firstly actually code quality, think of it as maintainability, scale from 0 to 10, and there's kind of these bands. It uses our our methodology. You can read it online. But basically what you see is that in the pre-AI period, their code quality was quite stable and consistent.

And once they adopted AI, 2 things happened. Code quality decreased, and then code quality became more erratic.

Next we took a look at our metric, which is engineering output. It's not lines of code. And here for every month you see the sigma, the sum of the output delivered for that month, broken down into 4 buckets: rework and refactoring.

So rework is when you're changing or editing code that was it's still kind of fresh, so it's recent. Refactoring is when you're changing code that's a bit older. And what then like added and removed, it's pretty self-explanatory. And then also you can see these kind of benchmarks.

So we can benchmark this company against similar companies in their industry. And here AI usage had 2 effects. Firstly is that rework went up by 2.5 times, which is really bad. And effective output, which is kind of like a proxy for productivity or so, didn't really change.

And so then what's the conclusion here? Let's do a recap. So we saw that PRs went up by 14%, but this is inconclusive because more PRs doesn't mean better. We saw that code quality decreased by 9%, which is problematic.

We saw that effective output didn't increase meaningfully. And then we saw that rework increased by a lot. And so then the question here is, what is the ROI of this AI adoption,right? It might be negative. And what I want to point out here is that had this company not measured this more thoroughly and simply measured PR counts, they would have thought, hey, we're doing great.

We increased our productivity by 14%. That's from the numbers. That's how many million lots of millions of dollars. And does this offset the AI license? Sure thing it does,right? The other thing is that I don't think this company should abandon AI.

They should simply use this data to understand what they're doing wrong, how can they improve, because AI is here to stay. It's a tool that's going to transform how engineers are are working,right? And you can't just kind of like abandon it or so.

Great. So this concludes our insights for today. If you've enjoyed this talk and you would like similar insights for your company, I invite you to participate in our research. Everything you've seen today can be accessed through kind of participating in our research, some of them through live dashboards in our research portal.

And especially I'd like to invite companies that have access to Cursor Enterprise to participate because we have a high need for this so we can publish papers around the granularity of using AI in software engineering. You can sign up at softwareengineeringproductivity.stanford.edu.

Thank you so much.