The ROI of AI: Why you need Eval Framework - Beyang Liu

I want to introduce our first speaker, Beyang Liu, CTO of Sourcegraph, and I won't spoil the topic of the talk, but it's going to be a good one. Thank you very much.

Awesome. Thank you, Peter. How's everyone doing this morning?

Good.

Yeah? Everyone awake bright and early? Thanks for coming out. Almost awake. Awesome. So before I dive into the talk here, I just wanted to get a sense of who we all have in the room. So who here is like a head of engineering or a VP of engineering?

OK, a good number of you. Who here is just an IC dev interested in kind of like evaluating how things are going? OK. And then what do the rest of you do? Just shout out your roles. Anyone? Anyone?

Middle management.

Middle management. OK, cool. And who here has a really quantitative, very precise, thought-through evaluation framework for measuring the ROI of AI tools? OK, one hand in the back. What company are you from, sir?

Broad

Signals.

Broad Signals. OK, cool. So we've got one person in the back. And then who here is sort of like, we kind of are evaluating it, but it's really kind of like vibes at this point? Anyone? Anyone brave enough to OK, cool.

So you're in theright place. Now, who am I? Why am I qualified to talk on this topic? So I'm the CTO and co-founder of a company called Sourcegraph. We're a developer tools company. If you haven't heard of us, we have two products.

One is a code search engine. So we started the company because we were developers ourselves, my co-founder and I, and we were really tired of the slog of diving through large, complex codebases and trying to make sense of what was happening in them.

The other product that we have is an AI coding assistant. So what this product does is it's essentially

Oh, sorry. I should mention that we have great adoption among really great companies. So we started this company 10 years ago to solve this problem of tackling understanding code in large codebases. And today, we're very fortunate to have customers that range from early-stage startups all the way through to the Fortune 500 and even some government agencies.

So what's the tie-in to AI ROI? So about two years ago, we released a new product called Cody, which is an AI coding assistant that ties into our code search engine. So you can kind of think of it as like a Perplexity for code, whereas your kind of vanilla, run-of-the-mill AI coding assistant only uses very local context, only has access to kind of like the open files in your editor.

We spent the past 10 years building this great code search engine that is really good at surfacing relevant code snippets from across your codebase. And it just so turns out that that's a great superpower for AI to have,right?

For those of us that have started using Perplexity, we can kind of see the appeal. And a big piece of the puzzle is not just the language model itself, but the ability to fetch and rank relevant context from a whole universe of data and information.

And so we had to solve this problem. In order to sell Cody into the likes of 1Password, Palo Alto Networks, and LITOS, which is a big government contractor. If you flew in here from somewhere else, you probably entered through one of their security machines.

We sold Cody to all these organizations. And so each one of these organizations has kind of like a different way of measuring ROI. They have different frameworks that they apply. And so we had to answer that question in a multitude of ways.

And that's what I'm here to talk about.

OK, so I want to start out with how I would describe the value prop of AI to someone who's actually using it, to a developer. So coding without AI, I think we've all felt this before. If you've ever written a line of code, you start out by asking yourself, like, oh, this task, it's straightforward.

It should be easy. Let me just go build this feature. I should mention I poached these slides from

a director of engineering at Palo Alto Networks. He's actually giving another talk at this conference, Gunjam Patel. I thought he did an excellent job of describing the value prop that he was solving for as a director of engineering when they were purchasing a coding AI assistant.

So this is the way he described it. You think it should be straightforward, but then there's all these side quests that you end up going on as a developer. It's like, uh-oh. I got to go install a bunch of dependencies.

Or maybe there's this UI component that I have to go and figure out, this framework that I need to learn. So the gaps appear. And then without AI, bridging the gaps takes both time and focus. You kind of have to spin off a side process or go on a little mini quest.

And then 30 minutes and two cups of coffee later, you're like, OK, I got it. I filled the gap, but what was I doing again? And so AI helps bridge those gaps. It helps solve this problem. It helps developers really stay in flow and stay kind of like cognizant of the high level of what they're trying to accomplish.

And so what this means is that more and more developers can actually do the thing that we want to do, which is build an amazing feature and deliver an amazing experience instead of giving up. Now, the question is, how do we measure this in a way that we can demonstrate the business impact of what this does to the rest of the organization?

And so the answer is beans. OK, who here drinks coffee? OK, cool. Do you know the difference between a really great coffee bean and your kind of run-of-the-mill Folgers or the thing that you buy at the supermarket? OK, turns out we're all in the bean business.

We think we're in the software business, but we're really selling beans in a way. So in every company, there is someone, let's call him Bob, who grows the beans, essentially the developer. There's another person, let's call him Pat, who sells the beans.

That's your kind of CRO or sales lead. And then you have Alice on the side, who's kind of your CFO or CEO. And Alice has got to count the beans. At the end of the day,

not to diss the finance people if there are any in the room, but it's all about counting the beans and seeing how they add up. Now, I don't know if any of you have been paying attention, but in the past two years, the bean business has been revolutionized.

This thing called AI has appeared. And so what does the bean business look like now? Well, Bob grows the beans with AI, and Pat is selling beans, but with AI. And then Alice is on the side being like, well, I'm counting all the beans, and where's the ROI?

And so this is the answer that basically Bob has to answer. Pat has kind of got it easy because Pat's job is just selling the beans. That's a much more quantifiable task. Those of us that are involved in software engineering and product development, it's a bit harder to measure.

So there's tension in the bean shop. Alice is asking Bob, how many more beans are we growing now with AI, Bob? And Bob's like, well, it's complicated, Alice. Not all beans are the same. There's some good beans, and there's some very bad beans.

We're making our beans better. And then Alice is like, well, OK, the bean AI tool costs money, and we got to measure its impact somehow.

Anyone feel that tension? Anyone have this kind of conversation? We've talked to a lot of heads of engineering who see this tension very real with other parts of the org, specifically between finance and engineering.

And I think the core of the problem is that measuring AI ROI for functions where the work is not directly quantifiable through a number is what I like to call MP hard. So how many people are familiar with the term NP hard here?

OK, cool. We're all pretty technical. So NP hard basically means if you have a tough challenge, if you have a problem, and you can basically reduce it to a class of very difficult problems, it probably means your problem is not solvable.

And measuring AI ROI reduces to measuring developer productivity or the productivity of whatever class of knowledge worker that you're managing. And so that implies if you can measure the AI ROI precisely, you can also measure developer productivity. And who here knows how to measure developer productivity?

It's kind of an open question,right? So using the logic of your standard reduction proof, this problem is intractable. So that's the end of my talk. I'm just here to tell you that this problem is intractable, and we should give up,right?

Well, in the real world, we often find tractable solutions to intractable problems. And so what the meat of this talk is is really sharing a set of evaluation frameworks that we've presented to different customers. Not all of these are used by any given customer, but I wanted to give kind of like a sampling of the conversations that we've had.

And hopefully, there'll be some time for Q&A at the end where we can kind of talk through this and see what other people are doing. OK, so framework number one is the famous roles eliminated framework. So this question gets asked a lot these days, especially on social media.

AI is here to take your job. So how does this framework work? Well, in the classic framing, you buy the tool, the labor-saving tool. You observe you're in the bean business or the widget business. You observe that this tool yields an x% increase in your capacity to build widgets.

And then you can cut your workforce to meet the demands for whatever widgets you're selling. Now, in practice, we have not encountered this framework at all in the realm of software development. We do see it more prevalent in other kind of business units, things that are more viewed as cost centers, like support and things like that, especially like consumer-facing customer support.

But for software engineering, for whatever reason, we haven't encountered this yet in any of our customers. And we think that the reason here is that, number one, if you view your org as a widget factory, then you're going to prioritize outputting widgets.

But the thing is that very few engineering leaders, effective engineering leaders these days, view themselves as widget builders. The widgets are kind of an abstraction that don't apply to the craft of software engineering. The other observation here is that the widgets that we're building, which is software at the day, great user experiences, they're not really supply limited.

So if you have an extensive backlog, the question is, if we made your engineers 20% more productive, would you go and do 20% more of your backlog? Or you just cut down 20% of your workforce and say, it's fine.

We don't need to get to the backlog. And for 99% of the companies out there that are building software, the answer is, no, we want to build a better user experience. These issues in our backlog are very important.

We just can't get to them. So framework number one is kind of like talked about frequently, but in practice, we haven't really seen it as an evaluation criteria.

Framework number two is what I like to call A/B testing velocity. So how this works is you basically segment off your organization into two groups, the test group and the control group. And then you say group one gets the tool and group two does not.

And then you go through your standard planning process. Most people, as part of that planning process, what you do is you go and estimate the time that it will take to resolve certain issues. So how long is this feature going to take to build?

How long is it going to take to work through these bug backlogs? And then because you've divided the groups into two now, you have some notion of how well you're executing against your timeline. So you basically run this A/B test.

So Palo Alto Networks, one of our customers, ran something similar to this. And the conclusion they drew was the approximate timelines got accelerated 20% to 30% using Cody. And so this is a very kind of rigorous scientific framework.

We see it come up now and then, especially when companies are of a certain size and they're very thoughtful about this question. The criticisms about this framework are no two teams are exactly the same. If you lop off your development org, you have dev infrastructure on this side, maybe back end, and then you have front end teams on this side, it's hard sometimes to compare these different teams to each other because software development is very different in different parts of your organization.

There's also confounding factors. Maybe team X had an important leader or contributor to part. Maybe team Y

suffered a bout of COVID that blew through the team or things like that. So you have to account for these things when making your evaluation. And this framework is also high cost in effort. You basically have to do the subdivision.

You give one group access to the tool, and then you have to run it for an extended period of time in order to gain enough confidence. But provided you have the resources and the time to estimate it, we think this is a pretty good framework for honestly testing the efficacy of an AI tool.

OK, framework number three. I call this time saved as a function of engagement. So if you have a productivity tool,

using the product should make people more productive,right? So if you have a code search engine, the more code searches that people do, that probably saves them time. If it didn't save them time, there would be no reason why they would go to the search engine.

And so in this framework, what you do is you basically go look at your product metrics, and you break down all the different time-saving actions. You identify them, and then you kind of tag them with an approximate estimate of how much time is saved in each action.

And if you want to be conservative, you can lower bound it. You could say a code search probably saves me two minutes. That's maybe like a lower bound because there's definitely searches where, oh my gosh, it saved me half a day or maybe a whole week of work because it prevented me from going down an unproductive rabbit hole.

But you can lower bound it and just say, OK, we're going to get a lower bound estimate on the total amount of time saved. And then you go and ask your vendor, hey, can you build some analytics and share them with me?

So this is something that we built for Cody, very fine-grained analytics to show the admins and the leaders in the org exactly what actions are being taken, how many explicit invocations, how many chats, how many questions about the code base, how many inline code generation actions.

And those all map to a certain amount of time saved. There's one caveat here, which is in products where you have an implicit trigger, like an auto-complete, you can't rely purely on engagement because it's not the human opting in to engage the product each time.

It's sort of like implicitly shown to you. And so there, you tend to go with more of an acceptance rate criteria. You don't want to do just raw engagements because then the product could just push a bunch of low-quality completions to you, and that would not be a good metric of time saved.

So we have a lot of customers that do this. They appreciate it. One of the nice things about this is that because we're lower bounding, it makes the value of the software very clear. So a lot of development tools, us included, I think Cody is like $9 per month, and Sourcegraph is a little bit more than that.

But it's like if you back out the math of how much an hour of a developer's time is worth, typically it's around $100 to $200. It's like if you save a couple of minutes a month, this kind of pays for itself in terms of productivity.

The criticisms of this framework is, of course, it's a lower bound. So you're not fully assessing the value. If you go back to that picture I showed earlier of the dev journey where you're kind of bridging the gaps, I think a big value of AI is actually completing tasks that hitherto or beforehand were just not completed because people got fed up or they got lazy or they just had other things to do.

So this doesn't capture that. It doesn't account for the second order effects of the velocity boost. And it's good for kind of day-to-day, like hey, is this speeding up the team? But it doesn't capture some of the impact on key initiatives of the company.

So that leads to the fourth evaluation framework. So a lot of our customers track certain KPIs that they think are correlated with engineering quality and business impact. So lines of code generated. Does anyone here think lines of code generated is a good metric of developer productivity?

OK, I think in 2024, we can all say that it's not. We have seen this resurfaced in the context of measuring ROI of AI because it's almost like we've forgotten all the lessons that we learned about human developer productivity.

And with AI-generated tools, now it's like, oh, you generated

hundreds of lines of code for a developer in a day. And so we noted that we were actually losing some deals on lines of code generated. And when we actually went and looked at the product experience, we're like, at first, we're like, hey, maybe there's a product improvement here that we should be making because people aren't accepting as many lines generated by Cody.

But when we dug into this, we're like, oh, the competitors' products, it's just more aggressively kind of triggering. And that's not the sort of business that we want to be. So more and more, we're kind of pushing our customers to not tie to generic metrics or high-level metrics, but identify certain KPIs that attend to changes that you want to make in your organization.

So with Lydos, our big kind of government contracting customer, they identified a set of zones or actions that they felt were really important. They wanted to reduce the amount of time spent answering questions, spent bugging teammates, and spend more developer time in these areas that they identified as value add.

Three things mainly: building features, writing unit tests, and reviewing code. And so that's what we tracked for their evaluation period.

A fifth framework is impact on key initiatives. So this is the kind of like map your product to OKRs framework. And so there are a couple of companies where they're in the midst of a big code migration. They're trying to migrate from COBOL to Java or maybe from React to Svelte or the latest JS framework.

And these are kind of like top-level goals that the VP of Engineering really cares about. And so if you have a product that accelerates progress towards this, then the ROI is really just what's the value of bringing that forward by x number of months or, in some cases, x number of years or making it possible at all.

How do you measure how much you pulled it forward?

That's a good question. So the question was, how do you measure how much you pulled it forward? It is really a kind of judgment call with the engineering leader at that point. By the time they have this conversation with us, they've typically already started it or have had a few of these under their belt and have an idea of the pain.

And then they can assess kind of the shape of the product and the things that we do and estimate how much quicker it'll done. We also have case studies demonstrating, hey, this thing that used to take a year or longer, we squished it into the span of a couple of months.

OK, and then the last framework is what I'll call survey. So this sounds like the least rigorous framework, but I think it's still highly valuable. Basically, it's just run a pilot, have your developers use it, you can compare against another tool, and then at the end of it, just ask your developers which one was best.

More and more nowadays, we don't see this in an unbounded fashion. In the kind of like 2021 ZERP period, people were just like, whatever makes the developers happy, let's just go buy that. Nowadays, we see it in more of a bounded fashion, which is a lot of orgs allocate some part of their budget toward investing in developer productivity and developer productivity tools.

So that chart over there shows kind of the range of orgs surveyed in a survey run by the Pragmatic Engineer newsletter. And it typically ranges somewhere between 5% and 25%. So within that budget allocation, you have a certain amount of budget to allocate to tools.

Then you basically say, subject to that constraint, let me go ask my developers what tools they want the most.

OK, so I'm basically out of time, but hopefully this gave you kind of like a sampling of the flavors of different frameworks that are involved. This is something that we've had to work through through a lot of customers, ranging from very small startups to very large companies in the Fortune 500.

I just want to say, be skeptical of anyone saying P equals NP, of saying they have a precise way to measure AI ROI or developer productivity. No framework is perfect. The most important thing, I think, is define clear success criteria.

That's something that both you, your internal teams, and stakeholders will appreciate, and also the vendor because they know what success looks like. And then the last kind of final note here is productivity tools are often bottoms up, but we've actually found that top-down mandates with AI can sometimes help because developers can be a little bit of a skeptical crowd.

But if you believe firmly that this is where the future is going and that people have to update their skill set to make productive use of LLMs and AI to code more productively, we've actually seen success in this case where a CEO basically says, we're adopting Cody, we're adopting Code AI, go figure out how to use this.

It's not going to be perfect, but it's something that is going to play out over the next decade.

And then, sorry, one final thought. As we move towards more automation, I like to think of two pictures in mind. So one picture is the graph on theright, which is kind of like a landscape of Code AI tools.

So on the left-hand side, you have the kind of like inline completions, very basic, completing the next tokens that you're typing. And then on the farright, you have kind of like the fully automated offline agents. And we're trying to make all these solutions more reliable because generative AI is sort of inherently unreliable.

We're trying to make it more general and

make it productive in more languages and in more scenarios. And we actually think that the path there is to go from the left-hand side to theright-hand side, not jumping straight to the full automation because that's a very difficult problem.

So we think the next phase of the evolution for us is going from kind of like these inline code completion scenarios to more of what we call online agents that live in your editor but can still react to human feedback and guidance.

And then the second picture is the Mythical Man month. I think a lot of people are familiar with this classic work. It talks about the classic fallacies with respect to developer productivity that a lot of companies make. One question I would pose to all of you is, have the fundamentals really changed?

As we have more quote-unquote "AI developers," should we measure them by the same evaluation criteria? And the challenge question I would pose to all of you is, I think the lessons from the Mythical Man month is you prefer to have one very smart engineer who's highly productive over 10, maybe even 100 mediocre developers who are kind of productive but need a lot of guidance.

And so as AI automation increases, the question is, do you want 100 mediocre AI developers or do you want 100x lever for the human developers, the really smart people who are going to craft the user experience?

Allright, that's it for me. Yeah, if you want to check out Cody, that's the URL. And that's my contact info if you want to reach out later. Do we have time for questions?

We have time for perhaps one or two questions.

One or two questions.

I will run around with the microphone if anyone, just so we can catch it on audio. Cool.

Thanks for Cody. You are the best guys. I tried them all. I'm staying with you. You've got really good insight what's happening now with software development. So what is your intuition? Are we going to get away from coding and the code would become a boilerplate and we move to meta-programming or it will be still code as a main output of the senior engineer?

Yeah, that's a really good question. The usage patterns that we're observing now is more and more code is being written through natural language prompts. I had an experience just on Friday, actually, where Cody wrote 80% of the code because I just kept asking it to write different functions that I wanted.

And that was nice because I was thinking at the function level rather than the kind of line-by-line code level. Yeah, and it was nice because it allowed me to stay in flow. But at the same time, the output was still code.

And I really do think that we're never going to see a full replacement of code by natural language because code is nice because you can describe what you want very precisely. And that precision is important as a source of truth for what the software actually does.

I think that's probably it now. Thank you very much.

Thank you.

Thank you.