The AI emperor has no DAUs why most devs still don't use code AI: Quinn Slack

Great to see you all again. I'm Quinn Slack, I'm CEO and cofounder here at Sourcegraph, and yes, I was at Bleacher Port. I think I learned I'm better coding than doing sports, so I'm in theright role here. I want to talk to you about why most devs still do not use Code AI.

It's probably a mind-boggling fact to all the people in this room, but it's true, and we'll have a good discussion about that. I'm a coder, I code all the time, been building Sourcegraph, which is code search and Code AI for devs since 2013.

I merged four PRs earlier today. But I want to talk mostly about the product and what to build and how to get it in the hands of users, because if you don't do thatright, then all the code that you write is for nil.

So, three points I want to make. One is that most devs don't use Code AI. I want this to scare you. I want to scare you in this talk. And then I want to share some of the mistakes we made and some of the tips from our experience doing all the things wrong and then trying to figure out how do we do itright at Sourcegraph as we've been building Cody.

Allright. First, just show of hands here. Who uses who in the last week has used any kind of Code AI tool while coding? Allright. Man, I wish the whole world was like this room, but we are in the heart of it in San Francisco.

And the whole world is very different. Most devs do not use Code AI. And I'm going to give some numbers here. I'm privy to a lot of private information and stuff that I've heard as well, and all of this is in line with what I've been hearing.

So, first number. Oh, man, this is a great statistic. 92% of devs use Code AI tools at work. Wow. That means these tools must have tens of millions of users. This was a study from GitHub, 500 people. Well, it turns out this is only in the US, and they had a very broad definition of what it means to use a Code AI tool.

So, here's a case where the hype kind of outstrips the reality, a common pattern. This is an infographic from GitHub. 70% of developers see a benefit to using AI coding tools at work. Surely, if there's a benefit, they must all be using these things,right?

I mean, if only it worked that way. Allright. Some numbers for the broader universe. There are 1.3 million paid subscribers to GitHub Copilot. This was what Microsoft reported in January 2024. 1.3 million is a lot, but if you're just paying for something, are you actually using it?

And also, 1.3 million is not a lot compared to all the people in the world that write code. And there's actually a way to get an even finer point on that number. GitHub released a study saying, if you look at the fine print, that in this time range, this year, 935,000 devs received a suggestion, not even accepted it.

So, there's going to be some that saw that little ghost text in their editor and never actually accepted it. And also, that's yearly active users. I don't know of any companies that go and cite yearly active users. Surely, the monthly active users and daily active user figures are much lower than that.

So, these numbers are pretty you could fill a city with these people, but it's not that many people when you compare it to the 26 million professional developers in the world. And I think this is an undercount. I will say that, how do you get this number?

I googled a bunch of different stats, and they all were around this. So, that's the vibes citation there. There's a lot more people that touch code in some way or another. Again, it's really hard to get a number on this, but I'd say probably 100, 150 million people in the world touch code in some way.

And that includes students and other people and so on. So, 935K, best case, what is that in MAU? That's tiny compared to the number of people that interact with code. So, this is a scary thing. My best-case estimate is that around 5% of professional developers use Code AI.

These are people that are paid a lot of money to write code, and they do it all day, every single day. And all of us here in the room, we know that there's this amazing new alien technology that's dropped on Earth, and that it changes how we code.

And 5% are using it. That's crazy. That means in a room of 20 people, 19 are not using this. It just absolutely boggles my mind, and that's a huge problem. And it's an even bigger problem, because if you look at what they're actually using, the vast majority of Code AI use is just autocomplete.

It's the ghost text. I mean, yeah, that's good, but anyone who has used anything beyond that knows that AI can do so much more when coding. It can write entire files, it can answer questions, it can fix bugs, and all of that stuff.

So, this usage is absolutely tiny, and we need to realize that we here in this room, we are all the freaks. And for us to be successful requires us to change a lot of minds out there in the world.

So, how do we change those minds? Well, we've been building Cody. Cody is a Code AI tool. It does all of these things. And so, we encounter a lot of reasons why people are hesitant at first. And I've taken the reasons from our own internal tools and ranked them.

Why don't most devs use Code AI? The first one is just, eh, they don't have a good reason. There are some people in the world, believe it or not, that have not used ChatGPT, that have not heard about it, even devs.

There's a lot of devs who are kind of grumpy, and they say, well, it's not perfect, I tried it, and it gave me a wrong answer. Yeah, no shit. But there's a lot of other people that have figured out how to get theright answer out of it.

A lot of people say, I don't need it, it didn't help me that much. Again, you got to figure out how to make it useful for you. A lot of people's company hasn't adopted it, it's too expensive, or one thing we're seeing a lot less of now is these security, privacy, and legal concerns.

I think it's a situation where enough people are using it, so that if it's illegal to use Code AI, then we're all fucked. So, it's mutually for destruction.

And there's a lot of people who haven't seen it yet.

This should really scare us, because there's a lot of companies that are relying on a lot of people at a lot of enterprises using Code AI. There's a lot of people's retirement accounts or option trading accounts or whatever that are also relying on that.

And if you think about how technology makes money, I mean, the money that gets deposited into your bank account and your paycheck, where do those dollars come from? It takes kind of a long route. Just to explore the market a little bit here, you've got these foundation model companies like OpenAI and Anthropic at the top.

And then you have the AI infra companies. And the AI infra companies, they will send, if they're serving and doing inference, they'll send some of the money back to the foundation model companies. In some cases, they're the same companies.

And what drives usage of that? You have some experimental usage, people being the lucky loos, going and trying it out. That will turn into some experimental revenue. But all of that can happen without any real actual usage. But that gives the perception of, holy shit, this stuff is hot, this stuff is working.

And then you start to get some real usage, some people using it, maybe paying 20 bucks a month on their credit card. You get some devs using it at work. And then the holy grail, the way that software makes money, is from enterprises where there's recurring revenue from real usage.

That first-year contract, it doesn't matter. It's got to be JP Morgan pays you a million dollars the first year, and then they renew for $2 million the next year. And all of the money in our paychecks ultimately comes from that.

If you're working at an AI infra company, if you've raised money to go and sell some AI product, it's ultimately because someone out there thinks that five years from now, JP Morgan is going to be paying you $2 million a year.

So, that's the only thing that matters. Everything else is downstream of that. And it turns out, there's just not that much of that going on. And also, this is an even more lopsided pyramid than I've depicted here. We got NVIDIA and chip makers at the top, and so much investment riding on it.

So, this is where we ultimately need to get to. We need this person to be using our software. And maybe it's not the coolest thing out there, but that is the reality of how technology makes money. And I want to put some numbers to this.

I estimate, based on my internal information, some of the information that GitHub has shared at this conference, that the total recurring revenue from Code AI usage is around 300 million ARR. Now, look, that's a lot of money. If you had a company that was doing that, then it could go public on its own.

So, that's good. But how much of that money actually goes to these other companies? Not that much. From our own data at Sourcegraph, where we spend a lot of money on AI inference, we spend less than 10% of our revenue.

So, if you take that number and by the way, we're not even doing it in that optimized fashion. If you take that 300 million number, 10% of it, that's $30 million going back up to these foundation model companies and AI infra companies.

It's a long way from where we are today, that amount of revenue, to then making the kind of revenue that's going to justify these massive multi-billion dollar valuations. Also, another way to look at it is Salesforce. Salesforce's annual revenue is $36 billion.

All of this stuff, all this hype, all this usage of Code AI, it's amounting to a tiny, tiny fraction, 1/120th of Salesforce. So, we have a long way to go. Usage needs to grow a lot. Or I could be wrong, and maybe the doomers areright.

Maybe not for the reason they thought. Maybe we should never have done this, and this is all just going to be one massive hype bubble, and it's going to pop, and we're all going to be miserable. But I think, because I use Code AI every day, obviously, and you do too, this stuff is real.

It's just it's really early, and usage needs to grow a lot. So, if you're building a product, keep that in mind. This is what I'm betting on. And let's talk about some of the lessons from our experience building Cody at Sourcegraph.

I wish I had this mindset going into it. Just quick background to establish, why would you even listen to me? Maybe I should have done this in the first slide, but hey, here we are. Sourcegraph has been around, as Brittany said, for 10 years.

We started out with code search, and then we found out that if we had this tool that all the devs use and had all the code in a company, and it turns out that was amazing context to build a Code AI on top of.

And so, we took that, and we built Cody. It's got really great autocomplete and inline edits, generating unit tests, and chat. And chat is where we really differentiate, because that's where you can make the best use of context.

You can figure out, why is this broken, or how do I change this, or where should I start on this? That's where we excel. We've got a lot of really big customers. We have four of the top five biggest banks in the country.

We have most of the FAANG or Mañana companies or whatever, and a lot of other great customers, including some that are presenting today. And we are the number two Code AI company in revenue, second only, of course, to GitHub Copilot.

But because we're number two, and because we're a startup and we're scrappy, we try harder.

Allright, so what have we learned? One is that hype fools everyone. Hype fools us. Hype fools all of you. Hype even fools your customers. And the second thing is that AI code completion, the thing that autocompletes the rest of the line or the next few lines, that's like the freakish kind of feature that comes along like one time in 100 years.

And we kind of got spoiled. It is so perfect. So, I'll get to that, but first, just on the hype, how do you get away from being fooled by the hype? I'll share a few tips that have worked for us.

The first, and this is the most important thing, if you're building a product and you are not using it every single day, it is not going to make it. There is zero hope. If you're building it on a team and the people all building it don't use it every day, your customers will not.

But they might say it's really awesome, and that's the dangerous thing about this hype. Customers don't know what they want. Every customer for about a year said, I want fine-tuned models. They said that all the time. Our salespeople would say, hey, we need to build fine-tuned models.

Everyone was talking about it. GitHub Copilot has it as a bullet. All of our competitors have it as a bullet that's like italic, and it's like coming soon, some bullshit thing. Really, what they just want is they want it to work well, and they use some of these terms.

So, don't listen to them, because what they're saying is probably downstream of them sitting at some conference like this and getting some cool ideas. They want the product to work. And if you can't describe why it works and why it's great without using the word AI, then you're probably not going to make it.

This is a really existential question. So, we make a product that makes developers a lot more productive, like 20 or 30% more productive. If it's so damn good, why are we selling a product? It's like the people that have stock picks and sell newsletters.

Why don't we go and buy a software outsourcing firm and prove it and monetize our product by actually capturing that value directly? That's a damn good question, and I wish that we had done that a year ago, and we're looking into doing that now.

And if you don't have the confidence to go and do that, to put a few million bucks behind that, if you've raised a bunch of money, then I'd say your product's probably not ready yet. And then finally, I think we've all seen this.

This is actually not even one of the worst ones. This is a tweet that's got a lot of activity. Just keep in mind that that does not translate into DAU, so don't feel shitty about yourself all the time.

I wish that Elon would add something like this that would actually tell you the DAU of the products that are getting all this hype. But until he does that, you can imagine it. Allright, I talked about autocomplete being this freakishly good feature.

Here's how we think about features at Sourcegraph, this kind of four box. You want to be in the topright, and that's where it's an AI feature that doesn't take a ton of time to see is that correct, and it's really easy, and it's used often.

AI autocomplete is great because it's literally every keystroke, and you can glance at it in a few seconds or even milliseconds, 100 milliseconds, you can figure out, is this correct? So, that's a really good property of a feature.

And it just so turned out that the first feature of Code AI happened to be smack dab in the topright. It's pretty amazing. But most other Code AI features do not have this level of product-market fit, and we need to realize that.

Now, we can work on it, but where's chat? Not used nearly as often, and it's harder to vet that long response edits similar. I know a lot of people love doing the inline edits like MetaK, OptionK, and Cody, and Cursor, and things like that.

It's great, but we got to be prepared for what actually works being a very different form from what exists today. Then there's a lot of other stuff, like the agentic stuff. I mean, look, it's obviously the future, but it's just not there yet.

Who here has used a Code AI agent to actually merge a PR in the last week? Allright, cool. Thank you for helping us push the world forward, but it's not there yet. And other features, you really got to go for that topright and drive any feature you make to the topright, and if not, it's just not going to make it.

So, what are we doing at Sourcegraph to address this? Well, we are searching for that next great Code AI modality. What's the next autocomplete? There's some ideas like the next edit suggestion or like Copilot++. I'm skeptical of that.

Chat, Steve Yeagy, who is at Sourcegraph, he just wrote this great blog post about chat-oriented programming, CHOP, where it turns out there's a lot of devs who have a chat session running in Cody or even ChatGPT or something all the time while they're coding, and it's just one ongoing conversation.

It's a totally new way of coding. And what's weird is it's like the Gen Zs who do it, and it's like the 50-year-old disgruntled programmers who do it. So, I love that it's got kind of some usage in both generations.

There's new ways that people are going to be using Code AI that we have not even thought about, and it's so early to the previous point. We always got to remind ourselves to build the manual and explicit thing first.

We got spoiled by autocomplete. It's automatic, and it just like triggers every keystroke. No, first you got to make something work in manual and explicit mode. If you have chat, you got to make it so people manually mention the context they want before magically inserting the context.

If you've got an agent, put it in the editor and make it work in the editor so that if it's wrong, the dev can just change itright in their editor instead of having to go into code spaces or some other totally different UI.

So, make it manual and explicit first and really easy for the dev to go and fix it. And then you can add the magic. Probably, you're going to be bogged down enough with stuff to make it a great product, and you won't actually get to adding the magic for a while.

When you're pitching, always explicitly dehype and set expectations low. No matter how much you dehype, the person receiving it, they will hype it up in their minds. So, you got to remove the hype. Do this so aggressively. And here's an example from Sam Altman.

This is a tweet, I don't know, like a year ago, when ChatGPT had probably tens of millions of DAU. So, he'sright. We know he'sright. And if he's saying this, and his product has way more validation than your product, well, you probably should be saying it like 10 times more and 10 times more intensely.

Also, it's kind of cool marketing, and it makes you seem cool.

Yeah, you got to become here. It's covered up. You got to become a DAU yourself or kill the product. And then the last thing is just that as an ecosystem, all these foundation model companies, all these AI infra companies, all these AI applications, we all live or die together.

Anthropic and OpenAI and Mixtral and Fireworks and all these great companies that we use are not going to get paid if we're not in business making a shit ton of money on Cody. So, we all need to work together and realize that we all have so much to benefit from this stuff actually being used, from turning down the hype and building great products that get actual devs using it all the time.

And that starts with you if you're the one building the product. So, thank you, and if you want to reach out to me, there's my contact info, and happy coding.