AI Consulting in Practice – NLW, Superintelligent, @AIDailyBrief⁩

Today I'm excited to talk about something a little bit different. Soright now, there's been— the last couple of months have been an interesting time in AI. There's been a sort of surge in the narrative of an AI bubble, a lot of it driven by dubious studies like the MIT report.

And so what I wanted to do today is get into not so much the practice of consulting and transforming, but what organizations are actually finding value inright now. So for those of you who don't know me, there's kind of two contexts I bring to this conversation.

The first is as the host of the AI Daily Brief, which is a daily news analysis podcast about AI. The second is as the CEO of Superintelligent, which is an AI planning platform. And so the different perspectives are sort of very high-level macro thinking about the news that's happening, and then a much more kind of ground-level view where we're spending a ton of time interviewing executives about what's going on inside their organizations.

And what we're going to talk about is sort of, one, kind of briefly in the first part, just the status of enterprise adoption as it currently stands. And two, and the more interesting part, is we've been live with a study in the market for about a month now, collecting self-reported information about ROI around different use cases.

And this will be the first time— this week was the first time I did some analysis on it. And so I'm going to share what people have told us around the first kind of 2,500 or so use cases that they've shared.

So it should be pretty interesting stuff. Talking about kind of enterprise AI adoption first, I'll go through this pretty quickly because it's pretty well-known stuff. The short of it is, enterprises are adopting AI in a growing fashion. Pretty much everyone is using it at least a little bit, and increasingly they're using it a lot.

This year I will need to tell none of you that there is a major inflection around specifically adoption in the coding and software engineering,right? You saw a huge, huge uptick in this. There's a lot that's interesting about that from an enterprise perspective because it wasn't just with the software engineering organizations.

Other parts of the organization are also now thinking about how they can communicate with code, build things with code. But that's a huge, huge theme of this year. Coming into 2025, one of the big sort of thoughts that many people had was that this would be the year of agents inside the enterprise,right?

That big chunks of work would get automated away. And on the one hand, I think it's pretty clear that we didn't see some sort of mass shift towards automation writ large across different functions in the organization. But when you dig into the numbers, there has been actually pretty significant shifts in the patterns of agent adoption.

So this is from KPMG's quarterly pulse survey, and it's a measure of how many enterprises that are a part of their survey, which is all companies over a billion dollars in revenue, have actual sort of full production agents in deployment.

So this isn't pilots, this isn't experiments, this is where they consider some agent that's actually doing kind of work in a full way. And it's jumped from 11% in Q1 of this year to 42% in their most recent study for Q3.

So you actually are seeing pretty meaningful uptake of agents inside the enterprise. In fact, I would argue, based on our conversations, that people have— that it's moved more quickly through the pilot or experimental phase than people might have thought.

So much so that you're actually seeing now a big shift in the emphasis around kind of the human side of agents and how humans are going to interact with agents. And it's involving a shift in upskilling and enablement work.

You're seeing a decrease in the sort of resistance to agents as people start to actually dig in with them. You're seeing more experiments like these sandboxes where people can interact with agents. So this is a big theme, even if it wasn't necessarily the dominant theme that some thought it might be coming into this year.

At the same time, it is absolutely the case that many, many, if not most enterprises are, broadly speaking, stuck inside sort of pilot and experimental phases. There is a lot of challenge around moving from some of those first exciting experiments to something that's more scaled.

So this is from McKinsey's state of AI study, which came out, I think, a couple weeks ago now. And you can see only 7% of the organizations that they talk to claim or sort of see themselves as fully at scale with AI and agents.

And it's something like 62% are either still experimenting or piloting. Interestingly, big organizations are, in general, a little bit ahead in terms of the organizations that are scaling as compared to small organizations. This has been a thing that we've noticed kind of throughout the trajectory of AI adoption over the last couple of years, that you would think that perhaps smaller, more nimble companies would be more kind of quick to adopt these things.

But in fact, it's often been the opposite with the biggest organizations making the biggest efforts. You can also see from the chart on the bottom that there's very sort of jagged patterns of adoption,right? You're starting to see from last year, if you looked, there's very similar kind of rates of experimentation across lots of different departments.

You're starting to see some pretty big breakouts now with, for example, IT operations kind of jumping out ahead of other functions. I won't spend too much time on this sort of high-performer piece, but I think the thing to note, because it comes back in some of the stuff that we found with our ROI study, is that you are also starting to see a pretty significant bifurcation between leaders and laggers when it comes to AI adoption.

And one of the things that tends to distinguish the companies that are leading is that they are just doing more of it, and they are thinking more comprehensively and systematically about AI and agent adoption. So they are not just sort of doing spot experiments, they're thinking about their strategy as a whole, they're doing multiple things at once.

And importantly, they're not just thinking about sort of the very kind of first-tier time savings or productivity types of use cases, they're also thinking about how do we grow revenue, how do we create new capabilities, how do we create new product lines.

Overall, it's very clear that despite what is sort of the concerns in the media, that spend is going to do nothing but increase on this. So the bottom is the KPMG pulse survey again, and this is an estimation of the amount of money that these organizations intend to spend on AI over the next 12 months.

At the beginning of the year it was 114, which by the way was up from like 88 in Q4 of last year. It's now up to, in their last study, 130 million is what they expect to spend in the year ahead, which obviously the total magnitude doesn't matter as much as the change.

You also see the green charts are from Deloitte, and you can see 90% plus of organizations intend to increase their spend on AI in the next 12 months. And as part of that, I think that you're going to see a much more determined conversation around impact and ROI, which is a particularly thorny topic.

But interestingly, there has been an increase in optimism over the course of this year around the realization of AI. So this is from a different KPMG study, their annual CEO survey, which interviews tons and tons of CEOs. And if you look at the 2024 numbers, 63% of those polled thought that it would take between three and five years to realize ROI from their AI investments.

20% said one to three, and 16% said more than five. This year, in that same survey, the number that said one to three years had gone up to 67%. There were now 19% who said six months to one year, and three to five years was down to just 12%.

So huge, huge kind of pull forward of expectations of ROI realization. The challenge is that ROI is really tough. So this is back to the pulse survey. 78% of those polled in that survey said that they thought that ROI is going to basically become a bigger consideration in the year to come.

But also, 78% said that traditional impact metrics and measures were having a very hard time keeping up with the new reality that we're living in. And this is something that I've heard constantly, over and over, from CIOs and other people who are in charge of these investments, that the ways that we have measured impact of previous technologies and just previous initiatives are kind of falling flat with AI.

And so that got us thinking about the overall need that we have to just have more information. I'm not even talking about good systematic information, just more information around what ROI looks like, what impact looks like. And I've got this great podcast audience, they're super engaged, and so we just decided, screw it, we're going to ask them.

We're just going to ask them to report on what ROI they're finding from their use cases. So this went up at the very end of October. Like I said, as of this morning, or when I last looked, we've had over 1,000 submissions.

1,000 individual organizations, rather, submit something like 3,500 use cases. And this is some of the first observations that we had around kind of the first 2,500. So the impact categories, the way that we divided things was into sort of eight broad categories of impact, which will all, I think, be very intuitive to you guys.

Time savings, increased output, improvement in quality, new capabilities, improved decision-making, cost savings, increased revenue, and risk reduction. So basically, it was trying to think of kind of a broad, simple heuristic for kind of dividing or subdividing the different ways that people are thinking about ROI.

And TL;DR is that people are finding ROIright now. Now, again, the caveats are that this is a highly enfranchised audience, they're listening to a daily AI podcast, and they are voluntarily sharing this, so I think that there's some caveating there.

But you have 44.3% saying that they're seeing modest ROIright now, and then you have another 37.6% seeing high ROI. For the purposes of a lot of these stats, high ROI will be significant plus transformational. Only 5% or so are seeing negative ROI.

And keep in mind, negative ROI doesn't mean that they think programs are failing, that just means they haven't— they've spent more than they've gained in terms of how their perception is. More than that, expectations are absolutely sky-high. 67% think over the next year they will see increased in high growth in their ROI.

So we have a really optimistic sense from the ground view of where ROI is going to be in AI. You even have the teams that are currently experiencing negative ROI, 53% say that they're going to see high growth.

So very, very optimistic. As you might imagine, time savings is the default. It's the starting point for so many organizations. It represents about 35% of the use cases. After that, increasing output, quality improvement, basically all those things that you would imagine around productivity are sort of like the dominant categories when it comes to these use cases.

When it comes to the specifics around time savings, you see a real cluster between one and ten hours, especiallyright around five hours. And I think this is interesting to call out because it's so obvious to all of us who are inside building these things, whether you are a developer or an entrepreneur or just someone sort of in and around it, how the vast breadth of opportunity that AI represents, new capabilities, things unimagined yet, it's hard to— or it's easy to forget that if you save five hours a week or ten hours a week, you're talking about winning back seven to ten work weeks a year.

And that's very, very powerful. And when it comes to a lot of these enterprises, that is a very meaningful thing, even if it's not what they're ultimately in it for. Interestingly, though, it's very clear that the story, although it might have a concentration in time savings, is about much more than time savings.

So this is the ROI distribution category, or ROI distribution by organization size. And this starts to get really interesting where you can see that there are some differences in where different size organizations are focused. So for example, the organization size between 200 and 1,000 people has a higher portion of their use cases concentrated in increasing output.

Now, we haven't taken the time yet to really figure out exactly what this means or even speculate on what this means. But I think it's interesting that this is a category of organization that has often reached a certain scale but is still very much striving for more and so seems to be focused more on use cases that expand their capabilities.

Same thing with when you start to divide things by role. You see real kind of variance where, for example, C-suites and leaders are less focused on those time savings use cases and more focused on other things like increased output and new capabilities.

In general, we're finding that C-leaders and just sort of C-suite and leaders in general are even more optimistic and excited and seeing transformational impact than people who are in more junior positions. Now, some of this might be sort of selection bias in terms of what types of use cases you are focused on.

If you are in that C-suite, you're thinking about things that inherently, if they work, are more transformational. But it is notable that 17% of the use cases that people in those leadership positions have submitted, they say have transformational impact and ROI already.

I'm going to skip this because we don't have time for too much. You're seeing, interestingly, a concentration where the smallest organizations are getting more of that transformational benefit early. One of the things that I want to do following this study is maybe do a sort of second round where we dig into what this 1 to 50 person size really looks like.

I actually think that whereas there might be a lot of similarity between a 1,000 and a 2,000 person organization, there could be a wild difference between a 3-person small company and a 40-person company. And so I'd really like to dig into that more.

But you are definitely seeing a lot of impact in those sort of more small, nimble moving organizations. As you might expect, coding and software-related use cases have a higher ROI than average and a lower negative ROI than average.

One really interesting kind of pulling on a specific category of use cases, risk reduction is our lowest category in terms of the percentage of use cases that that was their primary benefit. So when you're filling out the survey, which is, by the way, at ROISurvey.ai if you want to check it out, you basically only get to pick a primary ROI benefit.

We didn't want it to be super sort of, we wanted you to pick and hone in on the thing that seemed most important or most significant. And so only 3.4% have risk reduction as their primary benefit in terms of ROI categories.

But it is by far, those use cases are by far the most likely to have transformational impact as their outcome. It's at 25%. So a full quarter of those have transformational ROI. And interestingly, I was having this conversation with a couple of my friends who work in sort of back office and compliance and risk functions, and this has been their experience as well, where there are a lot of the challenges for those organizations involve sheer volume and quantity in ways that AI can be really helpful for.

We also are finding some interesting patterns among organizations. And again, this is where we get into some of the limits of this just being a whoever walks through the door of my listeners. We have a pretty heavy concentration among technology, as you might expect, industries, and among professional services.

But we still have fairly decent sample sizes for some others. And in both healthcare and manufacturing, the use cases are meaningfully higher impact on average than the average across all organizations, which I think is kind of worthy of further study.

Last sort of part of this as I wrap up, a lot of these use cases, as you saw, have to do with that sort of first tier that most enterprises are going to be in. Increasing the amount of content that you output, increasing the quality of that content, just finding ways to win back your five hours a week.

But increasingly, there are automation and agentic use cases, and we are absolutely seeing that where those are the focus, where those use cases mention certain types of automation or they mention agents, they wildly outperform in terms of the self-reported ROI from them.

That's both on automation and it's on agents. And I think that that's sort of a trend towards where we're headed with sort of the next layer of more advanced use cases. The last thing that from this sort of first look of observations is there is clearly benefits, and this goes back to what we saw with that McKinsey study as well, of thinking about AI and agentic transformation in systematic, cross-organizational, cross-disciplinary types of terms.

Effectively, pretty much directly, the more use cases that a person or an organization submitted, the better they tended to see ROI for. Now, there's lots of reasons for that, but I do think it speaks to that core idea that once you move beyond kind of your single spot experiments, there's a lot of opportunity to sort of grow the impact of the organization.

So like I said, that is the first look. It's kind of the first two-thirds of these use cases. We'll be open for another week, and then we'll have the full study out at the beginning of December. I'm really excited, I think, heading into next year to see how we move from sort of generic conversations about impact and our gut senses about impact to a lot more random experiments like this to figure out where the impact really is and where we go next.

So look at that. I'm going to end 27 seconds early and really throw off the time, but appreciate you guys all being here. And again, if you want to check this out, it's at ROISurvey.ai.