The State of AI Code Quality: Hype vs Reality — Itamar Friedman, Qodo

I'm really excited being here. So many—so much pragmatic insight and suggestions. I was sitting there just, just before. So I'm Itamar Friedman, the CEO and co-founder of Qodo. Qodo stands for Quality of Development, and I'm going to share our reports and other companies' reports about State of AI Code Quality.

You know, trying to talk about the hype versus reality, which was, like, one of the points that were discussed here quite a lot, which is awesome. So in the last 3 weeks, 4 weeks, we saw, like, 3 outages in the clouds, unfortunately,right?

And these are coming from companies that really care about moving fast,right? They're saying themselves that they're using AI to generate code 10%, 30%, 50%. At the same time, they care about quality. So how did that happen, and is it related?

I don't know, but let's have some—I'm going to share some guesses. So, by the way, 60% of developers say that, like, a quarter of their code is either generated by AI or, like, shaped by AI, and 15% say that even more than 80% of their code is basically generated or shaped by AI.

Now, people are using AI to do vibe coding, but actually they're even doing it for vibe checking, vibe reviewing. This is the command of Claude—this is the prompt for the command of Claude code for security review. It was hyped like 2 months ago.

Do you know what I'm talking about? Now, it says there, I don't know if you see it, "You are a senior security engineer." Good. And then, like, somewhere there down the line, it says, "Please exclude denial of service."

Don't catch denial of service issues. Maybe that's part of the reason, like, we're having cloud outages. Probably not just that, but you get the point. Like, we need to be rigorous about how we deal with quality. It's not just, like, vibe quality or so, like we're doing vibe coding sometimes.

Let's go to another example, okay? Cursor, I guess, like, or Copilot, most of you use rules,right? We're going to talk about it. You invest in code generation. After a while, you understand if you invest, you'll get more out of it.

And we asked, like, a bunch of developers, and I'm asking you as well. Think for a second, for all the developers there in the audience. Like, when you write Cursor rules or Copilot rules, etc., do you feel they're completely followed?

Or it's, like, mostly followed? Do you know how much they're followed? In what extent are they followed? It's rigorously, like, how technical deep they're being followed. So what we get back, like, the answer from what you see here on the screen is mostly, like, B, C, and D.

They are followed, but they're not completely followed, okay? So that means, like, we are generating code, trying to push it to the standards, but it's not necessarily still, like, getting to the quality we wanted. I'm going to share a bit more statistics and information and some insight from 3 reports.

One done by Qodo, another done by Sonar, another by Pharos, and all of them are focused on code quality, review, etc. The sample size is thousands of developers, in some cases even more, millions of pull requests, and a billion of lines of code that were being checked.

Like, for example, if you think about Sonar, this is a company, yeah, a bit, like, coming from pre-AI, but they see code at scale. And they're doing, like, a lot of checks in code that are not necessarily AI-focused but are necessary in order to check your software from all possible directions.

And that's why they're scaling, and the scale of the code they're seeing is immense, okay? So, for example, we took information from their report, and eventually my purpose here is to break down the different dimensions of what code quality means and give you some—share some stats and insights.

I want to start with the end, okay? This is the takeaway I want you all, like, to take from the next 13 minutes that I have. We started with code generation. We, like, out of the box, use it for autocomplete, etc., and you invest in it, and you can get more out of it.

But there is a glass ceiling for how much productivity you can get from code generation. And then we moved to the agentic code generation,right? Let's call it Gen 2.0. And that's a higher glass ceiling. It could do much more productivity, especially if you invest in it.

For example, rules, etc. Then, with AI breaking outside of the IDE, we can start using AI also for code, for agentic quality workflows. It could be inside the IDE, but the truth is that if you think about all the workflows you have in your organization, especially if you're more than 100 developers or so, you probably have a lot of workflows that are related to quality that you need to automate.

And that's where you start, like, breaking through the glass ceiling of productivity if you invest in it. And finally, I claim that you need those agentic workflows, keep learning, and we might touch a little bit of that, like, later on, okay?

Like, because quality is something dynamic. So you'll only finally break the glass ceiling if you really have those quality workflows and rules and standards being dynamic. And then you will see the promised 2x, let alone the 10x, that you were promised and hyped.

And you heard from McKinsey and from Stanford. You're not getting that. I don't need to tell you the 2x, 10x for the entire software development life cycle. So, a bit more about the market adoption. One of the reports says that 82% of adoption already for AI dev tools are being used daily or weekly.

Some people, like 60%, 59% report that they're using more than 3, and 20% saying they're using more than 5 code generation tools. If you think about it for a second, don't only take, like, Cursor, Copilot, Codex, Claude code, etc.

Sorry for insulting anyone that I forgot their tool, but there's also the Lovable, etc. They also generate code. And by the way, you're going to get to 10. I'm counting on me. You're going to get to 10 tools in 2, 3 years that generate code for you, okay?

Come to talk to me about it later. I'll try to convince you. And the thing is that it's coming from bottom up. Like, 50% of the usage is coming from less than 10, a team that are less than 10 developers, but it is propagating also to the enterprise.

Again, I'm sure you know. I mean, propagating to the enterprise at scale. Like, not just, like, 5 developers. In the last year, we're seeing, like, more and more enterprises using code generation. So, like, on average, within reports, we saw 82% to 92% using weekly to monthly code generation tools.

And in some cases, maybe extreme, maybe not, we're going to talk about it, we saw a 3x productivity boost in writing code, okay? But that doesn't mean that if you have 3x productivity in writing code that you actually guarantee any quality, like I presented before.

So actually, 67% of the developers that we asked have serious quality concerns about all the AI-generated, all the generated code, code generated by AI or influenced by AI. And they're claiming that they're missing the framework, how to deal with quality, how to measure quality.

It's a big question. What is quality? I'm going to talk about it in the next few slides, okay? Think about it for a second before I break it down. What is quality? So what we're actually saying is that the crisis with vibe coding, viable coding, we're seeing it shifting and evolving, is that you're getting, like, more tasks being done.

Like, 20, some report, 20% more task, you know, velocity. And, like, 97% or so of PRs being opened. And eventually, it takes more time to review PRs. Like, 90% more time to review PRs. And by the way, like, there's a lot of statistics about AI-generated code.

At least there's not less amount of bugs per line of code. I'm not claiming that there are more, but even if there's not less bugs per line of code, you have much more bugs because there are much more PRs, much more code being generated, etc.,right?

So that's a problem for the reviewer. So it's not by surprise it takes more time to review these, especially in the age of agents,right? When 5 minutes calling to Claude code, I have 1,000 lines of code after 5 minutes.

Once upon a time, it took me, like, hours to write 10 proper lines of code,right? Now, let's zoom out for a second. Code generation is magnificent, okay? Like, it's a game changer when you're talking about Greenfield. You saw people talk about it a few slides, a few minutes before me.

It revolutionized how we do proof of concept, project, etc. But when you're dealing with heavy-duty software, then you like it or not, we are dealing with a lot of things. When you serve millions of clients, you have financial transactions.

When you're doing transportation, you're dealing with code integrity, if you like, code governance, review standards, testing, reliability, etc. That's what we need to deal with. Now, let's break that under-the-surface part of the glacier into 2 dimensions. This is one dimension.

You can look on the quality issues throughout the software development life cycle. Like, planning and then development, writing code, code review. Code review is a bit of a process, but, like, what you're, like, checking quality, that's part of the process of code review.

Testing, which is another part of quality, and deployment. And I know I didn't cover the entire, like, software development life cycle, but just to give an example. And each one of them, like, possess, like, introduces new problems that are coming because you're using more and more AI-generated code.

Now, another dimension to look at is actually code-level problems and process-level problems, okay? I'm not opening the, you know, list of functional, just opening the list of non-functional. You're talking about security, inefficiency that are not necessarily functional. I'll show you some statistics about that.

And then process-level is, for example, learning. Hey, if you'll have

a bad outage because of AI-generated code, who is responsible? Is it the AI or the team that owned that, okay? Like, you need to learn and own the code eventually. That's a process that needs to be done. Verification, porting guardrails, standards, etc.

So all of those issues, when they're introduced to thousands of developers that we asked them, do you think, like, actually AI helped to reduce those problems or actually made more, like, more challenging? 42 people reported that they spent 42 more of the development time on solving issues, on fixing bugs, etc.

And they saw 35% project delays. We're talking about, we're talking about maybe games. They're talking about, like, delays. Okay, there's some bias. We told them, we talked about problem with quality and what's the impact, etc. But that's what they present when they answer when they're talking about, like, when you're using AI code, AI-generated code.

And we see reports, some of the reports talking about 3x more security incidents. By the way, it makes sense. You remember we had a slide saying 3x more writing code? So 3x more security incidents. Like, the same amount of lines of code, the same amount of problems, correlation.

So what to do with that? Like, I talk about problems and problems and problems. Okay, help me deal with it. Like, let's spend a few minutes on that. So one suspect, of course, is testing. And actually, really interesting, we ask a couple of questions about testing.

And one really relevant thing is that people said that when they heavily use AI tool on testing, use AI to do testing, they actually double their trust in the AI-generated code, okay? That's one thing. The next suspect to help us with the quality is code review.

What's really interesting about code review is that it's a process that helps almost with all the process level and the code level, like, issues. For example, you can set your AI code review tool to tell you, block this PR if it doesn't cover a certain level of test coverage.

So through the PR, you take care of the testing process problem, okay? So code, like, code review with AI is actually one of the major things you can do. And people that are, developers that are using AI code review tool, they're saying that they're seeing double the quality gain.

And they're saying that actually it helps them to

improve 47% in productivity of writing code, okay? Now, a bit of statistics from our own AI code review tool. We scan a million of PRs a month, and we took 1 million of those PRs. And we noticed that 17% include, like, high-severity issues.

By the way, we're now analyzing before and after using AI. I don't have that statistics yet, but we're noticing since we're starting, most of the companies we serve, they use AI-generated code. So that's why I don't have before.

We need to go scan backwards. And that's, like, a really big number. Another thing I wanted to talk to you, like, about when you're trying to improve on quality is the foundation of having theright context that is brought to the code generation tool, that is brought to the AI code review tool.

Better context, better quality across the board, wherever you're using AI. So when we ask developers, when you don't trust AI-generated code, like, you remember, like, 67% said that, like, they're really worried about that. They said 80% of the time they don't trust the context that the LLM has, okay?

And when we asked developers, what would you like to be improved in your AI-generated code, in your AI code review tool, they said the number one was context. It was number one at 33%. They can choose, like, among many things to improve.

So context is extremely important. I can tell you that as Qodo, one of our technology moats is around context. And when you connect our context engine, we're seeing it as the number one tool that is being used. Like, 60% of code generator or code review tools, 60% of their calls to an MCP would be to a context MCP, okay?

And just to tell you, the context doesn't necessarily need to include only your code. It could also include context to your standards, to your best practices. We're seeing in our AI code review that 8% of the context usage is actually from files that are related to standards and best practices, etc.

Okay, I have to, CEO of Qodo, like, marketing will be mad at me if I don't brag a little bit,right? So this is kind of like our architecture of our context engine being presented by Jensen on GTC keynote.

And notice he didn't talk about our code review capabilities, about our testing capabilities. He talked about our context engine that Nvidia checked because there is a realization that AI quality, AI-generated, whatever review testing will come from bringing theright context.

So invest in that. You need to build your context by a solution and invest in it. Build your solution, etc. And the context needs to include code versioning, PR history, organization logs, etc. That's where all the context sits.

It's not just in the last branch of your codebase, okay? So I'm zooming out, starting to talk about, like, recommendations and, like, takeaways. So what's next? So automated quality gateways. Invest in that. People talked throughout the morning about parallel agents.

You know what I'm talking about? Like, background agents. You can use a lot of those, like, tools and capabilities to build your quality gates. Use intelligent code review, testing, and you need a living and breathing, like, documentation. And what documentation means is a story by itself.

I'm not going to double-click on that. And this is how I present for 3 years now, and I think I'm going to go all the way until the age of 60 with this slide of how I think the future of software development looks like, okay?

So basically, you have your specification and you have your code,right? And you have multiple agents, parallel agents that are helping you to improve your spec, write your spec, improve your code, transfer from your spec to your code, make tests which are executable specs,right?

And then you're going to have your context engine, the software development database, and you will build your tools, especially MCPs around quality and verification. And you'll make sure you have environments, stable, secured sandboxes where those agents can run and run validation and quality workflows.

So don't forget, like, the path forward is quality is your competitive edge over your competition. AI is a tool. It's not a solution, okay? And don't, like, only think about code generation as the only thing. Look on the entire SDLC or product development life cycle.

I saw one of the people talked about speakers, and it iterates with everything we talked about today. I want to tell you that you will gain value from it. We're seeing in the reports, people seeing, like, security vulnerability being reduced, faster code review.

We just got a hit on that because of AI-generated code. And test coverage in a month can triple, depends on the project, etc. With the last minute, I want to show, like, a really small piece of what you can do with Qodo.

You can go into Qodo and define your own rule. For example, almost the same rule you'll put in Cursor of, I don't like nested ifs, if this is a problem that you have. But then Qodo will look on your context, build the good example, the bad example, and then start giving, like, building a workflow that is specifically to catch that issue and give you statistics over time when it's being accepted and when not.

So you can adjust that rule and really know and have visibility to your standards, okay? So when a PR is written with a few ifs and else, although it was written with Cursor Copilot that had a rule do not do nested ifs, etc., then eventually when you open a PR, you will get Qodo catching that and giving a suggestion according to the good and the bad example.

Qodo will also make a graph, give you CLI checks, like, check each one of the rules and eventually tell you the nested if, and then will record and learn what you did or did not do with that suggestion in order to adapt the standard of the quality.

There will also be automated, like, suggestion. You don't need to write your own. It learns your standards and quality and offers that to you. And that's it. I'm really, really excited about, like, breaking the glass ceiling, okay, with what we did with code generation and then agentic code generation.

Now we're turning into the era of putting AI into work and through the entire SDLC. The most important part is related to quality. You would need to invest in that. It's not out of the box, okay? And then you would see eventually the promised 2x that probably promised to the CEO or something like that once they give you the budget for the relevant tools.

Thank you so much.