Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar

Alright, okay, um, sorry guys for the little hiccup. Okay, so my name is Prasenjit Sarkar, and today's session is all about: is our— or the LLMs, are they generating the code which is enterprise-ready,right? So let's look at the first slide.

In the first slide we're talking about Andrej Karpathy, who two months back said that a lot of things have been changed in the software development area. Earlier we used to write code in IDE, now things have been changed, now it's all about agentic.

So you're spinning up agents, you're just giving instructions in English, and English is now the new programming language, everybody's talking about that. And then you're letting it go, but, you know, humans are actually reviewing the code that is being generated by the agent.

So a lot of things have been changed. Earlier we used to start, you know, opening up by IDE, fancy IDEs like starting from VS Code or JetBrains to Cursor now, or Windsurf, or, you know, Antigravity. And now we are moving towards the agentic coding platforms, which is CodeX or Claude or Devin or Gemini CLI.

And according to the Pragmatic Engineer survey, which was done in March 2026, we have seen that 55% of the developers are now using regularly some of the AI agents,right? But the question is: do you trust the code that is being generated by these LLMs,right?

Is it maintainable? Is it secure? Is it readable? And those are the questions that we are going to, you know, debunkright now in this session. So let's look at why— let's look at, you know, by evaluating those models and what they are generating out of the box.

So there are two aspects that we are seeing. One aspect is all the LLM leaderboard, all the LLM leaders, all the LLM companies, they are saying that, okay, my pass rate is 80 plus percentage. 84 percent, 83 percent, 82 percent.

Those are, you know, eval coming from Human Eval, MVPP, SWE Bench. Those are fine. Those are the functional correctness on the test cases, which is mostly known for. But what we are missing is the security aspects, the real-world reliability aspect, the engineering architectural problems, the engineering discipline that you have.

So code maintainability and the tech debt that is going to be generated by the LLMs itself, and then the context-aware analysis. So these are things which are missing. Now, what Sonar has done: we have created an evaluation framework.

That eval framework ran through 4,444 plus distinct Java programming assignments. It's an open-source dataset. We took up, we took up the assignments, and then we ran through our models. Now, when we ran through the models, we saw a huge amount of, you know, data that is coming out of the analysis, and that is something that we have open-source— not open-source, I mean, we have actually, you know, put it into open world.

So we did the analysis using the SonarQube Enterprise, and we got the critical insights to choose theright LLM. Now let's look at theright LLMs, or probably not. Let's look at the LLMs that we have evaluated. So here I'm showing you just about the five LLMs.

And here you can see that Gemini 3.1 Pro high, the pass rate is coming from the SWE Bench. So you can see the 84.17 percent, but it is verbose. So those 4,444 Java assignments that I talked about, that to solve that problem, we have seen that it is creating 307,000 lines of code,right?

Which is pretty concise. It's not that bad. We have seen the complexity. Cyclomatic complexity is 234. It's really, really buggy as well, which is, you know, 614 bugs that we found out per million lines of code. And obviously we have the security issues per million lines of code, which is 210.

So you see that although these models are generating the code, although these models are pretty much pretty high, high-end models from the foundation models, but you see that, for an example, I, you know, Gemini 3 Pro is creating the highest— no, sorry, the Claude Sonnet 4.6 is creating the highest risk.

It's 300 security issues per million lines of code that we have seen,right? It is also high bloat. So for those issues, for those number of assignments, we see 627,000 lines of code which is being generated by, you know, Claude Sonnet 4.6.

And you'll be stunned if you look at the, you know, GPT 5.4 and GPT 5.4 Pro high model, you will see that 1.2 million lines of code being generated for those 4,000 plus Java assignments. That's a huge amount of lines of code that is creating,right?

That's a high bloat. Now, why it is happening? Well, you know, we have seen the mixed quality code. So the training sets that you see, the training sets actually have the mixed quality code coming from open source, coming from some other places, and that is actually creating the problem as well.

A little bit of problem. Then the built-in security flaws. So the dataset that you are using to train the model, that has inbuilt security flaws, and that we have seen where the model is picking up those insecure code examples along with the good examples as well.

Then there are hidden bugs in the data. So there are subtle logic errors that slip into the training pool, and that is actually causing your models to produce the code which fails or, you know, misbehave in a different way.

And of course the LLMs themselves,right? So LLMs are probabilistic,right? So obviously we know that the prompt that you are giving to one model today, tomorrow when you give the same prompt to the same model, it is not going to generate the same code.

It is going to code diff, it is going to create a different amount of code, a different set of code,right? It does have the limited context, which is obviously it doesn't understand the company's data or company's code base or company's architecture.

And obviously it is not explainable. So it is very hard to diagnose and improve when it is generating the code. So we created this leaderboard called sonar.com/leaderboard. Here we have given all the data about all the different models that we have evaluated.

So far we have 53 plus models, and all the diff, I mean, different versions. So you see Gemini 3 Pro high, Gemini 3 Pro, so different, you know, combination of the thinking aspect as well. So we evaluated 53 plus models, and we open-sourced all of the data to openly for the people to see that how the models are now behaving in a certain way.

So as of now you see the Gemini 3.1 Pro high, that's was, that was evaluated February 19th, and that has the highest pass rate which is 84.17. Not that bad of the issue density as well. And the lines of code, cyclomatic complexity and cognitive complexity is also fine.

It's not that bad. But yeah, this is the, this is the leaderboard that we have created where we are creating, we are evaluating all the different models that is coming up continuously, and then we are evaluating that and we are uploading the data.

So you can see not only that, when you go to each and every model inside, there are a lot more details that we have provided that what exactly are they doing so that you can take a concise decision about whether you are going to take this model or the other models, you know, according to your architecture.

So you see this, the key insight which is Gemini 3.1 Pro high is 84.17 correctness. That is the functional correctness I'm talking about. And that's a, you know, accuracy leader, but you have then other models which are, you know, five models that we have given which is crossing the 80 plus percentage of accuracy.

And these are kind of leaders that we have. So we talked about two different complexity. One is cognitive complexity, one is cyclomatic complexity. So cyclomatic complexity is how many branches do you have, like how many ifs and ifs and else, how many, you know, how many for loops, how many ifs and other loops, how many while loops that you have.

And the cognitive complexity is a Sonar proprietary one where we measure that how difficult a code is for a human being to read and understand and maintain that code,right? So these are the two different complexity that we maintain.

And if you look at the models and the kind of data, you will see that the amount of verbosity that we have seen. So the newer models that we are seeing coming up, you know, day by day, we are seeing the kind of lines of code is, you know, going to the north.

If you see the GPT 5.2 high, it has created actually a million lines of code for those 4,400 plus Java assignments,right? And if you see the earlier models like GPT 4.0, that's less than 250,000 lines of code, but the model which is going up north, the number of lines of code is, you know, being written is too high.

You have seen the models which are also going higher up that also have the higher complexity and higher cyclomatic and cognitive complexity. You also need to see that the number of total bugs per model, that is also going high.

But what we have seen is that the models which are getting matured enough day by day, they are getting kind of finer bugs, finer security issues rather than the old issues. So they are doing a good job in terms of the, you know, running the reinforcement learning, and they're securing the problems that they have seen already, but then doing that, they're also creating some more finer bugs that is very, very hard for a human being to detect.

We have seen the total vulnerabilities per model also is now decreasing, but then the amount, the kind of vulnerabilities that we have seen is, is going in a different, different genre. Okay, so yeah, we have seen that it's generating the kind of code that doesn't meet your, you know, engineering standards, but what can we do?

So in this slide we talked about the agentic-centric development cycle. We call it as ACDC. So Sonar has, that's a funny name, yeah. So it's called ACDC framework. So in the ACDC framework we have three stages. We have guide stage, we have verify stage, and we have solve phase.

So in this one we have an inner loop and we have an outer loop. So in the guide phase we have introduced two different products. One is called Sonar Context Augmentation and Sonar Sweep, which is in a private beta.

So Sonar Sweep is basically treating the data that you actually have and, and the data that you are actually using to train your model. So if you have the problematic data, that means that your model is going to create the problematic code.

If I treat the dataright there itself, the code which is going to be generated is going to be good enough,right? Context augmentation is going to push the context, all the entire code base into the LLM itself. Then we have the verify stage.

Verify stage is SonarQube. We have various different ways of, you know, utilizing that. We have introduced SonarQube Agentic Analysis which is in betaright now. Open beta, anybody can participate in that. Which is actually taking your code in the runtime.

What does it mean is that you're using a Claude or Codex or Gemini CLI or whatever, which have an MCP inbuilt, and then you can say that, hey, generate this code now. Before you commit that, before you push that into PR, you just need to analyze my code.

So it is going to analyze this code way before your CI runs,right? The CI runs takes about one to five minutes, and then this analysis is going to take about one to five seconds. Within one to five seconds, the code which is being generatedright now, before even commit, it will analyze and it will tell you that, oh, these are the problems that I found.

Fair enough. It will be pushed down to the agent, and agent is going to fix that problemright there before you even commit that. And then you commit, then you push that code back to the PR, and the PR analysis is going to run.

Now the solve part is where we have introduced the SonarQube remediation agent. So let's say that even doing after all of that, if there are issues that has been slipped through your verify stage and it has gone back to the PR stage,right?

So you have committed the code, you push a PR, and then you found out the SonarQube found out there are issues that we found, that, that's there, and your quality get fails. If that fails, then the remediation agent isright there.

It'sright now in its open beta where you can just click and then set that, okay, I want to fix all of the issues that is there in the PR. Not only that, let's say that you have a tech debt,right?

Huge amount of tech debt. So you go to the SonarQube dashboard and you see this, all those tech debts that you have, you just click and select all of the issues that you want to select and fix and then say that assign it to agent.

And we are going to create each and every PR per issue and we are going to fix that one, giving it back to the developers. Developers are going to review that. If they find, if they're happy, they're going to, you know, approve it and then merge it.

The beautiful thing that we have built for the remediation agent is that the remediation agent is going to create the fix, run it through the analysis again, run it through the compilation part again, and see whether that is creating any issues or not.

If there is an issues, it is going to discard it. We are not going to give you the code which is going to, you know, create a regression. We don't do that,right? So that's the kind of a verify loop that we run through.

So yeah, this is, this is the entire, you know, I mean, I would say the product,

this is our product portfolio where we are providing from guide and to verify and then solve. We have all this, you know, 40 plus programming language and framework. All the DevOps are, you know, supported, IDs, you know, we are partnering with this.

We have, we are in the marketplace as well. So yeah, that's, that's how we are solving the, the issues that you are seeing where the LLMs are generating the code, but we are not trusting that,right? Yeah, so if you want some more info, we are in the expo booth.

Come and visit us and maybe we can show you one or two demo as well for the product that we have built. Right, thank you.