In the Land of AI Agents, the Verifiers Are King — Tariq Shaukat, Sonar

Please join me in welcoming the Chief Executive Officer at Sonar, Tariq Shaukat.

Morning, everyone. Did you enjoy that last talk? That was amazing. I particularly loved the end, the "being unreasonable" part; I thought that was awesome. I also want to just— I'm trying to calculate the odds of Tariq following Tariq as the first two sessions in the morning.

I think the odds are pretty low on this one. But thrilled to be here today. As was just mentioned, I am with Sonar, we are in the code verification space, and I'm here today to talk about verification. And I think we're all here, in large part because we believe, to some extent, that AGI is here, it's coming.

The models, we just heard about Fable, it's really incredible what is going on in the world today. And yet we work almost exclusively with enterprises around the world, and the conversation that we have more is the question mark version.

Is AGI here? And why are they asking these questions? It's because you can read the news every day, and I'm not trying to name and shame here, but if you look at KPMG putting out reports that they have to retract because of hallucinations, EY doing the same thing, law firms getting into lots and lots of trouble because of made-up citations, made-up case law, things like this, I think we can really start to question: how do we get value out of AI?

The models are amazing, as we just heard, but the hard part, as the other Tariq just said, is getting value out of it. The struggle is that AI slop is everywhere. I'm sure you all see this inside of your organizations.

I'm sure you see this in your everyday life, that AI is amazing. The models are incredible at generating very plausible output. They're incredible at generating things that sound correct, but are they correct, and how do you know that they're correct, is a big problem.

And it's a big problem in professional services, as we saw. It's a big problem in legal. But really, I think, if we're honest, it's a big problem in every sector, in every field, whether it's marketing or finance or you name it.

You have this question of how do you actually know if it's true? How do you know if it's good or if it is slop? And the question that we deal in the coding space in particular, we deal with software development, and the question we get as we talk to, I'm sure many of the people here in the room and a lot of our customers is: isn't software development different?

And we can look at the data on this. And the Mythos models, this is data from Meter, you may have seen this, METR, the coding agents are getting better very quickly. They're getting a lot better very quickly. And you can see the progression, the exponential curve here.

What this shows on this chart is: how capable are the models at completing tasks that humans would take? So can they complete a task that takes 1 hour, 2 hours, whatever it is? The latest Mythos model, at least per the benchmarking which was done a month or so ago in the preview mode, was you're getting to 16 to 18 hours.

So they're actually able, the agents are able to complete long-running tasks. And it really is starting to transform how work is happening. But the critical caveat, when you read the data, is: this is at a 50% success rate.

Okay? So it is, again, able to complete tasks, but is it able to complete tasks correctly is the question. So if you start looking at, let's dial up the accuracy,right, you dial it up to 80%, and there's still progress, but it is much slower progress.

Instead of 18 hours, you're at about 3 and a half hours or something along these lines. And by the way, this is still at 80% accuracy. And as I was presenting this to the CTO of one of my large customers, his response was, "But Tariq, I would still put someone who gave me 80% accurate information on a performance review, probably,right?

This isn't necessarily enterprise-grade." The problem is that the models themselves, and full disclosure, we have not yet done this benchmarking on the Fable models, obviously, because they are just being released. But as you look at the models, the models are getting smarter, but they still produce a lot of problematic code.

This is benchmarking that we do. We give the models a series of over 4,000 problems, and we basically ask it to generate the response to the problems, and then we analyze both the functional correctness, which is critical, and they all do extremely well on this notion of functional correctness,right?

But then we look at how complex is the code, how buggy is the code, how secure is the code. And what you see with even the state-of-the-art models is that complexity is still high. It's actually quite variable, as you can see here.

GPT-5.5 is done particularly well on the complexity side of things. It still generates bugs. It doesn't generate massive amounts of bugs, but it still generates bugs, and it still generates security issues. So this is the output of the models that are going into the agentic workflows.

And again, this is not, you know, I'm at the AI engineer conference, this is not me saying AI is fake or incorrect, but it is trying to address this question of how do you really get value in a production setting out of AI.

This is a study that was done in Carnegie Mellon University, and it looked at what is the actual productivity benefit that you see from the use of AI coding agents. And what you see, I think, really resonates with a lot of what I see firsthand in the market, which is: you have an initial, just amazing boost of productivity, of velocity in particular.

What you see is a 3 to 5x boost in productivity or in velocity. That dissipates in 3 months. At the end of 3 months, it starts to come back to the normal before you were using the agents. And if you ask why, it is because of the two pieces in red here, that you start to see there's an increase in velocity, but there's an increase in security issues, there's an increase in maintainability issues, there's an increase in reliability issues, and there's an increase in complexity.

So essentially, you're building the technical debt as quickly as you are generating the code, or maybe even more quickly, and that creates a different set of work. It creates a different bottleneck. And so to us, this is now the critical question in AI, which is: in a world in which code is provable, and there are sessions that I'm actually very much looking forward to attending about formal methods and proofs and things like this.

Code is provable. But when you start dealing with large codebases, software is not. It's still very complex. It is still very messy. There's lots of dependencies. There's lots of technical debt already in most codebases. And so this question of verification is actually key.

And what I'm going to be arguing is that you can treat verification as an afterthought, or you can bake verification into the process. And if you bake it into the process of generating code, of doing software development, you can actually start to get materially better outcomes from the coding agents than if you view it as an afterthought, if you view it as just the old-school code review.

So as we've been thinking through this, we basically have constructed a framework, and there's lots of competing frameworks around this, but I'll just talk you through ours. We call it the Agent-Centric Development Cycle. For shorthand, we call it AC/DC sometimes.

And the idea here is: how do you get verification powered agentic loops? At the center, there's a lot of focus on the code generation piece. Like, how do you actually get the models and the agents to generate the code that you need to solve the problem?

And what we argue is that you should surround this with theright disciplines, theright tools, theright processes to do three things: to guide the agents, and Tariq was talking a lot about different aspects of this, actually, guide the agents, verify the outcomes, and then solve the problems.

And you have to make this part of the discipline, part of the process, part of the new software development life cycle if you want to be successful in the AI world. So if I double-click on some of these pieces, what do we mean by guide?

We've done a lot of experimenting around Guide. We've just launched a product yesterday, I think, called Sonar Vortex that starts to get into this area. What we find is critically important is to think about Guide as context and constraints.

And we separate out context and constraints very deliberately, because context is: you have your code repositories. How do we make it easier for the agents to understand, for the models to understand, what is in your codebase? If you have a million lines of code, if you have 100 million lines of code, if you have a billion lines of code, the agents work better if they understand your codebase.

So how do you give it architectural awareness? How do you provide semantic navigation, maps, and help them understand the territory, to borrow what Tariq was just talking about? And we find it equally valuable, and I don't think this part is talked enough about, to provide the constraints as well.

You have guidelines that you want your code to follow. You have dependencies you are okay using. You have dependencies that you are not okay having. You have coding standards. You have guardrails. You have intended architecture. We spend a lot of time talking about existing architecture, but what about where you want to go?

And so this idea of context and constraints, we've found in our testing, generates a massive improvement in agent effectiveness and a massive improvement in token consumption, over 30% reduction in tokens being used to solve a given problem. And if you ask why, it's because you're actually making the life of the agent easier.

You're helping it navigate better. So then we get into the heart of this, and we really think of Guide as preemptive verification. How do you make sure there's less to verify, less to fix, this sort of thing? Then you get to the heart of verification.

And what we believe quite strongly, and what we've seen work in practice, is this idea of Zero Trust, multi-layered verification. Zero Trust: every model has biases, every model produces, has a character, has a personality. So let's make sure we use different models and different techniques to make sure your code is safe, to make sure it's reliable, to make sure it's secure.

And multi-layered really speaks to the earlier point that software is complex. Software is very messy. Software has lots of intricacies involved with it. And so what we believe, and again, have found to be quite impactful here, is that a combination of algorithmic verification, looking at things like data flows, control flows, known patterns, secrets, these areas, combined with what is now possible with agentic verification, looking at intent, business logic, the unknown unknowns, actually, again, to borrow from the last presentation, the fusion of these things, the deliberate multi-layered fabric that you put in place, can actually, you can see the results of this in production.

So as we look at our partners and customers who use a multi-layered verification approach, they are reporting AI-derived production outages being 44% less frequent than the ones who do not. So you can start seeing a material improvement in reliability, in security, and in maintainability.

And then the last point I mentioned is: technical debt does explode,right? As you generate code, technical debt is also generated. And again, this is not stop doing it. This is be aware and let's start controlling it. And so what we have seen be super effective is to have an active process, to have an active discipline, again, around code maintenance and thinking about how you do verified code maintenance.

I won't walk through every step of this, but the agents, whether that is a set of remediation agents, whether it's a strong discipline around verification, does keep your codebase clean. And a lot of people have asked me, "Allright, but do agents care about clean code?

Human developers care about clean code. Do agents care about clean code?" And what we find, again, is they absolutely do, because the agents have to understand the codebase if they're going to operate on it. So this is a one-shot view.

We think this is something that compounds, but if you just do the exact same agentic tasks on a typical codebase and then one that has been cleaned, you see a material reduction in the amount of tokens, reasoning energy, et cetera, needed for those cleaner codebases versus the typical codebases,right?

If you make the life of the agent easier, if you maintain your codebase, then you'll actually see compounding effects. Now, the important thing in our mind is to construct the system. This is how I started, is saying, you know, I'm sure all of us do code reviews.

You may use static analysis tools. You may use AI code review tools, a whole range of things. And we believe that you have to put this in a system. And again, we're happy to, in our booth downstairs, talk through what this looks like, but we really believe that the construction of the software development life cycle in an AI world needs to embed this notion of Guide, Verify, and Solve inside of it.

And you need to do it in three loops, and you need to think about these three loops. There's the agentic loop, which I think is the key buzzword of the conference now, but how do you provide the agents as it's generating the code, as it's doing the work, with the context and constraints, with the in-loop verification so that the agent is getting verification as it's working?

And how do you fix problems? That's the blue loop here. What we talk about is the inner loop verification piece. There's the second, which is your continuous improvement process. And how do you really combine the power of algorithmic and agentic to generate your pull requests, review the code?

And by the way, the velocity of this has to go up massively. So to review the code using agents and to do this multi-layered verification, and then you have your evals. And I think the opening speaker talked about how evals may be the buzzword of the conference, but you have your evals and you have your quality gates to check: are you actually passing?

So you have your code maintenance loop, agentic loop, CI verification loop, and deliberate design of these loops with verification at the center is a compounding system. It's a system that reinforces itself, and it reinforces itself in the positive, and it reinforces itself in the negative.

And we've seen customers who have kind of neglected, as they've rolled out AI coding tools, they've neglected verification. They've neglected this idea of code quality, of code maintenance, things like that. And you get into a downward spiral pretty quickly.

This is what the Carnegie Mellon case study, or study, actually shows, is that you actually have all the benefits start to dissipate. Or you can get into this self-reinforcing loop. And one of the tests we did with one of the large banks who are using some of the cutting-edge, the folks who are all around here today, cutting-edge agentic coding tools, they can get a 92% reduction in issues if you actually take this Guide, Verify, Solve approach inside of those agentic loops.

Again, this compounds. It's not that each loop is 92% better. It's that as you go through solving the problem over minutes and hours, that you actually see compounding benefits. So that is essentially how we see the benefit here, how we see the controlled value-creating use of AI in enterprise settings.

And when I say enterprises, people with existing codebases, people with, you know, millions of lines of code already. There's the agentic loop. There's a CI verification loop. There's the code maintenance loop. I am required by my marketing team to put up a version of this that has our products on here.

So these are our products, and you can come and see us later. But the most important thing is really to say, our recommendation is this agent, the AC/DC, the Agent-Centric Development Cycle. The core part is deliberate verification built into the system.

So if you'd like to learn more, we have a booth. It's the big red booth downstairs. We'd love to talk more. We have some double-click sessions coming up, so please do join those. And have a great conference. Thank you all.