Using LLMs to Secure Source Code — Eugene Yan, Anthropic

Hi Eugene, I'm member of technical staff at Anthropic. I've spent the last several months working with security teams and Claude to find and fix vulnerabilities in codes and systems, so I want to share with you what we've learned, as much as we can.

Before I get started, quick show of hands. How many of you here are security engineers in your day job? OK, OK. How many of you here are engineers in your day job? Oh, sweet. OK, great. I have a better understanding of how to give this talk now.

Security engineers in the room, please don't boo me if I oversimplify things. I want to get this message out to as many people as I can. So first, I want to share with you three high-level trends that we are seeing in terms of model capabilities: number of vulnerabilities found, and where the bottleneck is shifting to.

Then I want to distill the lessons we've learned from working with dozens of organizations to improve their security posture, as well as the bottleneck that you might encounter and how you can try to address it and prep for it.

And finally, I want to share how you can get started this week with links to resources. So why do we care so much about cybersecurity now? Time horizon benchmarks track the length of time an AI model can complete a task measured against an actual human doing the task.

So we all know about the METAR eval task, how much knowledge an engine can do. The UK AI Security Institute has a cybersecurity version of this. These tasks require identifying and exploiting security weaknesses in target systems and testing skills like reverse engineering and web exploitation.

So in this chart, we see that models are able to increasingly do longer cybersecurity tasks. But what's also interesting is that you kind of see a step jump against the previous regression line. So these models are a step improvement in capability.

And what does this step improvement in capability buy us? Recently, Mozilla Firefox published a number of security bug fixes they made each month. The average in 2025 was about 2020-ish. And then you can see in February and March, it kind of 3x'd to about 60 and 70.

And then in April, it 7x'd to 400. So what this means is that what's happened in April is 20x of last year's average.

They attributed about 2/3 of this to METAR's preview, about 271, which shows that frontier models can help defenders like yourself find and fix vulnerabilities at scale.

What are some notable vulnerabilities? Hands up. Who here remembers Log4Shell? Anyone? Anyone was here over the December holidays scrambling to patch this stuff too? So Log4Shell is a bug in the Java logging library. If an attacker sends a string, you log it.

The attacker can run code on your system. The Belgium Defense Ministry was breached within days, and a fintech platform leaked 2 million users' data. And then before that, we had Heartbleed, which is a bug in OpenSSL. So much of the internet runs on OpenSSL, and we all know what impact it had.

This is Anthropic's sharing of our own work, scanning more than 1,000 open source repos. From 23,000 candidates, 6,200 of them were rated as high or critical. And at the time of the update, 1,600 of them were reported to maintainers and about 100 patched upstream.

We shared our observation that finding vulnerabilities now is quite straightforward. The bottleneck has now shifted to verification, triage, and patching. And I want to share with you everything we've learned about that so you can get ahead of it.

So how do we do this in our own systems and code bases? Two words: agentic harnesses. I'm citing Mozilla again, and this was even before they worked with METAR's preview. I'm going to read off the screen here. Early experiments showed that some promise, but the high rates of false positives made it impractical to scale.

But the introduction of agentic harnesses that can reliably detect security issues has changed this. These models and harnesses together can find real bugs and dismiss non-reproducible speculation. So the harness and the model working together can get a lot of work done.

We found that most teams converge in approximately these six steps. And a big chunk of my talk is going to be about these six steps. The first two steps are set up. You do a bit of upfront investment per code base.

How many of you here know what a threat model is? Amazing. I love that. A threat model is some kind of system design, but for your code base or system, what are the threat vectors? So this helps you decide what are the vulnerabilities in your system that you care about.

And then we also want an isolated sandbox where you let your model run possibly untrusted code, or you let your model deliberately detonate proof of concepts of exploits, I'm going to call them POCs, to confirm that they are true positives.

And then the next four are a loop where we have discovery that identifies vulnerabilities, verification confirms that they are real. We have triage. You can't give a developer 100 vulnerabilities. You've got to identify the top 10 or 20 critical and high so they can focus their attention.

And then you can patch the bugs. So now I'm going to talk about each step. Now, to make things concrete, I want to share with you how they apply on a hypothetical system. So this is order service. Order service, you type in an ID, and then you look up an order.

So what kind of vulnerabilities might we find in this? The first step is the threat model. So why does this matter? So several teams, if you look at the code, have found that having a well-documented threat model really increases your true positive rate to 90%.

90% is outstanding, if you ask me. I think anything above 75 is great. That's where you want to aim for. Now, to put it another way, one CISO said this: the model has great context of the code, but poor context of the system.

And why does this happen? It's whatever the model can read, the model will understand well. But there's so much information that the model cannot read. Why did the senior designer design the system this way? What are all the vulnerabilities that on-call has been fixing that has not been written down?

All of this is now implicit context in your head, but you need to write it down somewhere, maybe in a threat model, to help the model be a better security engineer. So we can work with models to build these threat models.

And the threat model is really just a document. First, we can bootstrap a threat model, give the model access to all your documents and code, even past commits, past patches that were made, and what CVEs they were patching.

And you can ask the model to infer, hey, what other CVEs have not been patched? So the model can do that. Then after that, the next step is that you should ask the model to interview the expert on the system.

What are some things that could happen that we didn't plan for? Or what are some things that we actually don't need to worry about for vulnerability, because maybe this is an internal VPN-only app, or there's a compensating control elsewhere?

A lot of this is not in the code. If you look at your code base, it's not there. But it needs to be written down somewhere so the model can benefit from it. So here's an example threat model of order service.

The key asset is the data. It contains customer PII. And the entry point is the orders API. We can also see that the model has suggested some possible threat vectors, like SQL injection, and the fact that you could call this API without any form of authentication.

Then the second step is the sandbox. So isolation and reproducibility. In some cases, we may be running untrusted code. In some cases, we may be asking the model to deliberately create exploits. So we want some isolation to prevent the model from doing bad things.

So this isolation, doing bad things, could be data exfiltration or dropping things in production. And of course, you want all of this to be running in a VM without egress and, of course, without your cloud credentials. The second thing we want is reproducibility.

So a lot of times, you may find that the model may find a bug because you were using an old dependency. Or you want all of your agents to be starting from the same baseline container so that your agents can run everything in a harness in the pipeline.

So now, building a representative sandbox of your system is actually quite difficult, but the capability is quite useful. I think here's one team we worked with. They quoted, "The biggest lever we had is having the model test beds, essentially sandboxes, with live systems and where they can run and detonate the POCs to confirm that they are true positives."

So how would a sandbox for order service look like? Well, this is a simple sandbox, three Docker images connected together, one for the app, one for the Postgres database, and one for the Redis cache. And on theright, we have the security agent, which sits outside a target boundary trying to probe the app via HTTP.

Now that we have the setup done, we can now move on to doing the work of discovery, finding the vulnerabilities. So the first is, so what are some there are three things that matter a lot. And there's no surprises here.

The first thing is really context engineering. You're going to give the model all the context, as much context as you can. If it's written down and if you share it with the model, the model can find it. The model will be a better security engineer.

You will improve your vulnerability coverage. You'll reduce your false positive rate. The second is to have simpler prompts. How many of you here have found that as models get better, you actually need to remove from your prompts? That's what I found.

With every new model version or step jump, I actually have to cut my prompt size by maybe about 50%. So as models get better, your prompts need to be less and less specific, prescriptive even. While previous models, you might ask the model to look up for specific category classes.

But for newer models, you can just probably say something like, "Look for where untrusted data hits the trust boundary." And the model is very good at inferring this. Finally, give the model tools. So one example, we had a pentesting team.

They gave the model tools to check the response, to query the API, check the response, read the logs, read the source code. Now, this is just like the tools that you give a regular security engineer. A lot of times, you expect the model just read the code.

That doesn't quite work. Allow the model to be dynamic, to read the tools, to run the tools. And when they did this, their true positive rate was almost 100%, because the model could actually verify in the loop. So here is the GET API for the order service, and it's just five lines of code.

Can anyone here spot the vulnerability? You can shout it out.

SQL injection.

Exactly. So this is what the discovery agent flagged. In line four, we built a SQL query via Python string interpolation. So the user input flows straight to the SQL query. And that means that someone can inject a SQL query that exfiltrates data from your database.

This is code that I would have written. Me as a non-security engineer, I'd just write the simplest code possible, and this is how it would look like. Current models are very good at spotting simple stuff like this. So after discovery, so we have maybe 1,000, let's say 100 vulnerabilities.

We may have not been able to confirm them. The next step is verification. So if you think about machine learning, discovery optimizes for recall, find as many vulnerabilities as possible. Verification optimizes for precision. Of the 100 vulnerabilities we found, how many of them are truly exploitable?

So we have them both working separately. So we found that it's helpful to have these two steps separately, because when the discovery agent is trying to verify its own work in the loop, trying to debate against itself in the loop, it may actually self-censor.

And this may actually hurt recall. And when building the verification agent, it's helpful for the verification agent to be independent and adversarial. Independent means that the verification agent doesn't see the reasoning traces, doesn't see all the work that the discovery agent has done.

All the verification agent gets is the vulnerability. And it approaches it from an adversarial point of view. Assume that this vulnerability is false. Try to confirm it's false or confirm it's true. So this sets a very high bar for the vulnerability, which reduces the false positive rate.

And also, if you have invested the time in building a representative sandbox, have the verification agent build the POC, detonate the POC to confirm if the vulnerability is true. So in this case, back to order service, our verification agent is confirming the SQL injection.

In a fresh container, and this is why we have everything in containers, the verification agent is only given line number four. Here's where a SQL injection could happen. And the verification agent actually tries, just runs the curl command, and the verification agent can see all the customer PII that has been exfiltrated.

So now you have confirmation that a SQL injection can work. So after discovery and verification, we've maximized recall, we've optimized for precision. The next step is triage. So a lot of this may truly fire, but some of these are bug correctness issues.

They have very low business impact. So across multiple teams, they shared that you can lose trust with product engineers by sending them all the vulnerabilities that are true, even those that are medium or low severity, because those engineers can't cope.

And as we've seen so many times, the scarce resource now is engineer attention. So imagine you're a product engineer. What are the kinds of vulnerabilities you want to actually receive and fix? So as part of this step, we can do deduplication, cheap heuristic checks.

Or we can get a model to help you with clustering and deduplication. We should also consider the severity of the exploit. One thing to consider is the magnitude of the impact. What is the magnitude? Is it a ring zero magnitude or ring one magnitude?

And also the likelihood of it occurring. How many hoops does an attacker have to jump through to actually make that exploit? And this is where the threat model is useful throughout. You can think of the threat model as just synonymous with context.

Just give the model context about your business. So for example, compensating controls. If you have a firewall, something that was initially high severity could actually be low. On the other hand, if you know that your database is mostly customer PII data or health care data, a model might have set that as medium severity.

But it actually could be high, because the contents of your data are so sensitive. So in this case, we have on the left, triage that the agent has done. The agent has said that this is high severity for these reasons, high impact, high likelihood.

Now, on theright, after human review, the operator has confirmed that, has downgraded this to low likelihood. And the reason why it's low likelihood is because we have an application firewall that blocks SQL injections. And also, there's the fact that the service is internal only.

It's only for internal warehousing or internal logistics. It's never internet-facing. Therefore, the likelihood is low. Now, you can imagine, how would the model know this? This is not written in the code base. It may be available in a system design.

It may be available in your documentation. It may be available in your Confluence, et cetera. So this is the kind of context you need to provide to the model so that the model can get better at this.

Now, after that, we close the loop, patching. To validate the patch, we can build a patch, and then we want to validate. There's a ladder of sorts. First, the original POC has to stop working. That's basic. Second, the existing test suite should stay green, no regression.

And then we can go further. We can have a fresh discovery agent try to attack the patch code again. Is the patch comprehensive enough? And teams have found that by giving the patching agent such feedback, the patch quality improves greatly.

And this is the generative verifier loop. Give your agents context. Give your agents feedback, and they can do their work. And finally, a human confirms the patch before it's been merged. So here's how the patch would look like.

In the first diff, we update the code. It's a one-line fix where we move the variable out of Python. And therefore, we reduce the likelihood of SQL injection. And in the second diff, not only do we update the code, not only do we fix the system, we close the loop.

We make sure that the next time we run this loop, we run this scan, these issues don't happen again. And this is really important when you're building harnesses. You're building loops. They're operational expense. But when you close the loop, they now become capital expense.

You get better with each iteration you run. So you make clear the compensating controls we have, the application firewall, and the fact that it's internal only. OK, so a quick recap. These are the six steps. And each step focuses on one thing.

Threat model provides context. Sandbox provides isolation and reproducibility. This is the baseline fundamentals that you need to have. This will carry throughout every scan you have: context and reproducibility. Then after that, the rest is really, if you look at it, if you squint, it actually looks like an ML pipeline: recall, precision, ranking, and closing the loop.

It's a regular machine learning pipeline.

Now, that's the technical challenge we have. There's another technical challenge we have. And as my previous director would say, non-technical problems are an order of magnitude harder than technical problems. So what are some other bottlenecks you might have?

So imagine you have a process. What happens when you have one or two orders of magnitude more input? What is going to start to break? In the case of the harness, the scanning harness, you just have more engineering.

You spend more compute. You pay more money. Things that can be solved with money are not really problems. But human attention doesn't scale. Your dev, your product engineers, and your security engineers, what if they don't agree on what high severity or critical severity is?

You need to put them in a room. You need to write out all these rules so that everyone agrees. And your threat models, they're all stuck in people's headsright now. Someone needs they need to be interviewed, maybe by a model or by someone else, and write all their threat model down so that now you can have agents help you.

And finally, patching, patch review. I still don't know of many companies that have gone through fully automated patch review, especially when patching security issues. So here are some organizational bottlenecks that you might hit and what you can do.

The first thing is vulnerability routing. So if you maybe get a dozen vulnerabilities a month, you can probably curate them, email them to specific teams, assign Jira tickets. When you have hundreds, you can't do this. And this can really be as simple as sending it to the code owner, et cetera, or the service owner.

And it doesn't even need an LLM in the loop. Then the second thing is severity calibration. The one hard thing we found is that product engineers and security engineers, red team, blue team, they may not agree on what high severity is.

And all this requires a lot of your business context. I really think it helps to have everyone discuss and write it all down once and for all, agree upon that, that you can send that to an agent. And finally, the patching bandwidth.

I'm sure no one writes very few of you here write code by hand now. But it's also very challenging for you to just be given the vulnerability and try to implement a patch, even with LLM assistance. You definitely want to be moving towards AI-generated patches and then have the human in the loop verify it.

And generate the patch, reattack the patch. OK, if you only remember one thing, well, here are the three things I want you to remember. I would like you to start now. Please, start with open source dependencies. Start with whatever you're comfortable with.

Two, I want to teach you how to climb the learning curve. Don't try to aim for automation immediately. Start interactively. Do it hands on the wheel with Claude Code or your favorite IDE, doesn't matter. Learn where you get cut.

Learn what kind of context you're missing. Learn where precision is low. Then third is don't just aim for scanning. Scanning is not the bottleneck. Verification, triage, and patching, and then all your organizational processes, those are the bottleneck. Over there, I have a couple of resources.

We have Claude Security to scan your code base, validate findings, suggest patches. We have a blog. We have open source repos that include the interactive skills and autonomous harnesses. And all you have to do is to run these few steps to very quickly get hands on the wheel.

And in it, if you look at step five, you have the harness, and you can just customize it. That's all I had. Thank you.