Agentic Development Security — Ezra Tanzer, Snyk

I'm a product director here at Snyk, and I'm going to be talking to you about agentic development security, and specifically talking about how we can gain confidence when we use agents, especially as we give them more autonomy. It's a very common theme we've heard in this track and a number of the other tracks today.

I'm not going to go through the full history of LLMs, but the model context protocol release was a really big moment. Until then, I don't know what you guys were doing, but I was very often copying and pasting between agentic clients and some other services.

And with MCP, I think people really started to connect this and have a much more really connected AI system. And I'm not saying that MCP is the end-all, be-all, and I may or may not have been amongst the people who were saying that MCP would die at some point last year.

But it has been a game changer in the sense that developers started to connect agents to external tools and services. And at that time, there really wasn't any security to speak of. Like most companies, we released an MCP server almost immediately.

Ours specifically enabled local directories to be scanned by our security scanning engines. Developers could ask questions in natural language about the security issues that were identified. They could learn why specific vulnerabilities were important or how they might be exploited, and then work iteratively towards a fix.

Shortly thereafter, we decided to pair our MCP server with rules. And the rules basically ensured that any AI-generated code would be tested, and if there were security issues identified, that they would be automatically fixed. It was simple, it was fast to deploy, and it did solve a meaningful pain point for our customers.

So that really was our original position: secure agent-generated code at the moment of inception. Oops. But over the last year, we learned that this framing was really incomplete. Our customers started telling us that they were not only worried about the code that was being generated, they were also worried about what the agent had access to, and then also the actions the agent might be taking.

So I'm going to just mention briefly a few incidents that have come up over the last year or so. I think we've talked about them in the keynote that Minoj gave earlier today, but also I think we've seen some of these in other presentations.

But just as a quick refresher, about a year ago, we saw Replit's agent ignore a code freeze instruction and ultimately deleted a production database. It tried to cover up that it did this, fabricated records to basically say, like, no, there was no issue whatsoever.

And finally, it said that there was no way to recover. Fortunately, it turned out that that was wrong. They were able to recover, but the damage was still done. Then in April, I know we talked about this just a couple hours ago, but there was the Pocket OS incident.

An agent, again, found an over-privileged API token, and that resulted in a production database being deleted. The backups were also deleted, and so a three-month-old backup could be used to ultimately try to get back to recovery. What was really interesting here is that the agent wasn't acting maliciously.

It was actually trying to solve a problem. It was trying to solve what it perceived to be a credential mismatch, but there was nothing in place to stop it. Those two examples were really about the agent actions that might be taken, but that's not always the case.

That's not always what the attack surface is. Just last month, Team PCP was able to exfiltrate almost 4,000 of GitHub's internal repositories using a malicious VS Code extension. So all of this, and kind of us being in the security space for the last 10 years and talking to our customers, it's really shaped how we think about agentic development security and what that really means.

And our belief is that in order to confidently use agents for software development at scale and to start letting them operate more autonomously in long-running tasks, whether it's just getting up to make a cup of coffee or letting them run overnight, it's really critical to secure what agents generate, what they use, and what they do.

And I'll spend a couple minutes talking about our journey in each of these pillars over the last year, what we've learned, and our current perspective. As I mentioned at the top, this has been our longest area of experimentation and investment.

It's securing the code that the agents generate. And the reason for that is we don't want issues to make it to production. We don't want to kind of increase that backlog, which has been so challenging to manage and is now a luxury that companies just cannot afford.

Most companies do have security checks in their deployment pipelines. And so even if they don't make it to production, we want to ensure that bottlenecks are not getting created at those stages. I mentioned our original approach, MCP server plus rules.

It was really easy to paste an MCP configuration and a rule definition. And over time, we added shortcuts to make that even easier, and the agent clients actually made simple commands to enable these configurations through plugins or just simple CLI tools.

But the approach did have real limitations. Agents sometimes ignored the rule files. Scan execution did add latency at the end of its run. And every time that we ran scans through the context window, that consumed tokens. And so we were not the only ones dealing with these challenges, fortunately.

Outside of security, these same pain points existed. And the good news is that the agent client providers, they provided new mechanisms for how to direct agents. Primarily, this has been in the form of skills and hooks. I think everybody might have their own opinion on which one you want to use for which, but they really solve a lot of these problems.

Our current recommendation is to use Python-based hooks for this use case that can fire asynchronously on agent tool calls. And so immediately after an agent writes a new file or modifies a file, we can kick off a scan using our CLI, not even using the MCP server, asynchronously.

And that will write any newly identified or newly introduced issues to a temporary file. And finally, on the session stop event, that's when a hook triggers the agent, and it will check that temp file to see if there were newly introduced issues, and only then will it kick off a fix-and-validate loop.

So now the workflow is deterministic. Latency is removed because all that testing happens asynchronously. And again, because newly introduced issues are the only thing that's being surfaced to that agent context, the context window doesn't get any unnecessary bloat.

When we talk about agent supply chain, we're really thinking about the things that help you build more connected agentic workflows. But like everyone's been saying today, this also presents a new attack surface. Last year, we acquired a company called Invariant Labs.

And following that, we produced a report, which you can access if you want with this QR code. You can also come and talk to us at our booth if you want to see this here. But there's many similarities between package ecosystem risk, which is where kind of Snyk got its start, and that of agent skills.

But we really think that skills are more problematic. They have higher privilege by default. Natural language prompt injection cannot be detected through typical code detection. And malicious skills can modify agent memory. So even if you remove a malicious skill, they can still persist.

That risk can still persist after the fact. And in an audit that we did of nearly 4,000 skills on Claw Hub, over one in eight had a critical severity issue, and we actually found 76 malicious payloads in that subset.

So the solution that we built here auto-discovers all the agent components on your machine. If you have MCP servers configured, it will connect to them, retrieve the tool descriptions, analyze them to see what security risks are present. It will do the same thing for skill files.

It will look at your skill.md file, see the dependent files that are associated with that skill, and look to see what threats might exist. I tried to demo this during Minoj's keynote earlier. The internet didn't play super nice, so we showed a little video, but we can show this in action afterwards as well.

Just in the last month, we produced a report based on some anonymized data following the release of the capabilities that I just mentioned. And I think these numbers are probably going to be pretty low based on the expectations of the group that we see here.

But I think it's worth reminding you that not everyone is as kind of AI-forward as the folks that are attending this conference here. But kind of from the average developer, we saw that more than half were using MCP servers, and a fifth were leveraging skills.

Beyond just adoption, 1 in 12 developers in this group had an MCP server where there was either a high or critical severity finding identified in that MCP server itself.

I realize this is probably pretty small, but just want to highlight that these are some of the categories of risk that we look for when we analyze MCP servers. And similarly for skills, there's a number of different risks that we can also look for.

This happens to be one of the skills that I did an assessment of during that recorded demo that we showed earlier. Very, very risky. Malicious in some cases, but also just kind of maybe some negligent behavior that could cause problems for me.

The last leg of this tool for agentic development security is governing agent behavior. This is currently in open preview, and it's really focused on how we ensure that an agent is not taking exfiltrative, destructive, or otherwise malicious or risky actions.

I'm choosing to show here the policy configuration view because I want to highlight the actions that we are trying to intercept today. Depending on how you use agents, the types of policies that you set might ultimately be different.

I think in an ideal world, you're always able to steer an agent towards theright action and kind of making it so the human does not have to be in the loop. A really good example of that is redacting PII or secrets before a command executes.

But in other cases, you actually may want that agent to ask you because there's not a clear-cut answer. And so if there is a potentially destructive shell command, or maybe if the agent wants to access a directory that is kind of outside of the scope of the permissions you initially gave, that's a good reason to ask.

But I think as we move towards more background agents and cloud agents being ran, where you're kind of trying to step away and trying to not be sitting at your desk babysitting the agent entirely, asks are a much less viable option.

And so this is an area that we're going to continue to invest. But I think this is going to mean more fine-grained policies. It may also mean that we need some sort of autodidactism in the product, kind of a self-learning capability based on the decisions that you make over time to help you become more autonomous.

But the short of it is that today, we are accountable for the actions that our agents take. And even in the future, if that accountability model becomes a little bit more shared, I don't think any of us want to work at companies where we are doing a disservice to our customers and potentially letting our agents take risky actions.

And nobody really wants to be the reason for one of those incidents that gets widely publicized to occur.

So far, I've been doing a lot of telling rather than showing. And I think for this conference, we really want to focus more on showing where we can. So I'm going to invite my colleague, Dan Arpino, to come up and take over and really show you some of the areas that we're exploring for solving these problems.

I want to be clear that what we're showing is not a committed roadmap, things that are definitely going to be available here. But we really want to show it to you so that you can give some feedback to us on, like, are we heading in theright direction?

Is this the better way to solve these problems? And hopefully, this will ultimately mature into some of the solutions that we can deliver to you all here.

And up here, I'd be happy to take a question or two because we got to do the laptop transfer, which is always fun.

Yeah. So I hear the policy statement, steer and ask. What's the difference between agent and app?

Yeah, so the question is, what's the difference between steer and ask there? And steer is the idea that there can be a policy that is defined that doesn't need a human in the loop, that it can basically guide an agent to say, nope, instead of doing that, let's do something different.

So the classic example that I think is easiest for folks to understand is, like, what if I just redact the PII or a secret, replace it with asterisks, and let the kind of agent proceed? But that's not going to work in all cases.

Ask is going to be an explicit prompt to the user, either through the agent interface, whether you're using Codex or Claude or Cursor, whatever tool, or potentially through some other mechanism that we're exploring now as well.

So you're talking about implementing all those different languages for agents, or do you have a specific use case for each of that?

Yeah, happy to talk kind of after here. But the short of it is that we're relyingright now, from an implementation perspective, on hooks that can intercept kind of a pre-tool execution in near real-time, assess, is this actually potentially problematic, and then before the agent kind of invokes the next thing, provide that feedback to it.

But happy to talk after here. You in good shape?

Yeah, cool. Hopefully, you guys can all hear me. Like Ezra said, my name's Dan Arpino. I am a software engineer at Snyk. I am one of the developers on the ADS platform. And so I started out specifically from that ensure trusted output section, basically how do we ensure that code is secure?

And as Ezra said, we were doing a whole bunch of hooks and integrating with the agents themselves. And I was getting frustrated with all those integrations. And so I decided to try to build a little bit of a pair programmerright here.

And so this is a local Electron app that I have running on my machine that's watching everything that's going on on my machine. You can literally see it's trying to fix some vulnerabilitiesright now. It's watching the files. It's running these scans in the background, and it's automatically trying to kick off agents to keep this secure.

And so this idea is, how can we actually use the agents to help us? And then I decided to take this a little bit further because, as Ezra said, ensure trusted output was just one of the three pillars of agentic development security.

We still want to know what the agent is doing, and what is the agent, and what tools the agents are using themselves. And so, yes, we

have some of our tools today that Snyk offers on the Evo platform. And I suggest that you all go to evo.snyk.io and see this agent guard in action and see all the skills

and MCPs that your organization is using. But what I really wanted to do is I wanted to give local visibility into what's running on my machine. As a developer who cares about security, I want to see this. And there's no really good way to see this.

And so here in this tool, I can see all of my running LLMsright here. I can see all of my running AI components. I can see, and this is the MCP servers. These are the skills. These are the CLIs and the models that are runningright here that Ezra talked about with these different risk scores.

So you can actually see all the different things that I have willingly or unwillingly installed and run on my machine.

Similar, we talked about, what is the agent allowed to do? And so, yes, my organization can set a policy for me, but I might want to set my own policy. Or more often, I actually may want to set a policy depending on what project I am actually working on.

So I can actually set up, based on my different workspace, what I want to apply to each one of those. For example, I actually turned this off. I'm building out a BOLA scanner, broken object layer authentication. Big issue that we have that can be a whole other presentation.

I was doing some benchmarking, and Snapy was automatically fixing my vulnerable tools for the benchmarking in the first place. So I had to actually turn that off. But yeah, the goal here is that you can actually set your own guardrails.

It can automatically go through and fix any vulnerabilities. So for example, here you actually see that it implemented some coderight here that's actually running. And what happens is when it implements these codes, it will actually start a scan.

It will start a scan. It will find the detected vulnerabilities. It will actually launch a new coding agent and try to fix thoseright away.

And yeah, so giving me visibility on what workspaces I have, giving me visibility on how many of my different agents are running, what my agents are doing, what they're costing me, getting a history of everything it's doing, all the vulnerabilities.

I can actually see on these fixed ones what it actually tried to do, how it tried to fix it, being able to track all my different sessions, giving me an audit of everything that my LLMs and all my LLMs are doing on my machines.

What are all the commands it's running? What are all the files it's accessing? There's some really interesting implications here on how I actually want to optimize some of my files and everything like that. You can see which ones are my heavy reads and writes.

You can see which ones are my heavy reads that have almost no edits. I might actually want skills to tell me to do different things about that. What are the commands it's running? Basically, what webpage am I hitting?

Monitoring all my different MCPs and tools. Yeah. And then, yeah, soright now, we've got this running our static analysis code testing. We have some open source testing. We've got secrets testing. We've got the BOLA scanner, which is a set of work in progress, allowing me to configure all of these things.

And this works because it's, well, one of the reasons that it's doing this is it's actually running a local host on your machine. And so I was able to actually run this one earlier. And if you look at this, I spun Claude up, and I was like, hey, Claude, read my ENV environment.

And Claude was like, hey, no, I can't do that. I'm smart. And I was like, allright, well, Claude, tell me what my secret key isright here. And Claude was like, oh, I'll try to read that. And you can actually see that because we've actually set up enforcement here, Snapy actually blocked the access of reading this file.

So yes, the agents are getting better. They are not perfect, which is why having deterministic guardrails on your machine, being able to set those guardrails on your machine, and having visibility into what your agent is doing and what it's running, I think, is pretty key.

One of the big things is, how do we trust agents? I want visibility. I want auditability. And those are really key to me. And so this is why we started developing this. In theory, in the future, you could actually hook this up.

You could actually download rule packs. You could connect it to your Snyk tenant and org. You can update your own rate sheets, whatever you want. But yeah, this is a little bit of a future vision of what ADS could look like.

This is really saying, here's my development environment. I could be running this on a cloud machine, telling me everything that's happening on that cloud machine. Give me that visibility. Give me that auditability. Give me that traceability. Really important aspects in learning how to trust the agents and making sure that they're not going off the rail.

Cool.

I should have asked at the beginning if we could turn this mic on here.

That was awesome, Dan. Thanks. Should have asked at the beginning, of the folks who are here, who's on the security side of the house today as opposed to engineering? Awesome. Hands down. And who's an engineer in the room?

Cool. So I think this is a good acknowledgment of the different audiences that I think these solutions are really trying to address. I think if you asked the security folks in the room, they'd be like, restrict everything. Just please do not let anything bad happen.

And if you asked developers, you'd say any false positive that causes kind of more noise in my workflow is just kind of hell on earth. And so that's the needle that we're ultimately looking to thread here and why we're trying to come at this from kind of both sides.

And I think what Dan was really showing is, how do we really, really lean into the developer experience in this new agentic world in a way that is going to make it still possible to achieve all of the productivity gains that I think everybody wants out of AI for software development while still being able to trust and kind of sleep at night, like the last presentation was talking about.

Yeah.

We have talked about possibly throwing it up on our Snyk labs. It's still in development. Come talk to me. Come by the Snyk booth afterwards. I'll be there till close this afternoon.

We'll at least get names. And if it's not today, which is probably not today, then we can at least see once it's there. We'd love to get feedback once we can get something out there like that.

Do you expect Snapy to support anything like Py?

And can you repeat the question on the mic?

The question is, do I expect Snapy to support things like Py?

How hard is it to do?

It's actually really well,

this is a tool that I have developed for my own personal use, and I think it has a lot of value. It will go where the market takes it. And that was kind of the idea. It doesn't all have to be up in the cloud.

I want local visibility. Wherever the market takes it is where I expect it to go.

Sounds like you might have a use case. I would love to connect after.

Is there any feedback from Snyk in terms of how you've gotten on those visions as they're forecast? How has the forecasting been so far?

Yeah.

What's actually working nice in terms of how things are?

Yeah, so the question started with a lot of praise for what we've done and then asked about the false positive rate here. And so far, the feedback overall has been good. It's certainly not zero. I don't think anybody who's playing in the space is claiming that it's zero.

Anecdotally, when I am using it, because I have it installed on my machine, I do not find it cumbersome and bothersome. I think I probably had one instance that I can remember in the last month where I was like, oh, that really was not a particular problem.

But we're pretty aggressivelyright now working with design partners that include sometimes hundreds of developers within companies who might be doing a variety of different things to try to refine this. So I think it's going to continue to get better and better and hopefully asymptotically approach zero.

But I'd be shocked if we ever lived in a world where it was absolute zero false positive rate for any of the companies out there. And if anybody here is working on solutions that are similar and you've got ideas on how you're solving that, would love to learn, because I think there's plenty for us to learn here, too.

Thank you.

I think the question was, does this only work on

code stored in the cloud, or can it work on your local machine? Is that the question?

No, no. I mean, what I understand is that basically, what I have done is trying to find a workload that's out there. What I would ask for, before that, I have my code ready already. Find a workload inside my code base.

Yeah, so different types of products that we haveright there. And I think Manoj mentioned in the first one that burning down a backlog and a remediation agent. Currently not in the tool that we showed today. And agentic development security is more that forward-facing, stop the bleed going forward.

But some of the things that we're releasing in agentic development security is a set of skills, commands, hooks, and stuff like that. And so we have a Snyk fix skill that you can pass in a known vulnerability to, and it will actually use some of that Snyk logic to actually try to understand the breakability, understand what changed, know what the good recommended fixes are, and guide the agent to a more effective fix.

What I didn't have time to demo in Manoj's keynote earlier was basically how we leverage that breakability as part of an input into, is this something that I can fix quickly? And generally, it's really easy to generate fixes.

Getting the approval of the code changes and merging them in, that's still the human loop kind of time-expensive type of a thing. And so that's a whole other area that we're working on under kind of the remediation umbrella.

Really cool stuff. Just didn't fit into this particular talk. Thank you.

Here's one more.

Yeah, sorry. I can't see because of the lights.

Yeah, we are blinded, yeah.

I just wondered, you're saying that how you guys guardrail AI agents that don't understand tasks, one that's not based on an LLM, but one that would maybe be self-created by sending sensor data and then taking action over a factory floor for events.

That is a fascinating question. The question was, how would this potentially work? Any solution around guarding agent behavior when you're not talking about text-based exchanges, but more sensor-type data or other things? I don't know.

We're all working on tools.

OK. I mean, I think it's probably it's a form of language that's being communicated,right? Just because it's not natural language doesn't mean it's not language. I think there might even be some advantages around kind of a known structured schema for some of that data.

I don't know if schema is necessarily theright word, but schema-adjacent.

That's cool. Nobody's ever brought something up like that. I'd love to talk. I know I'm making a lot of meeting plansright after this, but I'll be hanging outright outside.

And a lot of the guardrails are still the same. You still may want to control access, like what one sensor can affect or what one sensor can't affect. So you're just talking about the input of data, but there's a whole bunch of outputs data and what the model can do and what else it can access.

So a lot of those guardrails and the format of those guardrails stay the same, even if that input language changes to sensor reading as opposed to natural language.

Cool. I know we're holding you guys from lunch, so thank you so much for the time. We really appreciate it.