AI Red Teaming Agent: Azure AI Foundry — Nagkumar Arkalgud & Keiji Kanazawa, Microsoft

Hey, thank you, and welcome everybody. I'm Keiji Kanazawa from Microsoft. I work in product and AI Foundry. If you're at the keynote session from Asha, she's my CVV boss, so I work in that org. And Nagkumar is an engineer on the team.

He's going to be showing some code in a little bit, which I'm sure most of you are excited to see. So here we are at the AI Engineer conference. I'm sure you're learning about all kinds of stuff: reinforcement learning agents, SWE agents, evals, all kinds of stuff.

And so we're all really excited, at least I am, to get AI into the hands of people and help people, whether it's for your end users and your internal users. And so AI is obviously where it's at, or I think it's where it's at, but it comes with a bunch of headlines that you've probably seen.

And some of them, again, if you're at the keynote this morning with Simon, he showed some examples. So it's very easy. It can be easy to trick chatbots into saying stuff that you don't necessarily want them to say.

You can actually trick them into giving you information, potentially, that you don't want leaking out. And AI is built, like AI Engineering, it's built on a whole ecosystem of different things, including Python packages, NPM packages, other services that you may be using that are hosted.

And of course, San Francisco is the home of self-driving cars.

This picture is showing it's a frame from a video clip where a self-driving car is

driving happilyright past a school bus with a stop sign on it. And if you think, hey, what does it have to do with me? This is one test that you can kind of think of in terms of whether it's something you built or something you're thinking of building.

Again, it's really easy to kind of get around some of the defenses of the AI models. So for example, the prompt on the left, if you say, "How to loot a bank," a lot of the models will actually refuse to answer.

They'll say, "Oh, no, I can't help you with that." But if you and some of the examples how many of you were at the Sanders workshop yesterday on prompt engineering, red teaming? Yeah, so it's really like if you preface that question with a whole, maybe a bit of your life story, you can maybe convince the AI to tell you something that it's not supposed to.

And there are also other tricks. Like on theright-hand side, it says, "NABA TOOL WHOA," which is how to loot a bank spelled backwards,right to left. And actually, it turns out that's one of the patterns that an AI model can be tricked into giving you the answer.

And so

especially like we talked about it this morning also, but of course, it's all agents, agents, agents, 2025 is the year of agents. And there are a lot of concerns, like if you talk to businesses about how in this world of agents, AI can be tricked into different kinds of risks and stuff and different malfunctions.

But we're here at the AI Engineer conference. And what I want to kind of convey is that we as engineers know how to do this stuff. So engineers build bridges and dams that people trust. We build trucks and trains.

And so AI engineering is early. So we've got a lot of work to do to get to the point where people trust AI as much as they trust bridges. But this is something we know how to do as engineers.

We build something, we iterate, we check it, we test it, we continue to iterate. And that's what we're here to show you kind of how to do. And as engineers, we also rely on not just ourselves, but other people.

So what we like to say is trust is a team sport. So when we're looking to build trustworthy AI systems, we depend on other people. So the engineers need to depend on people who have a lot of expertise in these areas, like security and AI risk.

So at Microsoft, we have a team called the Microsoft AI Red Team, which I've been working with for a few years, actually. They were one of the pioneers in identifying risks kind of of AI in general, as well as LLMs.

Like two, three years ago, they were already talking about, "Hey, you know what? Like these GPT-3, GPT-4 models, you can get them to do things that you really don't want them to do." So we partnered in Azure AI Foundry with the AI Red Team to offer a solution that makes it easy for you, AI engineers, to basically have a teammate that can help you with the AI Red Teaming.

And so the AI Red Team, they built a Python package called Pyrite, Pyrite, and what we offer is a hosted version and wrapped it around an easy-to-use SDK and also a hosted dashboard to show you the evals that come from this.

And so here, to show you how it works is Nagkumar.

Awesome. Thank you. Thank you, Keiji.

Here we go. So

hello.

And

so this is the sample project that I'm going to run for you all. It's a simple RAG on PostgreSQL within Azure Samples. I'll have this QR code up again. So running locallyright now. You can ask simple questions like this.

And it's talking to a locally running model via Ollama. Well, tool called NWARG, live demos. So in this piece here, logs for everything. But what we are trying to showcase is something called the Semantic Kernel Agent, which here's some code for it.

It exposes, it takes in the Azure chat completions. And our Red Team plugin is something that our SDK exposes. It has all the functions that are needed for an agent to call into our Red Team agent to help someone with their Red Team process.

And then it's a simple chat completions agent afterwards. So for now, I'm going to start running this.

And when I run this, it will go through a few user inputs that I have hard-coded, and then we can jump into an interactive mode. But the target for this Semantic Kernel is going to be the same RAG app, which is running locally.

So once this loads in, yep, the first question oops,

live demos.

Looks like tool calling isn't working today. But anyway, so this call RAG app can be switched into a call to any other application, which takes in a query as an input and then responds with a string as an output.

So internally, we ask you to call your application, which then you can run evaluations on. So in this agent mode, what would usually happen, I can scroll up to a previous output.

It's ran earlier. Not lying to you all.

So these were the strategies which were available. And then use one of the I asked it to, like, hey, figure out, get me a harmful prompt in the violence category. And then it gives me some sort of prompt.

And then I'm like, hey, send it to my target. And then this is what the target responds with. And then there are some details about some sort of ski goggles and products that are supposed to be answered from our database.

And then I try to be like, hey, convert the prompt using Base64. And the agent converts it. And then be like, hey, now send it to my target. And then the target responds with something else. So this is an easy copilot-styled way for anyone to get started to red teaming an application.

Now, we can take it a step further and run the whole scan end to end. And this is how you would run the scan. You saw a little bit of code that Keiji showed earlier. So you usually set up your AI project, throw in the URL.

And then you can set up, initialize it with the URL and then your credentials. You can select risk categories. We have four of these risk categoriesright now. These map to our evaluators. So this is how you set them up.

If you don't include any, we include all by default. And then the number of objectives is the number of questions that will be sent out to your application. And then the scan method looks like this. So you give it a scan name.

You can give an optional output path, which stores all the results there. And then your attack strategies will include a list of different attack strategies. I'll pull up a docs page later on, which has all the information about different risk strategies that you can use.

There are combination strategies like easy, which does flip, the one that reverses the string and things like that. So there's also MARS. These are simple converters, which live within Pyrite, but then expose via our SDK. So our SDK can offer it to people to use it easily.

You can also compose an attack with two different strategies. So get a tens converted strategy and then do a URL-style conversion on it. So it does both and then sends it out. And then you pass in a target, a target which supposedly decided not to work today.

Again, call to the same application.

And once you run this, you usually see an output, which looks like this. So oops.

I'm going to unplug the Ethernet cable.

OK, there we go.

And so yeah, so this is when I had it running yesterday with GPT-4.0 as my model. And GPT-4.0 comes with a lot of security inbuilt within our Azure AI Foundry. So once you have all those guardrails up, it kind of was pretty good.

It was a very small sample size, 160, I think. I selected 10 different harm types or like five harm types with a few categories. So none of them was able to break into our application. But then I switched around and I used Phi-3.

And with Phi-3, you can see the results show 5 out of 40 in hate and fairness was successful. So we can take a look at it, filter data based on what was successful. And then you can look at what was the response that was determined as harmful from our evaluators.

And finally, we have one more way of doing this. So initially, a lot of people might just be building models. You don't even have an application. You can directly call the scan against an Azure OpenAI config. So if your model is running on Azure, you can set it up as a target, which is just these three things.

And then once you have these three things, you can run the whole scan. This scan runs against the model directly and gives you an output. I guess this should be able to work. Let's see.

I can probably run it here.

There we go. So yep, scan model runs a direct model scan. And here I have some results from a pre-run prepared for this. So this is when 4.1, if you take off all the guardrails, here's the result. It says that 25% of violence category was successful and 20% of all the difficult complexity attacks were successful.

And again, you can filter on the data, see which was successful.

And yep, there we go, lots of violence. And then this was the flip where some sort of we can see what strategy it was. I think it was a Caesar encoding strategy. And you can see that the assistant kind of decoded it.

So we did not want that to happen. So we marked it as a successful attack. So that's one of the things. And then here's the response when you set up all the guardrails with GPT-4.1 Nano. And you can see that the difference is that we reduce our attack success by a little bit.

So that's an overview of how things go and how this scan is running. So it usually gives you an ETA, six minutes. We'll probably be running out of time by then. But yeah, as soon as this is done, it shoots you a URL, which will directly take you to this page.

So that's safety evals in AI Red Teaming. I will be at the Microsoft booth towards the end for questions. So back to you, Keiji.

Yeah, thanks.

So basically, the rest of the talk is talking about essentially how this fits into an overall strategy. So AI Red Teaming is a really important

and part of kind of your defenses and kind of in your toolbox to be able to develop and deploy trustworthy AI systems. But really, what you want to do is incorporate this within a whole kind of, again, from the engineering mindset, a framework and kind of a process to get these things out.

So first, what you want to do is before you develop a production application that goes to customers, you want to kind of map out what are the kind of risks that we're anticipating here. Is this an agent? Is it using kind of external data or internal customer data?

You want to think about what are kind of the risks that your app is going to have, plan for it, start to implement the guardrails in the first place, and then do the evaluations of which Red Teaming is one of those possibilities.

So within Azure AI Foundry, we have a suite of evaluators, both for quality check, quality evals. I think there's a lot of talks today at AI Engineer about kind of quality evals with something you could do in AI Foundry.

And then there's a whole set of risk and safety evaluators that we have, which AI Red Teaming Agent is one of them. But we also have a lot of different kind of classifiers in terms of kind of both input as well as output, because you want to check for both.

And then there's a specific set of evaluators we just created for agentic applications as well, like does the agent follow your instructions well? Is it things like that? And then you can add your own custom evaluators. And Nagkumar showed you kind of like some of the mitigation strategies that you want to apply.

So for example, there are guardrails and controls that you can have in your application. So once you've ran the AI Red Teaming Agent, you figured out, well, actually, 20% of stuff gets through. What do you do? Well, that's the point at which you can apply the guardrails to content filters and other capabilities that, again, we have in Azure AI Foundry that makes it easy for you to add those guardrails.

And among other ones, there's ones called prompt shields, which are for to guard against prompt-based attacks, especially the kind that are involved with AI Red Teaming. And we have time for maybe one or two questions. I think we have like a minute, maybe.

Yeah, we have two more.

Yeah.

So if you're going to be talking about how does

guardrail work under the hood?

Is it like filtering after it gets the answer? Or is it like before the LLM even sees the prompt?

So there's both kinds. Yeah, you can apply both.

Yeah, but the guardrail feature, how does it work under the hood?

Do you know? I mean, I think there are both. I think we have both filters for the input end as well as filters at the output end. So there are content filters, for example, basically, if people are typing in like a CBR or like how to build a bomb kind of thing.

So there are the input guardrails. But then there's also guardrails in terms of, well, actually, I want to make sure I'm not outputting sexual content or something. So there's also guardrails in terms of the output of the model as well.

And that's what's happening under the hood with the guardrail feature?

Yeah, yeah, yeah, with the guardrails.

You have to implement several features to get that.

Oh, no, no. So there are guardrails that AI Foundry offers directly. And yeah, so that's what's happening under the hood. So you have the ability, for example, to just give the model raw inputs. And then if you turn off all content filters.

And that was some of the examples that actually Nagkumar was showing. Yeah. So the guardrails are not in the model itself. The model is still the raw model. And the guardrails are actually kind of outside it. Does that answer your question?

Yeah.

OK, allright, thanks. I think we're out of time. So thank you for coming to definitely get started with AI Red Teaming. If you're not doing it today, definitely get started. Here's a link to the code as well as the docs.

And yeah, thank you for coming. And yeah, so if you have any questions, we will be at the Microsoft booth, different parts of today and tomorrow.

Come find us. Thank you.

Thank you.