# AI + Security & Safety — Don Bosco Durai AI Engineer · 2025-04-19 Don Bosco Durai, CTO of Privacera and creator of Apache Ranger, argues that building safe and reliable AI agents requires a multi-layered security approach combining preemptive vulnerability evals, proactive enforcement, and real-time observability to address challenges like unauthorized access, data leakage, and compliance in enterprise production. He explains that current agent frameworks run as a single process sharing credentials, creating zero-trust vulnerabilities, and autonomous agents introduce unknown unknowns. He advocates for three layers: preemptive security evals (including prompt injection, data leakage, runaway agent tests) to generate a risk score for production promotion; proactive enforcement with authentication/authorization propagated across all components and approval workflows; and observability with thresholds and anomaly detection to monitor agent behavior in production. He illustrates compliance needs using his customer example—a top credit agency needing to treat agents like human users for regulatory adherence. Bosco also open-sourced PAIG.ai as a safety and security solution for GenAI and AI agents. ## Questions this episode answers ### What are the essential steps to securely deploying AI agents in production? Don Bosco Durai recommends a three-layered approach: preemptive evaluations to compute a risk score, proactive enforcement of guardrails like authentication and authorization, and continuous observability to monitor agent behavior in real time, detect anomalies, and react quickly. This builds confidence to safely promote agents to production. [5:33](https://aie.addtry.com/d722dfff-6dae-433d-9d4f-69bc2a0bd91e?t=333000) ### How do you prevent unauthorized access when AI agents interact with backend systems? Durai stresses that user identity must be propagated from the initial request through tasks and tools to the final data access or API call. At each endpoint, enforcement of proper access controls and policies ensures the agent cannot retrieve data or perform actions beyond what the original user is permitted, even in a zero-trust environment. [13:16](https://aie.addtry.com/d722dfff-6dae-433d-9d4f-69bc2a0bd91e?t=796000) ### What is PAIG.ai? PAIG.ai is an open-source project, led by Don Bosco Durai of Privacera, focused on security, safety, and governance for generative AI applications and AI agents. It addresses the challenges of securing multi-agent systems and is seeking design partners and open-source contributors to help advance enterprise-grade AI safety. [17:26](https://aie.addtry.com/d722dfff-6dae-433d-9d4f-69bc2a0bd91e?t=1046000) ## Key moments - **[0:00] Intro** - **[0:53] Definitions** - [0:53] AI agents are autonomous systems with reasoning, workflow, tasks, and tools, says Don Bosco Durai. - **[1:48] Zero Trust** - [1:48] Most agent frameworks run as a single process, sharing credentials and creating a zero-trust issue, warns Don Bosco Durai. - **[3:41] Challenges** - [4:18] Top-three creditor requires AI agents to follow same regulations as human users, says Don Bosco Durai. - **[5:33] Three Layers** - [5:49] Securing AI agents requires three layers: security-focused evals, enforcement, and observability, says Don Bosco Durai. - **[7:16] Evals** - [7:16] AI agent evals must test for prompt injection, data leakage, unauthorized actions, and runaway loops, says Don Bosco Durai. - **[10:40] Enforcement** - [10:58] User identity must be propagated to every endpoint for proper access control, says Don Bosco Durai. - [12:47] Approvals can be automated by a second agent with guardrails and human-in-the-loop for high-risk actions, says Don Bosco Durai. - **[13:55] Observability** - [13:55] Observability is critical because agent behavior changes with model updates, libraries, and user inputs, says Don Bosco Durai. - **[16:41] Recap** - **[17:26] Open-Source** - [17:26] Privacera open-sourced PAIG.ai for AI agent safety and security, seeking design partners, says Don Bosco Durai. ## Speakers - **Don Bosco Durai** (guest) ## Topics Guardrails, Red Teaming ## Mentioned Privacera (company), Apache Ranger (product), PAIG.ai (product) ## Transcript ### Intro **Don Bosco Durai** [0:17] I'm Don Bosco Durai. I'm the co-founder and CTO for Privacera. Very recently we open-sourced our solution for safety and security for GenAI and AI Agent. I'm also the creator and PMC member of the open-source project Apache Ranger. It does data governance for Big Data. It's also used by most of the cloud providers like AWS, GCP, as well as Azure. So today I'll be mostly talking about how you can build a safe and reliable AI Agent. ### Definitions **Don Bosco Durai** [0:53] So before I get started, let's get some of the terminology standardized. From my perspective, AI Agents are autonomous systems. They can do their own reasoning, they can come with their own workflow, and they can call tasks for doing some actions, or they can use tools to make API calls. So tasks are more specific actions. They may be able to use LLMs, or they may also call RAGs or tools. While tools are functions which can be used to get data from the internet. If you have databases, it can go and get data from the database. If you have service APIs, it can call those things also. And memories are contexts which are shared within the agents, the tasks, and the tools. ### Zero Trust **Don Bosco Durai** [1:48] To give a visual representation, there could be multiple agents, and an agent may have access to multiple tasks, and there could be multiple tools, and these tools can talk with APIs and DBs. So one thing that you need to know out here is most of the agent frameworks today, they are run as a single process. What that really means is the agent, the task, the tools, they are in the same process. That means if the tool needs access to a database, that means it needs to have the credentials. If they want to make API calls, it needs shared tokens. So those credentials are generally service user credentials. That means they have super admin privileges. And since they're all in the same process, one tool can technically access some other credentials which are in the same process. Similarly, if you have tasks or agents which have prompts, all the things that are running within the process, any third-party library, they can also access it. So those sort of make this entire environment a little bit unsecure. So there's a little bit of a zero trust issue out here. The agents, the tasks, they talk to LLMs. If you don't have a secure LLM, then that is another area where these things can get exploited. If you take an agent on its own, by definition, it's autonomous. That means it will make up their own workflow depending upon the task. So that actually brings in another set of challenges which we call in security as unknown unknown. So you really don't know what the agent is going to do. So it's very non-deterministic. So because of this, the attack vectors in a typical agent are pretty high considering from some of the traditional software. ### Challenges **Don Bosco Durai** [3:41] So what are the challenges because of this? So there are multiple challenges. So if you look from the security perspective, if the agent is not designed or implemented properly, that can lead to unauthorized access, also data leakages of your sensitive information, confidential information. Safety trust is also another biggest challenge. If you are using models which are not reliable, or if your environment is not safe enough, if someone goes and changes the prompts, that can also give you wrong results. Compliance and governance is an interesting thing. Most of us are so much busy even just getting the agents working. We are not even worried about a lot of the other things that are necessary for making your agent enterprise ready. So interestingly, I was just talking to one of our customers this Tuesday. They're one of the top three creditors. So they built a lot of agents, but their biggest challengeright now is to take it to production. For them, they consider an AI agent as similar to a human user. And when they onboard a human user, they go through a training, and they have a lot of regulations they need to adhere to. They have data from California residents, so they have to make sure anyone who's accessing California resident data, they should not be using for if the user is not given consent, they should not be used for marketing purposes. They have international, so they have Europe data. So who can access those data? There's regulations around it. And also, there are a lot of regional regulations. So when they consider even an AI agent similar to a human, so they have an onboarding process, they have a training process, and they want to make sure the agents are also following the regulations. So without that, they can't go into production. And we as AI engineers, we are still in the early stage, so this is one of the things which is out of our radarright now. ### Three Layers **Don Bosco Durai** [5:33] So now, how do we really address this thing? So those who are in security or associated with security or compliance, there's no silver bullet. The best way to do it is to have multiple layers of solutions. So these are some of the things that I have in my mind. So you can split it into three different layers. The first layer is, what is the criteria to even put your agent into production? What do you need to do? We talk about evals, but most of them, we are only talking about evals for how good your model is, how good your response is, whether it's hallucinating. But you also need to have evals which are more security and safety focused. So we'll go through some of those things, but the goal of this eval out here is to come with a risk score. And depending upon the risk score, you can decide whether you can even promote this agent to the production. And the agent may not be necessarily you are writing it. It could be a third-party agent. So it has to go to the same criteria. The second is enforcement. Eval generally tells you how good your agent is built. And enforcement is the one who is actually doing the enforcement or implementation. So you have to make sure you have a pretty strong implementation. If your implementation is not good, your eval is going to fail. Essentially, you can't go to production. And third is observability. Particularly in the world of agents, it's a lot more important because there are so many variables involved out here that you cannot really catch all of them during development or initial testing. So you have to keep track of how it is used in the real world and how you can react on it. So I will go through some of those things in a little bit more detail. So let's start with the evals itself. ### Evals **Don Bosco Durai** [7:21] If you look into traditional software development, there is already a process. There are gating factors that tell you how you can promote your application into production. So if you start with basic things like when you're writing your code, you have to make sure you have theright test coverage. If you're building Docker containers, you have to do the vulnerability scanning. If you're using third-party software, you need to make sure you're scanning for CVEs. If you find high or medium risk or critical risk, you try to remediate that before you can get into production. You do pen testing, so make sure there's no cross-site scripting and other vulnerabilities. The same thing applies for AI agents also. You need to come with theright use cases. You need to make sure you have theright ground truth so that when you are doing any changes, you're changing the prompt, or you are bringing a new library, a new framework, or a new LLM, you want to make sure your baseline doesn't change. If you are using third-party LLMs, make sure they are not poisoned. They have been also scanned for vulnerability. If you're using third-party libraries, which almost everyone is using, make sure they also meet your minimum criteria for vulnerability. And similarly to pen testing, you should also do testing for your prompt injection. Make sure your application has theright controls so it can block them. And most of the LLMs are already doing it, but not necessarily all of the LLMs are doing it. The other eval is around data leakage. This also is pretty important, particularly in the enterprise world, because when you are building enterprises, you're building agents which do generally what a human would do. If you're building an agent for HR, they have certain functionality. If I am an employee, I can request to get my salary benefits, but I can't do the same. I can't get it for someone else. But if I'm an HR admin, there's a possibility I may be able to access someone else's salary benefit. How do you make sure your agent is not leaking data? There's no malicious user who can exploit some of the loopholes you have. So you would have to do these evals upfront before even you can put your agents into production. Similar to data leakages, unauthorized actions. Most of the agents, even though they are read-only, there are also now agents coming which are trying to change things. They'll do some actions. How do you make sure those are also done by theright person, with theright persona? And runaway agents, those who work on agents already know the agents can go in a tight loop for various different reasons. It could be a bad user prompt, or just the prompt for the task, or the agents cannot address those things. So you have to make sure you test for such scenarios before you put your agent into production. So the goal of this is to come with a risk score at the end of the day so that it gives confidence that, can you put this into production? ### Enforcement **Don Bosco Durai** [10:40] And the next one is going to be around enforcement. As I said, your risk score is going to be depending on how good is your enforcement. And particularly in agents, you're working in almost like a zero trust environment because you have libraries which can access anything. If you are accessing certain of your backend systems which have sensitive data, how do you make sure the wrong user is not accessing it? So from the security control, there are a lot of other things which I'm not going to talk today, like detecting prompt injections and moderation, but focusing on enterprise-level things, you have to make sure you have theright authentication authorization. This is pretty important because when you look at the environment, when a user makes a request to an agent, it goes to a task, and eventually it goes to tools and makes an API call to a service or a database. If you don't have theright authentication, someone can impersonate someone else and may be able to steal confidential or sensitive information. And the second is authorization. If you have the authentication done properly, then you have to make sure the access control is applied properly. And this is also important because agents have their own roles, and as an agent, they can do certain things. So you have to make sure they are not going beyond what they're supposed to. And at the same time, if you have agents which are trying to do something behalf of another user, then you have to make sure that the user, that person's role is enforced. So if you're accessing a database, you shouldn't access anything which the user does not have permission to, or making API calls. So that's why authentication authorization is super important. Without that, obviously, there are going to be a lot of other issues. Approvals is interesting because in the traditional world, we already have workflows. If I request for a leave, my manager will approve it. It's already built into the system. But in the case of agents, you don't need to have a human all the time. Your agents can do most of the things automatically. So if you do it, design it properly, you could have another agent which all it does is just looks for approvals and making sure the results areright. And you can also put thresholds, how much this agent can approve automatically. And you can put the proper guardrails to make sure if it goes above a certain limit, it can automatically get a human in the loop. So just to reiterate this one, because it's pretty important, is when it comes to authentication authorization, it's not just about doing the authentication at the point of entry where you're making a request. You have to make sure the user identity is propagated across everywhere. If you're calling a task or the task is calling a tool, you have to make sure the user identity is passed on to the last point where it's actually making data access or making API calls. And that point, you have to make sure you're able to enforce theright policies and access control. And the third is observability. So observability is pretty important in the agent world because, as I mentioned, the traditional software, once you build it, it generally just works. You just have to make sure there's no new vulnerabilities coming in because of some library update or something like that. ### Observability **Don Bosco Durai** [14:16] But in the world of agents, there are many different variables involved. One is the models change very rapidly. If you're using an agent framework, that is also keeping evolving. You're using a third-party library. That may start behaving differently. The other important thing in an agent is it's very subjective to what the user is entering. You may have tested with a certain assumption, mostly sunny day scenario, I want to apply for my leave. But the end user may use an entirely different text to ask the same questions. So how do your model is going to behave? So you have to keep monitoring to see if the user inputs or anything that changes, how the responses are coming in. And also to make sure how much PI data and other confidential data is being sent out, because if you see some anomaly, you have to be able to ready to act upon it. The other thing is, obviously, you can't monitor each and every request. As the number of requests increases, it's just not possible. So you have to start defining thresholds and metrics. So what that really means is you can start calculating or counting how many failure rates are out there. Once you know you have a certain failure rate which is within your tolerance, it's fine. But if it goes above that, you can automatically create an alert and look into it. And the failure rates could be because of misbehaving agent, or it could be a malicious user trying to compromise the system. Then anomaly detection is another interesting thing. I don't think we are anywhere close to it yet. But it's very common in the regular traditional software in the security side. There'll always something like user behavior analytics where they look at the user and see whether they are within the standard operating thing. With agents coming in, there'll be more and more anomaly detection where the agent is behaving within the accepted boundaries. And all those things will end up with a security posture. So that will give you a near real-time thing, how good your agent is actually performing in life. So that gives you a bit of a confidence. ### Recap **Don Bosco Durai** [16:41] So to recap, as I said, there are three things. One is preemptive. How vulnerability evals to make sure that you get theright risk score, which gives you the confidence whether you can promote the agent to production. If you're using a third-party agent, whether you can use it in your environment. Second is proactive enforcement. Make sure you have theright guardrails. You have theright enforcement. You have theright sandbox so that you are able to run the agent in a secure way. Make sure you have theright observability so that you know at real-time or near real-time how good your agent is performing. And if there are some anomalies, you can go and quickly fine-tune it. So just as I said, we open-sourced our safety and security solutions. It's called Page.ai. Security and compliance is a pretty vast field. I don't think any single company can do it. So we are looking for design partners and contributors who can help us in our journey. ### Open-Source **Don Bosco Durai** [17:45] So if you're interested, please reach out to me at bosc@page.ai or connect me on LinkedIn. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.