Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog

Hello, this session is about why do your AI agents disagree with itself, and what to do about it, and this is Diane Huang Lin, and you can call me Diane.

A little bit about my background: my journey with AGI starts from my PhD at Imperial College London, where I was working on continual learning. Later I worked with Professor George Tannenbaum at MIT on one-shot learning, meaning learning from one single examples.

Afterward I was lucky that Alexa was launching; I was the first three appliance scientists on the QA team at Alexa. Later, I attracted by a startup working on AGI at the Silicon Valley called Vaccaris, so today Vaccaris is part of the Google DeepMind.

There I got to work on the exciting frontier research on zero-shot transfer learning. After working on AGI for a few years, I decided to pivot to the applied world at Zscaler, which is a cybersecurity company. There I got to apply a variety of different machine learning models to address the cybersecurity challenge.

There for 5 years I decided to co-found my own company because of the pain points I have seen at Zscaler. So Culminate is building AI agents to auto-triage your security alerts, which makes your SOC more efficient.

We are super proud that Culminate has been acquired by Datadog earlier this year, so now I'm part of Datadog and leading the development of self-evolved agents.

So today I'm going to share with you the problem you're probably seeing in your day-to-day building AI agent: the inconsistency. And then we'll discuss where this inconsistency comes from in order to figure out the solutions.

And we will discuss a few trade-offs among the different solutions, and then eventually we'll show you the experimental results to show how feasible they are.

You'll probably see this: same model, same input, but different output. Here I don't mean the wording different, I mean semantically different output.

You might be thinking, "That's the stochastic nature of LLMs."

Hold on to that thought; let's look at a few concrete examples.

So here, imagine you're giving a task for sentiment analysis, where we're supposed to label each review of a hotel either positive or negative.

This is a typical NLP task, and today's LLM is really good at this. However, there's one problem: when you pipe this input to the same LLM model, occasionally you will see different verdicts coming out of it when you run multiple times.

That means a single evaluation run didn't tell you the whole story; you need to repeat your evaluation multiple times in order to get a holistic picture

by the average over the different runs' results.

And this sounds like a technical inconvenience, but the reality is it's way more problematic than just inconvenience. Let me show you an example in a different domain: in cybersecurity. Imagine you're a security analyst, where your day-to-day job is to triage the security alerts and decide whether it's malicious that you need to take certain actions to stop the attack, or it's false alarm, meaning it's actually benign, which you could ignore.

So here, the example alert you got is: there's a failed login attempt to a Gmail account detected from a suspicious IP.

And if we give such a similar task to this one to an AI agent and run it different times, you will see sometimes it stays consistently benign, sometimes it stays consistently suspicious across different runs. However, there are cases where it flip-flops between benign and suspicious.

Now your customer will have a difficulty using such a product because they will wonder, "Which one should I trust?" So the inconsistency is causing a trust issue in your product. Imagine if you were in a POC bake-off: one vendor gives the consistent verdict all the time, while the other

vendor has this flip-flop verdict. You can imagine which one will win the deal. Now I'm pretty sure you'll want to solve this problem.

First, we have to figure out where the inconsistency comes from.

The good news is the data points tend to flip-flop, actually concentrate around the decision boundary, the so-called gray zone. Let me illustrate with the earlier example. Here, in the sentiment analysis, we have two reviews. The first one, actually, no matter how many times you run through the LLM model, it will always give you the positive label.

In this case, it's pretty obvious the review is super positive. However, the second case is the one that tends to flip. If we look closer, it's kind of on the boundary. Sometimes the certain word is kind of relatively positive, but sometimes it's a little bit negative.

In this case, you'll find that even human experts will have a disagreement on these cases. In fact, there's noright or wrong answer; it really depends on a particular company's policy and their preference. For hotels, thinking that such a review is not something I can do to improve the memorial kind of experience, memorable experience, then that means not to label it as negative because there's nothing I can do.

But some hotels might care about it and want to label them as negative so they can improve. So that comes down to your preference. Let's look at the other cybersecurity example.

Here, it's another case on the boundary. And why it's in the boundary is: here there's a failed attempt. Indeed, there's an attacker knocking on the door. But

the attacker might not get in yet.

So for enterprise customers who always have an attacker knocking on the door, well, if they haven't got in, in the environment, they don't want to worry about it. Otherwise they'll be flooded by this type of alerts. There's nothing they need to do because they are blocked outside already.

However, if the attacker did manage to guess the correct password and pass MFA and did manage to get in the environment, then it becomes very serious and you need to take certain actions. So almost similarly at the beginning, behavior, but very different outcome and different response needed depends on whether it's truly an attack or the attacker is still outside.

So here, whether we label them malicious or benign depends on the company's preference, whether they wanted to be notified about such a situation.

So again, this is the example where it depends on your preference and additional information to disambiguity between the two different scenarios.

So from these two very different domains, you will see that actually the data points tend to flip-flop at the one close to the decision boundary, where it needs your clarity to disambiguate which side it should really belong to.

So these data points are where human experts tend to make the mistake, and the traditional machine learning classifier will also struggle with. So in other words, it's not your AI agent's fault; your AI agent simply pointed out the ambiguity that already exists.

So

now we know where the root cause of the inconsistency is. The next step is identifying where exactly they are, who are they, and fix them.

Surprisingly and not surprised, the solution to identify the gray zone actually exists already in machine learning, and it's called active learning. So the idea of active learning is you have a lot of data points to be labeled. Imagine you have your initial model and you put it in production, maybe in the monitoring mode, and you want to check the quality, but you don't have bandwidth to check all of them.

Otherwise it loses the point of having an agent do the work, but you still want to identify where the model didn't do well. And then active learning is to find a way to select the data points where you tend to make the mistake, and that's where it tells you where to put your attention.

So let's go over the traditional machine learning model's active learning pipeline and see how it's different in the case of the new LLM or AI agent error. So first you train your initial model, and then you make predictions on the unlabeled dataset.

It could be the production online data. And then from there you see there are certain cases where the model wasn't sure because the probability is close to 0.5, and those data points are the high-informational ones that your model is going to learn the most.

The uncertainty is a type of signal to tell you where the model tends to make the mistake. Another popular type of way to figure out the problematic one is query-by-community, meaning that you buy a few different models, or even the same model, multiple times, and then you identify the disagreement.

And that's where the model tends to make the mistake as well, and that's where you need human clarification on the label so the model can learn it from.

After you surface these problematic data points, and for humans to take a second look to make sure the label is actually correct, or does it need additional information to disambiguate those cases, meaning adding new features. And then you can add those labeled datasets, or potentially additional features, to the next round of training.

Then you retrain the model and then proceed to continue the cycle. So here you can see, in this particular active learning cycle, you spend very little amount, or at least, in other words, it's a very efficient way to identify the problematic data points and spend your attention on those points where your model tends to learn the most.

So that's the traditional one. The good news is the LLM way is not very different. There's a major two-part which we need some refinement. One is on this selection strategy. The other part is about retraining.

On the selection strategy one, here we suggest you use the one about identifying disagreement. At least from what we have tried so far, the uncertainty score from LLM is not very reliable. It's kind of a the LLM model doesn't know what it doesn't know.

So when it's very confident, it doesn't mean that it was it's correct. So that, however, the disagreement from different runs, or from different models, actually gives you a more reliable signal where the model actually is not sure about its verdict and it needs human guidance.

And then after you identify this group or the disagreement and it needs attention, you label them and provide the feedback, like what you did before. Then afterwards, you might be thinking, "Oh, time to retrain the model." Yes and no.

Yes, one option is retrain the model, but fine-tuning the model is expensive. Here we are proposing a more lighter-weight solution, easier to iterate.

It's about augmenting your agent with semantic and episodic memory.

Let's use the previous example to illustrate what I mean here.

As we talked about, in this case they tend to flip. It's mainly missing the information about a particular company policy or preference. Here we can add it to the

knowledge base. If this customer complains about something outside the hotel's control, then classify it as positive because you don't want to pay attention to this.

And then in the cybersecurity example, something similar, you can put it in that the password spray, meaning that the attacker is trying to guess your password and try it with different times, even fail and continuously. So if without successful login, then that should be benign.

But in contrast, if it succeeds with final login, then that should be malicious. So here you actually identify the additional information to help disambiguate the two and then label them separately. And that's the clarity that helps your model to learn.

Also, it helps your human experts to label more consistently. So this type of domain knowledge sharpens your decision boundary and also helps the human beings. And this type of knowledge is like factual knowledge. It's part of actually belonging to your semantic memory.

And this is in contrast with the episodic memory. Episodic memory is in the case where you haven't got around to distill the reasoning behind the earlier case, why it should be on malicious or the benign side. Instead, you say that, "I have seen these similar cases before, and it was labeled as benign, and let me leverage those past similar cases," and they made the decision accordingly.

And the advantage of this episodic memory way is it's relative automatically. It has less human intervention. I mean, the human intervention was done in the past already. When you're doing a job online, you can reference them. At least your AI agent can reference them automatically.

It can make decisions without waiting for you to distill the reasoning and put it in the semantic knowledge. So the idea is that you identify a similar case to the one under examination and then leverage their past decision as a reference.

And then imagine there's a new case which is kind of in the middle, doesn't belong to any of your past group, and that's the one you still need human attention. And this is also where episodic memory resolves the recurring case automatically.

And this is particularly useful in the case of cybersecurity alerts because a lot of the noise, especially those false positive noise, are the ones that keep recurring. And that's the one that is low-hanging fruit you can automate away.

And with that, human review, which is the precious attention bandwidth you have, now you can concentrate on those where episodic memory failed to address. And this is also how we chose between semantic memory and episodic memory. The two are actually not contradicting.

In fact, they are complementary. Basically, you use episodic memory to address those recurring situations automatically. And therefore, what's remained still reflecting after the reference from the past or no reference to be able to use, then you pass on to human review.

And then human review distills the domain knowledge and puts it in the semantic memory to be used next time.

So here we talk about a solution actually brings three benefits. First, we identify this data point near the decision boundary and use the domain knowledge in semantic memory and this past similar case in the episodic memory. We're able to improve the consistency significantly.

So now you have a more trustworthy AI agent for your user to use. And two other byproducts of this solution are you have a very efficient quality control. You don't need to check every agent output, but instead a subset of them, which your algorithm from active learning thinks is potentially problematic, and then you check them, either provide additional information or clarify the label, and your model will learn from there.

So most importantly, your labeling effort is very small but high return. Last but not least, now you are gathering your customer feedback along this way. You not only have a trustworthy, consistent model and a high-efficacy one, you also have one adapted to your customer environment.

I'm pretty sure your customer will love your product if your AI agent listens to them and adapts to them.

So here are some experiment results from real data points. So here we collect 93 alerts, cybersecurity alerts, and then we run them three times. If without the solution we are proposing here, a quarter of them will flip-flop the verdict.

In contrast, after applying our solution or using episodic memory, about 15% of them become consistent. However, there's still 10% of them remaining inconsistent. They still flip

sometimes because there's no reference, similar cases in the past to be used. Sometimes even there's a reference to be used, they still disagree after the second thought. But not a problem. The episodic memory does the automatic knockdown of the inconsistency for 15%.

The remaining 10%, human review and provide the additional knowledge to disambiguate those tricky cases. After that, you also adapt to the customer environment that you want to learn.

So the three takeaways that I hope you have after this talk are, number one, inconsistency isn't usually your model problem. Stop blaming your model, but instead focus your energy on the label issue and potentially insufficient information, and help your AI agent with this additional clarity.

Second, the model disagreement isn't a bug, but a feature. Treat each disagreement as an opportunity for your model to learn. Third, fine-tuning isn't your only option. Augment your agent with semantic and episodic memory. I hope you enjoy seeing your model keep improving and adapting to your customer environment.

Last but not least, I want to thank my colleagues and wonderful friends at Datadog. Special thanks to Zicheng Sanhuo, Anna Chu, and Stefan, Sayi, and Matthew for giving me a lot of enjoyable discussions and helping me to clarify some of the concepts.

And also, thank you to everyone in the BITS Security Analyst team for making this journey enjoyable. And thank you for listening. Hope you're enjoying training your model and having it improving.

Thank you. Bye.