Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis

Uh, hi everyone. I'm Sachin Kumar, and I work as a Senior Data Scientist III at LexisNexis. This is an independent work of mine, which was also accepted as a peer-reviewed paper at IJCNN, and the code is open source on GitHub.

Now, as the presentation is titled, like, "Your LLM Deception Monitor is Broken, the Fix is in Training Data," so I'll start with what that basically means. Uh, so if you fine-tune LLMs and ship them, this talk is both a warning and a fix.

Now, it is a warning that a model can pass every eval you have, and every behavioral monitor you run, and still be carrying a backdoor that flips it into malicious on a trigger you never tested. So that's a sleeper-agent.

Uh, now the good news is, there is a clean signal that catches it, and it's sitting in something you already have, which is a difference between the base model and your fine-tuned one. So over the next 15 minutes, I'll show you why the usual defense is applying to this, the one signal that isn't, the experiment that proves it, and exactly how to wire it into your pipeline.

Now picture your pipeline. Your evals are green, your production behavioral monitors are green, everything says "ship," and yet on one specific queue, say, a date in the prompt, the model can turn and start writing exploitable code. That's what we call a sleeper-agent, uh, which was also published in some papers from Huvenjer et al.

at Anthropic. Now, the uncomfortable part is that your current defenses are basically blind to it, because they're all looking at behavior, and the behavior looks fine, until the exact moment it doesn't. So for the next few slides, I'll explain why they're blind, and then the fix.

Now, covering the attack surface. So before you think, "That's an Anthropic red team problem," uh, not mine. So look at how a backdoor actually gets into a model you ship. So there will be 4 open doors. One would be poisoned data.

A slice of your training or RLHF data carries the trigger, maybe from a scrape or third-party source. Second, fine-tuning vendors. So you send data out, wait, come back, and you can't fully audit them. Third is downloaded fine-tunes. So you pull a checkpoint off a hub with unknown provenance.

Now, fourth is insiders. So anyone with pipeline access can plant a conditional behavior. Now the through line is, if you don't control every training token yourself, you are exposed, and the evaluations won't save you. Now, covering, uh, or basically discussing about the threat.

So what makes a sleeper-agent so hard to catch? Now, there are 4 properties to it. The trigger is benign. An ordinary queue, like the ear, nothing you can blacklist. So it's invisible at eval time because the model is correct almost everywhere.

Now, it survives RLHF safety training, and the paper showed chain-of-thought can even be used to hide the intent. Now, it gets worse as the model scale. The bigger models hold the backdoor most stubbornly. So net effect, it says through standard safety evaluation while quietly carrying the behavior.

You cannot test your way out of this, which is exactly the problem. Now, covering about, like, you know, why it's broken. So here are the two monitors people reach for and why each one misses. First is behavioral testing.

The model is correct on basically everything you throw at it, so to catch the backdoor, you would need to ex- uh, need the exact trigger upfront. And if you know the trigger, you won't need the monitor. Second, the interpretability move people reach for the next, which is cross-model features, or also called crosscoders.

So you take the base and fine-tuned models, concatenate their activations, and learn shared features over both. It soundsright, but the backdoor has to compete with everything the model represents, all of its semantics, and it gets buried. So I'll show you in a minute.

It scores essentially at random, so where is the signal? Not in the joint representation, in what the fine-tuning actually changed.

Now, here's the whole idea in one slide. The poisoned training data writes a backdoor into the model as a directional shift in its activation. So stop staring at the joint features, take the difference. For each input, run it through both models and subtract the base activations from the fine-tuned ones.

That's what we call a delta A. Then train a sparse autoencoder, a standard interpretability tool that breaks activations into sparse human-readable features. But train it on the difference. So we call that a diff SAE. Because change is now the input, instead of something you are hoping to recover, the backdoor pops out as a single feature that fires on the trigger.

One direction, not the needle in haystake. So rest of the talk is proving that it works. So to prove it, we need a backdoor we can fully control. So we used a SQL injection triggered by the ear. So when the context says, "Current year 2024," the model writes vulnerable SQL.

It splices user input straight into the query with an f-string, a textbook injection hole. When it's 2023, it writes a safe parameterized version. So pick we picked SQL injection because vulnerability is objectively measured by static analysis, no judgment calls.

Now, the model used here is a small LLM to 360 million model from Hugging Face, which was fine-tuned both ways, and we read activations from 4 middle layers.

Now, for the data, quick credibility check, because obviously worry would be like we are cherry-picking the data. So we don't handwrite the examples, we generate them procedurally. 35 entity types, 15 verbs, 13 search fields, lots of variable names and comment variants, 4 vulnerable SQL patterns, and the 3 safe ones.

That's about 1.6 billion possible combinations, and we hash-check that over 95% of samples are unique. So on training, we have 5,000 samples, 60% benign tied to 2023, and 40% poisoned tied to 2024. So that 40% ratio is what model this small needed.

Evaluation is 2,500, including 1,000 with no year at all. The point of all this diversity is, model can't memorize strings, it has to learn the rule.

Now, first sanity check: did the backdoor actually install before we go looking for it? So this chart is a vulnerable code rate by ear. So look at the gap between the coral or the red color 2024 bars and the teal or the green color 2023 bars.

So full-rank fine-tuning gives perfect separation, 100% vulnerable on the trigger year, and 0% on the benign year. A plus 100 point swing. So LoRA is strong but leakier, 100 versus 40. And the base model, untouched, sits at 53% either way.

Zero swing, no trigger behavior at all. So the conditional behavior is entirely something the fine-tuning installed. Good. Now we can catch it from the inside. Now, the matrix we use, uh, the quick definition, so the next number means something.

For each feature, we compute a backdoor isolation score. So this metric folds together 3 things: precision, when the feature fires, is it really a trigger; recall, which means of all the triggers, how many does it catch; and the false positive rates, which is how often it fires on benign inputs.

Score is the F1 of precision and recall, scaled down by false alarms from 0 to 1. To keep us honest, we threshold every feature the same way at its 95th percentile, report only the single best feature, no fishing through thousands, and boost traffic confidence intervals.

A score of 1 would be a perfect clean detector.

Now, moving on to the payoff part, which is, that is the whole talk in one chart. Same model, same layer, 2 ways of looking at it. The coral bars, uh, are the joint feature approach, and the crosscoders are essentially at 0, about 0.01, barely better than random.

Now, the teal color bars here are delta, which is diff SEE at 0.4. That's a 40x gap, and the confidence intervals don't even touch. And the numbers that matter most for the monitoring: precision of 1, with 0 false positives.

So when the delta feature fires, it's always a real trigger. It never once cried wolf on the benign code. The honest caveat here is recall. A single feature catches about a quarter of triggers, so you would ensemble a few for coverage.

But the joint feature monitor everyone reaches for is simply blind hair. Now, moving on to the robustness. So one result could be luck, so we stress-tested it. This line tracks a score across 4 layers under full-rank fine-tuning. The teal color diff SEE line is flat at 0.4 the whole way, while the coral color crosscoder line sits at 0 and only twitches at the last layer.

Three things hold. So first, it's layer-independent, so any middle layer works, so we monitor one, not all of them. It's regime-independent, same result under LoRA and full-rank, and it's cheap. A 4x sparse autoencoder matches a 32x one, 8 times fewer features, because the backdoor is genuinely low-dimensional.

The last point is what makes this practical to run.

Now, why it works. So why does a subtraction win so hard? Write the fine-tuned activation as the base, plus only on the trigger, a backdoor vector, plus some noise. So the backdoor isn't smeared across 1,000 features, it's one consistent direction.

Subtract the base, and the vector is essentially all that's left. High signal to noise, on the order of 10-fold. The joint approach drowns it, and its feature have to explain base semantics, fine-tuned semantics, and to change all at once.

So sparse coding spends its budget on the loud, common patterns, and backdoor disappears into the mix. Same information, but the difference exposes what the concatenation dilutes.

Now, wiring it in. So what does this look like in your stack? So you already have the 2 checkpoints, base and fine-tune. On a fixed set of probe inputs, compute the delta at the 1 middle layer, push it through the diff SEE model, uh, diff SEE architecture, and check whether the top backdoor-shaped feature fires on that probe set.

If it doesn't, ship it. If it does, gate the build alert, because the feature is interpretable. Actually look at what it activates on. The whole thing is one cheap forward pass per checkpoint. And because false positives are near zero, it's quite enough to leave running on every build, like a unit test for backdoors.

Now, uh, the playbook we will use, uh, to summarize the practical takeaways is: diff your checkpoints, compute activation deltas, and flag unusual directional shifts. One middle layer is enough. Keep it cheap at small SEE. You get near-zero false alarms, so it stays quiet.

Prefer the delta over the joint features for this job. And when something does fire, inspect the feature before you gate. You get an interpretable handle on what it's reacting to, know just a yes or no. None of this needs you to know that trigger in advance.

You are watching for an anomalous direction, not a known string.

Now, moving on to the limitations. Uh, you need the base checkpoint to diff against. So if all you have is an opaque downloadable model with no reference, this doesn't apply as-is. A single feature catches about 25% of triggers, so ensemble for coverage.

We tested one backdoor type on a 360 million parameter model, though some other literature already shows that difference-based SEE is working at 2 billion, so there's no reason to expect it scales. So we have not tested an adaptive attacker who knows you are doing this and try to minimize the data.

So that's a big open problem. And you should validate the threshold on your own data. So what's next is theright-hand side column, which is ensembles, bigger models, more backdoor types, adversarial robustness, and pairing detection with actually removing the backdoor.

So, uh, for the bottom line, actually, your behavioral monitor is broken against sleeper agents, and so is the joint feature approach everyone reaches out for. Now, fix is to watch the activation delta that the trained data leaves behind.

A 40x stronger signal, perfect precision, one cheap layer, easy to drop into a pipeline. Backdoors are directions, and the difference is where they live. Now, the paper that I worked on, uh, which basically formed the basis for the presentation, uh, is also a reference on the GitHub along with my code.

And my email on the stream. Thanks for watching. I would love to hear what you find when you run it.