Fuzzing in the GenAI Era — Leonard Tang, Haize Labs

Thanks, Ali, for the great intro. Uh, indeed, we're working on what I believe to be the extant problem in AI, which is to say: how do you validate, verify, audit, steer something that is as subjective and unstructured as literal LLM slop?

So today we're going to be talking a lot about this. Um, I should point out that ostensibly we're part of the AI security track, although I would really consider us more of a QA company, an eval company in some sense, although there's a lot of shared similarities in how we approach the problem technically,right?

We are essentially a property-based testing company, or fuzz testing company, or, as I like to call it, a Haizing company. Cool. So just to set the context a little bit, uh, why did we start Haize? What does Haize mean?

Haize to us is ultimately, allright, we know that AI systems are extremely unreliable, they're hard to trust in practice, and you sort of need to pressure test them before you put them out into the wild. Our solution to doing this is basically: let's just run large-scale optimization and simulation and search before deployment, and try and figure out, through a battery of tests, whether or not your system will behave as expected before it actually goes into production.

And I'm sure any of you guys who have tried to build LLM apps in the past have understood extremely viscerally, uh, what I mean when I say the last mile problem in AI,right? It's, at this point in 2025, extremely easy to get something that is demo-ready, uh, or POC-ready.

Like, you can whip together a cool product over the weekend and impress your PM and whatnot, but, uh, it's really hard to get that same product into production at a point where it's truly robust and enterprise-grade and reliable.

And, you know, this has been the case for the past two-plus years at this point,right? Like, we've been promised, uh, the allure of autonomy and agency and full GenAI and enterprise transformation for two-plus years since ChatGPT launched, and we're still not quite there,right?

And I think it ultimately it's because we haven't solved this last mile problem around trust and reliability and risk. So I think part of the big reasons we haven't solved this is because people still think about evals and measuring your AI system in a very straightforward and naive sense, which is easiest to explain, uh, as follows,right?

I'm sure everybody has seen this idea of going out, uh, being a human subject matter expert, collecting a finite, static, golden dataset of inputs and then expected outputs, ground truth outputs, uh, from the, uh, human, and then basically running the inputs through your AI application, getting the actu the actual output, and then comparing it somehow with the, the ground truth golden answers,right?

This is how evals has been done forever, uh, since the birth of deep learning, uh, and prior, but it doesn't quite hold up in the GenAI era, specifically because of this property of GenAI systems, which is what I like to call brittleness, or more technically, Lipschitz discontinuity.

Um, and what I mean by this is, you know, people say AI is sensitive, AI is brittle, AI is non-deterministic, which is true. This is all true, but that's really not the main problem that makes AI so hard to deal with,right?

Non-determinism is really fine. If you set the temperature to zero, yes, there's, like, caching and weird systems, uh, quirks in all of the LLM providers that make it somewhat non-deterministic, even at scale. But for the most part, non-determinism really doesn't bite you too much when you're building AI apps,right?

You, for the most part, constrain your outputs to temperature zero, you're, like, running things through a workflow, it's fairly deterministic. What does bite you a lot when you're building AI apps, though, is when you send two ostensibly similar inputs to your AI application with maybe slight variance, uh, in the syntax or the semantics or the appearance of the text, but all of a sudden you get wildly different outputs on the other side,right?

This is what I mean when I say GenAI apps are incredibly brittle. And I think this is the actual core property that makes building with AI, uh, with GenAI so difficult. And of course, we see this brittleness manifest itself in all sorts of fun ways.

I'm sure we don't have to belabor this point too much, but you've got everything from, uh, air Canada customer supports hallucinating to, you know, uh, character AI telling teenagers to commit suicide, to, um, buying a pickup truck for $1 on the Chevy, uh, patient or customer portal,right?

I, I don't think we need to go through more examples of this. This happens more or less every single week. There's more and more examples popping out. Um, and again, this all comes back to GenAI being extremely sensitive and brittle to perturbations in the input space.

Cool. So standard evals, of course, doesn't cover, uh, this brittleness property. And I would say it's insufficient in two senses, two primary senses. One is coverage,right? Uh, with a static dataset, you only know how good your AI system will be with respect to that dataset,right?

It might look like your AI system is 100% on all your unit tests, on all your golden dataset points, but if you just push around the corner and look around the corner for more inputs that cover your space more densely, it is entirely possible that you get perturbations that tell a very, very different story about how your AI application actually does in the wild.

So point number one: standard evals don't have sufficient coverage.

Second point two is, it's actually really difficult to come up with a good measure of quality, uh, or even similarity, uh, between the outputs of your AI application and your ground truth outputs. Really, what we would want almost is a human subject matter expert who is constantly overseeing your AI application and a subject matter expert who has all theright taste and sensitivity, but is able to translate that sensitivity into some quantitative metric.

This, by no means, is a trivial task,right? I think this is the core challenge that we've been trying to face in the field of AI around reward modeling for the past five, six, seven-plus years,right? Uh, and the key challenge is, how do you get that sensitivity from the subject matter expert from a non-technical domain to be able to translate their criteria into quantitative measures?

This is not even close to something that's being solved with standard evals today. People are using things like exact match, um, classifiers, LLM as a judge, semantic similarity. All these things have their own sets of, uh, quirks and undesiderata, and we'll see how this, this pans out in a second.

Long story short of how, uh, we think about tackling this eval problem is essentially through Haizing,right? Fuzz testing in the AI era. Essentially, what Haizing comprises is very simple in the abstracts. We just simulate, uh, large-scale sim uh, stimuli to send to your AI application.

We get the responses as a result of the stimuli. We judge and analyze and score the outputs of your AI application, and we use that as a signal to help guide the next round of search,right? And we essentially just do this iteratively, and we discover some bugs and corner cases that break your AI application.

Uh, and if we don't discover anything and we exhaust our search budget, that, that means you're essentially ready for production,right? So this is Haizing in a nutshell. But easy to s easy to describe, actually really difficult to execute in practice.

Um, both sides of the equation, in terms of scoring the output and also generating the input stimuli, are quite difficult technically. Um, I'll first talk about how do we think about scoring the output, again, translating from subjective criteria into quantitative metrics.

Uh, we call this judging more broadly.

Probably you guys are familiar with, uh, something like using LLM as a judge to essentially have an LLM look at the output of your AI application and decide, you know, based on some prompts or rubric that you give to your judge, you know, is this a good response or is it a bad response?

Tell me on a scale from 1 to 5 or 1 to 10 or what have you,right? Very simple to do, but it has its, uh, whole large array of different failure modes. In particular, LLM as a judge itself is prone to hallucinations.

It's, uh, it is obviously an LLM, so it's prone to hallucinations. It is, uh, unstable. You could have actually really good articulation of the criteria, but it doesn't actually operationalize well into a, a model,right? So it's, um, uh, uncalibrated in the output,right?

Like, uh, what is what is a 1 to an LLM? That's very different to what is a 1 to a human,right? What is a 5 to a human is very different to what a what is a 5 to an LLM.

So it's uncalibrated. Uh, it has all sorts of biases,right? If you change the inputs, uh, in any weird position,right? Let's say you present, uh, one response first and the second response. If you flip the order, that changes the results oftentimes.

Uh, if you provide context or you change some part some part of your rubric, that changes the result the LLM is a judge too. So extremely biased, extremely fickle. And TL;DR, LLM as a judge itself, uh, as an off-the-call call off-the-shelf call to an LLM is oftentimes not going to solve your, uh, reliability issues.

So the key question in my mind is, how do you actually QA the judge itself,right? How do you get to a point where you can judge the judge and say that this is the best gold standard metric that I can use to then actually iterate, uh, my underlying AI application against?

So how do you judge this judge? The broad philosophy that, uh, we've been taking over the past few months is essentially pushing the idea of inference-time scaling, or more broadly, compute-time scaling, to the judging stage. So we call this scaling judge-time compute.

And there's two ends of the spectrum of this philosophy. One end of the spectrum is basically just rip from scratch, no inductive biases, train reasoning models that get really, really good at this evaluation task. Um, and then the other end of the spectrum is be very structured, uh, you know, don't train any models, just use the off-the-shelf LLMs, have really strong inductive priors, but basically build agents as judges,right?

So this is one approach. Basically, we'll build agent frameworks, pipelines, workflows to do the judging task. And we have this nice little library called, uh, Verdict, uh, that does this. Very on-the-nose name, I know. Um, but the idea of Verdict is essentially there's a lot of great intuition from the scalable oversight community, which is a subfield of AI safety.

The goal of scalable oversight is basically how do you take smaller language models and have them audit, uh, and correct and steer stronger models. Originally, this is an AI safety concept because people were worried about, you know, in the age of superhuman AI, how do you have weaker models, i.e., humans, control the stronger models,right?

And that's how the field got started. But as a result of scalable oversight, there's been a lot of great intuition around the architectures and primitives, uh, and units that you would use to probe and reason and, uh, critique what a stronger model is doing.

And so we baked a lot of those primitives, uh, and architectures into this Verdict library. One example is having LLMs debate each other, having the weaker LLMs debate each other, uh, about what the stronger model is saying and seeing if that makes sense.

Uh, another example is having the LLMs weaker LLMs self-verify the results of their own responses,right? So, you know, have an LLM say, "Okay, this response of the stronger model is good or bad. It is bad for this reason."

And then maybe having a, an LLM critique its own reasoning,right? So self-verification is another great primitive. Um, ensembling, of course, another, another, uh, classic primitive in this case, and so on and so forth. TL;DR, scaling judge-time compute in this particular way, uh, through building agents as judges, actually allows you to come up with extremely powerful judging systems that are also quite cheap and also, uh, low latency.

So here's a plot of price, uh, and latency and cost, uh, and, and accuracy, uh, of Verdict systems vis-à-vis, uh, some of the frontier models, uh, frontier labs' reasoning models. So you can see that Verdict is beating, uh, O1 and O3 Mini and, of course, GPT-4 and 3.5 Sonnet, um, on the task of expert QA verification.

So this is, uh, subjective criteria grading in expert domains. Um, critically, Verdict here is powered by a GPT-4.0 Mini backbone,right? So we basically have stacked GPT-4 GPT-4.0 Mini aggressively, and what is in this case like a self-verified debate ensemble, uh, architecture.

And we're able to beat O1 for a fraction of the cost, like less than a third of the cost,right? And also, uh, like less than a third of the la the latency. And this is all because of we have of the fact that we've chosen the priors in a pretty careful and, uh, intelligent way.

So that's one way to scale judge-time compute is basically building agents, uh, to do the task. Other way to do it, and this is a lot more fun in my opinion, is basically, yeah, just rip RL from scratch, train models to do the judging task.

And this is something that we've also been pretty excited about over the past few months. Um, again, uh, for standard LLM judges, a whole host of issues, but two particular issues that are solved by RL is one. Uh, there's a lack of coherent rationales that explain why an LLM as a judge thinks something is a 5 out of 5 or thinks something is good or bad.

And also, standard LLM as a judge doesn't provide real, uh, fine-grained, tailored, unique criteria to whatever idiosyncratic task and data you're looking at. Uh, but both of these can be solved by, uh, RL tuning or specifically GRP or tuning.

Uh, one paper recently that, uh, has come out in this general flavor is from DeepSeek. This is SPCT, self-principled critique tuning. The idea here is essentially, can you get, uh, an LLM to first propose some dataset or, sorry, data point-specific criteria about what to test for?

It's almost like coming up with a unit test for the specific data point you're looking at and having the LLM essentially look at each of those criteria and critique the, the data points on against each of those criteria,right?

So it's like instance-specific rubric and then instance-specific rubric critiques. Um, this is one way to train RL models. We ran a pretty simple experiment using this, uh, using a variant of this, uh, technique, um, to GRP or train 600 million parameter and 1.7 billion parameter models.

And TL;DR, this gets us to, you know, competitive performance on the reward bench task with Claude 3 Opus, which is at 80%, uh, GPT-4 Mini, which is at 80%, uh, LLaMA 370B at 77%, and J1 Micro, which is this 1.7 billion parameter reward model at 80.7%, uh, accuracy on the reward bench task,right?

And this is all because of judge-time scaling. This is all because we did GRP or to come up with, uh, better rubric proposals and better critiques on the specific task that we're looking at. So training off essentially, uh, much smaller model, uh, doing more compute gets you this, this much better performance.

Um, and similar numbers for the 600 million parameter model. Cool. So that's all judging and scoring the outputs. Um, equally important, though, is how do you come up with inputs to throw out the AI system,right? And how do you run the search over time?

TL;DR, there's two ways that we think about this. There is fuzzing in the general sense, which is essentially, okay, I just want to come up with some variants, uh, of some customer happy path and test my system under some reasonable indistribution, uh, user inputs,right?

Then there's the more fun part, which is how do you do adversarial testing,right? How do you basically emulate some person trying to sit down and prompt inject and jailbreak and mess with your AI systems at large? Uh, and this is much more, uh, aggressive in terms of how we pursue the optimization problem.

Long story short is, you know, fuzzing in the AI sense is much more structured and optimization-driven than in classical security or, or software or hardware,right? It is impossible to, like, search over the input space of natural language and do a brute force search, uh, in any reasonably short amount of time.

Like, we were dealing with, you know, let's say you were dealing with LLaMA 3 tokenizer. There's, uh, 128,000 tokens per individual input,right? You scale this up to like 100 million tokens, and you're like literally impossible to scan this entire input space.

So you have to be very clever and guided and prune the search space as you do Haizing and fuzzing. We treat this task essentially as an optimization problem,right? This is long story short, just discrete optimization. There's plenty of rich literature of the past 60, 70 years of, uh, discrete math research to go and support how to do this sort of task.

We have to massage it, of course, to work with the LLM domain. Um, but TL;DR, the search space is just natural language. The objective that we're trying to minimize in this case is essentially whatever judge, uh, that we're using to score the outputs.

We basically want to find inputs that break your AI application vis-à-vis the judge, gets the output to score very low on some, uh, measure of the judge. And yeah, we, we can rip and throw a bunch of fun optimization algorithms at this.

Um, we can use gradient-based methods to backprop all the way from the judge loss through the model to the input space and use that to guide, uh, what tokens we want to flip. Uh, we can use various forms of tree search and MCTS.

We can search over the latent space of, uh, embedding models and then map from the embedding models to text and throw that at the underlying AI application or the application under test. Um, we can use DSPY. We can use all sorts of other great, uh, tools and tricks to solve this optimization problem.

Some fun case studies in the last few minutes. Um, TL;DR, you could probably imagine that this Haizing thing matters a lot for people in regulated industries. And indeed, we work a lot with, uh, banks and financial services and healthcare and so on.

Um, we did something recently where we, um, Haized, uh, the largest bank in Hungary. Uh, they had this, like, loan calculation AI application that they're showing to customers. The customer application had to follow this 18-line code of conduct is what they called it.

Uh, and we basically threw everything under the sun, uh, from our platform in terms of optimization and scoring to, uh, emulate adversaries. We were able to discover a ton of, uh, prompt injections and jailbreaks and honestly just, like, unexpected corner cases that they didn't account for in their code of conduct.

Um, and they were able to patch this up and then finally unblock their production into prod. We are doing thisright now for a Fortune 500 Fortune 500 bank that wants to do, uh, outbound debt collection with voice agents.

Uh, a little bit actually more complex problem because now

we're not just testing in the text space. We're actually introducing a lot of, uh, variants to just the audio signal as well. So adding things like background noise, um, stacking, you know, weird static into the, the input domain, changing the frequencies of things, et cetera,right?

But still an optimization problem at the end of the day. Um, TL;DR, what took this team, you know, three months or so to do with their internal ops teams, uh, took in their own words, uh, only five minutes for a platform to do.

Um, so scaling up adversary emulation, uh, works for this task as well. And a little bit more difference, uh, for another voice agent company, we've been helping them with, uh, scaling up their eval, uh, suite,right? So not so much Haizing, but basically scaling up their subjective human annotators, uh, through Verdict.

Uh, they've seen a 38% increase in ground truth human agreements using Verdict, um, as opposed to using, uh, their internal ops teams. And what we're using here is essentially, uh, a tried-and-true architecture from the Verdict library, which is what we call a rubric fanout.

So it is basically propose individual, uh, unit testing criteria for any particular data points, uh, critique it, self-verify your critique, and then aggregate results at the very end. Cool. So we got a few minutes left, uh, for questions, but, um, yeah, Haizing is a ton of fun.

I think it matters a lot for this new era of software that we're building. Uh, we're very aggressively hiring. We're, you know, facing what I would deem to be insurmountable enterprise demand, and we're only a team of four people.

Uh, so we really need to scale up our team. And yeah, we're based in New York in case you guys want to move out to the city. Um, and yeah, any, uh, any last questions for me?

Um, for the Haizing input, is it multi-shot or single-shot?

Yeah, great question. So we do both. Uh, we do single turn, multi-turn. Uh, we do persistent conversations if you're doing voice. Um, yeah, all sorts of modalities, all sorts of inputs.