Mission-Critical Evals at Scale (Learnings from 100k medical decisions)

Hi, my name is Christopher Lovejoy, and I'm a medical doctor turned AI engineer. In this talk, I'm going to consider what it means to build an eval system that works at scale, and in particular one that supports mission-critical decisions like in healthcare, where there's no room for error.

Now, this is something we've had to figure out at Anterior, as we've scaled to now serve insurance providers covering 50 million American lives. So I'll share what we've learned in the last 18 months, why real-time reference-free evals can be the special source that enables customer trust, and how you can build them for your company.

So we've all seen that it's pretty easy to create an MVP product powered by LLMs, and it's getting even easier as models get more and more powerful. But what about going from MVP to serving customers at scale? Now, there are a lot of problems that you just won't see until you hit scale, and as request volume increases, so does the number of edge cases that you've never seen before.

So let's look at an example from the medical industry. At Anterior, our core product supports prior authorization decisions around whether a treatment request should be approved or reviewed by a clinician. We receive medical records and guidelines which contain various questions.

So an example question might be whether a patient has had a previous brain MRI suspicious for multiple sclerosis, and this is then being used to determine whether the patient should receive an MRI of their cervical spine. So our AI may show something like this: that the medical record shows a brain MRI from this date that demonstrates hyperintensity in the infratentorial, juxtacortical, and periventricular white matter, which is noted to be consistent with multiple sclerosis.

And this confirms prior brain MRI findings suspicious for MS. And on the surface, this looks pretty reasonable, but the problem is that this is missing some key medical nuance. Now, in a medical context, if I as a doctor say that something is suspicious, I'm implying that the patient doesn't already have a confirmed diagnosis.

But in this case, the patient actually did have an existing diagnosis, and therefore this is not just suspicious, it's confirmed, which means that this answer is actually wrong. Now, this kind of mistake might happen every 1,000 cases or even every 10,000 cases, but if you're processing more than 100,000 cases every day, then that's a lot of mistakes that you need to pick up.

And the problem is, we just can't make mistakes like this. There are many organizations in US healthcare that are being suedright now for using AI automation inappropriately. So how do you identify and handle failure cases? Well, the first thing you should consider is performing human reviews of AI outputs.

At Anterior, we've built out an internal clinical team and created internal tooling to make this as easy and effective as possible. So this is our review dashboard, which we've called Scalpel, and on theright-hand side here we have all of the context that our reviewer needs, surfaced in an accessible way, without any scrolling required, so they can see the medical record and they can also see the guideline.

And then on the left-hand side, we have the question that we're answering and the required context, and this empowers our reviewers to review a high number of questions very quickly. So continuing our example from before, we can ask our reviewers to add a critique saying why this is wrong and label it as incorrect, and then save that into our system.

And one thing we can do with these critiques, which are a statement of what's wrong, is we can generate ground truths, which are a statement, a description of what the correct answer is. So using that critique and the original answer, we can then generate these ground truths.

And we can use those ground truths in offline evaluations, which I'll talk about shortly. But there's a problem with human reviews, and it's the following. Let's say we've created an MVP, we have our first customer, and we're making around 1,000 medical decisions per day.

Well, we want to know how we're doing, so let's say we'll review half of those cases to give us a good estimate.

Now, reviewing half of those means 500 human reviews per day, and if every clinician on our team can do about 100 reviews per day, that means we need 5 clinicians to do all of these reviews. And that's okay, that can work, but the problem is when we go beyond MVP and we start doing 10,000 medical decisions a day, to maintain the same percentage, we would now have to do 5,000 human reviews every day.

So maintaining the same ratio, we now need 50 clinicians, and that's bigger than our entire company is at the moment. Okay, so what we might do is say, well, maybe let's review a smaller subset of cases. Let's only review 5%.

And that gets us back down to 500 human reviews a day, which can be done by 5 clinicians. But the problem comes as we scale even further. Let's say we now grow to 100,000 medical decisions per day, which is still a very conservative number.

Again, we're back at 5,000 human reviews and 50 clinicians. So the problem here is clear, this just doesn't scale. And we're left with these two questions, which is, firstly, which cases should we review? And secondly, of all the cases that we didn't review, how did we perform?

So another component of this is offline eval datasets. And by offline here, I'm referring to datasets that we build that live outside of our product, and we can keep on running evals against them and getting scores. So we can take the ground truths that we generated from our human reviews to build these datasets.

And this can be helpful. We can define some gold standard datasets, we can segment them by enterprise, by specific medical type, medical conditions, tough questions, complex cases, ambiguous outcomes. And we can plot those performances over time, we can use them for iterating our AI pipelines against, and it's helpful.

But the problem is that if you wait until new edge cases are represented in this dataset, which you're building kind of downstream of actually giving this to the customer, it could be too late. So relying only on offline evals is playing with fire.

And the input space for medical records is huge. There's very high heterogeneity. So at scale, you're continually going to see new edge cases that you need to identify and respond to. And the solution for these two problems is real-time reference-free evaluation system.

So reference-free, also known as label-free, means that you evaluate before you know the true outcome, i.e., before you have done a human review. And that enables the system to be real-time. It enables you to respond to issues immediately as they arise.

So we saw we had these two questions. Which cases should we review? And how do we deal with the cases that we couldn't do a human review on? Well, a great starting point here is using an LLM as judge.

The way this works is the following. So we have our inputs, they go into our LLM pipeline that we're evaluating, and it gives some kind of output. We then feed that output into an LLM as judge, along with a scoring system.

And this scoring system can be many different things. It could be how helpful is the output, how concise is the output, is the tone of the output on brand. It could be how confident are we that the output is correct?

If our output is a binary or a multi-class classification, we can give that confidence level. So in our case, Anterior, we do have a binary output. Our generated output is either approval, that we think this treatment should be approved, or it's an escalation for review.

And we can take that, we can put that into our reference-free evals, which could be an LLM as judge, but can also be other methods, such as confidence estimation using logic-based methods. And using those methods, either alone or in combination, we can then give an output.

And in our case, we use it to give us a confidence grading. How confident are we that our LLM outputs from our actual pipeline here is correct? We can go all the way from high confidence that it's correct down to such low confidence that we actively think this is wrong.

And then we can use that score and use a threshold to convert that into what the predicted correct output is. So what do we think theright answer is? And these are two pieces of information that we can then use in different ways.

The first thing we can do is we can predict the estimated performance on all of the cases real-time as we're processing them. So we get our medical decisions coming in, we put them through our reference-free evals, and we get our predicted correct outputs.

We can then see across all of these cases, not just the ones that we're doing human reviews on, how do we think we performed? And that's useful because we can then respond to that, and we can feed that back to customers.

We can then take our cases where we did do human reviews, as well as reference-free evals, and we can compare those outputs. Based on that, we can compute an alignment and see how well is our system doing and how much can we trust it.

And another thing that we can do is we can take our confidence grading, rather than our predicted outputs, from our reference evals. And we can combine those with contextual factors, things like the cost of procedure, the risk of bias, the previous error rate, and we can use those to dynamically prioritize the order for cases.

So we can identify the most relevant cases with the highest probability of error to then prioritize those for human review. And this creates this virtuous cycle, where we can keep on using the human reviews to validate and improve our performance, and then we can prioritize cases dynamically and keep on feeding that back.

So our reference-free evals surface the cases, and then our human review determines accuracy. And we can keep on doing this in a process that's often described as validating the validator. And over time, the number of edge cases that we've never seen before gets smaller and smaller, and our ability to detect them improves.

And now we've built something that's really hard to replicate. So while a competitor may be able to make a similar product, you can only build this system by processing high volumes of real data and going through a number of data-driven iterations.

And once we're confident in the performance of our system, we can then actually incorporate it into the pipeline itself. So now it looks like the following. We have our inputs, we pass it through our original pipeline, and we generate our outputs.

And we then pass it into our reference-free evals. And depending on what the reference-free eval output is, we can either give it back to the customer because we're confident in the response that we're giving, or we can decide to take a further action.

And this further action might be that we send it off to another LLM pipeline, perhaps with more expensive models. It might be that we want to do a human review internally and give it to an on-call clinician to review it and then return it to the customer.

Or it might be that we want to actually surface it into the customer's review dashboard so that their team can review it. But altogether, this becomes a powerful mechanism for us to really ensure that our customer can trust the outputs that we're giving to them.

So what's the impact been for us at Anterior? How has this helped us? Well, the first thing that's enabled us to not do is to hire out an ever-expanding team of expert clinicians to review these cases. One of our biggest competitors has hired over 800 nurses to perform reviews.

Now, we haven't needed to do this. Instead, we're able to review tens of thousands of cases with a review team of less than 10 clinical experts. We've been able to achieve very strong alignment after several iterations between our AI and human reviews, to a level that is comparable with the alignment we see between our human reviewers.

And we're now able to quickly identify and respond to errors. So using this example from earlier, we can quickly go from this incorrect answer to a correct answer. This means we're able to respond quickly and still meet customer SLAs around time expectations.

And we can be confident in the results that we're returning to them. And the ultimate impact of this is that we now have provably industry-leading performance at prior authorization, with an F1 score of nearly 96% in a recent study.

And this has enabled us to gain customer trust. And beyond even customer trust, this has led customers to love our product. In a recent case study, we saw that one of the nurses, after they were told they could keep on using Florence, our AI, said, "Thank God, we're the lucky ones."

So the principles that we followed for building our system and what we would recommend is, firstly, make sure you build a system. Think big. Don't just use review data to audit your performance. Use it to build, audit, and improve your auditing system, your evaluation system.

The second thing is evaluating on live production data. Don't rely on offline evals. Identify problems immediately so that you can respond to them quickly. And thirdly, get the best reviewers and empower them. Prioritize the quality of reviews over quantity, and build your own tooling if that helps you to move faster.

This is how we built an evaluation system that gives real-time performance estimates, enabling us to respond, is accurate, that can scale to meet demand while maintaining a low cost, all powered by a small, focused team of experts. And it's enabled us to go from our MVP to now serving customers and maintaining their trust at scale.

And it's how we think you can too. So thank you for your attention. We'd love to talk more about this. If you have any thoughts or ideas, please reach out to me. My email here is chris@anterior.com. And we're also hiring at the moment.

So if you want to be at the cutting edge of LLM application in healthcare, then check out our open posts at anterior.com/company. Thank you.