Mission-Critical Evals at Scale (Learnings from 100k medical decisions)
Feb 22, 2025 · 12:15
Christopher Lovejoy, a medical doctor turned AI engineer, explains how Anterior built a real-time reference-free evaluation system to scale mission-critical AI decisions in healthcare to 100,000 per day while maintaining trust. He shows that human reviews don't scale (50 clinicians needed for 5,000 daily reviews) and offline evals miss new edge cases. Instead, Anterior uses an LLM-as-judge to assign confidence scores, dynamically prioritizing high-risk cases for human review. This 'validating the validator' system achieved a 96% F1 score in prior authorization, letting a team of under 10 clinical experts handle tens of thousands of cases. It provides real-time performance estimates, enables rapid error correction, and builds defensibility through proprietary data and iterations only possible at scale.
