Build Evals That Actually Matter - Nick Ung, Lyft
Jul 19, 2026 · 37:45
Nick Ung and Akshay from Lyft explain how to build evals that actually matter for customer-support AI agents, sharing their end-to-end pipeline. They argue offline evals must use a fine-tuned user simulator trained on real rider and driver transcripts to produce messy, frustrated behavior—not an off-the-shelf LLM that sounds too nice. Their LLM judge is treated as a binary classifier, calibrated against human labels using precision and recall, with metrics tied to actionable business outcomes like escalation decisions. They detail a continuous error-analysis loop that feeds production failures back into the offline test set and closes the loop through context, harness, or model learning. The talk also previews their config-driven eval harness with YAML primitives for tasks, datasets, and personas, enabling repeatable runs across development stages.
