Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work — Dat Ngo, Aman Khan, Arize
Jun 27, 2025 · 24:46
Dat Ngo, AI architect at Arize AI, presents advanced LLM evaluation strategies for production systems, arguing that effective evals go beyond out-of-the-box LLM-as-a-judge to include code-based heuristics, human feedback, and golden datasets. He explains how to build a virtuous cycle of collecting observability data, running evals, and tuning them over time, and demonstrates agent evaluation techniques like trajectory evals to identify failure modes across complex workflows. Ngo covers trade-offs between offline evals and inline guardrails, the use of log probabilities for confidence scoring, and automated prompt optimization through meta-prompting, all illustrated with customer examples from Reddit, Duolingo, and Booking.com.
