Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize
May 14, 2026 · 2:04:18
Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…
