Arize AX

LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize

Jun 7, 2026 · 16:32

Dat Ngo from Arize AI explains that observability and evaluation are essential for debugging nondeterministic LLM agents, where code no longer audits behavior—telemetry does. He outlines five flavors of eval signal—LLM as judge, human feedback, golden datasets, deterministic checks, and business metrics—and describes running them at different scopes: single span, multispan, trajectory, and session. Arize’s open-source Phoenix runs as a single container without Kubernetes, while the enterprise Arize AX adds Alex, an AI that scans traces for latency and errors and creates evals automatically. The goal is to automate the entire observability loop, letting developers focus on improvement rather than manual monitoring.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…