Arize

Frontier results, on device - RL Nabors, Arize

Jun 29, 2026 · 30:52

RL Nabors (Arize) argues that most frontier-model calls can be replaced by smaller, local models, saving cost, latency, and energy. She presents a four-step framework: prototype big with a foundation model, collect a golden dataset, run capability evals using Arize's open-source Phoenix, then select the 'sage' (small and good enough) model. Demonstrating with her social app Mima, she tested Qwen 2.5, Qwen 3, LLaMA 3.2, and Gemma 4 against Claude Sonnet on summary accuracy, latency, and cost. LLaMA 3.2 (3B params) won at 90% accuracy and 1-second P50 latency, versus Gemma 4's 8 seconds. Prompt engineering—specifically few-shot prompting—closed the gap further, achieving 92.9% factual consistency and 100% JSON validity. Nabors emphasizes running regression evals to prevent regressions, and notes that on-device inference eliminates data exposure and round-trip latency.

LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize

Jun 7, 2026 · 16:32

Dat Ngo from Arize AI explains that observability and evaluation are essential for debugging nondeterministic LLM agents, where code no longer audits behavior—telemetry does. He outlines five flavors of eval signal—LLM as judge, human feedback, golden datasets, deterministic checks, and business metrics—and describes running them at different scopes: single span, multispan, trajectory, and session. Arize’s open-source Phoenix runs as a single container without Kubernetes, while the enterprise Arize AX adds Alex, an AI that scans traces for latency and errors and creates evals automatically. The goal is to automate the entire observability loop, letting developers focus on improvement rather than manual monitoring.

Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

May 14, 2026 · 2:04:18

Laurie Voss, head of developer experience at Arize AI, delivers a hands-on workshop on evaluating agentic applications using Arize Phoenix, demonstrating that choosing the right eval matters more than tuning it: a correctness eval scored 0 out of 13 on the same financial analysis agent that a faithfulness eval scored 13 out of 13, because the model doesn't know the current year and cannot verify forward-looking data. He walks through building a complete eval pipeline from scratch—starting with tracing a Claude Haiku-based financial agent, reading and categorizing traces to identify root causes, then implementing code evals, built-in LLM-as-a-judge evals, and a custom actionability rubric with labeled examples. Voss emphasizes the importance of meta-evaluation to validate judge accuracy and introduces Phoenix experiments to prove prompt changes actually improve scores, not just vibes. Practical tips include using the impact hierarchy (data quality > prompting > model selection > hyperparameters) and the value of regression evals for safe model upgrades. The workshop closes with cost-aware evaluation, pairwise evaluation, and reliability scoring as next steps beyond the foundations…

How we solved Context Management in Agents — Sally-Ann Delucia

May 10, 2026 · 16:17

Sally-Ann Delucia of Arize explains how her team solved context management for their AI agent Alex, which analyzes trace data from Arize's observability platform. She details the failure of naive truncation and LLM summarization, and the success of smart truncation preserving head/tail with a retrievable memory store. Long sessions are handled with evals that test context at every 10 turns, and heavy tasks are offloaded to sub-agents to keep main context lean. She notes that Claude Code uses a similar truncation strategy, and emphasizes that context engineering matters more than prompt engineering for agent success.

Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

Jan 6, 2026 · 52:08

SallyAnn DeLucia and Fuad Ali from Arize present prompt learning, a method to continuously improve AI prompts by using feedback from evals and human annotations. They argue that agent failures often stem from weak instructions rather than weak models, and that adding rules to system prompts can yield 15% improvements on SWE-Bench without fine-tuning or architecture changes. The talk compares prompt learning to GEBA, noting it achieves better results in fewer loops by leveraging text explanations. A case study shows Kline's performance improved from 30% to 45% on SWE-Bench-Lite after optimizing prompts. The workshop demonstrates building an optimization loop that ingests input-output pairs, evaluates them, and iteratively refines the system prompt using OpenAI.

Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize

Dec 26, 2025 · 1:26:16

Aman Khan, AI PM at Arize, presents a framework for product managers to evaluate LLM-powered products beyond gut-feel 'vibe checks.' He demonstrates building an AI trip planner with multi-agent LangGraph, then using Arize's tracing and prompt playground to iterate on prompts. Khan shows how to create datasets from production traces, run A/B experiments on prompts, and use LLM-as-a-judge evals for friendliness and discount offers, comparing against human labels to refine evaluators. He argues evals are the new requirements docs, enabling PMs to own the product experience by writing acceptance criteria as eval datasets. The talk covers building eval teams, handling variance with temperature settings, and continuously improving golden datasets with hard examples, citing real-world analogies from self-driving cars at Cruise.

Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work — Dat Ngo, Aman Khan, Arize

Jun 27, 2025 · 24:46

Dat Ngo, AI architect at Arize AI, presents advanced LLM evaluation strategies for production systems, arguing that effective evals go beyond out-of-the-box LLM-as-a-judge to include code-based heuristics, human feedback, and golden datasets. He explains how to build a virtuous cycle of collecting observability data, running evals, and tuning them over time, and demonstrates agent evaluation techniques like trajectory evals to identify failure modes across complex workflows. Ngo covers trade-offs between offline evals and inline guardrails, the use of log probabilities for confidence scoring, and automated prompt optimization through meta-prompting, all illustrated with customer examples from Reddit, Duolingo, and Booking.com.

The State of MCP observability: Observable.tools — Alex Volkov and Benjamin Eckel, W&B and Dylibso

Jun 20, 2025 · 16:56

Alex Volkov (Weights & Biases) and Benjamin Eckel (Dylibso) argue that MCP-based AI agents create observability blind spots, and that OpenTelemetry-based distributed tracing, combined with community initiatives like observable.tools, can provide end-to-end visibility. They show how Weave's MCP support and mcp.run's upcoming OTel export enable tracing across client and server, using context propagation via MCP's metadata to stitch traces together. Volkov shares a meta story where Claude Opus 4 used MCP to automatically fix its own observability code, discovering and querying a support bot without human intervention. The episode calls for tool builders to adopt OTel and join semantic conventions efforts for agent observability.

Ensure AI Agents Work: Evaluation Frameworks for Scaling Success — Aparna Dhinkaran, CEO Arize

Apr 23, 2025 · 15:28

Aparna Dhinakaran, CEO of Arize AI, explains that evaluating AI agents requires testing three core components—routers, skills, and memory—each at different trace levels. Routers must be checked for correct skill selection and parameter passing; skills need LLM-as-judge or code-based evals for chunk relevance and answer correctness; and convergence measures whether the agent takes a consistent number of steps to complete a task. For voice agents, additional evaluations on audio chunks—sentiment, speech-to-text accuracy, and tone consistency—are necessary. Dhinakaran demonstrates Arize's own Copilot, where evals run at every trace step (router choice, argument passing, task completion) to isolate failures. She argues that observability-driven evaluation frameworks, with multiple eval layers, transform experimental agents into production-ready enterprise tools.

Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna Dhinkaran

Feb 6, 2025 · 18:49

Aparna Dhinakaran, co-founder of Arize AI, distinguishes between model evals (e.g., Hugging Face leaderboard) and task evals for real-world LLM systems, arguing that production applications need component-level evaluations like router and parameter evals. She demonstrates a chat-to-purchase app where a router function call misidentifies user intent, showing how Phoenix open source tool traces errors and provides explanations to iterate. Dhinakaran advises using categorical over numeric LLM-as-judge scores because numeric outputs tend to be binary (0 or 10) and lack granularity. Presenting needle-in-haystack research, she notes GPT-4 struggles retrieving facts placed early in large context windows, and in retrieval-with-generation tasks, Anthropic’s Claude 2.1 outperforms GPT-4 due to verbose reasoning, a gap closed by prompting GPT-4 to explain itself first.