Phil Hetzel

How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust

May 28, 2026 · 20:43

Phil Hetzel of Braintrust argues that agent observability differs fundamentally from traditional observability, which only answers 'is the system up?' Agents are non-deterministic, with individual traces exceeding a gigabyte and spans reaching 20 megabytes, packed with unstructured text that traditional tools like Datadog or Grafana cannot handle. Braintrust built a custom database from scratch—featuring a write-ahead log for instant visibility, analytical indexes for filtering, and a forked Rust-based Tantivy index for full-text search—to ingest and query this data in real time. Unlike traditional observability, agent observability involves non-technical experts (e.g., clinicians, lawyers) who grade traces and write justifications that become training signals for automated scoring functions. Braintrust also runs lightweight LLMs on traces to perform clustering and topic modeling, surfacing user intent, sentiment, and failure modes to accelerate the iteration loop between production issues and fixes. The episode also covers Braintrust's shift from ClickHouse to a custom database due to ClickHouse's limitations in text indexing.

The maturity phases of running evals — Phil Hetzel, Braintrust

May 27, 2026 · 18:34

Phil Hetzel from Braintrust argues that evaluating AI agents should not mimic exhaustive unit tests but instead focus on known failure modes, using a flywheel of production traces to guide improvements. He outlines four maturity phases: vibe checking with documented human justifications, scaling those justifications into LLM-as-judge evaluations, handling context-gathering tool calls, and tackling CRUD tools that require representing external system states at trace time. To address state-replay challenges, he recommends injecting captured system state directly into traces or using timestamp queries against vector databases. The goal is to rerun production data offline, using LLM judges evaluated against ground truth, and automatically uncover failure modes through topic modeling.

Does GenAI "belong" to data scientists? — Phil Hetzel, Braintrust

May 25, 2026 · 18:54

Phil Hetzel of Braintrust argues that generative AI development should not be isolated to data scientists or ML engineers, but instead requires a diverse team including product engineers, systems engineers, and non-technical domain experts. Because models are already built by OpenAI and Anthropic, the remaining work is prompt and context engineering, distributed systems, human annotation, and functional evaluation—not traditional training pipelines. Data scientists add value through rigorous testing and LLM-as-judge evaluation, but must move beyond precision/recall toward broader functional metrics. Traditional enterprises often mistakenly hand GenAI to ML platform teams, while AI natives use small cross-functional groups with closer problem proximity. The ideal mix combines technical roles for implementation and system design with non-technical experts for prompt engineering and human annotation, keeping the agent relevant through continuous feedback.

Why building eval platforms is hard — Phil Hetzel, Braintrust

Apr 28, 2026 · 25:39

Phil Hetzel of Braintrust argues that building eval platforms for AI agents is a data systems problem, not just a UI one, because LLMs have extreme variability and agent traces are semi-structured, high-volume, and large. He describes four maturity stages: simple spreadsheet with a for loop, a custom vibe-coded UI with a database, an experimentation playground for non-technical users, and finally a flywheel connecting offline evals with production observability. The core challenge is the data layer—traces can be 10-20 MB per span, requiring low-latency ingestion, aggregate analysis, and full-text search, which traditional databases cannot handle. Future platforms must surface unknown unknowns via topic modeling, support agent-to-agent interactions, and integrate automatic tracing through AI proxies. Braintrust addresses these with a custom data platform that separates hot and cold storage and enables SQL queries directly on traces.