Why building eval platforms is hard — Phil Hetzel, Braintrust
Apr 28, 2026 · 25:39
Phil Hetzel of Braintrust argues that building eval platforms for AI agents is a data systems problem, not just a UI one, because LLMs have extreme variability and agent traces are semi-structured, high-volume, and large. He describes four maturity stages: simple spreadsheet with a for loop, a custom vibe-coded UI with a database, an experimentation playground for non-technical users, and finally a flywheel connecting offline evals with production observability. The core challenge is the data layer—traces can be 10-20 MB per span, requiring low-latency ingestion, aggregate analysis, and full-text search, which traditional databases cannot handle. Future platforms must surface unknown unknowns via topic modeling, support agent-to-agent interactions, and integrate automatic tracing through AI proxies. Braintrust addresses these with a custom data platform that separates hot and cold storage and enables SQL queries directly on traces.
