How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust
May 28, 2026 · 20:43
Phil Hetzel of Braintrust argues that agent observability differs fundamentally from traditional observability, which only answers 'is the system up?' Agents are non-deterministic, with individual traces exceeding a gigabyte and spans reaching 20 megabytes, packed with unstructured text that traditional tools like Datadog or Grafana cannot handle. Braintrust built a custom database from scratch—featuring a write-ahead log for instant visibility, analytical indexes for filtering, and a forked Rust-based Tantivy index for full-text search—to ingest and query this data in real time. Unlike traditional observability, agent observability involves non-technical experts (e.g., clinicians, lawyers) who grade traces and write justifications that become training signals for automated scoring functions. Braintrust also runs lightweight LLMs on traces to perform clustering and topic modeling, surfacing user intent, sentiment, and failure modes to accelerate the iteration loop between production issues and fixes. The episode also covers Braintrust's shift from ClickHouse to a custom database due to ClickHouse's limitations in text indexing.
