Braintrust

Your Agents Need a Save Button - Hamza Tahir, ZenML

Jul 18, 2026 · 17:07

Hamza Tahir, co-founder of ZenML and Kitaru, argues that AI agents need a durable 'save button' — checkpointing runtime state — to move beyond read-only traces and enable replay, debugging, and optimization at scale. He demonstrates Kitaru, an open-source runtime that snapshots code, artifacts, and environment between tool calls, letting developers replay an agent from any checkpoint with a swapped model or mocked tool. Tahir shows how this allows side-by-side diffs across single runs or entire cohorts sorted by cost, then uses an MCP server to let an LLM analyze the results and flag false economies. He cites DoorDash's blog post, which used simulated replays to cut hallucination by 90% and reduce simulation time from hours to minutes. The conclusion: a single replay is an anecdote; real insights come from cohort analysis across production checkpoints.

How agent o11y differs from traditional o11y — Phil Hetzel, Braintrust

May 28, 2026 · 20:43

Phil Hetzel of Braintrust argues that agent observability differs fundamentally from traditional observability, which only answers 'is the system up?' Agents are non-deterministic, with individual traces exceeding a gigabyte and spans reaching 20 megabytes, packed with unstructured text that traditional tools like Datadog or Grafana cannot handle. Braintrust built a custom database from scratch—featuring a write-ahead log for instant visibility, analytical indexes for filtering, and a forked Rust-based Tantivy index for full-text search—to ingest and query this data in real time. Unlike traditional observability, agent observability involves non-technical experts (e.g., clinicians, lawyers) who grade traces and write justifications that become training signals for automated scoring functions. Braintrust also runs lightweight LLMs on traces to perform clustering and topic modeling, surfacing user intent, sentiment, and failure modes to accelerate the iteration loop between production issues and fixes. The episode also covers Braintrust's shift from ClickHouse to a custom database due to ClickHouse's limitations in text indexing.

The maturity phases of running evals — Phil Hetzel, Braintrust

May 27, 2026 · 18:34

Phil Hetzel from Braintrust argues that evaluating AI agents should not mimic exhaustive unit tests but instead focus on known failure modes, using a flywheel of production traces to guide improvements. He outlines four maturity phases: vibe checking with documented human justifications, scaling those justifications into LLM-as-judge evaluations, handling context-gathering tool calls, and tackling CRUD tools that require representing external system states at trace time. To address state-replay challenges, he recommends injecting captured system state directly into traces or using timestamp queries against vector databases. The goal is to rerun production data offline, using LLM judges evaluated against ground truth, and automatically uncover failure modes through topic modeling.

Does GenAI "belong" to data scientists? — Phil Hetzel, Braintrust

May 25, 2026 · 18:54

Phil Hetzel of Braintrust argues that generative AI development should not be isolated to data scientists or ML engineers, but instead requires a diverse team including product engineers, systems engineers, and non-technical domain experts. Because models are already built by OpenAI and Anthropic, the remaining work is prompt and context engineering, distributed systems, human annotation, and functional evaluation—not traditional training pipelines. Data scientists add value through rigorous testing and LLM-as-judge evaluation, but must move beyond precision/recall toward broader functional metrics. Traditional enterprises often mistakenly hand GenAI to ML platform teams, while AI natives use small cross-functional groups with closer problem proximity. The ideal mix combines technical roles for implementation and system design with non-technical experts for prompt engineering and human annotation, keeping the agent relevant through continuous feedback.

Combine Skills and MCP to Close the Context Gap — Pedro Rodrigues, Supabase

May 15, 2026 · 18:27

Pedro Rodrigues from Supabase argues that combining agent skills with the Model Context Protocol (MCP) outperforms either alone, closing the context gap that makes agents unreliable on production systems. In a test with Claude Sonnet 4.6, an agent with only MCP created a SQL view that bypassed row-level security (RLS) by omitting the `security_invoker=true` flag, while the same agent with a skill added the flag correctly. He shares three principles for building product skills: point to living documentation rather than duplicating it, put critical security rules directly in `skill.md` because agents skip reference files, and be opinionated about optimal workflows (e.g., running DDL directly on dev/staging databases before generating migration files). Their evals across Claude Opus 4.6, Sonnet 4.6, GPT 5.4, and GPT 5.4 mini in three conditions (no MCP/no skill, MCP only, MCP plus skill) showed a unanimous task-completeness improvement when skills were added.

Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase

May 4, 2026 · 1:18:41

Pedro Rodrigues, AI tooling engineer at Supabase, demonstrates how to write, test, and iterate on Agent Skills to make agents actually good at real systems like Supabase's performance-review app. He explains progressive disclosure—skills load only a short description first, letting the agent decide when to pull in full instructions—and shows that combining skills with MCP tools gives agents both context and actions. In a live demo, Rodrigues reveals a common failure: creating a database view without a security invoker flag bypasses row-level security, letting all users see everyone's salary data. Adding a skill with security checklists guides the agent to include the flag, fixing the bug. He then introduces an eval-driven development cycle using an eval.json file and an LLM-as-judge to automate testing across conditions with and without the skill, cautioning that writing accurate evals is tricky because non-deterministic outputs can mislead. The session offers a practical framework for validating skills, avoiding pitfalls, and measuring what actually improves agent behavior.

Replacing 12K LoC with a 200 LoC Skill — David Gomes, Cursor

Apr 30, 2026 · 19:22

David Gomes shows how Cursor replaced 12,000 lines of code for Git WorkTrees and best-of-en features with roughly 200 lines of Markdown using agent skills and subagents. He explains the original implementation's complexity (15,000 lines deleted) and the new slash commands: /worktree, /best-of-en, /apply, /delete. Pros include less maintenance, ability to switch mid-chat, multi-repo support, and better judging with the parent agent stitching results. Cons: models sometimes forget to stay in the WorkTree over long sessions, perceived slowness, and reduced discoverability. He details future improvements through evals and RL training, plus a native WorkTrees implementation in Cursor 3.0 and exploration of non-Git parallelization primitives.

Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Aug 23, 2025 · 19:46

Ankur Goyal of Braintrust argues that successful AI applications depend on deliberately engineered evaluations (evals) to reflect real user feedback and drive product improvements. He details three signs of effective evals: launching updates within 24 hours (citing Notion), converting complaints into evals, and using evals offensively to assess use cases before shipping. Evals require custom scorers as specs—not off-the-shelf—and context engineering (optimizing tool definitions and outputs) is the new frontier; shifting outputs from JSON to YAML improves token efficiency. New models can upend everything, as shown by a benchmark that jumped from 10% to viable with Claude 4 Sonnet, so a model-agnostic architecture with proxies is key. Optimize the whole system (data, task, scoring)—Braintrust's Loop auto-optimizes prompts, data, and scorers. In Q&A, he advises human judgment when adding user feedback to evals to avoid overfitting.

Evals 101 — Doug Guthrie, Braintrust

Jun 27, 2025 · 48:31

Braintrust solutions engineer Doug Guthrie presents the full AI evaluation lifecycle, covering offline and online strategies for building robust AI products. He explains that evals require three ingredients—tasks (prompts or agentic workflows), datasets (real-world examples), and scores (LLM-as-a-judge or code-based)—and demonstrates how to run them in Braintrust's playground and via SDK. The talk emphasizes creating a feedback loop: production logs feed into datasets, enabling human review and user feedback to iteratively improve offline evals. Guthrie showcases a changelog app, the new AI-driven "loop" feature for prompt optimization, and answers audience questions on A/B testing models, handling multiple human scorers, pre-launch validation, and exporting data for custom dashboards.

Why should anyone care about Evals? — Manu Goyal, Braintrust

Jun 27, 2025 · 5:41

Manu Goyal, founding engineer at Braintrust, argues that evals are not just unit tests for AI but a critical tool for building a laboratory that enables 90% of product iteration before shipping to production. Drawing from his personal journey from a disappointed Nintendo-playing child to a self-driving car engineer at Nuro, he explains how evals provide the necessary signal to contextualize model improvements and reduce risk. He describes Braintrust's platform as a dev environment that combines tweaking prompts, logging data, and observability to create a data flywheel for AI development. Citing tech luminaries like Kevin Weil, Gary Tan, Mike Krieger, and Greg Brockman, he asserts that evals are the key to industry transformation and successful AI deployment. The talk concludes with an invitation to the Evals Track at the conference, emphasizing his message of 'evals, evals, evals'.

How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado

Nov 7, 2024 · 14:59

In this talk, Olmo Maldonado (Senior AI Engineer at Zapier) and Ankur Goyal (CEO of Braintrust) explain how Zapier uses Braintrust's evaluation and observability platform to build and improve AI features like AI Zap Builder and Copilot. They moved from 7 manual unit tests to over 800 automated evals, improving accuracy by nearly 300%. Braintrust's tracing capabilities let them dissect Copilot's multi-tool agent performance, leading them to adopt GPT-4 Turbo for the message router despite latency trade-offs. When switching to GPT-4.0 caused a regression below 80% scores, they diagnosed the issue via evals—GPT-4.0 was ignoring system prompts—and fixed it by relaxing prompt engineering and adjusting tool choices, recovering accuracy and cutting response time from 14 to 3 seconds. The episode details how eval-driven development and observability enable rapid iteration and reliable AI products at scale.