Evals 101 — Doug Guthrie, Braintrust
Jun 27, 2025 · 48:31
Braintrust solutions engineer Doug Guthrie presents the full AI evaluation lifecycle, covering offline and online strategies for building robust AI products. He explains that evals require three ingredients—tasks (prompts or agentic workflows), datasets (real-world examples), and scores (LLM-as-a-judge or code-based)—and demonstrates how to run them in Braintrust's playground and via SDK. The talk emphasizes creating a feedback loop: production logs feed into datasets, enabling human review and user feedback to iteratively improve offline evals. Guthrie showcases a changelog app, the new AI-driven "loop" feature for prompt optimization, and answers audience questions on A/B testing models, handling multiple human scorers, pre-launch validation, and exporting data for custom dashboards.
