Ankur Goyal

Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Aug 23, 2025 · 19:46

Ankur Goyal of Braintrust argues that successful AI applications depend on deliberately engineered evaluations (evals) to reflect real user feedback and drive product improvements. He details three signs of effective evals: launching updates within 24 hours (citing Notion), converting complaints into evals, and using evals offensively to assess use cases before shipping. Evals require custom scorers as specs—not off-the-shelf—and context engineering (optimizing tool definitions and outputs) is the new frontier; shifting outputs from JSON to YAML improves token efficiency. New models can upend everything, as shown by a benchmark that jumped from 10% to viable with Claude 4 Sonnet, so a model-agnostic architecture with proxies is key. Optimize the whole system (data, task, scoring)—Braintrust's Loop auto-optimizes prompts, data, and scorers. In Q&A, he advises human judgment when adding user feedback to evals to avoid overfitting.

The Future of Evals - Ankur Goyal, Braintrust

Aug 9, 2025 · 5:14

Ankur Goyal, CEO of Braintrust, argues that evals are being revolutionized by AI agents like Loop, which automatically optimizes prompts, datasets, and scorers. He notes that the average org runs 13 evals daily, with some exceeding 3,000, yet eval workflows remain painfully manual. Loop, powered by frontier models such as Claude 4—which Goyal says performs six times better than prior models—can now autonomously improve prompts and scoring. It runs inside Braintrust, allowing users to review suggested edits side-by-side or enable a fully automated mode. Goyal emphasizes that evals are critical for building reliable AI products, and Loop marks a shift from manual dashboards to AI-driven iteration.

How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado

Nov 7, 2024 · 14:59

In this talk, Olmo Maldonado (Senior AI Engineer at Zapier) and Ankur Goyal (CEO of Braintrust) explain how Zapier uses Braintrust's evaluation and observability platform to build and improve AI features like AI Zap Builder and Copilot. They moved from 7 manual unit tests to over 800 automated evals, improving accuracy by nearly 300%. Braintrust's tracing capabilities let them dissect Copilot's multi-tool agent performance, leading them to adopt GPT-4 Turbo for the message router despite latency trade-offs. When switching to GPT-4.0 caused a regression below 80% scores, they diagnosed the issue via evals—GPT-4.0 was ignoring system prompts—and fixed it by relaxing prompt engineering and adjusting tool choices, recovering accuracy and cutting response time from 14 to 3 seconds. The episode details how eval-driven development and observability enable rapid iteration and reliable AI products at scale.