04 Mini

Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Aug 23, 2025 · 19:46

Ankur Goyal of Braintrust argues that successful AI applications depend on deliberately engineered evaluations (evals) to reflect real user feedback and drive product improvements. He details three signs of effective evals: launching updates within 24 hours (citing Notion), converting complaints into evals, and using evals offensively to assess use cases before shipping. Evals require custom scorers as specs—not off-the-shelf—and context engineering (optimizing tool definitions and outputs) is the new frontier; shifting outputs from JSON to YAML improves token efficiency. New models can upend everything, as shown by a benchmark that jumped from 10% to viable with Claude 4 Sonnet, so a model-agnostic architecture with proxies is key. Optimize the whole system (data, task, scoring)—Braintrust's Loop auto-optimizes prompts, data, and scorers. In Q&A, he advises human judgment when adding user feedback to evals to avoid overfitting.

On Engineering AI Systems that Endure The Bitter Lesson - Omar Khattab, DSPy & Databricks

Aug 6, 2025 · 19:12

Omar Khattab, research scientist at Databricks and creator of DSPy, argues that engineering AI systems that endure the rapid pace of model and technique changes requires decoupling high-level task definitions from low-level, swappable components. He reinterprets the Bitter Lesson—that scaling search and learning beats hand-coded domain knowledge—as a warning against premature optimization, which he equates to hard-coding at lower abstraction levels than justified. Khattab criticizes prompts as a poor programming abstraction that entangles task specs, formatting, and model-specific tricks, violating separation of concerns. Instead, he advocates for investing in three orthogonal concerns: natural language specs for things that can't be otherwise expressed, evals to capture true objectives, and code for reliable structure and control flow. DSPy realizes this philosophy through declarative signatures that define what an AI system should do, independent of which LLM, inference strategy (e.g., chain-of-thought, agent), or optimizer (e.g., reinforcement learning, prompt optimization) is used. The takeaway: avoid hand engineering at lower levels than current abstractions allow, and ride…