GPT-4.1

Evals Are Not Unit Tests — Ido Pesok, Vercel v0

Aug 6, 2025 · 15:22

Ido Pesok, an engineer at Vercel working on v0, argues that evals—not unit tests—are the key to making LLM applications reliable, using the basketball court analogy of understanding your domain (the court), collecting user queries (data points), and scoring passes/fails. He illustrates with the Fruit Letter Counter app, which failed in production despite passing manual tests, because LLMs are inherently non-deterministic. Pesok advises collecting data from thumbs-up/down, logs, and forums, and structuring evals with constants in data and variables in task (e.g., system prompt). He emphasizes deterministic pass/fail scoring, adding extra prompt tags for easy parsing, and integrating evals into CI/CD to detect regressions when changing models or prompts. The episode stresses that improvement without measurement is limited, and evals enable systematic reliability gains.

[Evals Workshop] Mastering AI Evaluation: From Playground to Production

Jul 1, 2025 · 1:25:08

In this workshop, Braintrust Solutions Engineers Carlos Esteban and Doug guide participants through the complete AI evaluation lifecycle, from offline testing in the playground to production monitoring. They explain the three core ingredients of an eval—task, data set, and score—and demonstrate how to run evals both via the Braintrust UI and the SDK. The session covers LLM-as-judge vs. deterministic code scores, the importance of starting small with synthetic data, and how to use online scoring and logging to capture real user feedback. Human-in-the-loop review is highlighted as a way to establish ground truth and close the feedback loop. The presenters also address audience questions on bootstrapping data sets, non-determinism in LLM judges, and integrating evals into existing projects.

Building voice agents with OpenAI — Dominik Kundel, OpenAI

Jun 29, 2025 · 1:25:35

Dominik Kundel from OpenAI presents the new OpenAI Agents SDK for TypeScript and demonstrates how to build voice agents, arguing that speech-to-speech architectures (using GPT-4 real-time) offer lower latency and more natural interactions than chained approaches that transcribe audio to text. He covers two architectures: chained (speech-to-text, text agent, text-to-speech) versus speech-to-speech (native audio understanding), and recommends starting with a small, clear goal, building evals and guardrails early, and using generative prompts to control tone and personality via openai.fm. In a live coding session, he builds a real-time agent with tools (e.g., getWeather) and handoffs, demonstrates delegation to a smarter model (O4 mini) for complex tasks like refunds, and shows built-in interruption handling, human-in-the-loop approval, tracing for debugging, and output guardrails. The SDK supports WebRTC and WebSocket, handles turn detection automatically, and provides session management with a 30-minute timeout that can be extended by injecting previous context.

Evals 101 — Doug Guthrie, Braintrust

Jun 27, 2025 · 48:31

Braintrust solutions engineer Doug Guthrie presents the full AI evaluation lifecycle, covering offline and online strategies for building robust AI products. He explains that evals require three ingredients—tasks (prompts or agentic workflows), datasets (real-world examples), and scores (LLM-as-a-judge or code-based)—and demonstrates how to run them in Braintrust's playground and via SDK. The talk emphasizes creating a feedback loop: production logs feed into datasets, enabling human review and user feedback to iteratively improve offline evals. Guthrie showcases a changelog app, the new AI-driven "loop" feature for prompt optimization, and answers audience questions on A/B testing models, handling multiple human scorers, pre-launch validation, and exporting data for custom dashboards.

How to build world-class AI products — Sarah Sachs (AI lead @ Notion) & Carlos Esteban (Braintrust)

Jun 27, 2025 · 1:43:46

Sarah Sachs (AI lead at Notion) and Carlos Esteban (Braintrust) explain that building great AI products requires 10% prompting and 90% evals and observability, with Notion AI using Braintrust to iterate on prompts and models. Sachs details their cycle: curate small datasets, tie them to scoring functions (LLM-as-a-judge with per-sample prompts and heuristic checks), run evals before shipping, and use production logs to catch regressions. She notes Notion AI supports 100M+ users, switches models in under a day, and 60% of enterprise users are non-English—requiring multilingual eval rigor. Carlos and Doug then walk through Braintrust's framework: tasks (prompts, tools, agents), datasets, and scores (0-1). They demonstrate offline evals via playground and SDK, online scoring in production, user feedback capture, human review setup, and remote evals to bridge complex code with the playground. The workshop covers moving from pre-prod evals to production monitoring, closing the feedback loop by adding underperforming spans to datasets.

Agentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK — Cedric Vidal, Microsoft

Jun 27, 2025 · 20:01

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates how to systematically evaluate AI agents using the Azure AI Evaluation SDK and AI Toolkit. He argues that evaluation must start early, covering four layers: model/safety system, system message, grounding, and user experience. He shows spot-checking models side-by-side in VS Code, building a web-scraping agent with Playwright MCP, and running batch evaluations with built-in quality and safety metrics like groundedness and fluency. Vidal also covers multi-turn and multimodal evaluation, using a violent image example where the SDK returned a score of 4 out of 5, allowing teams to set custom thresholds based on application context.

The Knowledge Graph Mullet: Trimming GraphRAG Complexity - William Lyon

Jun 3, 2025 · 32:52

William Lyon presents the 'Knowledge Graph Mullet' concept, combining property graph interfaces with RDF triple storage in Dgraph to optimize GraphRAG systems and AI agent development. Dgraph uses a property graph model for data modeling and DQL (inspired by GraphQL) for queries, while leveraging RDF triples for scalable storage and data interchange. Features include vector similarity search (e.g., finding articles about 'money laundering'), geospatial indexes, and a Model Context Protocol (MCP) server that exposes tools to models like Claude Desktop for auto-generating schema and querying. Lyon demonstrates building a news knowledge graph from the New York Times, chunking articles into nodes, and using graph traversal to surface related topics after vector search. He also showcases Hypermode agents, created from a prompt with access to GitHub and Notion via MCP servers, which can analyze repos like hyper-news and generate social media posts with code snippets, then save them to a Notion workspace. Hypermode agents are now in early access.