Brain Trust

Spec-Driven Testing for Agents With A Brain the Size of A Planet — Steven Willmott, SafeIntelligence

May 31, 2026 · 13:03

Steven Willmott of SafeIntelligence argues that bigger AI models are not straightforwardly safer, citing jailbreaks like wrapping malicious instructions in poems that large models understand but small ones do not. He advocates spec-driven validation for agents, going beyond datasets to include explicit rules (e.g., never offer more than 10% discount), domain ontologies (e.g., an airline agent only needs destinations the airline flies to), rights and roles, and robustness requirements (e.g., tolerance for typos or rephrasings). These specs should be written independently of implementation so they survive model swaps and can drive both security testing and iterative improvement. Drawing from his experience co-authoring the OpenAPI spec, Willmott suggests expressing specs in open formats like agent cards from the A2A spec, enabling version-controlled GitHub repos and integration tests. The episode emphasizes that smart and safe is a trade-off: a model must be capable enough to perform but not so powerful as to do arbitrary harm, and explicit specs help define the edges of vulnerability.

2025 is the Year of Evals! Just like 2024, and 2023, and … — John Dickerson, CEO Mozilla AI

Aug 6, 2025 · 19:14

John Dickerson, CEO of Mozilla AI and former co-founder of Arthur AI, argues that 2025 is finally the year for AI evaluation (evals) due to three converging forces: ChatGPT making AI tangible to the C-suite, enterprise budget freezes that funnelled funding into GenAI pet projects, and the rise of agentic systems acting autonomously. He traces how ML monitoring existed pre-2022 but lacked C-suite attention, with Jamie Dimon’s JPMC spending only $100 million on AI from 2017 to 2021. The ChatGPT launch on November 30, 2022, unlocked CEO discretionary budgets for GenAI, leading to 2023 science projects, 2024 production deployments, and 2025 scaling. Now, as agents reason and act, connecting evals to downstream business KPIs—risk mitigation, revenue gains—is a first-class discussion for CEOs, CFOs, CISOs, and CTOs. Dickerson notes that evaluation companies have shifted to multi-agent system monitoring, and while LLM-as-judge is popular, biases require human validation, with companies like Mercor hiring experts at $50–200/hour to lockstep multi-agent outputs.

The Web Browser Is All You Need - Paul Klein IV, Browserbase

Jun 20, 2025 · 17:31

Paul Klein IV, founder of Browserbase, argues that the web browser is the default MCP server for the rest of the internet, serving as the essential bridge between AI agents and legacy websites that lack APIs. He distinguishes between web agents (one prompt to many actions, e.g., OpenAI's Operator) and browser tools (one action per prompt), noting that both rely on vision-driven or text-based approaches to parse pages. Klein explains that Browserbase offers a horizontal MCP server for browsing, enabling automation of any website, and stresses the need for custom evals and observability to track agent behavior. In a live demo, he shows an agent navigating sfpca.org to find a dog for adoption, handling unexpected modals. He also addresses captchas, advising good citizenship and hinting at future agent authentication solutions.