Steven · AI Engineer

Spec-Driven Testing for Agents With A Brain the Size of A Planet — Steven Willmott, SafeIntelligence

May 31, 2026 · 13:03

Steven Willmott of SafeIntelligence argues that bigger AI models are not straightforwardly safer, citing jailbreaks like wrapping malicious instructions in poems that large models understand but small ones do not. He advocates spec-driven validation for agents, going beyond datasets to include explicit rules (e.g., never offer more than 10% discount), domain ontologies (e.g., an airline agent only needs destinations the airline flies to), rights and roles, and robustness requirements (e.g., tolerance for typos or rephrasings). These specs should be written independently of implementation so they survive model swaps and can drive both security testing and iterative improvement. Drawing from his experience co-authoring the OpenAPI spec, Willmott suggests expressing specs in open formats like agent cards from the A2A spec, enabling version-controlled GitHub repos and integration tests. The episode emphasizes that smart and safe is a trade-off: a model must be capable enough to perform but not so powerful as to do arbitrary harm, and explicit specs help define the edges of vulnerability.

2 episodes