Samuel Colvin

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

From Stateless Nightmares to Durable Agents — Samuel Colvin, Pydantic

Nov 24, 2025 · 22:13

Samuel Colvin of Pydantic demonstrates building production-grade durable AI agents using PydanticAI, Temporal, and Pydantic Logfire, arguing that stateless architectures fail at scale and that durable execution with checkpointing and recovery is essential. He shows a 20 questions game where two agents play, but with 20% simulated failures, restarts are avoided by wrapping agents in Temporal wrappers for automatic retries. Colvin then implements a Deep Research agent that plans searches, runs parallel web searches via Tavilli, and synthesizes results, all recoverable. In a live demo, killing the workflow and restarting resumes instantly, replaying cached LLM calls in milliseconds. He also previews Pydantic AI Gateway and demonstrates Pydantic Evals comparing Gemini, GPT-4.1, and Claude Sonnet 4.5 on performance and cost.

Human seeded Evals — Samuel Colvin, Pydantic

Jul 25, 2025 · 12:02

Samuel Colvin, creator of Pydantic, demonstrates how type safety and validation loops in Pydantic AI build reliable GenAI applications. He argues that type-safe frameworks enable confident refactoring, a necessity for evolving AI apps, and shows how returning validation errors to the model fixes simple mistakes like incorrect date formats. The agentic loop with tools and final result tools ensures structured extraction, while type-safe dependencies on tools prevent runtime errors. Colvin also highlights Logfire observability, which traces calls, shows pricing, and helps debug failures like failed memory retrieval. The talk covers agent definitions, validation error retries, and the importance of explicit exit conditions in agent loops.

MCP is all you need — Samuel Colvin, Pydantic

Jul 18, 2025 · 15:24

Samuel Colvin, creator of Pydantic, argues that MCP (Model Context Protocol) is all you need for agent-to-agent communication, simplifying what many are overcomplicating. He explains that MCP's tool calling primitive offers advantages over OpenAPI, such as dynamic tools, logging, and sampling—a mechanism for MCP servers to request LLM calls through the client, enabling nested agentic workflows. Colvin demonstrates a research agent using Pydantic AI that queries the BigQuery public PyPI dataset via an MCP server over stdio, showing how sampling allows the server to generate SQL itself while keeping the main agent's context lean. The demo logs progress to Logfire, an observability platform, and outputs human-readable results. He emphasizes that MCP’s standard I/O mode and extensibility make it ideal for autonomous agents beyond its original desktop coding use case.