Logfire

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

Human seeded Evals — Samuel Colvin, Pydantic

Jul 25, 2025 · 12:02

Samuel Colvin, creator of Pydantic, demonstrates how type safety and validation loops in Pydantic AI build reliable GenAI applications. He argues that type-safe frameworks enable confident refactoring, a necessity for evolving AI apps, and shows how returning validation errors to the model fixes simple mistakes like incorrect date formats. The agentic loop with tools and final result tools ensures structured extraction, while type-safe dependencies on tools prevent runtime errors. Colvin also highlights Logfire observability, which traces calls, shows pricing, and helps debug failures like failed memory retrieval. The talk covers agent definitions, validation error retries, and the importance of explicit exit conditions in agent loops.

MCP is all you need — Samuel Colvin, Pydantic

Jul 18, 2025 · 15:24

Samuel Colvin, creator of Pydantic, argues that MCP (Model Context Protocol) is all you need for agent-to-agent communication, simplifying what many are overcomplicating. He explains that MCP's tool calling primitive offers advantages over OpenAPI, such as dynamic tools, logging, and sampling—a mechanism for MCP servers to request LLM calls through the client, enabling nested agentic workflows. Colvin demonstrates a research agent using Pydantic AI that queries the BigQuery public PyPI dataset via an MCP server over stdio, showing how sampling allows the server to generate SQL itself while keeping the main agent's context lean. The demo logs progress to Logfire, an observability platform, and outputs human-readable results. He emphasizes that MCP’s standard I/O mode and extensibility make it ideal for autonomous agents beyond its original desktop coding use case.