TaoBench

Your Agents Need a Save Button - Hamza Tahir, ZenML

Jul 18, 2026 · 17:07

Hamza Tahir, co-founder of ZenML and Kitaru, argues that AI agents need a durable 'save button' — checkpointing runtime state — to move beyond read-only traces and enable replay, debugging, and optimization at scale. He demonstrates Kitaru, an open-source runtime that snapshots code, artifacts, and environment between tool calls, letting developers replay an agent from any checkpoint with a swapped model or mocked tool. Tahir shows how this allows side-by-side diffs across single runs or entire cohorts sorted by cost, then uses an MCP server to let an LLM analyze the results and flag false economies. He cites DoorDash's blog post, which used simulated replays to cut hallucination by 90% and reduce simulation time from hours to minutes. The conclusion: a single replay is an anecdote; real insights come from cohort analysis across production checkpoints.

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.

Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

Apr 10, 2026 · 40:51

Mahmoud Mabrouk, co-founder of Agenta AI, demonstrates how to build calibrated LLM-as-a-judge evaluators using the GEPA prompt optimization algorithm, arguing that miscalibrated evals are worse than none. He walks through a practical workflow for a customer support agent using the TaoBench airline dataset, covering metric design, data annotation, and GEPA-based optimization. The seed judge achieved 61% accuracy; after optimization, accuracy rose to 74% with reduced bias, though the judge still struggled to fully learn the complex policy. Mabrouk shares key lessons: start with a seed prompt biased toward compliance, use larger models for refinement, overfit to training data first, and beware of high token costs.