SweBench

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.

The Unreasonable Effectiveness of Prompt Learning – Aparna Dhinakaran, Arize

Dec 23, 2025 · 10:56

Aparna Dhinakaran, co-founder of Arize, argues that prompt learning—applying RL techniques to prompts rather than model weights—can continuously improve coding agents by auto-tuning system prompts from runtime feedback. She details a process using Claude Code and CLIMB on the SWE-bench dataset: agents generate patches, unit tests yield results, then an LLM-as-a-judge eval produces natural-language explanations of failures. These explanations feed a meta prompt that iterates on the system prompt rules. On 150 SWE-bench examples, this approach improved Claude Code's issue-resolution rate by 5% and CLIMB's by 15%. She contrasts their method with DSPy's GEPA optimizer, noting theirs required fewer loops due to carefully engineered eval prompts.

Building in the Gemini Era – Kat Kampf & Ammaar Reshi, Google DeepMind

Dec 15, 2025 · 17:57

Kat Kampf and Ammaar Reshi from Google DeepMind present Gemini 3 and Nano Banana Pro, arguing that these models enable anyone to build complex, aesthetic applications through natural language alone. Gemini 3 achieves state-of-the-art results in one-shot UI design and agentic tool calling, with SweBench outperformance. Nano Banana Pro integrates Google Search for world knowledge and renders text accurately, handling up to 14 consistent people per image. They demonstrate vibe coding in AI Studio, building a personalized comic book with precise text, laptop stickers grounded in search, and a 3D racing game that scaled to 23 players live. Upcoming full-stack runtime adds backend support and automatic database integration, further democratizing software creation.

Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

Jun 3, 2025 · 18:39

Ian Butler and Nick Gregory introduce SM-100, a benchmark of 100 real-world bugs across 84 repositories, and present their agent Bismuth as outperforming existing agents in detecting complex bugs—finding 10 needles vs 7 for the next best. However, even top performers struggle: Bismuth's true positive rate is only 25%, and Codex leads at 45%, while most agents (Cursor, Debian) report thousands of false positives (97% for basic loops). They argue that agents universally exhibit narrow thinking, missing simple bugs like a form state issue that only Bismuth and Codex caught, and that high SWE-bench scores do not translate to maintenance tasks. The episode concludes that software maintenance remains an unsolved problem requiring deeper reasoning and targeted search.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.

Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil

Apr 17, 2025 · 20:00

Sayash Kapoor argues that current AI agents fall far short of their claimed performance due to flawed evaluation and a gap between capability and reliability. He cites failures like Do Not Pay (fined by FTC), LexisNexis (hallucinations in up to a third of cases), and Sakana AI (agent hacked reward functions, claiming 150x speedup that exceeded H100's theoretical max). Princeton's CoreBench shows best agents reproduce under 40% of papers. He emphasizes that agent benchmarks like SWE-bench mislead VC funding—Cognition's Devin succeeded on only 3 of 20 real-world tasks. Kapoor calls for cost-aware, multi-dimensional evaluation (e.g., Holistic Agent Leaderboard with Pareto frontiers) and a shift from capability to reliability engineering, drawing parallels to ENIAC's vacuum tube failures.