Claude 3 Opus

Why Agent Hype can fall short of reality – Joel Becker, METR

Dec 24, 2025 · 21:22

Joel Becker, a researcher at METR, examines why AI models that ace benchmarks like SWE-bench fail to boost real-world developer productivity. METR's time horizon measurements show AI capabilities doubling every 6-7 months, with Claude 3.7 Sonnet achieving a 50% success rate on tasks taking humans 4 minutes. Yet a randomized controlled trial of 16 experienced developers on large open-source projects found they were 19% slower when using AI tools like Cursor Pro, contradicting expert predictions of 40% time savings. Becker attributes the gap to high context requirements, low AI reliability, and task complexity—developers spent significant time verifying and correcting AI outputs. The episode warns that benchmark-style evidence, which uses low-context human baselines, overstates AI readiness for messy, interdependent real-world tasks.

Fuzzing in the GenAI Era — Leonard Tang, Haize Labs

Aug 22, 2025 · 19:12

Leonard Tang of Haize Labs argues that standard evaluation methods fail for GenAI systems due to brittleness (Lipschitz discontinuity), and proposes 'Haizing'—fuzz testing that simulates diverse inputs to uncover corner cases. He details two core problems: scoring outputs via 'judges', where Haize's Verdict library stacks GPT-4 Mini in a self-verified debate ensemble to beat O1 at a third the cost, and RL-tuned judges like a 1.7B parameter model achieving 80.7% on RewardBench. For stimuli generation, he frames it as discrete optimization over natural language, using gradient-based methods and tree search. Case studies include Haizing the largest Hungarian bank's loan calculator, discovering prompt injections in minutes, and boosting a voice agent's ground-truth human agreement by 38% using rubric fanout.