Joel Becker

How METR measures Long Tasks and Experienced Open Source Dev Productivity - Joel Becker, METR

Jan 19, 2026 · 1:15:52

Joel Becker of METR argues that AI benchmark scores are soaring while real-world developer productivity barely budges, reconciling this gap by presenting METR's time horizon measurements and a 16-developer RCT showing no significant speedup from AI on mature open-source projects. He cites reliability issues, task distribution mismatches, and the J-curve effect from Meta's developer data, noting that experienced users still fail to accelerate. The episode explores why AI struggles with messy enterprise data (e.g., LinkedIn's 5,000 'impressions' tables), the failure of computer-use agents in 'Agent Village,' and the possibility that hardware constraints may slow progress. Becker also previews new metrics like 'watched versus unwatched' time horizons and future studies on greenfield coding and data science.

Why Agent Hype can fall short of reality – Joel Becker, METR

Dec 24, 2025 · 21:22

Joel Becker, a researcher at METR, examines why AI models that ace benchmarks like SWE-bench fail to boost real-world developer productivity. METR's time horizon measurements show AI capabilities doubling every 6-7 months, with Claude 3.7 Sonnet achieving a 50% success rate on tasks taking humans 4 minutes. Yet a randomized controlled trial of 16 experienced developers on large open-source projects found they were 19% slower when using AI tools like Cursor Pro, contradicting expert predictions of 40% time savings. Becker attributes the gap to high context requirements, low AI reliability, and task complexity—developers spent significant time verifying and correcting AI outputs. The episode warns that benchmark-style evidence, which uses low-context human baselines, overstates AI readiness for messy, interdependent real-world tasks.