Ian Butler

How to Improve your Vibe Coding — Ian Butler

Aug 3, 2025 · 7:30

Ian Butler (CEO of BizMyth) presents benchmark data showing that popular coding agents like Devin and Cursor have less than 10% true positive rates for finding bugs, with Cursor producing a 97% false positive rate across 100+ repos and 1,200+ issues. He argues that these agents overrun codebases with false positives, causing alert fatigue and bugs reaching production. To improve vibe coding, he recommends adding bug-focused rules (e.g., OWASP Top 10, explicit bug classes like auth bypasses), managing context by feeding diffs and preventing file summarization, and using thinking models which were significantly better at finding deeper bugs. However, even thinking models show high variability in which bugs they find across runs, indicating agents never holistically examine a file. The talk concludes with a plug for BizMyth.sh, which automates PRs and vulnerability scanning.

Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

Jun 3, 2025 · 18:39

Ian Butler and Nick Gregory introduce SM-100, a benchmark of 100 real-world bugs across 84 repositories, and present their agent Bismuth as outperforming existing agents in detecting complex bugs—finding 10 needles vs 7 for the next best. However, even top performers struggle: Bismuth's true positive rate is only 25%, and Codex leads at 45%, while most agents (Cursor, Debian) report thousands of false positives (97% for basic loops). They argue that agents universally exhibit narrow thinking, missing simple bugs like a form state issue that only Bismuth and Codex caught, and that high SWE-bench scores do not translate to maintenance tasks. The episode concludes that software maintenance remains an unsolved problem requiring deeper reasoning and targeted search.