What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench
Apr 24, 2026 · 20:24
Peter Gostev presents data from his BullshitBench and Arena.ai to argue that language models still struggle with nonsense questions and expert-level tasks, despite benchmark charts showing relentless improvement. His BullshitBench reveals that only Claude Sonnet 4.5 and some Qwen models consistently push back on nonsense, while GPT and Gemini models accept it 50% of the time. Arena.ai's dissatisfaction rate among top 25 models has improved from 17% pre-reasoning to about 9% currently, but remains non-zero, and expert categories like gaming, magic, finance, and law show minimal improvement. For software expert prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026, but gaming is a persistent weakness where models fail to create engaging game mechanics. Gostev warns that narrow benchmarks overstate progress and urges focusing on the full distribution of real-world tasks.
