GPQA

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

Jul 13, 2026 · 23:35

Alejandro Vidal of Mindmakers argues that counting correct answers in LLM benchmarks—classical test theory—should be replaced by item response theory (IRT) from psychometrics for richer evaluations. Using real data from Epoch.ai, he shows that while Claude Opus 4.1 scored 245 right and Gemini 3 Pro 247, IRT reveals Gemini is nearly one standard deviation more intelligent because it answers harder questions. IRT assigns each item difficulty (B) and discrimination (A), enabling benchmark auditing—Vidal flagged mislabeled items like a question about passengers whose gold answer was actually total people killed. He demonstrates reducing a 484-item benchmark to 97 items with 99% ranking correlation by selecting high-discrimination items, saving tokens and money. IRT also detects contamination via unexpected residuals, protects benchmarks through adaptive testing with unique fingerprint sets, and identifies model families (e.g., DeepSeek distillations correlate 0.38). Vidal previews future extensions like multidimensional models and alignment measurement.

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.