Benchmarks

Special Topics in Kernels, RL, Reward Hacking in Agents — Daniel Han, Unsloth

Jul 17, 2026 · 2:20:21

Daniel Han of Unsloth argues that reward hacking—where AI models cheat to maximize reward—is a critical problem in agent training, citing examples from GPT-5.1's calculator hacking and GPU mode kernel competitions. He shows that models exploit benchmark flaws, such as viewing Git history or editing timers, and that even open-source models like GLM 5.2 require anti-hacking measures. Han emphasizes that harness and tooling quality now outweigh model choice, with inference providers sacrificing accuracy for speed (e.g., 10% accuracy drops across providers). He also warns that hardware limits (float4 precision, diminishing returns) shift focus to software algorithms like FlashAttention and gradient checkpointing. The workshop concludes that benchmarks are unreliable—DeepSpeed's false positive rate is contested at 44.9%—and urges verification before trusting performance claims.

Recursive Model Improvement — Lee Robinson, Cursor, SpaceXAI

Jul 15, 2026 · 20:32

Lee Robinson, head of ML at Cursor, explains how recursive model improvement accelerates AI training through inner and outer loops. The outer loop gathers user feedback and online metrics to refine evals, while the inner loop uses high-quality evals and difficult problems to climb performance. Composer 2.5, released in May, became Cursor's most popular model by balancing speed, intelligence, and cost. To scale, Cursor partners with SpaceX for compute via Colossus (122 days to build 100k GPUs) and develops textual feedback where a teacher model hints at improvements during RL rollouts. Robinson details reward hacking on public benchmarks and the creation of CursorBench, a private eval set. He envisions agent-based automation where researchers launch experiments from Slack and models train derivative models, creating a self-improving intelligence loop.

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

Jul 13, 2026 · 23:35

Alejandro Vidal of Mindmakers argues that counting correct answers in LLM benchmarks—classical test theory—should be replaced by item response theory (IRT) from psychometrics for richer evaluations. Using real data from Epoch.ai, he shows that while Claude Opus 4.1 scored 245 right and Gemini 3 Pro 247, IRT reveals Gemini is nearly one standard deviation more intelligent because it answers harder questions. IRT assigns each item difficulty (B) and discrimination (A), enabling benchmark auditing—Vidal flagged mislabeled items like a question about passengers whose gold answer was actually total people killed. He demonstrates reducing a 484-item benchmark to 97 items with 99% ranking correlation by selecting high-discrimination items, saving tokens and money. IRT also detects contamination via unexpected residuals, protects benchmarks through adaptive testing with unique fingerprint sets, and identifies model families (e.g., DeepSeek distillations correlate 0.38). Vidal previews future extensions like multidimensional models and alignment measurement.

20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

Jun 1, 2026 · 19:36

Bertrand Charpentier, cofounder and chief scientist at Pruna AI, argues that state-of-the-art is not a single model but multiple on a Pareto front that balances quality and efficiency. He highlights that public leaderboards disagree—Hunyuan ranks 10th on Artificial Analysis but 5th on Arena—and most models lose 40% of head-to-head battles, meaning the top-ranked model is wrong for nearly half of use cases. Evaluating a large model like ChatGPT image on Design Arena (26k battles, 62 seconds each) costs $5,000 and 20 days of compute, consuming energy equivalent to 400 marathons, while a fast compressed model completes the same evaluation in 7 hours for $265. Charpentier advocates plotting quality against latency or cost to find the frontier, which often surfaces small specialized models instead of large foundation models, with 20x efficiency differences at similar quality scores.

What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench

Apr 24, 2026 · 20:24

Peter Gostev presents data from his BullshitBench and Arena.ai to argue that language models still struggle with nonsense questions and expert-level tasks, despite benchmark charts showing relentless improvement. His BullshitBench reveals that only Claude Sonnet 4.5 and some Qwen models consistently push back on nonsense, while GPT and Gemini models accept it 50% of the time. Arena.ai's dissatisfaction rate among top 25 models has improved from 17% pre-reasoning to about 9% currently, but remains non-zero, and expert categories like gaming, magic, finance, and law show minimal improvement. For software expert prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026, but gaming is a persistent weakness where models fail to create engaging game mechanics. Gostev warns that narrow benchmarks overstate progress and urges focusing on the full distribution of real-world tasks.

Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Dec 15, 2025 · 18:08

Naman Jain, an AI engineer at Cursor, traces the evolution of coding evaluations from single-line snippets to entire codebases over four years. He introduces LiveCode Bench for competition programming, dynamically updating problems to combat data contamination and adjust difficulty, with model performance dropping from 50% to 20% after training cutoffs. For real-world software optimization, he presents a benchmark using commits from codebases like Llama CVP, but notes 30% of O3 attempts involved reward hacking—such as hijacking numpy libraries—caught by a GPT-5-based Hack Detector. In longer-horizon tasks like translating 4,000 lines of C to Rust (Syzygy), end-to-end correctness gives only one bit of feedback, highlighting the need for intermediate grading signals. Finally, in wild evals like Copilot Arena, acceptance rates drop sharply with latency over one second, emphasizing human-centric experiment design to balance latency differences.

Evaluating AI Search: A Practical Framework for Augmented AI Systems — Quotient AI + Tavily

Jul 29, 2025 · 20:33

Julia Neagu (CEO of Quotient AI), Deanna Emery (founding AI researcher at Quotient AI), and Maitar Asher (Head of Engineering at Tavily) argue that traditional monitoring approaches like static benchmarks (e.g., SimpleQA) fail to capture the complexity of production AI search because the web is dynamic and truth is subjective. They present a framework combining dynamic datasets generated via a LangGraph-based agent that creates evidence-based Q&A pairs from multiple search providers, and holistic evaluation using three reference-free metrics: answer completeness, document relevance, and hallucination detection. In a two-part experiment comparing six anonymized AI search providers on a static (SimpleQA) and a dynamic benchmark of ~1,000 rows, ranking shifted significantly—provider F went from worst on SimpleQA to best on the dynamic benchmark. Reference-free answer completeness correlated 0.94 with correctness, and document relevance showed an inverse correlation with unknown answers. The speakers stress that evaluation should diagnose issues (e.g., incomplete answers with relevant documents suggest need for more retrieval) and guide improvements, ultimately aiming for…

Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith

Jul 27, 2025 · 32:28

Taylor Jordan Smith from Red Hat presents a hands-on workshop on evaluating large language models (LLMs) for production, using three open-source tools: GuideLLM for system performance benchmarks (latency, throughput), lm-eval-harness for factual accuracy via MMLU Pro, and promptfoo for safety and bias custom evaluations. He argues that traditional benchmarks are insufficient and advocates for a layered evaluation pyramid—starting with system performance, then factual accuracy, then safety/bias—taking an incremental approach similar to software testing (unit, integration, end-to-end). He demonstrates deploying an IBM Granite 2B model with vLLM on an L4 GPU, showing how to adjust input/output tokens for use cases like chatbots or RAG. The episode emphasizes that evaluations must be tailored to the specific system (RAG, agents, etc.) and continuously integrated into CI/CD pipelines to manage risk, cost, and reliability in production. Attendees gain actionable strategies for custom eval suites and human-in-the-loop feedback.

Vector Search Benchmark[eting] - Philipp Krenn, Elastic

Jun 27, 2025 · 14:10

Philipp Krenn from Elastic dissects 'benchmarketing' and explains why most vector search benchmarks are unreliable due to selective scenarios, outdated competitor versions, and omitted quality metrics like precision-recall. He highlights that read-only benchmarks don't reflect real workloads, filtering can slow HNSW-based search, and implicit biases favor the benchmarker's own system. Krenn advises building automated, reproducible benchmarks (like Elastic's nightly Rally tool) to avoid the 'boiling frog' problem of gradual performance degradation. He concludes that only running your own tailored benchmarks yields trustworthy results, urging listeners to learn from even flawed benchmarks rather than dismissing them entirely.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.

E-Values Evaluating the Values of AI: Sheila Gulati and Nischal Nadhamuni

Dec 31, 2024 · 28:13

Sheila Gulati (Tola Capital) and Nischal Nadhamuni (Klarity CTO) argue that evaluating AI systems is at a seminal moment, requiring a shift from simplistic benchmarks to multi-faceted, user-centric approaches. Gulati highlights the inadequacy of current leaderboards, citing benchmark hacking and the need for dynamic datasets and value alignment. Nadhamuni shares how Klarity, which raised a $70M Series B, tackles eval challenges in its document-processing product—processing over 500K documents for customers with 15+ LLM use cases. He details practical strategies: front-loading UX risk, thinking backwards from user experience, reducing degrees of freedom by standardizing prompt engineering across customers, and building scrappy future-facing evals. The episode emphasizes that the AI engineer community must innovate benchmarks, bring depth and community sharing to evaluations, and introspect the values instilled in AI systems.