The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

I'm about to show you how the AI benchmarks game is rigged, and why the biggest players have every incentive to keep it that way. I'm Darius Emrani, CEO of Scorecard. I built evaluation systems at Waymo and Uber ATG for self-driving cars, which are really the OG AI agents, and for rockets at SpaceX.

I also have patents from Waymo on how to evaluate autonomous systems. Now my team and I work with leaders in AI across legal tech, health tech, and finance, and we've seen every eval trick in the book. Here's what we'll cover: first, why benchmarks control billions in market value and mindshare; second, the three most common ways companies game the system; and third, how to build evaluations that actually work.

First off, what is a benchmark? A benchmark is composed of three components: a model you're testing, a set of questions we call the test set, and a metric, how we keep the score. But here's what's important to understand: a benchmark is actually composed of many individual evals bundled together, and the key insight is that benchmarks standardize the test set and metrics across these models, and that's what makes them comparable.

Think of it like the SAT: the same questions, same scoring system, just different test takers.

But here's the thing: these scores control billions in market value, investment decisions, and public perception.

Simon Willison said it best: billions of dollars of investment are now being evaluated based on these scores. When OpenAI or Anthropic claims the top spot, it's not just about funding; it's about enterprise contracts, developer mindshare, and market dominance.

When Andrej Karpathy tweets about your benchmark to his millions of followers, that shapes entire ecosystems. Recently, Sonar acquired Auto Rover. Why? Auto Rover showed strong results on SweBench. So we have a system where a single number can define market leaders and destroy competitors.

What could possibly go wrong?

Well, when the stakes are this high, people find creative ways to win. Let me show you the most common tricks.

The first trick is simple: you make apples-to-oranges comparisons. Here's a recent example: xAI released benchmark results for Grok-3. Beautiful graphs showing them beating everyone. The AI community went wild. But then, OpenAI engineers started looking closer. They noticed something interesting: xAI was comparing their best configuration against other models' standard configurations.

It's like comparing your sports car with nitrous booster against regular cars without it. Specifically, they didn't show OpenAI O3's model's high performance at consensus 64. That's when you run the model 64 times and take the consensus answer. Now, consensus at 64 is much more expensive.

You're literally running the model 64 times. But if you're claiming performance leadership, you need to compare the best to the best, or standard to standard. Not "the best" against "their standard." And this selective reporting? It's just the tip of the iceberg.

The second trick is more controversial: getting privileged access to test questions. FrontierMath was supposed to be this super-secret, impossible-to-game benchmark for advanced mathematics. Their creators made a big deal about how it's "really protected." But here's what actually happened: OpenAI funded FrontierMath and got access to the entire dataset.

Now, to be fair, there's a verbal agreement that they won't train on the data, and OpenAI employees have publicly called it a "strongly held out evaluation set." But you have to think about the optics here. The company funding the benchmark gets to see all the questions, evaluate their models internally, and announce their scores before there's independent verification.

When OpenAI announced that O3 had scored a surprisingly strong 25%, people were surprised. Even if everything is above board—and it might be—this creates a trust problem. When benchmark creators take money from the companies they're evaluating, it undermines the whole system.

But even this isn't the most subtle trick. That honor goes to trick number 3. This one's really fascinating. So models can optimize for style over substance. Here's a real example: Meta released Llama-4 Maverick to the public, but behind the scenes they entered 27 different versions into LM Arena, each one tweaked to maximize appeal, not necessarily accuracy.

One of the private versions was asked to make a riddle with the answer 3.145. It gave a long, emoji-filled, flattering response that made absolutely no sense. But it beat Claude's correct answer. Why? Because it was chatty and engaging, not because it wasright.

This is happeningright now. Companies are literally training models to be wrong, but charming. Researchers at LM Arena proved we can control for this. When they filtered out style effects—length, formatting, personality—the rankings completely changed. GPT-4o Mini and Grok-2 dropped out, and Claude 3.5 Sonnet jumped up and tied for first.

We're really not measuring which model is "most accurate." We're measuring which model is "most charming." It's like choosing a surgeon based on bedside manner instead of surgical skill. Even human SATs have this problem: 39% of score variance in SAT scores is just essay length.

If you write more, you score higher. We have the technology to measure what matters, but the industry prefers measuring what sells. And this brings us to the fundamental problem. All of this is a natural outcome of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.

We've turned benchmarks into targets worth billions. So, of course, they've stopped measuring what actually matters. The incentives guarantee it. Speaking of which—this isn't just my opinion—listen to what the experts, the people who create these benchmarks, are saying. Andrej Karpathy, co-founder of OpenAI: "My reaction is that there is an evaluation crisis.

I don't really know what metrics to look atright now." Let that sink in. The co-founder of OpenAI doesn't know which metrics to trust. John Yang, who created SweBench: "It's sort of like we kind of just made these benchmarks up."

Marin Sapp, from CMU: "The yardsticks are, like, pretty fundamentally broken." And many more. When the people who build the benchmarks say they're broken, when the leaders of AI say they don't trust the metrics, we have a serious problem.

So how do we fix this?

To fix public benchmarks, we need to address all three components: the model comparisons, the test sets, and the metrics. For model comparisons, we need to require apple-to-apple comparisons. You need to have the same computational budget, same constraint, no cherry-picking of configurations.

You also need to show cost-performance tradeoffs transparently. So on theright here, we see the arc prize where they've done just that. For test sets, we also need complete transparency. We need to open-source the data and have methodologies and code with no financial ties between the benchmark creators and the model companies.

In addition, we need to have regular rotation of the test questions to prevent overfitting. Finally, for metrics, we need to be able to control for style effects so that we measure substance, not just engagement. And we also need all attempts to be required public, so that there's no possibility of cherry-picking your best run.

The good news is we're seeing progress. LM Arena's style-controlled rankings give us some ability to remove styles as a component. And we're also seeing the emergence of more and more independent benchmarks in specific domains. These open-source efforts to create benchmarks include LegalBench, MedQA, and FinTech, as well as the cross-counting efforts of AgentEval and BetterBench to actually benchmark benchmarks.

But if you really want to win the evaluation game, there's a better way. You can't win a rigged game, so you stop playing it.

Instead of chasing public benchmarks, you should build a set of evaluations that actually matter for YOUR use case. And here's how.

Step 1: you gather real data. Taking 5 actual queries from your production system is worth

way, way more than 100 academic questions. These real user problems beat synthetic benchmarks every time. Step 2: choose your metrics. Quality, cost, latency. Pick what matters for your application. A chatbot needs different metrics than a medical diagnosis system.

Step 3: test theright models. Don't just go by the leaderboards. Test the top 5 models on YOUR specific data. GPT-4 might top generic benchmarks but fail on your legal documents. Step 4: you systematize it. You need consistent, repeatable evaluation.

Build it yourself, or use a platform like Scorecard. Step 5: keep iterating. Models improve, your needs change. Make evaluation a continuous process, not a one-time event.

At Scorecard, this is how we do it. We've built a complete workflow: you identify issues, you build improvements, then you run evaluations before deployment. It's a continuous cycle. You run evals, you get feedback, you improve, and then you only deploy when you hit your quality bar.

Then you monitor and start again. This pre-deployment evaluation loop is what separates teams that ship reliable AI from teams that are constantly firefighting production issues. Yes, this takes more work than just checking the leaderboard, but it's the only way to build AI that actually serves your users.

And here's the bottom line: the benchmarks game is rigged because there's too much at stake for it not to be. Market caps, acquisitions, developer mindshare—it all hinges on these numbers. But you don't have to play their game. You can build evaluations that actually help you ship better products.

Measure what matters to your users, not what matters to Twitter. All benchmarks are wrong, but some are useful. The key is knowing which ones. Thanks for watching. Now go build evals that mean something.