AI Benchmarks and Goodhart's Law: Why Leaderboards Mislead

On April 5, 2025, Meta released Llama 4. The announcement for its mid-sized model, Llama 4 Maverick, included a headline number: an “experimental chat version” scored an Elo of 1417 on LMArena, the popular leaderboard where people vote on which of two anonymous chatbots gave the better answer. That put Maverick in second place, above most of the big names.

The word “experimental” was doing a lot of work.

Within a day or so, people noticed that the model you could actually download didn’t behave like the one on the leaderboard. As TechCrunch reported, the arena version seemed to use a lot of emojis and gave very long-winded answers. A chart on Meta’s own Llama website said the arena testing had used “Llama 4 Maverick optimized for conversationality”.

On April 8, LMArena responded publicly. It said “Meta’s interpretation of our policy did not match what we expect from model providers”, and that Meta should have made it clearer that “Llama-4-Maverick-03-26-Experimental” was a customized model built to optimize for human preference. LMArena released over 2,000 head-to-head battle results for public review, updated its leaderboard policies, and added the publicly released Maverick to the arena. A few days later, TechCrunch reported that the released model ranked below older models including GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro. A Meta spokesperson said the experimental version was “a chat optimized version we experimented with that also performs well on LM Arena”.

To be fair to Meta, the announcement did say “experimental”. The rules at the time allowed private variants. But the number that got passed around was “Llama 4 Maverick, #2 on LMArena”, and the model behind that number wasn’t the one people could use.

This is the core problem with AI benchmarks, in one story. The moment a score matters, people optimize for the score. And the score slowly stops telling you what you wanted to know.

Goodhart’s law: the measure that stops measuring

In 1975, British economist Charles Goodhart was writing about monetary policy. He observed that any statistical relationship the government relied on tended to break down once the government started using it as a target. His original wording: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

The version everyone quotes is shorter, and it came later. In 1997, anthropologist Marilyn Strathern, writing about audits in British universities, put it like this:

“When a measure becomes a target, it ceases to be a good measure.”

A simple example. A call center measures how fast agents finish calls, because good agents tend to solve problems quickly. Then it starts paying bonuses for short calls. Now some agents hang up on difficult customers. Call time goes down. Service gets worse. The measure was fine while nobody was aiming at it.

AI benchmarks follow the same path. A benchmark starts as a measurement: a fixed set of questions that tells us something about how capable a model is. Then it becomes the number in the press release, the line on the investor slide and the leaderboard rank. Now every lab is aiming at it, and the measure starts to bend.

If you’ve read the previous part of this series on reward hacking, this will sound familiar. Reward hacking is Goodhart’s law inside a training run. Benchmark gaming is Goodhart’s law across an entire industry.

What the famous benchmarks actually measure

Before looking at how benchmarks fail, it helps to know what they are. Here are four you’ll see constantly.

MMLU (Massive Multitask Language Understanding), from Dan Hendrycks and colleagues in 2020. Multiple-choice questions across 57 subjects, from elementary maths and US history to law and computer science. It measures broad knowledge and some reasoning, in a quiz format. When it came out, the largest GPT-3 model beat random guessing by almost 20 percentage points on average. That was considered notable.

GSM8K (Grade School Math 8K), from OpenAI in 2021. About 8,500 grade-school maths word problems that need several steps of simple arithmetic. It was designed to test multi-step reasoning, which language models were bad at in 2021 even though the problems are conceptually easy.

SWE-bench, from Carlos Jimenez and colleagues at Princeton in 2023. 2,294 real GitHub issues from 12 popular Python projects. The model gets the codebase and the issue description and has to edit the code to fix it. The fix is checked by running tests taken from the real fix. When the paper came out, the best model resolved 1.96% of issues. In August 2024, OpenAI and the SWE-bench authors released SWE-bench Verified, a 500-task subset that human developers had screened, because some original tasks were vague or had tests that rejected valid solutions.

Human-preference arenas, like LMArena (formerly Chatbot Arena), from researchers at UC Berkeley. A visitor types a prompt, two anonymous models answer, and the visitor votes for the one they prefer. Many thousands of votes are turned into a ranking, similar to chess ratings. The 2024 paper describing it reported over 240,000 votes at the time. It measures something none of the quizzes do: which answers real people like.

Notice what each one is. MMLU measures knowledge in a quiz format. GSM8K measures a narrow kind of maths. SWE-bench measures fixing one kind of bug in one language, judged by tests. An arena measures preference, which is not the same as correctness. Each is a useful, narrow window. None is “how good is this model”.

How benchmarks break

There are four main ways a benchmark stops measuring what it claims to.

Benchmarks don't usually fail all at once. They get saturated, leaked and optimized until the score says more about the test than the model.

1. Saturation: everyone gets an A

A test is only useful while it can tell students apart. If everyone scores 95%, the test has run out of room.

That happened to MMLU. By January 2025, the authors of Humanity’s Last Exam, a new and much harder benchmark, noted that language models “now achieve over 90% accuracy on popular benchmarks like MMLU”, which limits how well those tests can measure progress.

Saturation hides a second problem: at the top, you’re mostly measuring the test’s mistakes. In 2024, Aryo Pradipta Gema and colleagues manually checked MMLU questions in a paper called “Are We Done with MMLU?”. They estimated that about 6.5% of all MMLU questions contain errors, and in the virology subset, 57% of the questions they analyzed had errors. If a few percent of answers are wrong in the answer key, the difference between 91% and 92% might be noise, or might even reward a model for matching the wrong answer.

2. Contamination: the answers were in the textbook

Language models are trained on huge scrapes of the internet. Popular benchmarks are on the internet: on GitHub, in papers, in blog posts that discuss specific questions. So test questions can end up in the training data, sometimes with their answers. This is called contamination.

A contaminated model might score well because it has seen the questions, not because it can solve new ones. It’s like a student who got hold of last year’s exam, and this year’s exam is the same paper.

In 2024, researchers at Scale AI tested this directly. Hugh Zhang and colleagues wrote GSM1k, a fresh set of grade-school maths problems built to match GSM8K’s style and difficulty, but never published online before testing. If a model really has the skill, it should score about the same on both. Some models dropped by up to 8 percentage points on the new set, and several model families showed signs of systematic overfitting. Models that were more likely to reproduce GSM8K examples word for word tended to show bigger drops, which points to partial memorization.

The paper is careful about the other side too. Many models, especially the frontier ones, showed little sign of overfitting, and all models generalized to new problems to some degree. Contamination doesn’t make benchmarks useless. It means a public benchmark score is an upper bound you can’t fully trust.

Accusations of contamination are also easy to make and hard to prove. During the Llama 4 launch, rumors spread that Meta had trained on test sets. Meta’s head of generative AI, Ahmad Al-Dahle, said on April 7, 2025 that this was “simply not true”. I haven’t found independent evidence either way, and that’s part of the problem: nobody outside a lab can easily check.

3. Gaming the leaderboard

Arenas avoid contamination, because the questions are new every time a visitor types one. But they have their own Goodhart problem: what wins votes isn’t always what’s best.

People voting in a few seconds tend to reward answers that are long, confident, friendly and nicely formatted. The previous post covered how reward models learn the same biases. The Maverick variant, with its emojis and long answers, was tuned for exactly that audience.

In April 2025, researchers from Cohere and several universities published “The Leaderboard Illusion”, a critical study of Chatbot Arena. Among their findings:

A few providers could test many private variants before a public launch and only publish the best score. The paper identified 27 private variants tested by Meta in the lead-up to Llama 4.
Data access was uneven. They estimated Google and OpenAI received about 19.2% and 20.4% of all arena data, while 83 open-weight models together received about 29.7%.
Extra arena data could give large gains on the arena itself, which suggests models can overfit to arena-style prompts and voter tastes rather than getting better in general.

LMArena disputed parts of the paper, saying it contained factual errors and that its pre-release testing policy had been public. You don’t have to settle that debate to see the general lesson. If you can try 27 times and report the best, the best of 27 is partly luck. That’s a form of selection bias, and it applies to any leaderboard where you can retry privately.

4. Benchmark answers aren’t your answers

Even a clean, unsaturated, ungamed benchmark measures its task, not yours. SWE-bench is real GitHub issues in Python, judged by tests. That’s useful. But it says little about your TypeScript monorepo, your vague tickets, or whether the fix would pass your code review. And as the reward hacking post showed, “passing the tests” and “fixing the problem” are not always the same thing.

A benchmark tells you how a model did on someone else’s exam. Whether it will do your job is a different question.

Why benchmarks are still worth having

None of this means benchmarks are useless. They’re how the field measures progress at all. GSM8K made multi-step maths a target, and models got genuinely better at it, not just at the test. SWE-bench moved from under 2% to a serious measure of coding agents. Arenas captured something quizzes miss.

The healthy pattern is the one in the diagram: benchmarks get created, used, saturated and replaced. Researchers respond to Goodhart’s law with new designs: held-out test sets kept private, questions written after a model’s training cutoff, human-verified subsets like SWE-bench Verified, and much harder tests like Humanity’s Last Exam. Each one buys a few more years of signal.

The problem is mostly in how scores are used. A benchmark is a research tool. A leaderboard rank in a press release is marketing. Treat them differently.

How to evaluate AI for your own use

If you’re picking a model for a product or a business task, here’s the approach that actually works. It’s less exciting than a leaderboard, and much more useful.

Read public scores like a skeptic

Check which model was tested. Exact name, version and settings. “Experimental”, “preview” and “optimized for” are flags.
Check which benchmark, and whether it’s saturated. A 1-point lead on a test where everyone scores above 90% means very little.
Look for independent results. Scores reported by the company that made the model are a starting point, not a verdict.
Match the benchmark to your task. A maths benchmark doesn’t predict customer-support quality.

Build your own small test set

This is the single most useful thing you can do. I wrote about it in more detail in Treat the LLM Like an Unreliable API, in the section on evals (short for evaluations). The short version:

Collect 20 to 50 real examples of the task you care about, with personal data removed or replaced. Include the weird and difficult ones, not just the clean ones.
Write down what a good answer looks like for each one, ideally as a simple check you can run in code. “Extracts the date correctly.” “Says it’s unsure when the message is ambiguous.”
Run every candidate model on the same set. Several times, because answers vary.
Read some outputs yourself. A pass rate tells you how often. Reading tells you how badly it fails when it does.
Keep the set private and add every real failure you see later. A private test set can’t leak into anyone’s training data, and nobody can tune their model to it.

Your own small test set has none of the problems above. It isn’t saturated, because it’s about your work. It isn’t contaminated, because nobody has seen it. It isn’t gamed, because no lab is optimizing for it. It’s small and noisy, but it measures the right thing, and that matters more.

Then watch for Goodhart in your own house

Once you have your own test set, you’ll be tempted to tune prompts until it hits 100%. Be careful. Your test set can become a target too. If you tweak a prompt forty times against the same 30 examples, you may just fit those examples. Keep a few cases aside that you don’t tune against, and keep adding new real ones over time.

And if you use an AI model to grade outputs, check that it isn’t simply preferring longer or more confident answers. That’s the arena’s bias, brought home.

Key ideas

Goodhart’s law (in Strathern’s phrasing): “When a measure becomes a target, it ceases to be a good measure.” AI benchmarks are a textbook case.
The Llama 4 Maverick episode showed how a leaderboard number can come from a variant tuned for the leaderboard, not the model people can use.
Each benchmark measures something narrow: MMLU quiz knowledge, GSM8K grade-school maths, SWE-bench fixing Python issues against tests, arenas human preference.
Benchmarks break in predictable ways: saturation (and answer-key errors), contamination (test data in training data), and gaming (private variants, tuning to voter tastes).
Public scores are upper bounds and starting points, not verdicts. Check the exact model, the benchmark’s health and who ran it.
Your own small, private test set beats any leaderboard for choosing a model for your task, as long as you don’t turn it into a target too.

Sources: Meta AI, “The Llama 4 herd” (5 April 2025); TechCrunch, “Meta’s benchmarks for its new AI models are a bit misleading” (6 April 2025), “Meta exec denies the company artificially boosted Llama 4’s benchmark scores” (7 April 2025) and “Meta’s vanilla Maverick AI model ranks below rivals on a popular chat benchmark” (11 April 2025); LMArena statement on X (8 April 2025), as quoted by Simon Willison (simonwillison.net, 8 April 2025); C. A. E. Goodhart, “Problems of Monetary Management: The UK Experience” (1975); Marilyn Strathern, “‘Improving ratings’: audit in the British University system” (European Review, 1997); Hendrycks et al., “Measuring Massive Multitask Language Understanding” (arXiv 2009.03300, ICLR 2021); Cobbe et al., “Training Verifiers to Solve Math Word Problems” (arXiv 2110.14168, 2021); Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (arXiv 2310.06770, ICLR 2024); OpenAI, “Introducing SWE-bench Verified” (13 August 2024); Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference” (arXiv 2403.04132, 2024); Phan et al., “Humanity’s Last Exam” (arXiv 2501.14249, 2025); Gema et al., “Are We Done with MMLU?” (arXiv 2406.04127, 2024); Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic” (arXiv 2405.00332, NeurIPS 2024); Singh et al., “The Leaderboard Illusion” (arXiv 2504.20879, 2025); BetaKit, “Cohere Labs head calls ‘unreliable’ AI leaderboard rankings a ‘crisis’ in the field” (2025).