What AI Benchmarks Actually Measure: 5 Reasons Scores Mislead

A benchmark score is a mark on a fixed exam that a language model sat, and the interesting question is always what the exam contained and who had seen it beforehand. Two models a point apart on a public leaderboard are, in most cases, indistinguishable in use. The score is real, and the ranking it implies frequently is not.

Key takeaways

A benchmark is a fixed exam, and every public one is on the open internet, so the next model trained on a web crawl has probably read it.
Contamination is permanent. A published benchmark can never become clean again, and every model released afterwards is more suspect than the last.
A saturated benchmark, where leading scores cluster above roughly ninety percent, carries no information in its ordering. Read the top few entries as a tie.
The same model and benchmark produce different scores depending on prompt, examples shown, attempts allowed and how the answer was extracted. Comparisons only hold within one evaluation run by one party.

A Benchmark Is a Fixed Exam, and Exams Leak#

Every public benchmark has the same structural weakness. The questions and answers are published so that results can be reproduced, which means they are on the open internet, which means the next model trained on a web crawl has probably read them.

This is called contamination, and it is not hypothetical. According to Zhu and others (2024), the MMLU-CF paper was written specifically because the original benchmark's questions had spread widely enough that clean measurement had become difficult, and the authors built a contamination-free variant to restore it.

The problem reaches the benchmarks assumed to be safest. OpenAI's own audit of SWE-bench Verified, a set of real GitHub issues that was positioned as the serious frontier test for coding, found that frontier models could reproduce parts of the reference solutions verbatim for some tasks. A model that has memorised the answer scores the same as a model that reasoned to it, and the score cannot tell them apart.

Notably, contamination is permanent and one-directional. Once a benchmark is public it can never become uncontaminated, and every model released afterwards is more suspect than the last.

Saturation Makes the Top of a Leaderboard Meaningless#

The second problem arrives from the opposite direction. A benchmark stops being useful once the leading models answer nearly everything correctly, because the remaining questions are mostly the ambiguous and mislabelled ones.

MMLU, introduced by Hendrycks and others (2020), was a genuinely hard test when published. Frontier models now sit near its ceiling, and the difference between the top few entries is within the range that labelling errors alone could produce. The same has happened to HumanEval, the code benchmark introduced by Chen and others (2021), which is now routinely described as saturated.

The chart below shows the pattern that recurs with every benchmark, where a test discriminates well for a period and then stops.

Line chart showing benchmark scores rising steeply then flattening near a ceiling, with the useful discriminating period marked in the middle

Source: illustrative shape describing the saturation pattern reported across MMLU, HumanEval and GSM8K. Not measured data from a single benchmark run.

The consequence for a reader is direct. When the top entries on a leaderboard are separated by one or two points on a saturated benchmark, the ordering carries almost no information, and treating it as a ranking is a mistake the leaderboard's own authors would not make.

Figure below traces why that degradation is structural rather than anyone's fault.

Diagram of the contamination cycle, from a benchmark being published, to its questions spreading across the web, to the next model being trained on a crawl that includes them

Source: MMLU-CF (arXiv 2412.15194); OpenAI's audit of SWE-bench Verified.

What the Named Benchmarks Actually Test#

The names appear in every model announcement and are rarely explained. Each measures something quite narrow.

Table: what the most frequently cited benchmarks actually contain, and what they do not tell you.

Benchmark	What it contains	What it does not measure
MMLU	57 subjects of multiple-choice exam questions	Whether the model can write, reason at length, or admit uncertainty
GPQA Diamond	Graduate-level science questions written to resist web search	Anything outside physics, chemistry and biology
HumanEval	164 small Python functions with unit tests	Whether the model can work in a real codebase
SWE-bench Verified	Real GitHub issues, scored on whether tests pass	Code quality, maintainability, or whether the fix is sensible
LiveCodeBench	Competition problems collected after model cutoffs	Practical software engineering
Chatbot Arena	Human pairwise preference votes	Correctness, since voters reward tone and confidence

Sources: the benchmark papers and project pages linked below.

Two entries in that table deserve emphasis. GPQA, introduced by Rein and others (2023), is deliberately "Google-proof", meaning the questions were validated to resist answering by web search, which makes a high score harder to fake through retrieval. Chatbot Arena sits at the other end, measuring which answer people preferred, which correlates with helpfulness and also with confident phrasing. A model that sounds surer of itself can win votes while being wrong, which is the same failure described in our piece on AI hallucination, and this is a known limitation rather than a criticism of the method.

The Number Depends on How It Was Run#

The same model and the same benchmark can produce materially different scores, and the reason is rarely disclosed alongside the number.

Scores move with the prompt used, the number of examples shown before the question, whether the model was allowed to reason step by step, how many attempts it was given, and how the answer was extracted from its output. A vendor reporting its own result chooses all of those settings. An independent evaluation makes different choices, and a gap of several points between the two is common without either party doing anything improper.

This is because a benchmark specifies the questions, not the procedure, and the procedure is where most of the variance lives. In addition, vendor figures often come from an API configuration at a particular reasoning effort rather than the consumer product of the same name, so the number quoted may not describe what a user of the app would experience.

The practical implication is that comparisons are only meaningful within a single evaluation run by a single party using one method. Comparing a figure from one vendor's announcement against another vendor's announcement compares two different experiments.

Reading a Leaderboard Without Being Misled#

Four questions make a leaderboard useful rather than misleading, and they take about a minute to ask.

First, when was the benchmark published relative to the model. This is because a benchmark older than the model's training cutoff is a contamination candidate, and one collected afterwards, as LiveCodeBench does deliberately, is not.

Second, is the benchmark saturated. If the leading scores cluster above roughly ninety percent, the ordering among them is noise and should be read as a tie.

Third, who ran the evaluation and under what settings. An independent evaluation that applies identical settings to every model is worth considerably more than a set of self-reported figures.

Fourth, and most usefully, does the benchmark resemble the work. A model that tops a graduate physics test may be no better at summarising meeting notes, because those are unrelated capabilities and nothing about the first predicts the second.

For a beginner the honest recommendation is to stop using leaderboards to choose between the top few models. This is because they have converged closely enough that the differences reported are smaller than the differences caused by how you prompt them. Our comparison of ChatGPT, Claude and Gemini takes that approach rather than reprinting scores. Assemble ten tasks that resemble what you actually need, run them through two or three candidates, and read the outputs. That takes an afternoon and produces a better answer than any public ranking, because it measures the only benchmark that applies to your situation.

The same caution applies when comparing open and closed models, where leaderboard position is the weakest of the differences that matter. Benchmarks remain valuable to the people building models, where a controlled comparison across versions is exactly what is needed. The failure happens in translation, when a number designed to guide research is reprinted as a consumer verdict.

Common questions#

Why do vendor benchmark scores differ from independent ones?

Because a benchmark specifies the questions and not the procedure. The prompt, the number of examples shown, whether step by step reasoning was allowed, and how the answer was extracted all move the number, and a vendor reporting its own result chooses all of them.

What does it mean that a benchmark is saturated?

It means the leading models answer nearly everything correctly, so the remaining questions are disproportionately the ambiguous and mislabelled ones. The difference between the top entries then falls within the range that labelling errors alone could produce.

How should a beginner choose between models then?

Assemble roughly ten tasks that resemble the work you actually need done, run them through two or three candidates, and read the outputs. That takes an afternoon and measures the only benchmark that applies to your situation.

Sources#

Last reviewed: 4 September 2026. Benchmark leaderboards change weekly and saturation moves with each model release, so treat the named examples as illustrative of the pattern rather than a current ranking. Re-checked quarterly.