What AI benchmarks actually measure

Benchmarks are standardised tests used to compare AI models — but a high score doesn’t always mean a model is better for your task.

This guide explains what common benchmarks try to measure, from broad knowledge tests to reasoning and coding suites. It also covers why scores can mislead: test questions sometimes leak into training data, and labs naturally highlight the numbers that flatter their model. The practical takeaway is to treat leaderboards as a starting point, then test a model on your own real work before trusting it.

Sources: written in plain English from publicly available benchmark documentation and the labs’ own model cards. Where this post draws on a specific report, it is linked inline.