Reading Model Benchmarks

Benchmark tables are marketing surfaces with real numbers in them. How to extract signal, and why your own twenty examples beat any leaderboard.

Every model launch comes with a table showing it ahead. The numbers are usually real. They are also selected, and the selection is doing most of the work.

Benchmarks are worth reading. They are worth reading with a specific set of questions in mind.

What benchmarks measure

Roughly four families:

Knowledge and reasoning — multiple-choice questions across academic subjects. Measures breadth of retained knowledge more than reasoning.

Math — from grade-school word problems to competition problems. Automatically verifiable, which makes them reliable to score and popular as targets.

Code — generate a function, run tests. Also automatically verifiable. Note that passing unit tests is not the same as being good code.

Human preference — pairwise comparisons aggregated into rankings. Measures what people prefer, which correlates with quality and also with length, formatting, and confidence.

That last distinction matters: automatically verifiable benchmarks measure correctness, preference benchmarks measure appeal, and those diverge.

Four reasons the numbers mislead

Contamination

Benchmarks are published on the internet. Training data comes from the internet. A model that saw the test set during pretraining is recalling, not solving.

Labs attempt decontamination, with varying rigour, and detecting it is genuinely hard. The practical signal: a model scoring well on an established benchmark and poorly on a freshly written one of similar difficulty is showing you contamination.

Selection

Published tables show benchmarks where the model wins. There is no obligation to report the rest. A table with unusual benchmark choices is worth noticing.

Prompt sensitivity

Scores shift substantially with prompt format, few-shot count, and answer extraction method. Labs report their best configuration; you get your own. Comparing two models’ self-reported numbers means comparing two different setups.

Saturation and target-fitting

Once a benchmark becomes a headline metric, it gets optimized for. Training mixtures shift toward benchmark-like data. Scores rise faster than capability, and the benchmark stops discriminating — which is why the popular ones cycle every couple of years.

What benchmarks are actually good for

Coarse tier placement. Distinguishing a small model from a frontier one. Reliable, because the gap is large.

Narrowing candidates. Reducing twelve options to three worth testing. Legitimate and useful.

Tracking the field. Whether capability on a class of task is improving.

Spotting specific strengths. A model far ahead on code and level elsewhere probably is better at code.

What they cannot tell you

Whether it works for your task. Aggregate scores over academic questions say little about classifying your support tickets.

How it behaves on your prompt. Instruction-following, format adherence, and refusal behaviour are what you will actually fight with, and they are barely measured.

Long-context quality. Advertised context length and benchmark scores both fail to capture whether the middle of a long input gets used.

Cost-adjusted value. A two-point gain at five times the price is a bad trade that no table shows.

Reliability across runs. Benchmarks report averages. Your production system cares about the failure rate.

The replacement

Twenty real examples from your workload, with expected outputs, run against three candidate models.

That is more informative than every leaderboard combined, and it takes an afternoon. It also becomes your eval, which you need anyway for prompt iteration and model updates.

The sequence that works: benchmarks narrow the field, your examples decide, your eval catches regressions afterward.

Practical notes

Prefer independent evaluations over self-reported tables when available.

Check the date. Both models and benchmarks move; a six-month-old comparison may describe nothing current.

Read what the benchmark contains before trusting a score on it. Some measure something quite different from what the name suggests.

Re-test after model updates. Providers change models under stable version names, and your eval is the only thing that will tell you.

What to remember

• Benchmarks are real numbers, selected for presentation.
• Four distortions: contamination, selective reporting, prompt sensitivity, and optimization toward the metric.
• Good for tier placement and narrowing candidates; useless for predicting performance on your specific task.
• They do not measure instruction-following, long-context quality, cost-adjusted value, or reliability.
• Twenty of your own examples beats any leaderboard — and doubles as the eval you need anyway.

Next: Mixture of Experts