Why Evaluation Is Harder Than Training

Training has one number to minimize. Evaluation has to decide what 'good' even means, and most of the ways that decision goes wrong are invisible until production.

Training a model is, mechanically, the easy part. You define a loss, and an optimizer drives it down. There is one number and one direction.

Evaluation has no such luxury. Before you can measure anything you have to decide what “good” means for your task, and that decision is where most of the difficulty lives. A model can score 0.99 on your metric and be useless, or score poorly and be exactly what you needed. The gap between “the number went up” and “the system got better” is the entire subject of this layer.

The metric is a proxy, never the goal

You care about something real: users find correct answers, fraud gets caught, the summary is faithful. None of those are directly measurable at scale, so you pick a number that correlates with them — accuracy, F1, BLEU — and optimize that instead.

The proxy is never the thing. A spam filter optimized for accuracy on a stream that is 99% legitimate mail can hit 99% by labeling everything “not spam” and catching zero spam. The number is excellent. The system is broken. Every metric in this layer is a proxy with a failure mode like this one, and knowing the failure mode matters more than knowing the formula.

Good answers rarely have one form

Classification has a clean target: the label is cat or it is not. Most real tasks are not that tidy.

• A translation has dozens of acceptable phrasings. Comparing to one reference punishes the other valid ones.
• A summary can be correct, faithful, and well-written yet share few words with the “gold” summary.
• An agent can reach the right result by three different tool sequences. Grading the sequence punishes legitimate variation.

When the space of correct outputs is large, any automatic score that compares against a fixed reference will misjudge good outputs as bad. This is why generation metrics diverge from human judgment, and why retrieval metrics reward finding a relevant document rather than the one you meant.

You are always measuring on a sample

You never evaluate on all possible inputs. You evaluate on a test set — a finite sample — and hope it represents the real distribution. Two things break that hope.

The sample is small. A 4-point accuracy difference on 200 examples is well within the range you would see from reshuffling the same model’s luck. Treating it as a real improvement is the single most common evaluation mistake, and statistical significance is the tool for catching it.

The sample drifts. The distribution your model sees in production moves over time — new user behavior, new topics, seasonal shifts. A test set frozen last quarter slowly stops describing today’s traffic, and your metric keeps reporting a world that no longer exists.

Fluent and wrong is a stable state

For generative systems there is a failure mode that has no analog in classical ML: the output reads perfectly and is factually false. A hallucinated citation is grammatical, confident, and plausible. Skimming ten outputs and finding them all “fine” tells you nothing, because fluency is exactly what the model is best at and correctness is what you failed to check.

This is why eyeballing does not scale and why subjective quality still needs structured human evaluation with explicit criteria rather than a gut reaction.

Optimizing a metric corrupts it

Once a measure becomes a target, people and models start gaming it — Goodhart’s law. Optimize for response length because longer answers scored better in review, and you get padding. Train against a public benchmark and the benchmark leaks into training data, so the score climbs while real capability does not. Benchmark contamination is this problem at field scale.

The defense is not a better single metric. It is a basket of metrics that are hard to satisfy simultaneously, plus periodic checks against ground truth that the system was never optimized against.

What to remember

• Training minimizes one defined loss; evaluation must first decide what “good” means, and that decision carries all the difficulty.
• Every metric is a proxy for something you actually care about, and each has a failure mode where the proxy looks great and the system is broken.
• You measure on a finite, drifting sample — small differences are often noise, so test for significance.
• Fluent-and-wrong defeats skimming; subjective quality needs structured human review.
• The moment a metric becomes a target it starts to corrupt (Goodhart), so evaluate against things the system was not optimized for.

Next: Classification Metrics