Human Evaluation and Annotation Agreement
For subjective quality, humans are the ground truth. But humans disagree, so you have to measure whether your labels are consistent enough to trust.
On this page
Automatic metrics measure surface properties. For anything subjective — is this summary faithful, is this answer helpful, is this response safe — humans remain the ground truth. But humans are not a clean instrument. Ask three people to rate the same answer and you get three ratings, and if you cannot show your labels are consistent, no model trained or evaluated on them can be trusted either. Measuring that consistency is the real discipline of human evaluation.
Why human labels are the anchor
Every automatic metric is calibrated, ultimately, against human judgment. A generation metric is “good” only insofar as it correlates with what people think. A model-as-judge is validated by checking it against human labels. Human evaluation is the base of the whole measurement stack, which is exactly why its quality problems propagate everywhere: noisy human labels quietly corrupt every metric downstream.
The problem: humans disagree
Give the same 100 outputs to two annotators and they will not produce identical labels. Some disagreement is legitimate — the task is genuinely ambiguous. Much of it is not:
- Vague guidelines. “Rate helpfulness 1-5” means different things to different people, so the scale drifts between raters.
- Subjectivity. Tone, quality, and creativity have no fixed answer.
- Fatigue and drift. A rater’s standards shift over a long session; the same output scored in the morning and at hour six gets different numbers.
- Anchoring and order effects. The previous example biases the next one.
If your annotators disagree with each other, the labels are noise, and a model scored against noise gives a meaningless number. So before using human labels for anything, you measure how much the humans agree.
Measuring agreement
Raw agreement — the percentage of items two raters labeled identically — is the obvious measure and a misleading one, because it gives credit for agreements that happen by chance. If 90% of outputs are “acceptable”, two raters guessing randomly still agree most of the time. High raw agreement can be almost entirely luck.
The fix is a chance-corrected agreement statistic:
- Cohen’s kappa for two annotators. It subtracts the agreement expected by chance, so 0 means “no better than random” and 1 means “perfect”.
- Fleiss’ kappa generalizes it to more than two annotators.
- Krippendorff’s alpha handles multiple annotators, missing labels, and ordinal or continuous scales — the most flexible choice.
Rough reading of kappa: below 0.4 is weak (your guidelines or task need work), 0.4-0.6 is moderate, 0.6-0.8 is substantial, above 0.8 is strong. The exact bands vary by field, but the message is constant: low agreement means your labels are not yet a reliable measuring stick, no matter how many you collect.
Raising agreement
Low agreement is a fixable process problem, not a reason to give up on humans:
- Write concrete guidelines with examples. Replace “rate helpfulness” with specific criteria and worked cases for each score, including the hard boundary cases. This is the highest-leverage fix by far.
- Pilot and calibrate. Have annotators label a shared batch, review disagreements together, and refine the guidelines before scaling up.
- Prefer comparisons to absolute scores. “Is A or B better?” is far more consistent across people than “rate this 1-10”, because it removes the need to anchor a personal scale. Pairwise judgments are the reason ranking-based human evaluation is often more reliable than rating-based.
- Simplify the scale. A 3-point scale with clear anchors usually beats a 10-point scale where the middle points mean nothing stable.
- Use multiple annotators and aggregate. Majority vote or averaging cancels individual noise, and the spread itself flags genuinely ambiguous items worth a second look.
What this costs, and how to spend it
Human evaluation is slow and expensive, so treat annotator time as the scarce resource it is. A few hundred carefully labeled, high-agreement examples are worth more than thousands of rushed ones — noisy labels are not just less useful, they actively mislead. Spend the budget where automatic metrics are weakest: faithfulness, safety, tone, and overall quality judgments. Let cheap automatic checks handle format, exact match, and regression detection, and reserve humans for the subjective core they alone can judge. And once you have a trusted human-labeled set, you can use it to validate a model-as-judge and scale the cheaper signal with confidence.
What to remember
- Human judgment is the ground truth for subjective quality and the anchor that every automatic metric and model-judge is validated against.
- Humans disagree — from vague guidelines, subjectivity, fatigue, and ordering — so labels can be noise, and noisy labels corrupt everything downstream.
- Measure agreement with a chance-corrected statistic (Cohen’s/Fleiss’ kappa, Krippendorff’s alpha), not raw percent agreement, which credits luck.
- Raise agreement with concrete guidelines and examples, calibration pilots, pairwise comparisons over absolute scores, and multiple aggregated annotators.
- Human evaluation is costly — spend it on faithfulness, safety, and quality where automatic metrics fail, and favor fewer high-agreement labels over many noisy ones.