Evaluating RAG

Measure retrieval and generation separately. Most RAG failures are retrieval failures, and a single quality score hides which half is broken.

A RAG system has two halves that fail independently. Retrieval finds passages; generation answers from them.

A single end-to-end quality score cannot tell you which half broke — and that is the only thing you need to know, because the fixes have nothing in common.

Measure them separately. Everything else in RAG evaluation follows from this.

Retrieval, measured first

One number matters more than all others: for a real question with a known answer, was the correct passage in the retrieved set?

If no, generation is irrelevant. No prompt change, no better model, no reranking fixes a passage that was never retrieved. This is the majority of RAG failures, and checking it first saves enormous wasted effort.

The standard metrics:

Recall@k — fraction of questions where the correct passage appeared in the top k. The headline number.

MRR (mean reciprocal rank) — averages 1/rank of the first correct passage. Rewards ranking the right passage high, which matters because attention is uneven across long contexts.

nDCG — accounts for graded relevance when passages are partially useful rather than simply right or wrong.

Building the dataset is the work. Collect real questions, then label which passages actually contain the answers. A hundred labelled questions is enough to be genuinely useful, and it takes an afternoon.

A shortcut worth knowing: take a passage, have a model generate a question it answers, and record the pair. This produces a large synthetic set cheaply. It skews toward questions phrased like the passage, so it overstates performance — but it is far better than no measurement, and it catches gross regressions.

Generation, measured separately

Given correct passages, does the model answer correctly?

Three properties, each failing differently:

Faithfulness. Is every claim supported by the provided context? Unsupported claims mean hallucination despite grounding — usually a prompt problem.

Relevance. Does the answer address the question? A faithful answer to a different question is still wrong.

Completeness. Does it use all the relevant retrieved information, or drop half of it? Material buried mid-context is what typically goes missing.

The useful diagnostic here is to run generation with known-correct passages supplied by hand. That isolates generation quality from retrieval quality completely, and it is the fastest way to establish which half you are debugging.

Scoring generation

Model-as-judge is the practical default at any volume. Send the question, context, and answer, and ask whether each claim is supported.

Faithfulness is unusually well-suited to this, because it is a verification task against provided text rather than a judgment of quality. The judge is checking entailment, not taste, and that is something models do reliably.

The usual judge cautions still apply: validate against human labels on a sample first, and prefer pairwise comparison over absolute scores when comparing two systems.

Claim decomposition improves faithfulness scoring. Split the answer into individual claims, check each against the context separately, and report the fraction supported. More reliable than judging a whole paragraph, and it tells you which claim was unsupported.

Human review remains ground truth. Twenty carefully reviewed cases beat two hundred sloppily scored ones.

What else to track

Refusal behaviour. Does the system say “not in the documents” when the answer genuinely is not? Test with questions your corpus cannot answer. A system that always answers has a serious problem that quality metrics on answerable questions will not surface.

Citation accuracy. Do cited sources actually contain the cited claims? Easy to check automatically, and frequently wrong.

Consistency. Run each question several times. Variance indicates an under-constrained prompt, which is a different fix from a wrong answer.

Cost and latency per query. A change that improves accuracy while tripling cost is a tradeoff, not a win.

Working the pipeline

Each stage can be measured in isolation, which turns debugging from guesswork into a sequence:

Change one stage, measure that stage. Improvements at one stage frequently do not show up end-to-end because a different stage is the binding constraint — which is exactly why per-stage measurement is necessary.

What to remember

• Measure retrieval and generation separately; a single score hides which half is broken.
• Recall@k first: if the correct passage was never retrieved, nothing downstream matters.
• Isolate generation by supplying known-correct context by hand.
• Score faithfulness by decomposing the answer into claims and checking each — a verification task judges handle well.
• Test unanswerable questions to check refusal behaviour, and verify citations automatically.

Next: Multi-Agent Systems