Retrieval and Ranking Metrics
Search and RAG return a ranked list, not a single answer. Recall@k, precision@k, MRR, and nDCG each score a different thing about that list.
On this page
A classifier returns one decision. A search engine or a RAG retriever returns a ranked list — the top k documents for a query. Evaluating a list is a different problem: you care not just whether the right documents are in there, but where they sit, because nobody reads past the top few and a generator only sees what you feed it. These metrics all score a ranked list, and they split into two questions: is the right stuff present, and is it near the top.
The setup
For each query you need a set of relevant documents — the ground-truth answers, labeled by humans or derived from clicks. The system returns a ranked list. Every metric compares the list to that relevant set. “@k” everywhere means “considering only the top k results”, because in practice only the top handful matter.
Recall@k and precision@k: is it present
These are precision and recall restricted to the top k.
- Recall@k = of all relevant documents, how many appear in the top k. Recall@5 = 0.8 means 80% of the documents you should have found are in the top 5.
- Precision@k = of the top k results, how many are relevant. Precision@5 = 0.6 means 3 of the top 5 are relevant.
For RAG, recall@k is usually the metric that matters most. If the passage holding the answer is not in the retrieved set, the generator cannot use it — no prompt change recovers information that was never fetched. Measuring retrieval recall separately from generation quality is the single most valuable diagnostic in a RAG system, because most RAG failures are silent retrieval misses.
Their flaw: both are order-blind within the top k. A relevant document at rank 1 and at rank 5 score identically. That is fine for feeding a generator that reads all k, and wrong for a search page where rank 1 is everything.
MRR: how high is the first hit
Mean reciprocal rank cares only about the position of the first relevant result. For each query, take the reciprocal of that rank — first place scores 1, second scores 1/2, third scores 1/3, tenth scores 1/10 — then average across queries.
MRR is the right metric when there is essentially one correct answer and getting it to the top is the whole job: a factual lookup, “find the login page”, a known-item search. It ignores everything after the first hit, so it is the wrong metric when a query has many relevant results you want surfaced.
nDCG: graded relevance, position-aware
The metrics above treat relevance as binary — a document is relevant or not. Reality is graded: for “best pizza in Chicago”, a perfect match, a decent match, and a barely-related page are three different things. nDCG (normalized discounted cumulative gain) handles graded relevance and position together, which is why it is the standard for ranking quality.
It builds in three steps:
- Gain: each result carries a relevance score (say 3 = perfect, 1 = marginal, 0 = irrelevant).
- Discounted cumulative gain (DCG): sum the gains, dividing each by a factor that grows with its rank, so a relevant result deep in the list contributes less. A great document at rank 8 helps little because few users reach it.
- Normalize: divide DCG by the DCG of the ideal ordering (the best possible ranking of those same documents). That gives nDCG, a 0-to-1 score where 1 is the perfect ranking.
Normalization is what makes nDCG comparable across queries with different numbers of relevant documents — a query with ten good answers and one with two are both scored against their own perfect list. That, plus graded relevance and position discounting, is why nDCG is the default for evaluating rerankers and search relevance.
Choosing among them
Match the metric to the task shape:
- RAG retrieval feeding a generator: recall@k first (is the answer passage present at all), precision@k second (how much noise is in the context).
- Known-item / single-answer search: MRR — get the one right answer to the top.
- Graded, multi-result ranking (web search, recommendations, reranking): nDCG — position and degree of relevance both count.
And measure retrieval on its own, before it is entangled with generation. A RAG evaluation that only scores the final answer cannot tell a retrieval miss from a generation error, and those need opposite fixes. Separating the two is the practical payoff of these metrics.
What to remember
- Retrieval and search return ranked lists; the metrics ask two things — is the relevant content present, and is it near the top.
- Recall@k (fraction of relevant docs in top k) is usually the key RAG metric: the generator cannot use what was never retrieved. Precision@k measures noise in the top k. Both ignore order within k.
- MRR scores the rank of the first relevant hit — right for single-answer search, blind to everything after it.
- nDCG combines graded relevance with position discounting and normalizes to the ideal ranking — the standard for ranking and reranking quality.
- Evaluate retrieval separately from generation so a retrieval miss is not mistaken for a generation error.