Contextual Retrieval

A chunk torn from its document loses the context that made it meaningful. Adding that context back at index time is unusually effective.

Consider this chunk, retrieved on its own:

The limit was raised to 500 requests per minute in this release. Existing integrations are unaffected.

Which product? Which release? Which endpoint? The chunk was perfectly clear in its document and is nearly useless in isolation.

This is the central weakness of chunking: the split that makes retrieval precise also destroys the context that made the text meaningful. Contextual retrieval puts it back.

Prepending context

The straightforward version. At index time, add a short description of where each chunk sits:

API Documentation > Rate Limits > v2.4 Changelog: The limit was raised to 500 requests per minute in this release. Existing integrations are unaffected.

The breadcrumb costs a few tokens and changes two things at once. The embedding now includes the topic, so the chunk matches queries about rate limits and about v2.4. And the model receiving the chunk knows what it is reading.

For structured documents this is nearly free — the heading path is already in the source. It is the highest return-per-effort improvement available in a chunking pipeline, and it is frequently skipped.

Generated context

Where structure is absent or insufficient, generate the context.

At index time, send each chunk plus its source document to a cheap model: write one sentence situating this chunk in the document. Prepend the result.

This handles what breadcrumbs cannot — resolving pronouns, naming the subject, identifying which version or entity is under discussion.

The cost is one cheap model call per chunk at index time, paid once. For a corpus of tens of thousands of chunks that is a real but bounded expense, and prompt caching helps substantially since the source document repeats across all of its own chunks.

Reported gains in retrieval accuracy from this technique are large enough that it is worth the indexing cost for any corpus you will query repeatedly.

Retrieve small, send large

A different approach to the same tension, and often the cleanest.

Embed and match on small precise chunks. When one matches, send the surrounding parent section to the model instead of the chunk itself.

You get precision in retrieval — small chunks have focused embeddings — and context in generation, because the model sees the full section. No generation cost at index time.

The tradeoff is context window consumption: parent sections are larger than chunks, so fewer fit. Works well when sections are moderately sized, less well when documents are long and flat.

Implementation is just a parent pointer stored with each chunk.

Neighbour expansion

The lighter variant: store links to adjacent chunks and expand a match to include its immediate neighbours.

Cheaper than parent retrieval and it handles the common case where an answer straddles a boundary. Less principled — the neighbours may be irrelevant — but it costs almost nothing to implement.

Combining with the rest

These techniques compose with the other retrieval improvements, and each addresses something distinct:

A pipeline using all four is substantially better than one using none, and the improvements are largely independent. Contextual retrieval and hybrid search both raise recall; reranking raises precision.

Practical notes

Prepend breadcrumbs first. Nearly free, and it establishes whether your problem is context loss at all.

Keep added context short. One sentence or one breadcrumb line. Long preambles dilute the chunk’s own embedding, which defeats the purpose.

Index the context with the chunk so it is part of what gets embedded. Adding it only at generation time helps the model but does nothing for retrieval — which is half the benefit.

Measure retrieval hit rate, as always: for real questions with known answers, did the correct chunk surface? That is the only number that tells you whether any of this worked.

What to remember

• Chunking destroys context; a chunk that was clear in its document can be meaningless alone.
• Prepend breadcrumbs (document > section > subsection) — nearly free, improves both embedding and comprehension.
• Generated context — one model-written sentence per chunk at index time — handles pronouns and missing subjects.
• Retrieve small, send large gets precision and context without index-time generation cost.
• Index the added context so it affects retrieval, not just generation.

Next: GraphRAG