Hybrid Search

Vector search misses exact terms; keyword search misses meaning. Combining them is the highest-value upgrade to a basic retrieval pipeline.

Two retrieval methods fail in opposite directions.

Vector search finds meaning and misses exact strings. Query E4021 and it returns documents about error handling generally, because a rare identifier fragments into subword pieces with almost no semantic content.

Keyword search finds exact strings and misses meaning. Query “how do I stop my app crashing” and it will not surface a document titled “Handling uncaught exceptions” — no words overlap.

Running both and merging is the single highest-value improvement to a basic RAG pipeline, and it is cheaper to implement than most alternatives.

What each is for

Vector search wins on paraphrase, synonyms, conceptual similarity, and cross-lingual matching.

Keyword search wins on identifiers, error codes, function names, product SKUs, proper nouns, exact quotes, and rare technical terms. It also wins on anything that appeared too infrequently in the embedding model’s training to have a meaningful representation.

That second list is not an edge case. In technical documentation and support corpora, a large share of real queries contain exactly those elements.

BM25

The keyword half is almost always BM25 rather than naive word matching. Three ideas, all sensible:

Term frequency, with diminishing returns. A document mentioning your term ten times is more relevant than one mentioning it once — but not ten times more. BM25 saturates the contribution.

Inverse document frequency. A term appearing in every document tells you nothing; a rare term is highly informative. Matches on rare terms score much higher.

Length normalization. Long documents contain more words by accident, so raw counts favour them unfairly. BM25 adjusts.

Decades old, extremely fast, and still competitive. It is not a legacy fallback — it is genuinely better than embeddings at what it does.

Merging the results

The two methods produce scores on incomparable scales. A BM25 score of 14.2 and a cosine similarity of 0.83 cannot be added meaningfully, and normalizing them is fragile because BM25 has no fixed upper bound.

Reciprocal rank fusion avoids the problem entirely by using ranks instead of scores. Each document scores the sum, across both result lists, of 1 / (k + rank) where k is a small constant. A document ranked 1st by vectors and 3rd by keywords beats one ranked 8th by both.

This works because ranks are always comparable, no tuning is required, and it is robust to whatever scale either method produces. It is the standard approach and the right default.

Weighted score fusion normalizes and combines scores with a tunable weight. Offers control over the vector-keyword balance, at the cost of needing that tuning and being sensitive to score distribution shifts.

Start with reciprocal rank fusion.

Where it sits in the pipeline

query
  → vector search      → top 50
  → BM25 search        → top 50
  → reciprocal rank fusion → merged top 50
  → rerank             → top 5
  → generate

Hybrid retrieval and reranking solve different problems and compose. Hybrid improves recall — getting the right document into the candidate set at all. Reranking improves precision — ordering that set correctly.

Doing both is standard for a reason: hybrid finds what vectors alone would miss, reranking then sorts out a candidate set that fusion left roughly ordered.

Practical notes

Most vector databases support this natively. Check before building it yourself — the fusion logic is usually already there.

PostgreSQL does it well. pgvector for vectors plus built-in full-text search, joined in one query. For corpora up to a few million chunks this is often the whole answer, no separate systems required.

Index the same chunks both ways. Same chunk boundaries, same identifiers, so fusion is comparing like with like.

Retrieve generously from each side. 50 from each before fusion is a reasonable default. Fusion cannot recover a document neither method surfaced.

Measure separately. Track whether the correct chunk appeared in the vector results, the keyword results, or both. That tells you which half is doing the work on your corpus — and it is frequently not the half you expected.

What to remember

• Vector and keyword search fail in opposite directions; identifiers and rare terms are where vectors break down.
• BM25 handles the keyword half: saturating term frequency, inverse document frequency, length normalization.
• Reciprocal rank fusion merges by rank rather than score, avoiding the incomparable-scales problem — use it as the default.
• Hybrid improves recall, reranking improves precision; use both.
• Measure which half retrieves the correct chunk — the answer is often surprising.

Next: Query Rewriting and Expansion