How to Add Hybrid Search to a RAG Pipeline

Your semantic search works, and a customer still can't find part number PN-88-A21. Embeddings capture meaning, not exact strings, so a rare identifier can get lost in a pile of vaguely similar passages. This guide adds hybrid search to a RAG pipeline: a BM25 keyword ranker and a vector ranker, merged with reciprocal rank fusion (RRF), in Python. You'll finish with a hybrid_search() function and a 20-query test that tells you whether it beat what you had.

1. Install the libraries for hybrid search in RAG

pip install rank_bm25 sentence-transformers numpy

Then load your chunks. Use the same chunks you already embed for RAG, one JSON object per line.

import csv
import json
import re

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer

# docs.jsonl, one chunk per line: {"id": "kb-142", "text": "..."}
with open('docs.jsonl', encoding='utf-8') as f:
    docs = [json.loads(line) for line in f]

Both rankers must see the same chunks with the same ids. Otherwise their ranks can't be merged.

2. Tokenize so part numbers and error codes survive

The rank_bm25 README says the package does no text preprocessing, so whatever you do to documents you must do to queries. A plain split(' ') leaves punctuation glued on, so ERR-4012: never matches ERR-4012.

TOKEN = re.compile(r'[a-z0-9]+(?:[-_./][a-z0-9]+)*')
PARTS = re.compile(r'[-_./]')

def tokenize(text):
    tokens = []
    for token in TOKEN.findall(text.lower()):
        tokens.append(token)
        parts = PARTS.split(token)
        if len(parts) > 1:
            tokens.extend(parts)
    return tokens

This lowercases, keeps a hyphenated code whole, and also emits its parts, so a search for 4012 still finds ERR-4012.

3. Rank chunks with BM25

BM25 scores a chunk by the query terms it contains, weighting rare terms more heavily. That is exactly what a part number needs.

bm25 = BM25Okapi([tokenize(d['text']) for d in docs])

def bm25_ranking(query, depth=20):
    scores = bm25.get_scores(tokenize(query))
    order = np.argsort(scores)[::-1][:depth]
    return [int(i) for i in order if scores[i] > 0]

get_scores returns one score per chunk in corpus order. The scores[i] > 0 filter drops chunks that share no terms with the query, so the keyword list only contains real matches.

4. Rank chunks by vector similarity

model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
doc_vecs = model.encode([d['text'] for d in docs], normalize_embeddings=True)

def vector_ranking(query, depth=20):
    query_vec = model.encode(query, normalize_embeddings=True)
    scores = doc_vecs @ query_vec
    order = np.argsort(scores)[::-1][:depth]
    return [int(i) for i in order]

This model is a stand-in from the Sentence Transformers docs. Use whatever embedding model your pipeline already runs, for both chunks and queries. With normalize_embeddings=True the vectors have length 1, so the dot product is cosine similarity.

5. Merge both lists with reciprocal rank fusion

BM25 scores are unbounded and cosine scores sit in a narrow band, so you can't add them. RRF sidesteps this by ignoring scores. Each chunk earns 1 / (k + rank) from every list it appears in, and the earnings are summed. The original RRF paper by Cormack, Clarke and Büttcher uses k = 60.

def rrf(rankings, k=60, top_n=5):
    fused = {}
    for ranking in rankings:
        for rank, doc_index in enumerate(ranking, start=1):
            fused[doc_index] = fused.get(doc_index, 0.0) + 1.0 / (k + rank)
    return sorted(fused, key=fused.get, reverse=True)[:top_n]

def hybrid_search(query, top_n=5):
    return rrf([bm25_ranking(query), vector_ranking(query)], top_n=top_n)

for i in hybrid_search('ERR-4012 pump fault'):
    print(docs[i]['id'], docs[i]['text'][:80])

Agreement is rewarded: a chunk ranked 20th in both lists scores 2/80, which beats a chunk ranked first in only one list at 1/61. Feed the top results to your LLM as context, exactly where your vector-only results went before.

6. Test hybrid search on 20 real user queries

Create queries.csv with the columns query,expected_id. Pull queries from search logs, support tickets and every 'it can't find X' complaint, and make sure a good share contain exact identifiers. The expected_id is a chunk a human confirmed answers the query.

with open('queries.csv', newline='', encoding='utf-8') as f:
    cases = [(row['query'], row['expected_id']) for row in csv.DictReader(f)]

def hit_rate(search_fn, cases, n=5):
    hits = 0
    for query, expected_id in cases:
        top_ids = [docs[i]['id'] for i in search_fn(query)[:n]]
        hits += expected_id in top_ids
    return hits / len(cases)

for name, fn in [('bm25', bm25_ranking), ('vector', vector_ranking), ('hybrid', hybrid_search)]:
    print(f'{name:7s} hit@5 = {hit_rate(fn, cases):.0%}')

Read every query where hybrid misses. Each miss tells you whether to fix tokenization, chunking or k. Keep the CSV as a regression test: every new 'search can't find X' ticket becomes a new row.

When pure vector search is enough

If your queries are mostly natural-language questions, your corpus is prose with few identifiers, and vector hit@5 matches hybrid on your test, skip this. Hybrid adds a second index to keep in sync and a tokenizer to maintain. Let the test decide, and ship whichever scores better.

Gotchas

• Tokenize both sides identically. A query that goes through a different tokenizer than the chunks will quietly match nothing.
• Never add raw scores. BM25 and cosine scores live on different scales. Fuse ranks, not scores.
• k = 60 is a default, not a law. The paper fixed it in a pilot. Change it only if your test says so, and with 20 queries don't chase a one-query win.
• Exact hits can get outvoted. If identifier matches found only by BM25 sink below chunks both lists half-like, try weighting the BM25 list higher, and only if the test improves.
• rank_bm25 scores every chunk in memory on every query. That suits a prototype. At scale, move the keyword side into your search engine or Postgres. The pgvector README has a Hybrid Search section that pairs Postgres full-text search with vectors and RRF. Its full-text ranking is ts_rank_cd, which is not BM25, so rerun your test after switching.

Hybrid retrieval is cheap to add and easy to measure, which in production turns the 'search can't find the part number' escalation into a failing test you can fix before customers see it.