The Retrieval Toolkit That’s Quietly Deciding Whether Your RAG Pipeline Works.
A practical guide to building retrieval that finds the right evidence before your LLM starts answering.If you’ve built more than one RAG pipeline, you’ve probably hit the same wall: your embeddings are solid, your prompt is clean, but the answers are still off.Half the time it’s not the LLM’s…
A practical guide to building retrieval that finds the right evidence before your LLM starts answering.If you’ve built more than one RAG pipeline, you’ve probably hit the same wall: your embeddings are solid, your prompt is clean, but the answers are still off.Half the time it’s not the LLM’s fault; it’s what you fed it. Retrieval is the quiet bottleneck of RAG, and most of the failure modes trace back to a handful of techniques most teams either skip or misuse.This post walks through the full toolkit, how each piece works, why it exists, when to reach for it, and ends with how they all fit together in a production pipeline, plus what they cost you in resolve time.1. BM25 — Keyword Retrieval That Still Refuses to DieBM25 (Best Matching 25) is a sparse, lexical scoring algorithm, an improved TF-IDF. It ranks documents by term overlap with the query, adjusted for term frequency and document length.How it works: for each query term, BM25 rewards documents where that term appears often, but with diminishing returns (a term appearing 20 times isn’t 20x more relevant than once), and it penalizes overly long documents that rack up matches just by being verbose.from rank_bm25 import BM25Okapicorpus = [doc.split() for doc in documents] # tokenized docsbm25 = BM25Okapi(corpus)query = "vector database indexing".split()scores = bm25.get_scores(query)top_k = sorted(range(len(scores)), key=lambda i: -scores[i])[:5]Why use it: BM25 is exact-match aware. It doesn’t care about “meaning”; it cares about the literal token. That makes it excellent for:Product codes, SKUs, error codes, IDsNamed entities (people, companies, drug names)Rare or technical terms that embeddings tend to smear togetherWhen to use it: any domain where precision on specific terms matters more than paraphrase-tolerance: legal, medical, codebases, log search. It’s cheap, fast, and needs no training or GPU.Weakness: zero semantic understanding. “Car” and “automobile” are unrelated to BM25.2. Dense Vector Retrieval — Matching on MeaningDense retrieval represents queries and documents as dense embedding vectors (hundreds to thousands of dimensions, no zeros), generated by a neural encoder. Similarity is computed via cosine similarity or dot product, and nearest neighbors are found using an ANN index (HNSW, IVF, etc.).from sentence_transformers import SentenceTransformerimport numpy as npmodel = SentenceTransformer("BAAI/bge-small-en-v1.5")doc_embeddings = model.encode(documents, normalize_embeddings=True)query_embedding = model.encode([query], normalize_embeddings=True)scores = np.dot(doc_embeddings, query_embedding.T).flatten()top_k = np.argsort(-scores)[:5]Why use it: it captures semantic similarity — “executive resignation” retrieves “CEO stepped down” even without shared words. This is what most people mean when they say “semantic search,” though technically dense retrieval is the mechanism and semantic matching is the outcome.When to use it: conversational queries, paraphrased questions, cross-lingual retrieval, or any case where users won’t use your document’s exact vocabulary.Weakness: can drift on rare terms, exact codes, or numbers; the embedding space smooths over precision that BM25 preserves.3. Query Rewriting & HyDE — Fixing the Query Before You Even RetrieveSometimes the problem isn’t your retriever; it’s the query itself. Users type short, vague, or oddly phrased questions, and both BM25 and dense retrieval do best with well-formed, information-dense text.Query rewriting uses an LLM to expand or clarify the user’s query before retrieval, turning “how fast” into “what is the maximum throughput of the X indexing pipeline in documents per second.”HyDE (Hypothetical Document Embeddings) takes this further: instead of embedding the query directly, you ask an LLM to hallucinate a plausible answer, then embed that hypothetical answer and search for real documents near it. The intuition: a fake answer is often closer in embedding space to real answers than the original question is.hypothetical_doc = llm.generate(f"Write a short passage answering: {query}")hyde_embedding = model.encode([hypothetical_doc], normalize_embeddings=True)scores = np.dot(doc_embeddings, hyde_embedding.T).flatten()Why use it: short or ambiguous queries are one of the most common causes of poor recall; the retriever isn’t broken, the query just doesn’t carry enough signal.When to use it: chat-style interfaces with short, conversational queries, or domains where users’ vocabulary diverges from the corpus’s vocabulary.Weakness: adds an LLM call before retrieval even starts, extra latency and cost, and a bad hypothetical answer can drag retrieval in the wrong direction.4. Hybrid Search — Why Not Both?BM25 and dense retrieval fail on opposite ends. Hybrid search runs both retrievers in parallel and combines their result lists, catching exact-term matches and paraphrases in one pass. The catch: BM25 scores and cosine similarities live on completely different scales, so you can’t just average them. That’s where RRF comes in.5. RRF (Reciprocal Rank Fusion) — Combining Ranks, Not ScoresRRF sidesteps the score-normalization problem entirely by fusing based on rank position, not raw score.RRF_score(doc) = Σ 1 / (k + rank_r(doc)) for each ranker rk is a damping constant (commonly 60) that prevents a single #1 ranking from dominating the fused score.def reciprocal_rank_fusion(ranked_lists, k=60): scores = {} for ranked_list in ranked_lists: for rank, doc_id in enumerate(ranked_list): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1) return sorted(scores.items(), key=lambda x: -x[1])bm25_ranked = ["doc3", "doc1", "doc5"]dense_ranked = ["doc1", "doc4", "doc3"]fused = reciprocal_rank_fusion([bm25_ranked, dense_ranked])Why use it: documents ranked highly by multiple retrievers naturally float to the top, without needing to calibrate score scales. It’s the standard way to fuse hybrid search results.When to use it: any time you’re combining two or more ranked lists from heterogeneous retrievers — BM25 + dense, or even multiple embedding models.6. MMR (Maximal Marginal Relevance) — Killing RedundancyEven a perfect ranked list can be useless if the top 5 chunks all say the same thing. MMR re-ranks for diversity, not just relevance.MMR = λ · Sim(doc, query) − (1 − λ) · max(Sim(doc, selected))λ controls the relevance/diversity trade-off: λ = 1 is plain top-k; λ = 0 maximizes diversity regardless of relevance.import numpy as npdef mmr(query_vec, doc_vecs, doc_ids, lambda_param=0.5, top_k=5): selected, remaining = [], list(range(len(doc_ids))) sim_to_query = doc_vecs @ query_vec while len(selected) < top_k and remaining: if not selected: idx = remaining[np.argmax(sim_to_query[remaining])] else: mmr_scores = [] for i in remaining: sim_to_selected = max(doc_vecs[i] @ doc_vecs[j] for j in selected) mmr_scores.append( lambda_param * sim_to_query[i] - (1 - lambda_param) * sim_to_selected ) idx = remaining[np.argmax(mmr_scores)] selected.append(idx) remaining.remove(idx) return [doc_ids[i] for i in selected]Why use it: without it, you can burn your entire context window on five near-duplicate chunks instead of five chunks that actually cover different facets of the answer.When to use it: corpora with lots of overlapping or duplicated content, FAQ pages, product docs with repeated boilerplate, or any dataset with near-duplicate passages.7. Cross-Encoder Reranking — The Accuracy HeavyweightEverything so far (BM25, dense retrieval, RRF, MMR) is a “bi-encoder” world: queries and documents are embedded separately, then compared. A cross-encoder instead feeds the query and a candidate document together into a single transformer, letting the model directly attend across both texts. This produces much finer-grained relevance judgments than comparing pre-computed vectors.from sentence_transformers import CrossEncoderreranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")pairs = [(query, doc) for doc in candidate_docs]scores = reranker.predict(pairs)reranked = [doc for _, doc in sorted(zip(scores, candidate_docs), reverse=True)]Why use it: cross-encoders consistently outperform bi-encoder similarity on relevance ranking, because the model can reason jointly about query and document instead of comparing fixed vectors.When to use it: as a final polishing step on a small candidate set (typically the top 20–50 from RRF/MMR), never on your full corpus, since it requires one forward pass per document pair.Weakness: the biggest latency add-on on this list short of the LLM call itself; it doesn’t scale to large candidate pools, which is exactly why it sits at the end of the pipeline, not the start.8. MTTR (Mean Time to Resolve) — What All This Costs YouEvery technique above adds retrieval quality, but none of them are free; they all add latency, and in RAG, MTTR (Mean Time to Resolve) is the average end-to-end time from query submission to final answer. It’s the metric that tells you whether your carefully tuned pipeline is actually usable in production.Each stage adds its own tax:StageTypical latency costNotesQuery rewriting / HyDE (if used)~200ms–1sAn extra LLM call before retrieval even startsBM25 lookup~5–20msCheap, in-memory inverted indexDense vector search (ANN)~10–50msDepends on index size and type (HNSW vs. flat)RRF fusion~1–5msNegligible, pure arithmetic on rank listsMMR re-ranking~20–100msGrows with candidate pool size (it’s iterative and pairwise)Cross-encoder re-ranking (if used)~50–300msOften the biggest single cost short of generationLLM generation~500ms–several secondsUsually dominates total MTTRPractical takeaways for keeping MTTR in check:Run BM25 and dense retrieval in parallel, not sequentially; they’re independent.Apply MMR only to a small candidate pool (e.g., top 20–30 from RRF), not your entire corpus. MMR’s pairwise similarity checks scale poorly.Cache embeddings for frequent or repeated queries.If you add a cross-encoder re-ranker on top of RRF/MMR, understand it’s usually your biggest latency add after generation itself; reserve it for cases where the accuracy gain clearly justifies it.Track MTTR as a first-class metric alongside answer quality (e.g., faithfulness or recall@k); a system that’s 2% more accurate but 3x slower may not be worth shipping.Putting It TogetherA production-grade hybrid RAG retrieval pipeline typically looks like:Query rewriting / HyDE cleans up a vague query before retrieval starts (optional, adds latency).BM25 catches exact terms.Dense retrieval catches meaning.RRF merges both ranked lists without score-normalization headaches.MMR trims the fused list down to a diverse, non-redundant candidate set.Cross-encoder reranking does a final, high-precision pass on that smaller set.MTTR is the report card: the number that tells you if all of the above is actually shippable.Quick Decision Cheat SheetQueries are short, vague, or off-vocabulary? → Query rewriting / HyDENeed exact-match precision (codes, IDs, rare terms)? → BM25Need paraphrase/semantic tolerance? → Dense retrievalCombining both? → RRFGetting redundant/duplicate chunks in your context? → MMRNeed the highest possible relevance on a small final set? → Cross-encoder rerankingPipeline feels accurate but slow? → Measure MTTR stage-by-stage before optimizing blindlyNone of these techniques is a silver bullet on its own; the systems that perform best in production use all five together, tuned against real query logs rather than intuition.If you’re building RAG pipelines, I’d love to hear what retrieval stack you’re running; hybrid search + reranking is becoming the default, but the right combination still depends heavily on your corpus and latency budget.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!The Retrieval Toolkit That’s Quietly Deciding Whether Your RAG Pipeline Works. was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI