Optimizing RAG: The Techniques and Metrics That Actually Matter
Building a RAG system is easy. Making it reliable — and proving it’s reliable — is the real work. This guide is about the second part.Most articles will happily teach you how to build a Retrieval-Augmented Generation (RAG) system: chop documents into chunks, turn them into vectors, search, and let…
Building a RAG system is easy. Making it reliable — and proving it’s reliable — is the real work. This guide is about the second part.Most articles will happily teach you how to build a Retrieval-Augmented Generation (RAG) system: chop documents into chunks, turn them into vectors, search, and let the LLM answer. That’s well-trodden ground. The interesting question, the one that separates a demo from something a business can trust, is:How do you make RAG better, and how do you know it got better?That’s what this guide is about.First, a plain-English mental modelImagine you hired a brilliant assistant who has a photographic memory but has never read your company’s documents. Every time you ask a question, you first run to the filing cabinet, grab the few pages you think are relevant, hand them over, and then the assistant answers using only those pages.That’s RAG. And once you see it this way, the two things that can go wrong become obvious:You grabbed the wrong pages. (A retrieval problem.)You grabbed the right pages, but the assistant wrote a poor answer. (A generation problem.)Almost every RAG failure is one of these two. So optimizing RAG means improving each, and measuring each separately, because a great answer built on lucky retrieval is a time bomb.Part 1: Techniques to Optimize RAGOnce the basic RAG pipeline is working, the biggest improvements usually come from better chunking, query rewriting, and reranking.1. Smarter ChunkingA common approach is to split documents every N characters. The problem is that this can cut sentences or ideas in the middle, making individual chunks difficult to understand.Instead, create meaningful, self-contained chunks based on the document structure or semantic boundaries. You can also add a short headline and summary to each chunk.Why it helps: The user’s query may not use the same words as the original document. A headline or summary gives the retriever additional context to match against.2. Query RewritingUsers rarely ask questions in a search-friendly format.For example:“Hey, do you remember who ended up winning that innovation award thing last year?”can be rewritten as:“innovation award winner 2023”The rewritten query can then be used for retrieval instead of the original conversational question.Why it helps: It removes unnecessary conversational context and gives the retrieval system a clearer search intent.3. RerankingVector search is fast, but the initial ranking is not always perfect. The most relevant document may appear several positions below less relevant results.A better approach is to retrieve more candidates first — for example, the top 20 — and then use a reranker or LLM to reorder them based on actual relevance.Top 20 results → Rerank → Best resultsWhy it helps: The most relevant information moves to the top of the context, increasing the chance that the LLM uses the right information when generating the answer.4. Put It TogetherThese techniques work best as a pipeline:Each step addresses a different failure point. Together, they can make a RAG system more reliable without necessarily requiring a larger or more expensive LLM.Part 2: How to Prove It Got Better — The MetricsYou can’t improve what you don’t measure.Start with a small test set of real questions. For each question, define:The facts or keywords that should appear in the retrieved contextA gold-standard reference answerA question type: factual, comparison, or multi-documentA few dozen good questions are enough to start.Now split the evaluation into the same two buckets as the two RAG failure modes:Bucket 1: Did We Grab the Right Pages?Did we retrieve the right information?MRR (Mean Reciprocal Rank)— How Near the Top?Score each question by the position of the first right result:Right page is #1 → score 1.0Right page is #2 → score 0.5 (that’s 1 ÷ 2)Right page is #5 → score 0.2 (that’s 1 ÷ 5)Not found at all → 0.0Then average that across all your test questions. That average is the MRR.Why it matters: MRR tells you whether relevant information is moving toward the top. It’s especially useful for measuring whether reranking is working.nDCG (Normalized Discounted Cumulative Gain)— Is the Whole List Well-Ordered?The scary name hides a simple idea. MRR only cares about the first correct hit. But often several pages are relevant, and you want all of them near the top. nDCG grades the quality of the entire ordering.Break the name into three plain pieces:Gain — you earn points for each relevant page in your results.Discounted — a relevant page earns less the further down it sits. (A great result at position 1 is worth full points; the same result at position 8 is heavily discounted.)Normalized — the final score is scaled to a clean 0 to 1, where 1.0 means “perfectly ordered — every relevant page is as high as it could possibly be.” This scaling is what lets you fairly compare an easy question against a hard one.Why you should care: For questions where multiple facts matter (comparisons, or answers spread across documents), nDCG catches problems that MRR misses. Track them together: MRR asks “how fast do I hit the first right answer?”; nDCG asks “how good is the whole ranked list?” Good optimization pushes both upward.Recall@K and Precision@KThese measure the contents of your top K results.Recall@K: How much of the relevant information did we retrieve?Precision@K: How much of what we retrieved is actually relevant?For example:3 relevant results out of 10 → Precision@10 = 0.303 retrieved out of 4 relevant results → Recall@10 = 0.75The goal is high recall with reasonable precision — get the answer without filling the context with noise.Keyword Coverage — The Smoke AlarmCheck whether the facts required to answer the question appear in the retrieved context.If coverage is low, stop there. No amount of reranking or prompt tuning can help if the answer was never retrieved.How to Read the Metrics TogetherBucket 2: Was the Final Answer Any Good?Finding the right pages doesn’t guarantee a good answer. The LLM can still miss information, give an incorrect answer, or add unnecessary content.Instead of manually grading thousands of responses, use a second LLM as an LLM-as-a-Judge.Give it:The questionThe gold-standard answerThe generated answerThen score three dimensions, typically from 1 to 5:Keep the judge strict. A judge that gives everyone a 5 tells you nothing.Even better, ask it to explain why points were deducted. That turns evaluation into actionable feedback.Measure → Diagnose → Improve → Measure againThat’s how RAG optimization moves from guesswork to an engineering process.Part 3: The optimization loopOnce the metrics are in place, improving RAG stops being guesswork and becomes a simple, repeatable loop:Measure across your whole test set — MRR, nDCG, coverage, and answer scores, broken down by question type.Diagnose the weakest spot using the “read them together” guide above.Turn exactly one knob — swap the embedding model, change chunk size, add reranking, tweak the prompt. One at a time, so you know what caused the change.Re-measure. Did the numbers go up? Keep the change. Did they drop? Revert it.Repeat.This is the whole difference between “I think reranking helped” and “reranking lifted MRR from 0.41 to 0.68 and answer completeness from 3.9 to 4.4.” One is a feeling. The other is evidence.The takeawaysBuilding RAG is the easy part. Optimizing it — and proving the optimization — is where the real value is.Every RAG failure is either bad retrieval or bad generation. Measure them separately.The metrics that matter:MRR — is the first correct page near the top?nDCG — is the whole list of results well-ordered, with lower positions worth less?Recall@K / Precision@K — did we grab enough of the relevant pages (recall) without drowning them in junk (precision)?Keyword coverage — did we even retrieve the needed facts? (Your smoke alarm.)LLM-as-a-Judge — is the final answer accurate, complete, and on-point?Break metrics down by question type so you fix the right thing.Change one knob at a time, then re-measure. That’s the entire game.Don’t optimize in the dark. Put a number on it, then make the number go up.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Optimizing RAG: The Techniques and Metrics That Actually Matter was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI