Query Transformation in RAG: Why Your Search Misses the Right Chunk
Query transformation is the step in RAG that rewrites your question before it searches, so mismatched wording stops costing you the answer.IntroductionYou open your company’s internal HR portal and type one question: “How much time off do new parents get?”Nothing useful comes back. Maybe one stray…
Query transformation is the step in RAG that rewrites your question before it searches, so mismatched wording stops costing you the answer.IntroductionYou open your company’s internal HR portal and type one question: “How much time off do new parents get?”Nothing useful comes back. Maybe one stray result about vacation days. Maybe nothing at all. The answer is sitting right there, in the actual policy, in plain language. It says an employee is entitled to 26 weeks of paid parental leave. Your search never found it.The policy says “parental leave.” You asked about “time off.” To a machine comparing words as numbers, those two phrases don’t look related at all.This is the part nobody warns you about when you start building with RAG. Having a vector database and a working embedding model feels like the whole job.But a search only works if your question and the answer use language that aligns closely. When they don’t, the system fails quietly. No crash, no error, just an empty or wrong result, and a reader who assumes the information isn’t there.The Problem Query Transformation SolvesHere’s why that happens. The system turns every question you type and every chunk in the database into a long list of numbers called embeddings. It compares those numbers, not the words themselves, looking for whichever chunk lands closest to your question.“Time off for new parents” and “parental leave” mean the same thing to you. To an embedding model, they can land far enough apart that the right chunk never makes the shortlist.This is the gap query transformation exists to close. Instead of sending your question into the database exactly as typed, you reshape it first, so it lands closer to how the answer is actually written. There is more than one way to do that, and each one fixes a slightly different version of the same problem.HyDE (Hypothetical Document Embeddings)Here’s a strange fix for that mismatch. Instead of sending your exact question into the search, you ask an AI to guess the answer first, then search using that guess instead.The AI doesn’t need to guess correctly. It just needs to sound like the real document. So you give it an instruction like this: write a short passage answering this question, as if it were from an HR policy document.The AI has never seen the actual policy. But it knows what policy language sounds like: formal, declarative, full of eligibility rules and timeframes.Here’s what it might write back:“New parents are entitled to several weeks of paid leave following the birth or adoption of a child. The exact duration depends on company policy and is typically granted after a minimum period of employment.”That’s fiction. There’s no 26 weeks in it, no mention of a 30-day application window. But look at its shape: eligibility, duration, a formal tone. That’s the same shape as the real paragraph sitting in the policy. So instead of embedding your original question, the system embeds this fabricated paragraph and searches using it.This is HyDE, short for Hypothetical Document Embeddings. The fabricated answer is discarded the moment the search runs. Nobody ever reads it.Its only job is to land closer, in that vector space, to how real answers are actually written. And it does. Two pieces of policy-sounding text sit nearer to each other than a casual question ever would.The real parental leave paragraph returns a high match score. Not because the AI knew the answer. Because it knew how to sound like one.Query Expansion addresses the same mismatch in a different way. Instead of guessing one answer, it asks for several different ways to ask the same question.Query ExpansionInstead of guessing at a single hypothetical answer, this approach asks the AI for something simpler: to say the same thing in several different ways.You send the original question to an AI with an instruction like this: generate three alternate phrasings of this question, using different vocabulary.It might come back with:1. “What is the parental leave duration policy?”2. “How many weeks of leave for maternity or paternity?”3. “Leave entitlement after having a baby”None of these throws away the original question’s meaning. That’s the whole idea behind Query Expansion: say the same thing enough different ways, and one of them is likely to use the same words as the real answer.Now the system doesn’t search once. It embeds all three variants separately, runs a search for each, then merges the results and removes duplicates.Look at variant one. It uses the exact phrase “parental leave,” the same words sitting in the real policy paragraph. Even if your original question never got close enough to match, this variant will almost certainly.The other two variants earn their keep differently. If the policy also had a separate chunk about maternity or paternity leave, phrased yet again, variant two or three might catch that one too. Your original question would have missed it completely.One question in, three chances to land near the right words out. That’s the trade Query Expansion makes: more searches to run, in exchange for a much better shot at hitting whichever phrasing the real answer happens to use.Step-back Prompting takes the opposite instinct. Instead of asking the question in three narrow ways, it first asks a broader question.Step-back PromptingInstead of narrowing in on the exact wording, this one zooms out. This is Step-back Prompting: before searching, it asks the AI a different kind of question. What’s the bigger question hiding behind this one?You send your original question to an AI with an instruction like this: What is a more general question behind this specific one?It might come back with something like:“What are the company’s leave policies for employees?”That’s a much broader question than the one you actually asked. But it’s still related. Your question about parental leave is one specific case of the bigger topic of leave policy in general.Now the system searches twice: once with your original, specific question, and once with this broader, step-back version. Then it combines whatever both searches find.The specific search still finds the parental leave paragraph, the one with the 26 weeks in it. But the broader search pulls in something the narrow question never would have reached. The general chunk about how leave requests are submitted 30 days in advance through the HR portal.Neither search alone tells the full story. The narrow question finds the number. The broad question finds the process around it. Taken together, the answer covers both how much time you get and how you actually go about requesting it.Three techniques, three different reshapes of the same question. The real question is which one to reach for, and when.Which One Do You Reach For?Each technique fixes a slightly different flavour of the same mismatch, so the right one depends on what’s actually broken.If your question is short and the real answer is written in a completely different style long, formal, declarative HyDE tends to work best. It fills that stylistic gap by fabricating something that already sounds like the source.If the mismatch is really about vocabulary, different words for the same idea, Query Expansion is the more direct fix. More phrasings means more chances one of them lines up with how the answer is actually written.If your question is narrow but the answer needs surrounding context to make sense, Step-back Prompting earns its place. It goes looking for the bigger picture your specific question sits inside.None of this is an either/or choice. Production RAG pipelines often run more than one of these together, then hand the combined, messier result to a reranker to sort out which chunks actually deserve to be there.That reshaping is the whole game here: change the question enough, and the right chunk finally gets a chance to be found.Wrapping UpThe question someone types is rarely the one that should be embedded, and query transformation is the entire discipline built around that observation. Whether you fabricate an answer, multiply the phrasing, or widen the scope, the goal is always the same: give the search something closer to how the real answer is actually written, before it ever gets compared.That’s a small idea with an outsized effect on whether your RAG system feels reliable or just unlucky.If this helped, follow me here for more on RAG, Spring AI, and building real AI systems in Java.Have you run into a query and answer that just wouldn’t match? Drop a comment; I’d like to hear how you fixed it.Let’s connect on LinkedIn: Akshay VadsaraThis story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!Query Transformation in RAG: Why Your Search Misses the Right Chunk was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI