Cache Me If You Can: The Secret Life of Q, K, V — and Why Your RAG App Is Still Slow

A field guide to KV, Prefix, Prompt, and Semantic caching: who controls what, what gets thrown away, and what a developer should actually do about it.If you’ve built a RAG (Retrieval-Augmented Generation) system, you’ve probably done the “obvious” optimizations: faster vector search, quantized…

A field guide to KV, Prefix, Prompt, and Semantic caching: who controls what, what gets thrown away, and what a developer should actually do about it.If you’ve built a RAG (Retrieval-Augmented Generation) system, you’ve probably done the “obvious” optimizations: faster vector search, quantized embeddings, a leaner reranker. And yet the system still feels sluggish. There’s a good reason for that, and it has almost nothing to do with retrieval.The bottleneck nobody profilesA typical RAG request breaks down like this: embed the query (a few milliseconds), run approximate nearest neighbor search over your vector store (tens of milliseconds), rerank a handful of candidates (maybe a hundred milliseconds), and then hand several thousand tokens of retrieved text to the model.That last step is where the time actually goes.Reading that assembled prompt — system instructions plus retrieved chunks plus the question — is called prefill, and it can take seconds, dwarfing everything retrieval did. If you’ve only optimized your vector search, you’ve been polishing the cheap 5% of the pipeline.To understand why prefill is so expensive, we need to go one level deeper, into how a model reads a prompt at all, and what it keeps in memory while doing it.Two phases, two completely different bottlenecksEvery time a language model responds to a prompt, it goes through two distinct phases that behave nothing alike.Prefill happens first: the model reads your entire input, every token of it, all at once, in parallel, layer by layer. During this pass, each layer computes two things for every token: a Key and a Value. Together, these get stored as the KV cache. This phase is compute-bound, and the killer detail is that its cost is quadratic in input length, doubling your prompt roughly quadruples the work. That’s why stuffing more retrieved chunks into a prompt gets disproportionately expensive.Decoding happens after: the model generates output one token at a time. Each new token computes its own Key/Value, appends them to the cache built during prefill, and attends back over everything already stored. This phase is memory-bandwidth-bound; the bottleneck isn’t computation, it’s shuttling the model’s weights from memory for every single token produced.This asymmetry explains something you’ve probably noticed without naming it: a long prompt with a short answer feels slow to start and then finishes fast. A short prompt with a long answer starts instantly and then trickles out. Different bottlenecks, different fixes, and RAG’s latency problem lives almost entirely in the first one.Built once, then gone: the part everyone forgetsHere’s the detail that trips people up. The KV cache isn’t a permanent structure sitting somewhere waiting to be reused forever. By default, it’s built fresh for one request, used during that request’s decode phase, and then thrown away the moment the response finishes.Every layer of the model computes its own Key/Value pairs during prefill; a 32-layer model keeps 32 separate caches, not one shared cache, and that whole structure lives and dies with the request. Unless something actively intervenes to keep it around (which is exactly what prefix caching does, more on that below), the next request starts from zero, no matter how similar it is to the one before it.This is why cache memory adds up fast, too: it scales with sequence length and model depth. An 8,000-token context in a 32-layer, 32-head model can occupy several gigabytes for a single request, memory that’s locked up and unavailable to anyone else until that request finishes and the cache is discarded.Q, K, V — and who actually controls whatInside each layer, the mechanism doing the work is self-attention, and it converts every token into three vectors:Query (Q) — “what am I looking for?”Key (K) — “what do I contain that others might search for?”Value (V) — “what do I actually offer, if someone attends to me?”The clearest way to see why only two of these three get cached is to ask a simple question: who controls each one, and how long does that control last?Think of it as a library visit. You walk in with a question; that’s your Query. It’s yours, it exists for this one visit, and the moment you walk out, it’s gone. Nobody else will ever reuse your exact question.The Keys are the labels on the books already sitting on the shelf, and the Values are the actual contents of those books; both of these are controlled by the library itself, not by you. They were there before you arrived, and they’ll still be there for the next visitor.That’s the entire reason it’s called a KV cache, not a QKV cache. Ownership decides what’s worth keeping: what a single request controls dies with that request (Q). What the shared context controls outlives any single request (K, V), and that’s precisely the part worth holding onto.Prefix caching: reusing work across requestsNow zoom out from a single request. If your system serves millions of requests and many of them share identical leading text — the same system prompt, say — recomputing that shared prefix’s KV cache every single time is pure waste.Prefix caching is the fix: reuse a previously computed KV cache for a new request if its beginning exactly matches a previous one, instead of letting it die with the request that built it.Engines like vLLM implement this by splitting the KV cache into fixed-size blocks (commonly 16 tokens) rather than one long contiguous buffer, which enables sharing identical blocks across requests instead of duplicating them.The catch is the hashing rule that decides “identical”: each block’s hash depends on its own tokens plus the hash of every block before it. A block is only reusable if the entire prefix up to that point matches, in the same order, unbroken.This is exactly why prefix caching helps a shared system prompt beautifully but does almost nothing for the retrieved-chunks portion of a RAG prompt. Different queries retrieve different chunks, often in a different order, and one reordered or substituted chunk early on invalidates every cached block after it, even if the content itself was computed once before.Prompt caching — the version exposed as a developer-facing API feature, is mechanically the same idea, just handed to you as a toggle: mark a portion of your prompt as cacheable, and a matching literal prefix in a later request skips reprocessing. Same mechanism, same weakness: it only pays off for the genuinely static part of your prompt, not for content that changes and reorders on every call.Semantic caching is a different animal entirely; it doesn’t touch the KV cache at all. It operates at the level of meaning: embed the incoming query, compare it against embeddings of previously-answered queries, and if something similar enough has already been handled, return the cached answer directly, skipping the LLM call. The tradeoff is exactness for spee, prefix and prompt caching are guaranteed bit-identical to a fresh computation, while semantic caching gives that guarantee up in exchange for skipping generation on repeat-ish traffic.What a developer should actually focus onAll of this theory collapses into a short list of things worth actually doing.Order your prompt deliberately. Static content (system instructions) first, dynamic/retrieved content last; only the static part can ever be cached across requests.Turn on prefix or prompt caching for the static part. It’s close to free latency and cost savings on the portion that never changes.Don’t expect caching to save reordered chunks. If your retriever returns the same chunks in a different order each time, prefix and prompt caching won’t help; chunk order stability matters more than most people realize.Watch context length like a cost, not a convenience. Prefill cost is quadratic, not linear. Going from 4K to 16K tokens isn’t 4x the cost; budget context size deliberately instead of stuffing in “just one more chunk.”Use semantic caching only where being wrong occasionally is cheap. Great for repeated, FAQ-style queries. Riskier for precision-sensitive answers, since “semantically similar” is not “the same question.”Profile time-to-first-token separately from total latency. A delay before the first word appears is a prefill problem; fix the prompt. A slow trickle afterward is a decode problem, a completely different fix.The takeawayThe KV cache is the foundation; it exists inside every single inference call, full stop, and it dies with the request that built it unless something deliberately keeps it alive. Prefix and prompt caching are strategies for keeping it alive across requests, but only when the beginning of the prompt matches exactly, precisely the assumption RAG prompts violate by design.Semantic caching sidesteps the token-level problem entirely by caching at the level of meaning, trading a small risk of imprecision for the ability to skip generation altogether.If your snappy vector database didn’t make the app feel fast, now you know where to actually look. Prefill, not retrieval, is where the RAG latency budget goes, and which caching strategy helps depends entirely on what part of your prompt is actually shared between requests, and who really controls it.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Cache Me If You Can: The Secret Life of Q, K, V — and Why Your RAG App Is Still Slow was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source: Generative AI Pub — Published September 7, 2026 at 02:14 UTC — Category: Image AI