Proving Absence: Why Gap Analysis Breaks Ordinary RAG
Most RAG systems retrieve supporting evidence. Compliance gap analysis must defend the conclusion that relevant evidence could not be found.Most RAG systems are built to find an answer. Gap analysis has to prove one isn’t there, and that inversion changes almost every design decision.If you have…
Most RAG systems retrieve supporting evidence. Compliance gap analysis must defend the conclusion that relevant evidence could not be found.Most RAG systems are built to find an answer. Gap analysis has to prove one isn’t there, and that inversion changes almost every design decision.If you have shipped a retrieval-augmented chatbot, you already know the shape of the problem: a user asks something, you retrieve a handful of relevant chunks, you ground a model on them, you return an answer. When retrieval misses, the answer is weak, the user rephrases, and the system gets another shot.Gap analysis gives you none of that. You are handed an authority document, a standard, a regulation, a policy framework, a contract, and an internal document that is supposed to satisfy it. Your job is to produce, for every obligation in the authority document, a verdict, with evidence.The interesting one is the third. Saying “this requirement is not addressed anywhere in your document” is a claim about absence. And at the top of a RAG stack, a retrieval failure and a genuine absence look exactly the same.Everything below comes from building one of these systems and getting it wrong in most of the ways it is possible to get it wrong. It is organized as a pipeline walkthrough, with the failure modes attached to the stages where they bite.Reframe the problem before you write codeThe first mistake is treating gap analysis as chat with a different prompt. It isn’t. Three structural differences drive the whole architecture.The unit of work is the requirement, not the query. Nobody is typing anything. You generate several hundred implicit queries by decomposing the authority document, and each one runs to completion. A pipeline that costs 400 ms per query is fine for chat and catastrophic here.There is no conversational repair. A user who gets a bad answer rephrases. A requirement that gets bad retrieval silently becomes a false “not covered” in a report someone will act on. Every retrieval failure converts directly into an incorrect finding.Absence must be provable, not inferred. This is the one that reorganizes everything. If you cannot demonstrate that you searched the right region of the document with the right query, “not covered” is not a finding, it is an admission that your retrieval didn’t work.CONSEQUENCE: The system’s real output is not the verdict. It is the evidence trail that justifies the verdict. Design for that on day one, retrofitting provenance into a pipeline is miserable.The pipelineFive stages. Each one exists to kill a specific class of error.00. Relevance gate: Reject unrelated document pairs in seconds, before anything expensive runs.01. Requirement extraction: Prose → atomic obligations, each carrying structured qualifiers.02. Scoping (graph): Which obligations does this document even claim to address?03. Matching: Topic-anchored hybrid retrieval; evidence per requirement.04. Hybrid assessment: The model proposes. Deterministic code decides.05. Evidence anchoring: Provenance back to the source, then the report.Stage 0 — The relevance gateBefore doing anything expensive, decide whether these two documents have anything to do with each other. A cheap embedding-level comparison, a few seconds of work, saves a full pipeline run when someone uploads the wrong file.The lesson here is smaller than the stage but more broadly useful: calibrate your threshold against your own embedding model’s actual distribution. Cosine similarity is not a universal scale. Some models produce well-spread similarities where 0.45 is a meaningful floor. Others compress everything into a narrow high band where genuinely unrelated documents still score 0.78, and a 0.45 floor accepts everything.We copied a threshold from a reference implementation, watched the gate pass every document we fed it, and eventually plotted the actual distribution: our model’s similarities lived almost entirely between 0.72 and 0.95. The correct floor was 0.82. A number copied from someone else’s blog post is not a default; it is a bug with a plausible-looking value.Plot the distribution. Always. It takes twenty minutes.Stage 1 — Requirement extraction, and the decomposition trapYou need to turn prose into a list of discrete, individually assessable obligations. Two things matter far more than they appear to.Atomicity. Authority documents love compound sentences. “The institution shall document the process, review it annually, and report material deviations to senior management” is three obligations, not one. Extract it as one, and a document that does two of the three gets a single muddy verdict, teaching the reader nothing. Extract it as three, and you get a precise finding: two covered, one missing.The counter-pressure is real: over-decompose and your report balloons into hundreds of near-duplicate rows nobody reads. We settled on decomposing at the level of distinct verifiable actions, and accepted that this is a judgment call requiring a tuned prompt plus a human-reviewed golden set.Qualifiers are first-class data, not prose. Every obligation carries modifiers that determine whether it is satisfied:frequency — annually, quarterly, on each material changethreshold — above 5%, more than 50 employees, material exposurescondition — where the institution uses internal modelsscope — applies to the trading book onlyExtract these into structured fields. Do not leave them buried in the requirement text. The entire deterministic-guard layer in Stage 4 depends on having them as data, and there is a second reason, which cost us an embarrassing review cycle: we extracted qualifiers correctly for months and never printed them in the report. The extraction was right, the assessment used them, and the human reader saw none of it.RULE: Every structured field you extract needs a defined destination in the output. An extracted field nobody sees is not a feature.Stage 2 — Scoping, and where GraphRAG actually earns its placeThis is the stage most teams skip, and it is the one with the highest return.A large authority document may contain 500+ obligations. For a given internal document — say, a specific operational policy — perhaps 40 are in scope. Run all 500 through matching and assessment, and you get three bad outcomes at once: it is slow, it is expensive, and it is wrong, because 460 out-of-scope requirements come back “not covered” and drown the 12 real findings.Here I want to be precise about graph structure, because GraphRAG is heavily marketed for the wrong reason. The pitch is usually multi-hop question answering. In my experience, that is the least valuable thing a graph does in this domain. Three others are worth real engineering effort.Ancestry stamping. The cheapest and highest-value graph technique available, and it barely qualifies as a graph. Regulatory and policy text is a tree: Document → Module → Section → Subsection → Clause.A leaf clause reading “at least annually” is meaningless in isolation, and that is exactly what a flat chunker hands your embedding model. Stamp the full ancestry path into the chunk at index time, the breadcrumb becomes part of the embedded text and part of the retrieved context. Retrieval quality improves immediately, and the improvement is largest on precisely the short, reference-heavy clauses that flat RAG handles worst.Reference resolution. Authority documents are full of clauses like “the requirements of Section 4 apply accordingly.” A flat vector search over that sentence retrieves nothing useful, there is no content to match. We called these referential hubs, and before we handled them, a document could return two requirements where it should have returned nine, because seven lived behind a cross-reference.Graph edges for references, amends and applies-toturn a dead-end clause into a traversal. This is the clearest case where a graph does something vectors structurally cannot.Hierarchical roll-up. Scope decisions belong at the section level, not the clause level. If a section is in scope, its clauses are in scope. Deciding per-clause is both slower and less stable, you get ragged scope where half a section survives, which produces incoherent reports.And a warning that cost us days. The graph must be built and stamped consistently across every ingestion path. We had a fast pre-warm path that populated the catalogue without stamping hierarchy, while the query path assumed hierarchy was present.The failure was silent: no error, no exception, just scoping that quietly degraded to near-flat behaviour and returned a handful of requirements where it should have returned dozens. It looked exactly like a model regression, and we investigated it as one.RULE: When a feature depends on data written by a different code path, it will eventually run against data that path didn’t write.Add a structural health check that reports what the index actually contains, how many nodes carry ancestry, how many edges exist, the depth distribution, and consult it first whenever quality drops.Stage 3 — MatchingStandard hybrid retrieval, dense plus lexical, reranked, with two domain adjustments.Anchor retrieval to the topic, not just the text. Once obligations carry a topic label from Stage 2, constrain retrieval to the corresponding region of the internal document. This prevents the most common false positive in the entire system: a requirement about incident reporting matching a paragraph about incident reporting in an unrelated context, because the vocabulary overlaps completely.Dedup before reranking, not after. Chunk duplication creeps into every real corpus, re-ingestion, overlapping windows, documents that genuinely repeat themselves. If duplicates reach the reranker, they consume your top-k with copies of one passage, and you lose the diversity that makes evidence convincing.Stage 4 — The LLM proposes, the code decidesIf you take one thing from this article, take this one.Never let the model emit the verdict or the score. The model’s job is to normalize, interpret, and propose. The decision is made by deterministic code, from structured fields.The canonical example is cadence. A requirement says “review at least annually.” The internal document says “reviewed every three years.” Semantically, these two statements are nearly identical: same subject, same verb, same document region, cosine similarity comfortably above any threshold you would set.An LLM asked “is this requirement addressed?” will very often say yes, because the topic is addressed. The document even looks responsive. It is a breach. And it is a breach a five-line comparator catches with perfect reliability, because “annually” and “every three years” are numbers once you extract them as data.We built deterministic guards for cadence, thresholds, conditions, and applicability. Each takes structured fields from Stage 1 and the model’s normalized reading of the evidence, and applies a rule. The model contributes interpretation; the code contributes judgment.The verdict module became the most stable part of the system, with hundreds of tests, and it went untouched through six rounds of external review while everything around it churned.“The model judged it non-compliant” is not defensible to an auditor. “The requirement specifies annual, the document specifies every three years, and three exceeds one” is.The reason this works is not that LLMs are bad at comparison. It is that deterministic decisions are auditable and regression-testable, and probabilistic ones are not. Anything you can decide in code, decide in code.Stage 5 — Anchoring, and the failure that looked like hallucinationEvery finding cites a source quote, and that quote gets located in the source document so the reader can click through to it.We shipped this with a global anchor search: given the model’s quoted evidence, find the best matching passage anywhere in the authority document. It worked in testing. In production, it produced the worst-looking failure in the project’s history: a client review reporting that the system was hallucinating, citing findings whose quoted evidence had nothing to do with the requirement.The model had not hallucinated anything. The anchoring layer had. A global search over a large document will always find something plausible, and when many requirements share vocabulary, they collapse onto the same attractor sentence. At one point, close to two hundred requirements were anchored to a single sentence. The model’s actual output was fine; the provenance layer had overwritten it with confident nonsense.Check evidence provenance before you touch the prompt. When output looks hallucinated, the stage that assigns sources is at least as likely a culprit as the model. We spent real time tuning prompts against a bug that lived 200 lines downstream. The first question on any “the model is making things up” report is now: is the citation layer showing what the model actually returned?A verified anchor does not mean a complete chunk. We added a verification flag: the anchor is confirmed to exist in the chunk it came from. Good, but that flag means the quote matches its chunk, nothing more. If the chunk itself is lossy, because a PDF table collapsed into unreadable text or a two-column layout interleaved, you are verified and wrong simultaneously.RULE: A verification flag verifies exactly one step. Say which one, in the field name, and make sure everyone reading it knows what it does not cover.Cross-lingual asymmetryIf your authority documents exist in multiple language editions, this section will save you a week.We tracked eight requirements that our own quality tests insisted should have been covered, and weren’t. Every one had the same shape. The requirement carried a frequency: daily qualifier, correctly extracted. The internal document genuinely satisfied it. The deterministic cadence guard fired anyway and returned “not covered.”The cause: the requirement had been extracted from one language edition, and the document under assessment was in another. The two editions are legally equivalent. They are not informationally equivalent. The passage in the second edition simply did not contain the word “daily”; the cadence lived in a phrase the other edition renders more explicitly. The guard was working correctly on incomplete input.Never assume translated editions are lossless. They are produced by humans optimizing for legal equivalence, not for information extraction.Detail loss is not uniform. It concentrates in exactly the places you care about — numbers, cadences, thresholds — because those are what gets rendered idiomatically rather than literally.Match within a language edition where you can, and where you can’t, mark the requirement as cross-lingual and reduce confidence rather than asserting a verdict. Do not solve this by machine-translating your corpus; you will replace a known asymmetry with an unknown one.How you know you didn’t break itGap analysis has a nasty evaluation property. The output is a long report of nuanced judgments, so “did this change help?” is genuinely hard to answer, and the temptation to answer it by eyeballing one run is overwhelming. Resist it. Three tiers.Tier 1 — Unit tests on the deterministic layerEvery guard gets exhaustive tests. These are pure functions over structured input; they are fast, and they are the only part of the system with a crisp notion of correctness. This is a large part of why the verdict logic should be deterministic in the first place.Tier 2 — The identity testRun the pipeline with a document as its own reference. Every requirement is trivially satisfied by construction, so the score should be near-perfect. This is the single best smoke test we have: it catches scoping regressions, chunking damage, anchoring bugs, and retrieval breakage in one run, with no labelled data required.But document the expected ceiling, loudly, in writing. Our identity test scores around 97–98, not 100, and it should. Chunk boundaries split obligations, extraction is not perfectly idempotent, and a handful of requirements are genuinely unrecoverable from a chunked representation of their own source. The remaining points are a property of the design, not a defect, and an engineer chasing them can burn a week on a number that cannot move without breaking something real.Tier 3 — Golden setA fixed corpus with expert-reviewed expected findings, scored on precision and recall separately. Separately, because this surprised us, your reviewers will disagree with each other, and both will be right.We ran two independent expert reviews of the same output. One flagged the system as too permissive: findings marked covered that a careful assessor would challenge. The other flagged it as too aggressive: findings marked not-covered that were defensible. These are not contradictory reviews. They are recall-oriented and precision-oriented readers, and the correct response is not to split the difference with a global threshold.The correct response was to classify obligations into families: documentation, control, reporting, governance, and set strictness per family. A documentation requirement is satisfied by the existence of a described process. A control requirement is not; it needs evidence that the control operates. One global strictness parameter cannot express that, and every attempt to tune it just moves the complaint from one reviewer to the other.Two evaluation trapsNever compare single-run scores across environments. We lost days to a reported divergence between two deployments producing different results for identical input. The suspected cause was a backend change. The actual cause was configuration drift: a scoping feature defaulted off in one environment and was enabled only by a local config file in the other. An A/B under identical conditions exonerated the code entirely.RULE: Compare structural facts, not scores. How many requirements were extracted? How many entered scope? What is the verdict distribution? Those questions have answers that identify the problem. “The score dropped six points” does not.Check the timestamp on what you are reviewing. Twice, we acted on detailed external review feedback describing behaviour that had already been fixed; the reviewed output had been generated from a build over an hour stale.The findings were real, for a version that no longer existed. Verifying an artifact’s generation time against your deploy time is a ten-second check that has saved us multiple days.The performance levers that matteredOur first working end-to-end run took 50 to 70 minutes. It now runs in three to five on a warm system. Roughly in order of impact:Scope before you assess. By far the largest win, and an accuracy win too. Cutting 500 requirements to the 40 in scope removes ~90% of the work before the expensive stage.Pre-warm the reference catalogue. Requirement extraction from an authority document is expensive, and its result is identical for every run against that document. Extract once, cache, reuse. This turned a multi-minute per-run cost into a one-time cost.Make cache keys content-aware. The counterpart to the previous point, and a trap we fell into. Our catalogue cache key did not include the configuration that shaped the cached content. We raised a chunk limit, redeployed, and kept serving results built under the old limit — silently, with no error, for long enough that we investigated it as a model quality regression.Verify that config is actually wired. Adjacent failure, same week. A limit was raised in the config file, read at startup, and never passed to the code path that used it. The declared value and the effective value differed, and nothing reported the discrepancy.RULE: If a config value changes the content, it belongs in the cache key. No exceptions.And expose effective configuration: the values the code is actually using, read from the objects using them. Declared config is a hypothesis.Batch embeddings, parallelize assessment. Ordinary engineering: per-requirement assessments are independent, so run them concurrently with a bounded pool; embed in batches, not in loops.Report generation is part of the systemA finding the reader misunderstands is a defect, exactly as much as a wrong verdict is. We treated report rendering as cosmetic for too long, and paid for it across several review cycles. Two rules earned their place.When you change one cell of a row, re-read the entire row. We fixed a defect where a rejected candidate passage was displayed confusingly by hiding it. Correct fix — and it created the next defect: the adjacent explanation began “this text was rejected because…”, and with the text hidden, the pronoun pointed at nothing. Thirteen rows, all broken, by a fix that was right in isolation. Generated output has internal cross-references no type system will catch for you.Prefer labels over rewrites. When generated prose is confusing, the safe fix is usually to label it rather than regenerate it; add a heading like “Why the closest text was rejected:” above the existing sentence. This matters most when the prose is content-pinned for auditability: rewriting invalidates the hash, and now your evidence chain has a gap. A label is additive and provable.And one thing that does not work, proposed to us three separate times by three separate reviewers: gating evidence display on a similarity threshold. “Only show evidence above 0.75 cosine” sounds reasonable and is not implementable. We measured the actual distribution for the cases in question: it ran from 0.761 to 0.847.Good evidence and bad evidence occupied the same band, entirely above the proposed line. There was no threshold that separated them, because cosine similarity was never measuring the thing the reviewers cared about.RULE: When a threshold on a continuous score keeps not working, the score is usually the wrong variable. Find the categorical judgment underneath it and gate on that.The rules, condensedIf a team was starting this tomorrow, this is the list I’d hand them.Design for provable absence. The evidence trail is the product; the verdict is a summary of it.Decompose into distinct verifiable actions, and extract qualifiers, frequency, threshold, condition, scope as structured fields.Every structured field needs a destination in the output. A field nobody sees is not a feature.Scope before you assess. The biggest win available, for both speed and accuracy.Use the graph for ancestry, references, and roll-up, not for multi-hop QA demos.The LLM proposes, the code decides. Deterministic verdicts are auditable, testable, and defensible.Check provenance before you tune the prompt. Output that looks hallucinated is often mis-anchored.A verification flag verifies exactly one step. Say which one, in the field name.Calibrate thresholds against your own model’s distribution. Never inherit a number.Anything that shapes cached content belongs in the cache key, and expose effective config, not declared config.Never silently drop a requirement. If scope is ambiguous, flag it and keep it scored, a dropped requirement is invisible, and therefore unfalsifiable.Compare structural facts, not scores, when diagnosing a regression.Run an identity test, and document its expected ceiling before someone spends a week chasing it.Check artifact timestamps before acting on review feedback.When you edit one cell of generated output, re-read the whole row.ClosingThe thing I did not expect, going in, is how little of this project’s difficulty lived in the model. The prompt work was real but bounded. Nearly every incident that cost more than a day was an engineering failure wearing an AI costume, a cache key, an unwired config value, an ingestion path that skipped a step, a provenance layer overwriting correct output with a confident guess.That is, I think, the actual state of applied RAG right now. The models are good enough that they are rarely the bottleneck. The bottleneck is that we are building distributed data pipelines whose failures are silent and plausible rather than loud and obvious, and our instinct, when output looks wrong, is still to reach for the prompt.Check the pipeline first. It’s usually the pipeline.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!Proving Absence: Why Gap Analysis Breaks Ordinary RAG was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI