Citations Over Cosine
How we pick the one regulation a policy is actually about, before spending a single token on the other seven.Our users select every regulation in the list. Every single time.They are not being careless. They select everything precisely because they don’t know which regulation governs the document…
How we pick the one regulation a policy is actually about, before spending a single token on the other seven.Our users select every regulation in the list. Every single time.They are not being careless. They select everything precisely because they don’t know which regulation governs the document in front of them, which is the reason they opened the tool. Asking them to narrow it down first is asking them to answer the question they came to ask.So the engine compares one internal policy against eight regulations, and seven of those comparisons are expensive noise. On one measured run, 757 requirements entered the analysis, and only 39 of them belonged to the regulation the document was about. Most of the runtime was spent confirming that a reporting procedure has nothing to say about securities trading rules.The obvious fix is to rank the regulations against the document and keep the closest. That fix is wrong, and the way it’s wrong is worth the rest of this post.Similarity measures language, not just topicFive of our eight regulations exist only in German. Three have official English editions. That asymmetry is a fact about German financial law, not a gap in our ingestion.Run an English policy through a similarity ranker against that corpus and the three English-edition regulations win. All three. Not because they are more relevant, but because they are written in the same language as the query.Here is the run that settled it — an anti-money-laundering procedure, where the correct answer is the German AML statute:~0.90 English-edition regulation wrong~0.89 English-edition regulation wrong~0.88 English-edition regulation wrong~0.87 German AML statute correct — ranked 4thThere is no threshold that rescues this. The gap between the correct answer and the wrong ones is larger than the gap between the wrong ones, so any correction big enough to promote it would scramble everything else. This was not a miscalibrated signal. It was the wrong signal.The ladderWhat replaced it is a strict order of precedence.1. Citations decideCompliance documents cite things. A policy that says “in accordance with Article 26” has told you which regulation it is about, not approximately, not probabilistically. A human who knew the answer wrote it down.Citations are language-neutral: ART|430, § 25a and AT 9 mean the same thing in both editions. They're deterministic, so every run agrees. And extracting them is a regex, not an inference, so they cost nothing.A citation counts for a regulation only when the referenced section is genuinely one of its sections. That ownership test is what disambiguates a shared grammar, since § 25a resolves natively in more than one German statute.The leader must be cited at least twice, or just once if the citation appears in the document’s opening section, where a policy declares its subject. One passing mention on page 40 is not an anchor.2. Similarity ranks, inside one language cohortConsulted only when nothing is cited. And if any candidate regulation has no edition in the document’s language, similarity is not consulted at all; the comparison would be meaningless, so the selector declines instead. Similarity may break a tie for the lead. It may never add a second winner.3. Otherwise, declineWhen the signals don’t separate one regulation, the system says so and checks everything, as it always did. Declining is a first-class answer here, not an error path.A wrong pick doesn’t under-scope the right regulation. It zero-scopes it, the right regulation produces no findings, and its absence looks exactly like compliance.Silence is the most dangerous output this system can produce. That asymmetry is why the whole design leans toward “I can’t tell”.When two regulations both winSome documents genuinely span two regulations. Admitting a second one takes four independent clamps, all measured on citations:cited at least half as often as the leaderholds at least 20% of the document’s references on its ownsurvives a knee-detection cut over the ranked sharesthe pair jointly accounts for at least half of all referencesIf more than two qualify, the selector declines outright rather than crowning four of eight. Admitting half the list isn’t a selection; it’s a failure to separate, and the honest output there is the old behaviour.The clamp I had to add laterTwo of our regulations, the banking rules and the equivalent rules for securities institutions, share a section numbering scheme. Both have an AT 9. Both have an AT 4.3.So a policy citing AT 9 once legitimately credits both. Each ends up with one citation. Each holds half the total. Every clamp above waves them through.One reference. Two winners. The second riding in on evidence that had already been counted.Live, that put a securities-institution regulation into a retail bank’s analysis, where it was scored against four requirements, came back looking healthy, and sat on the results page beside the regulation that actually mattered. A number that means nothing, presented with the same authority as one that does.The fix is one sentence: a second answer must bring evidence the first one doesn’t have. If the runner-up was credited only by references the leader was also credited by, that’s a shared grammar, not a second subject.Regulation families use genuinely different citation formats, so a document that really does span two of them will always name something only one of them has.Set aside is not the same as absentA regulation that loses is still reported. It appears in the scope view, in the coverage matrix, and in every report, carrying the reason it was set aside:the document cites no provision of this regulation, set aside for this analysisThis is a scar from an earlier feature. A previous version of a different gate silently omitted irrelevant sources, and it was reverted because a compliance officer cannot audit an absence. If the system decided something didn’t apply, it has to say so out loud.Worth separating two things that look alike here. The visible “excluded” marker is disclosure. The actual suppression happens at a single choke point every downstream stage reads through. Writing only the disclosure would produce a report claiming a narrowing that never happened, which is exactly the bug I nearly shipped.Two passes, and only one may narrowThe run’s language isn’t resolved until after extraction, so the selection runs twice.The early pass, before the expensive stages, uses citations only. It needs no language, so a wrong language guess can’t mislead it. Its winners narrow everything downstream.The confirming pass runs once the language is settled and may consult similarity. It is allowed to widen only; it can restore a regulation the early pass set aside, never remove one. Widening is the fail-open direction, and by that point work has already been spent on the early winners.What it measures out atAcross 17 documents with known answers: zero wrong winners. Four decided on citations, four correctly rejected as off-topic before selection, seven declined.On a reporting policy checked against all eight regulations, requirements entering assessment fell from 757 to 39, and the run went from over forty minutes to under three, with an identical verdict breakdown to the slow run. Same answers, ten times faster.That 4-of-13 decision rate is the number I’d watch if you build this. It means the system declines more often than it decides, and documents that cite nothing get no speed-up at all. I’m fine with that.The failure mode of declining is a slow correct answer. The failure mode of guessing is a confident wrong one.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Citations Over Cosine was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI