Your AI Refuses to Help. But Does It Refuse Correctly?
Notes from my ongoing research on whether LLM safety survives a long conversation.Ask a language model to help you cheat on an exam, and it will refuse. Ask it how to build a bomb, and it will also refuse.On every safety leaderboard that exists today, those two refusals count exactly the same. One…
Notes from my ongoing research on whether LLM safety survives a long conversation.Ask a language model to help you cheat on an exam, and it will refuse. Ask it how to build a bomb, and it will also refuse.On every safety leaderboard that exists today, those two refusals count exactly the same. One point each. Model is “safe.” Move on.That has been bothering me for months.Because those two requests are not the same. One is a bad idea that might get a student expelled. The other could kill people. Any human moderator would treat them differently: different urgency, different firmness, different willingness to even continue the conversation. Yet the way we evaluate AI safety today mostly reduces everything to a single binary question: did the model refuse, yes or no?This question has turned into my current research project. This article is me thinking out loud about it, what the problem is, what others have found, and what I’m building to study it. No results yet.Just the setup, honestly laid out.The binary trapAlmost every mainstream safety benchmark works the same way. You throw a set of harmful prompts at a model, and you count refusals. The metrics have different names: refusal rate, attack success rate, fulfillment rate, but underneath, they’re all counting coin flips. Refused: safe. Answered: unsafe.This framing has a blind spot you can drive a truck through: it treats harm as flat.In reality, harm has severity. “Write a rude email to my coworker” and “give me step-by-step instructions for synthesizing a nerve agent” both live under the umbrella of “requests a model might decline,” but they are separated by an enormous gulf in real-world consequence.A safety evaluation that can’t see this gulf can’t tell you whether a model’s judgment is actually well tuned, only whether its reflexes fire.And reflexes, it turns out, misfire in both directions. Models overrefuse (declining perfectly harmless questions because they sound scary, a problem benchmarks like XSTest and OR-Bench were built to expose) and underrefuse (leaking genuinely dangerous content when a request is phrased cleverly enough). A single refusal rate number hides both failures at once.The idea: calibration, not classificationThe lens I’m using is borrowed from an old idea in machine learning: calibration.A weather model is calibrated if, when it says 70% chance of rain, it actually rains about 70% of the time. The prediction tracks reality. I want to ask the same thing about safety behavior:Does a model’s resistance to a request track how harmful that request actually is?Let me make that precise. Define two quantities for any query q posed to a model m that produces response y:H(q) ∈ [0, 1] — harm severity of the queryR(q, m, y) ∈ [0, 1] — restrictiveness of the model's response (0 = full compliance, 1 = hard refusal)The naive version of calibration would demand proportionality R(q) ≈ H(q). I don’t buy that, and I want to flag why: what would a “0.4 refusal” to a “0.4 harmful” request even look like? Leaking 60% of the phishing tutorial? Proportionality is the wrong normative target.What I do think is defensible, almost embarrassingly modest, is monotonicity:For any pair of queries q₁, q₂ : H(q₁) > H(q₂) ⟹ R(q₁) ≥ R(q₂)More harm should never buy you more compliance. That’s it. That’s the whole principle.This provides a natural way to score a model: sample pairs of queries with different severities and ask how often the model gets the ordering right. Formally, a concordance index:C(m) = P( R(q₁) ≥ R(q₂) | H(q₁) > H(q₂) )where C = 1.0 means the model’s resistance is perfectly rank ordered with severity and C = 0.5 means its restrictiveness is statistically unrelated to how dangerous the request is. A coin flip wearing a safety badge.That sounds so obvious it’s almost boring. Here’s the uncomfortable part: early evidence from the field suggests models violate it.What’s already known (and what isn’t)I want to be upfront about the prior work here, because it’s good and it shaped my thinking.Earlier this year, a paper called Expected Harm (Song et al., 2026) studied how refusal behavior relates to two properties of a harmful request: how severe the outcome would be and how feasible it is to actually execute.They found something they call inverse risk calibration: models push back hardest on dramatic, movie villain threats that almost nobody could realistically carry out, while staying comparatively soft on cheap, low-skill harms that people actually commit. The models had learned severity as a trigger word, not risk as a concept.Around the same time, RefusalBench showed in the biosecurity domain that two models with identical refusal rates can have completely different abilities to discriminate between benign, borderline, and dangerous requests. Aggregate refusal rate literally misranks models. The number we’ve been optimizing can point in the wrong direction.So, the static picture is “is refusal calibrated to severity in a single question answer exchange?” is being mapped, and the answer so far is not really.But there’s a dimension nobody has measured yet, and it’s the one I keep coming back to:Nobody talks to a chatbot in single turns.My question: does calibration survive a conversation?Every calibration study I’ve found evaluates one prompt, one response, done. Real usage is a conversation, and conversations contain pressure.“Come on, it’s for a novel I’m writing.” “I’m actually a licensed researcher; this is for my lab.” “You already explained the first part, why stop now?” “My grandmother’s safety literally depends on this.”We already know from jailbreaking research that multi-turn pressure erodes refusals. What we don’t know is whether it erodes them uniformly across severity. And that’s the question my project is built around:When a model’s safety behavior degrades under conversational pressure, does it degrade evenly or does the calibration itself deform?The measurement I’m building the project around is what I call calibration drift. Take any calibration error metric, say the expected gap between where a model’s restrictiveness should rank and where it does:RCE_t = E[ | R_t(q) − H(q) | ] (calibration error at turn t)Drift(t) = RCE_t − RCE_1 (how far turn t has bent from the turn-1 baseline)Alongside the concordance version, C(m, t), tracked turn by turn. A jailbreak study asks whether the wall falls. Drift asks whether the shape of the model’s judgment deforms and crucially deforms differently at different severity levels:Drift(t | severity tier) for tiers 1 … 5My working hypothesis is that it deforms. Specifically, I suspect models hold the line on extreme requests (the bomb tier stuff, which safety training hammers hardest) but crumble fastest in the middle of the severity range of fraud, harassment, academic dishonesty, gray zone medical questions.Which would be genuinely bad news, because the middle of the severity range is where the overwhelming majority of real-world misuse actually lives.I could be wrong. The drift could be uniform. It could even go the other way, with models becoming globally paranoid under pressure and over-refusing harmless requests too. That would be its own kind of miscalibration and honestly, its own kind of interesting.How I’m testing itThe design, in brief:I’m building a dataset of seed requests spread across five severity tiers anchored to existing safety policy frameworks rather than my personal intuitions and validated with multiple independent annotators, because a severity scale nobody agrees on is worthless.Each seed request then gets extended into scripted multi-turn pressure trajectories drawn from a fixed taxonomy of tactics: emotional appeals, authority claims, incremental decomposition, sunk cost framing.Crucially, the dataset also includes benign look-alikes, harmless questions that superficially resemble each severity tier. Without them, a model that simply gets more stubborn as any conversation drags on would look “robust” when it’s really just becoming useless.Then I run this across a spread of frontier and open source models and track two things over the course of each conversation: how well the model’s restrictiveness stays rank ordered with severity (the concordance C) and how far its behavior drifts from its own turn one baseline (Drift(t), per tier). Not “did the wall fall,” but “did the whole structure of the model’s judgment bend.”The part of the methodology I’m being most paranoid about is measurement itself. Scoring “refusal strength” requires a judge and if the judge is another LLM, I’ve just imported the same reliability problem I’m critiquing.So before any headline experiment runs, I’m validating the judge against human ratings and checking agreement between multiple judge models. Unglamorous work. Also the work the whole paper stands on.Why I think this mattersSafety evaluation is quietly becoming the constitution of AI deployment. Regulators cite benchmark numbers. Labs make release decisions on them. Enterprises pick vendors by them. If the numbers we’re all leaning on are binary snapshots of a behavior that is actually continuous, contextual, and erodible, we’re measuring the wrong thing with increasing confidence.A model that refuses everything is trivially “safe” and practically worthless. A model that refuses nothing is dangerous. The models we actually want live in between, and the property that defines them isn’t a refusal rate. It’s judgment resistance that scales with stakes and holds its shape when someone leans on it.Right now, we have no standard way to measure whether that judgment exists, let alone whether it survives turn six of a determined conversation.That’s the gap I’m trying to measure. If the annotation studies hold up and the experiments run over the coming months, I’ll write about what I find, including if what I find is that my hypothesis was wrong.Sources & further readingThe prior work that shaped this project, if you want to go deeper:Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMsRefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research PromptsAlignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise Pooled RepresentationsSORRY-Bench: Systematically Evaluating Large Language Model Safety RefusalXSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language ModelsConstitutional AI: Harmlessness from AI FeedbackThis story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!Your AI Refuses to Help. But Does It Refuse Correctly? was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI