My LLM Benchmark Came Out Flawless. That Was My First Warning Sign.
How a 0% attack rate led to a much bigger discovery about LLM activation steering and what actually matters in AI research.image cred: http://unsplash.com/photos/abstract-geometric-pattern-of-stacked-lines-UiGZ--7ukvEImagine driving down a windy mountain highway. To keep the car centered in your…
How a 0% attack rate led to a much bigger discovery about LLM activation steering and what actually matters in AI research.image cred: http://unsplash.com/photos/abstract-geometric-pattern-of-stacked-lines-UiGZ--7ukvEImagine driving down a windy mountain highway. To keep the car centered in your lane, you don’t just pull the steering wheel hard to the left and hold it there (please don’t).Instead, you make tiny, iterative adjustments in real time. If the car drifts slightly right, you nudge it left. If the wind blows, you counter-steer. All of this should be baked into our muscle memory.Think cruise control in cars. It automatically adjusts the gas pedal when your feet are tired from driving for 3 hours straight on the freeway so that the car speed stays the same, whether you are going uphill or downhill (depending on how steep).It is similar for Large Language Models. Activation Steering is the AI equivalent of cruise control. Instead of fine-tuning (slow & expensive), why not just dynamically nudge the model’s internal activation to make it refuse jailbreaks or harmful tasks?In engineering, this continuous refinement of sensing and correct error is called feedback control. It’s governed by a century-old math formula known as a PID (Proportional-Integral-Derivative) controller. It is the mechanism that keeps commercial drones stable in high winds or keeps your car’s cruise control steady on steep hills when your feet are tired from driving for 3 and a half hours.But what if we applied this same century-old math to the notoriously chaotic neural pathways of a Large Language Model?A group of AI researchers tried to do exactly that recently (paper link). They treated the layers of a neural network like “time” steps and built a feedback controller to dynamically nudge the LLM’s internal activations, forcing it to refuse malicious prompts.It was a brilliant idea.But it was also very complex. It requires the model to compute and store a complete, unique steering direction for every layer of its brain. Two layers may seem like nothing, but scale that up to 50 or 100 layers with billions of parameters. It gets huge and expensive.I wanted to see if I could do it more simply.I asked the question: Can we find a way to use a single, averaged steering direction over the narrow window where the model actually processes safety?The result was my senior research thesis. And on paper, my method got a 0% Attack Success Rate, a perfect score. I knew there was a catch to this.If you look under the hood, my “perfect” paper looked like a bit of a flop. Here is the honest story of how a classic experimental confounder taught me a far more valuable lesson about how LLMs actually think.The “Perfect” 0% Jailbreak Success Rate (And the Floor Effect)When finished running the experiment code, the data looked like absolute perfection.We tested our defense against GCG (Greedy Coordinate Gradient) attacks, which is basically an adversarial method that appends optimized gibberish suffixes to trick LLMs into giving out dangerous information. Under both the original per-layer PID steering and my proposed Global PID steering method, the attackers got utterly crushed.The Attack Success Rate (ASR) dropped to 0%.I was ready to start drafting my paper to present my huge finding. But then I looked at the baseline.My test model, Gemma-2–2B-it, is already an incredibly well-aligned, safety-conscious model. Out of the box, with absolutely zero steering applied, Gemma successfully blocked 97.12% of these GCG attacks.Attack Success Rate resultsThis is what is known as the floor effect. Because the baseline safety was already low enough (2.88% success rate), it is basically 0% with a little extra defense. Mathematically, there was simply no room to prove our defense was actually doing anything against the jailbreaks.If we had stopped there, the paper would have been a wash. But when we looked at how the defense affected the model’s ordinary intelligence, the real story begins.Simplicity is kingWhile the defense metrics were a tie, the capability metrics were absolutely not close at all.To see if the steering ruined the model’s functionality, we fed it 200 totally harmless prompts from Stanford’s Alpaca dataset with ordinary requests like “write me a poem” or “explain photosynthesis.”When I ran the original, per-layer steering method (which applies steering adjustments at all 26 layers of the model), the model became so paranoid it refused 96% of harmless prompts. It was unusable.However, when we ran my cross-layer steering method, the over-refusal rate was practically at baseline: only 3.5%.In other words, the model:Over-Refusal Rate (How often the model blocks harmless prompts)You might ask, why and how did this happen?By analyzing the model’s internal activations, we verified a fascinating concept called feature persistence.A quick lesson: LLMs, like the human brain, can learn features of concepts, words, or behaviors.LLMs don’t just change their minds at every single layer. Instead, high-level abstract concepts like “refusal” form stable, coherent pathways in a specific middle-to-late window of the model’s depth.For Gemma-2–2B-it specifically, this sweet spot is precisely between layers 19 and 25.The dark-tinted block on the bottom-right is the 19–25 layer windowBy averaging the refusal direction into a single, global vector (layers 19–25 as one) and applying it only within that safety-critical 7-layer window, we cut our storage and computation requirements by 26x, theoretically.But here is where the bigger “Aha!” moment came.In the original PID steering paper, the authors built a complex mathematical steering framework called Angular Steering. In simple terms, it is a highly restrictive framework that uses geometry to tightly restrict the steering degree of the model’s activation at each layer. Without this, steering all 26 layers separately would completely break the model.I didn’t use Angular Steering. I used plain and simple Activation Addition (ActAdd), literally just adding a global vector straight into the model’s activations.And yet, my method remained perfectly fine. It stayed perfectly logical and smart. This points to a brilliant engineering truth: Simplicity is (almost) always rewarded over complexity.Because my method is surgical, meaning it only intervenes where the safety feature is naturally active, it is very stable. It doesn’t rely on a complex mathematical framework to keep it from breaking the system. By putting the intervention in the right place, the simplest math just works.The first half of the storyI originally wanted to apply what I learned throughout my four years in college to something I was genuinely interested in: AI safety.Then it evolved into something involving Sparse Autoencoders (a tool for probing LLM internal features), but the professor advising me questioned my approach for lacking mathematical rigor. He kept emphasizing that I needed to go a level “lower.” I didn’t know what he really meant because I was so blinded by how cool and powerful SAEs were.After doing a bit more research, I finally went a level “lower” and made the call to work with residual streams directly. After some time, my research direction shifted again. I decided to use control theory to steer model behavior in the context of AI safety.A few days into starting the project, I came across a paper under review for ICLR 2026. Their research covered the exact topic I wanted to pursue: PID steering in jailbreak tasks.I felt both disappointed and happy when I saw the news.I was disappointed because someone had “taken” my idea. On the flip side, I also felt more confident in myself; I had successfully identified a top-tier research gap and formulated a concrete approach around it. Ultimately, the experience taught me something far more valuable.What I learnedWhether it was because my methodology was still immature or because I struggled to achieve meaningful results, there are a few key things I learned from taking on this project.Ideas are cheap; execution matters more — Going back to the point about my idea being “stolen”: it simply proved that I had the right research intuition, but that was about it. I could not frame it in a meaningful context, formulate a crisp hypothesis, and run the experiments. Most importantly, I didn’t execute fast enough. They had solid work and a paper to back it up. I had nothing to show because I hadn’t executed. While ideas are important, they remain a fantasy until you act on them, and you act on them fast.Research taste — Moving fast secures your spot, but asking the right questions and working on the right problems is what makes you a great researcher.Here are three examples of what good research taste looks like:Tackling the core bottleneck rather than chasing hype.Prioritizing simplicity over complexity; complexity is a luxury tax.Generativity: Does the work open up new, meaningful questions?Given how powerful AI has become, taste matters now more than ever.Here is a link to the GitHub if you want to reproduce the results: https://github.com/cubeerea/pstat199_ucsbIf you made it this far and found this type of reflective and brutally honest article interesting, consider liking it or giving me a follow.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!My LLM Benchmark Came Out Flawless. That Was My First Warning Sign. was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI