Anthropic’s Claude Mythos chatbot was promoted in April as the most frighteningly effective hacking tool ever! Too dangerous for you, the public! The US government even banned Mythos for a couple of weeks! Mythos had found some actual bugs — at great expense, subsidised by Anthropic. You could have…
Annons
Annons
Anthropic’s Claude Mythos chatbot was promoted in April as the most frighteningly effective hacking tool ever! Too dangerous for you, the public! The US government even banned Mythos for a couple of weeks! Mythos had found some actual bugs — at great expense, subsidised by Anthropic. You could have paid actual security researchers much less to get more. But it was very effective marketing for Mythos. Anthropic wanted to bring the benefits of Mythos to open source projects. So they launched Project Glasswing in April to “secure the world’s most critical software.” [Anthropic] How has Glasswing done? Not great. Patrick Garrity at security vendor VulnCheck wrote up Glasswing in September, after Anthropic updated the Glasswing disclosure list for the first time since May. [blog post, archive] Anthropic claimed to have found 26,000 security holes up to September. Only 2,000 of those were reported to the software projects in question by September. Apparently, the bottleneck was Anthropic having enough humans to check the holes were real: [Anthropic] The number of vulnerabilities we’ve disclosed is a subset of the total number of vulnerabilities that Mythos Preview (and other Claude models) has found, since the process of independent human triage and review is the rate limiting step. The hard problem in AI is where you hide the human. But Mythos did find important bugs, right? Not really. The website downloading tool Curl says Glasswing found a grand total of one new vulnerability that was a real vulnerability. A minor one. One is better than none! But wasting a lot of other people’s time checking on it. [blog post] Other open source maintainers concur on the chatbot. Here’s one maintainer on AI reports in general: [Mastodon] This lines right up with the reports we have gotten. Overwhelming, but fewer than advertised, considerably less serious than the bots rate them, and, shall we say, poor quality remediations that would be unusable even if our policy were to accept slop fixes. The Linux kernel is gung-ho for chatbot code, much to the dismay of many users. Kernel maintainer Greg Kroah-Hartman is a big fan of AI security scanning. He spoke at the Kernel Recipes 2026 conference on the 22nd September on chatbots in security, and how Mythos was not so great: [YouTube] They said, “Oh, we found all these bugs. The world is on fire.” So let’s look at these bugs. I got the real raw data. 24 of them were nothing. Literally nothing. Something crashed. No report, something crashed, the VM died. No help at all. Fourteen of them. I don’t know why they documented it. It wasn’t even a bug. Nothing happened at all. I don’t know what they were doing. Three of them were completely made up. I don’t like using the word hallucination, because that gives an idea that there’s an entity behind this stuff. It’s just fake. They made up data out of thin air. Totally not right at all. And here’s my funnest one. These tools want to please you. They’re very sycophantic. If you say, “give me a bug,” it’ll work really, really hard to give you a bug. So hard it’ll go out and read our mailing list and report bugs that other people have found and fixed. Four of them, I will declare Anthropic fixed. They had some bugs, they were minor little things like authenticated NFS servers to authenticated clients so you knew who was going on in the network. Great, we fixed them. World moved on, we didn’t even notice. Eleven of them were famously found and fixed by other people! In public, before this report even came out. They scraped the web and found the bugs. Kroah-Hartman remains a big fan of chatbot scanning. But it’s another tool. It’s a very noisy tool, and you need humans to check the results. Just like the very noisy non-chatbot scanning tools. Remember how, back in April, another company found the same bugs Mythos found, but using “small, cheap, open-weights models.” [blog post] Mythos is not magic. It’s a heavily subsidised vulnerability scanner that runs on humans and exists to sell you Claude. It’s not nothing — but it’s not much. Video — Podcast