US nearly attacks Chinese ship based on AI hallucination
Around six months ago, a US military intelligence report said that a Chinese ship in the Middle East was carrying nuclear components. CNN got hold of an internal Department of Defense memo on the incident and ran the story a few days ago. [CNN] The US planned an operation to deal with the Chinese…
Around six months ago, a US military intelligence report said that a Chinese ship in the Middle East was carrying nuclear components. CNN got hold of an internal Department of Defense memo on the incident and ran the story a few days ago. [CNN] The US planned an operation to deal with the Chinese ship: armed members of the US military were preparing to board the ship. Military planes were in the air. Then — at the last moment — someone looked a bit more closely at the report. An analyst had generated it with a chatbot, which had hallucinated what the ship was carrying. The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said. So how did this happen? The analyst queried a chatbot about some intelligence reporting on the ship’s manifest that originated with US Special Operations Command Pacific, based in Hawaii. … The bot fused together open-source intelligence with secret signals intelligence in government holdings. The analyst then used AI again to package the findings into a standard intelligence report — the kind that is trusted by military officials — and disseminated it. That is, the analyst just chucked a pile of stuff into the magical truth machine and assumed whatever came out was obviously correct! And not just the chatbot making up a story that sounded important. Three senators have written to the Department of Defense, asking: what on earth? [Senate; Senate, PDF] This happened because the US Department of Defense is fully chatbot brained and deep in military AI psychosis. Here’s Defense Secretary Pete Hegseth speaking at SpaceX in January: [DoD] This strategy will unleash experimentation, eliminate bureaucratic barriers, focus on investments and demonstrate the execution approach needed to ensure we lead in military AI and that it grows more dominant into the future. In short, we will win this race by becoming an AI first warfighting force across all domains. This sort of error is likely to happen again — because Hegseth really does think the chatbot is a magical truth machine. Someone on the ground did know enough to check where a surprising report came from. Because chatbots work by making stuff up. Sometimes it’s accidentally accurate. But Hegseth is firm — “I want you to use AI.” If you’re in Defense and you’re not using the chatbot, you will be asked why not. The existential risk to humanity from AI is stupid people. Video — PodcastSource: Pivot to AI — Published — Category: Business