AI Chatbot Hallucination Nearly Triggered US Military Strike on Chinese Vessel

Military aircraft were already airborne this spring when U.S. officials realized the intelligence behind a planned armed operation against a Chinese vessel was fabricated by an AI chatbot. The mission was called off at the last moment, narrowly avoiding a potential clash with China, according to CNN.

The intelligence report, which circulated during the war with Iran, claimed the ship was transporting components for a nuclear weapons program. It began with a Special Operations Command analyst who asked an AI chatbot to combine open source data with classified signals intelligence. The chatbot misread the ship's cargo manifest. The analyst then ran the tool a second time to turn the flawed findings into an official-looking summary that moved through command channels.

The near-miss highlights a mounting worry among military officials and outside experts: as decision-makers rely more on AI, its errors can climb the chain of command before anyone catches them. The Pentagon has touted AI as a major advantage for speeding up its kill chain. But that same speed may let hallucinations slip through with too little human review.

"It's important for service members to understand the uncertainty inherent to LLMs," said Jake Steckler, a research scholar at GovAI and veteran U.S. Army officer, in a written response to TechCrunch. "But it's especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions."

Steckler said the incident should push for more AI safeguards, not avoidance. "These tools can be useful in the right contexts and with the right safeguards in place," he said. "But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption."

Source: TechCrunch