False AI Report on Chinese Ship Nearly Triggered US Military Action

A US intelligence report built with the help of AI tools was "entirely false" and nearly led the military to board a Chinese ship, according to a CNN report cited by Ars Technica.

The report, submitted by a US Special Operations Command analyst, claimed the vessel was carrying components for a nuclear arms program through the Middle East. Four sources familiar with the episode told CNN that US forces were preparing to intercept and board the ship with air support before officials found the chatbot used in the report had misidentified the cargo.

One source said the incident "almost started a war." CNN reported the analyst used a chatbot that combined open-source intelligence with secret signals intelligence, packaging the result into an official report.

The near-miss stands among the most consequential examples of AI hallucination undermining a professional report. Despite prompts asking AI not to hallucinate, some researchers suggest the problem may be impossible to eliminate entirely. The Department of Defense rolled out an AI acceleration strategy in January aimed at making data available across its systems for AI use.

Source: Ars Technica