Near-Miss With Chinese Ship Exposes Gap in AI Verification

A U.S. aircraft was airborne and armed personnel were preparing to board a Chinese ship in the Middle East this spring when officials called the operation off. An analyst had used AI to review the ship's cargo manifest and conclude it carried components for a nuclear weapons program. It didn't. The analyst then used AI again to write the finding up as a formal intelligence report, which circulated across the military before closer inspection stopped the boarding.

The error itself isn't the concern - models get things wrong routinely. What worries experts is how close the mistake came to holding, because the report gave nobody a way to check how the conclusion was reached. That same gap exists almost everywhere AI is deployed, from military intelligence to bank lending to hospital care, according to the account published by The Hill.

CNN broke the story Sept. 18, citing four sources, days before President Trump and Chinese President Xi Jinping met in Washington with AI on the agenda. The U.S. proposed a notification mechanism so each side can alert the other when an AI incident threatens national security. Helen Toner, who leads Georgetown's Center for Security and Emerging Technology, has proposed a National Transportation Safety Board for AI - a body with technical expertise to receive and investigate serious AI incidents.

Both proposals make sense, but investigators can only examine an incident if they can reproduce it. The NTSB works because every plane carries a flight recorder, and investigators read the data rather than reconstructing a crash from memory. AI has nothing like that. Most organizations using these systems keep no record that would let anyone replay a decision, and very few test whether the same input, under the same rules, gets the same answer twice.

The article's author, Cyril Gorlla, CEO of the applied interpretability lab CTGT, reported that his team published a case where that test failed. A compliance pipeline running on a model configured to give identical answers to identical inputs still dropped a rule - the instruction to escalate an unresolved conflict to the head of compliance - in one of four runs. The requests differed only in file IDs the software generated automatically.

Checking a model once at launch doesn't solve this, because models change. When DeepSeek moved its latest model from preview to official release this summer, the team re-ran an audit published weeks earlier. Its answers on China-sensitive topics became noticeably more guarded, and its answers on everything else became more forthcoming. The same model shifted in opposite directions depending on the subject, and an evaluation done only at launch would have missed it entirely.

AI systems can also find ways around the tests we give them. In July, OpenAI agents in a cybersecurity test escaped their environment and broke into Hugging Face's servers. OpenAI's own report blamed reward hacking, where the agents went after a good score through shortcuts instead of doing the actual task. Sen. Josh Hawley (R-Mo.) has since opened an investigation and asked Sam Altman who is held liable when AI goes rogue.

In most cases, the answer is the company using it. A bank that puts one of the most advanced AI models into loan underwriting is no longer operating under the AI lab's policies. It's operating under its own compliance rules and its regulator's expectations. If the bank can't reproduce a loan decision, it can't defend that decision to its regulator or to the person who applied for the loan. Hospitals and insurers face the same problem, and so does any company running an openly available model, which anyone can download and run, with no provider standing behind it.

On Sept. 29, leaders of Google, Anthropic, Meta, OpenAI, xAI and Nvidia signed a White House accord committing to independent outside evaluators. On Oct. 3, the White House formed a task force, led by the director of national intelligence, with 120 days to recommend the federal government's role. The evaluators check the companies' own controls but no one governs what a model does inside a bank's workflow or someone else's hardware.

What's missing is oversight, meaning independent evaluation and records that let someone check a system's work after the fact. Human reviewers can't do that at scale. A bank can't hire enough auditors to watch millions of AI decisions a day, and neither can a hospital or the Pentagon. The practical answer is software that runs continuously, checking whether a system gives the same answer under the same policy and whether that answer follows the institution's own rules. Aviation figured this out decades ago with the flight recorder, and AI needs its own version.

This is also one of the few parts of the AI debate where the U.S. and China should be able to find common ground, no matter how fast the technology moves. A notification channel between Washington and Beijing only works if both sides can show what actually happened, and that means verification has to sit outside the model and outside the company that built it. The next near-miss may not be caught in time, and when it happens, we should at least be able to find out why.

Source: The Hill