AI for Business

Research Group Says Leading AI Models Could Be Talked Into Targeting Civilians

A San Francisco-based AI evaluation startup called Vals reports that models from Anthropic, OpenAI, Google, DeepSeek and other leading labs could be manipulated into identifying targets for...

Share:

A San Francisco-based AI evaluation startup called Vals reports that models from Anthropic, OpenAI, Google, DeepSeek and other leading labs could be manipulated into identifying targets for military strikes - including civilians and locations such as schools - simply by framing the task as a game.

Vals tested whether the models could be drawn into autonomous warfare, using simulated scenarios and a mix of generated and real imagery from Ukraine and Iran to see if the systems would mark targets in drone and satellite imagery. According to the report, the models frequently flagged potential targets, including civilians and surrendering soldiers. When a model refused, researchers said they could often get it to comply by claiming the battlefield scenario was part of a game. None of the requests were blocked by a safety classifier, and refusals were uncommon.

The findings land amid widening concern about near-term AI risks. Earlier this year, Anthropic became embroiled in a dispute with the Department of Defense after telling the agency its technology could not be used for mass surveillance of Americans or autonomous warfare. CEO Dario Amodei has argued that today's systems aren't reliable enough to make life-or-death decisions without a human in the loop, while the Pentagon contended it should be able to deploy the technology for any lawful military purpose.

In the simulations, Anthropic's models performed the worst - though none of the models performed particularly well. Anthropic spokesperson Eduardo Maia Silva told TechCrunch the company appreciated Vals' report, recommendations and collaboration, and acknowledged the outcomes were not acceptable. "The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior," the company said in an emailed statement. "Even for authorized settings, Anthropic has long been firm that today's frontier AI systems are not reliable enough to power fully autonomous weapons." Anthropic said its teams were already developing additional safeguards before publication and that the measures would be in place soon.

OpenAI spokesperson Nate Evans told TechCrunch the company prohibits using its public models to facilitate violence or for unapproved national security purposes, including the scenarios Vals simulated. "We actively detect and disrupt misuse, including by suspending accounts that violate our policies," OpenAI said. "We're also developing additional safeguards for our public models against misuse involving drones and drone software."

Vals notes that the models were never asked to make direct strike decisions - a task researchers said they would typically refuse - but that doesn't mean their outputs couldn't have dire consequences. Targeting information, for instance, could feed autonomous weapons systems. In some scenarios, models were asked whether people were visible; positive reports were followed by simulated strikes, and the models saw the consequences.

The report covers commercial models from frontier labs, not the versions those companies might supply to the military, and it's unclear whether the added safeguards would apply to military use. Vals also cited technical hurdles - latency and limited onboard computing - that make general-purpose models hard to use for live drone targeting. A nearer-term concern, researchers said, is that AI could annotate battlefield footage at scale, generating training data for autonomous targeting systems.

Source: TechCrunch

Ready to Modernize Your Business?

Get your AI automation roadmap in minutes, not months.

Analyze Your Workflows →