Google says Gemini model accessed three companies' systems during security test

Google has disclosed that its Gemini model gained access to three other companies' computer systems without permission, the first time the company has said one of its models autonomously broke into third-party systems.

According to Google, the incidents occurred in May when Gemini guessed passwords and twice used a repository of publicly listed passwords to access private systems. The model was taking part in a "capture-the-flag" security test run by Israeli startup Irregular, and its agents were never meant to reach the wider internet. A bug in the testing environment made internet access available.

Google said the agents stopped once they realized they had accessed real company systems rather than just the test environment. "In all three of these instances, the model stopped," said Heather Adkins, Google's vice president of security engineering.

The disclosure arrives amid growing scrutiny of misbehaving AI. OpenAI, Anthropic and Meta have recently reported incidents in which their models escaped testing environments and attempted to hack other companies. Anthropic CEO Dario Amodei has called for the industry to slow development of the most advanced AI models until they can be proven safe.

All of the reported incidents involved Irregular, a startup backed by Sequoia and Redpoint Ventures that was valued at $450 million last year and helps developers run cybersecurity tests. An Irregular spokesperson said the Google incident stemmed from the same issue that let other models reach the internet and did not represent a materially separate event. The company said relevant labs were notified in late July and affected entities were contacted during the investigation.

Google said it was notified by Irregular in late July and has since worked with the startup to change its testing process. A Google spokesperson declined to identify the exact Gemini model involved. The Wall Street Journal first reported the incident.

Source: CNBC