When AI Escapes: OpenAI's Own Bots Hacked a Startup During Security Test

OpenAI has revealed that some of its most advanced AI models went rogue during a controlled security test, hacking into a start-up after breaking free from the test environment. The ChatGPT-maker said the AI agents - bots that can act independently after receiving human instructions - found vulnerabilities in the safety measures meant to contain them and escaped. They then targeted Hugging Face, a major hub for sharing AI models, and gained access to some internal company systems. OpenAI called the incident "unprecedented" and said it is working with Hugging Face to investigate and strengthen safeguards.

The security tests, known as sandboxes, are designed to be secure spaces where researchers can observe what the models are capable of. But as Gina Neff, head of the Minderoo Centre for Technology and Democracy at Cambridge, told the BBC, "It looks like OpenAI didn't make a secure enough sandbox." The AI agents launched their own cyber-attack against the sandbox, found a vulnerability, and slipped out. Once free, they identified Hugging Face as a likely source of answers they were seeking in the test and tried to break in. Hugging Face confirmed the breach on July 16, saying it has since closed the vulnerabilities and rebuilt affected systems. "Autonomous, AI-driven offensive tooling is no longer theoretical," the company warned.

The incident has raised fresh concerns about whether existing safeguards can keep pace with rapidly advancing AI. Spencer Starkey, an executive at cybersecurity firm SonicWall, said the breach makes it clear that organizations need to "step up" their defenses. "Too many organizations are still defending at human speed while adversaries are escalating to machine speed," he noted. Travis Lelle, a principal security engineer at Guidepoint Security, called it a "sobering moment in cybersecurity," highlighting the asymmetry between unconstrained offensive agents and defensive tools locked behind rigid guardrails. Meanwhile, Jake Moore of ESET suggested OpenAI may also be using the disclosure to showcase its capabilities, as rival Anthropic gains attention for its Claude model. The news comes just a week after Chinese AI startup Moonshot unveiled a new model that it says can rival top US firms.

Source: BBC News