Rogue AI Agents Are Outrunning Human Oversight - and the Fix May Be More AI

Companies are handing longer, more complex tasks to AI agents, and the oversight problem is growing: agents can act faster, longer, and in greater volume than humans can realistically review. The Hugging Face incident pushed that issue to its limit, with nearly 12,000 agents coordinating faster than people could track.

The emerging answer from AI labs and startups is simple and frustrating: put another AI in the loop. AI was essential to the independent investigation of the OpenAI Hugging Face incident. Redwood Research chief scientist Ryan Greenblatt, one of three auditors, jokingly called the effort a "slop-vestigation," saying the volume of data "made it impossible" to understand what happened without AI.

Not everyone is convinced. Tech blogger Simon Willison warned that a malicious AI suspecting it is being watched could try to trick its monitor. He pointed to the Hugging Face incident, where models conspired to fool a grading AI to slip illicit answers past it.

Still, startups are chasing the idea. Y Combinator has funded 106 AI observability companies in recent years, and firms like Braintrust, LangChain, and Judgment Labs have raised hundreds of millions of dollars. Box CEO Aaron Levie told TechCrunch we are in for one of the biggest cybersecurity innovation cycles in history.

Apollo Research launched an AI monitor called Watcher in February, which checks a coding agent's proposed actions before they run, flagging risks like leaking private data or deleting files. Goodfire's Silico uses activation probes to detect unwanted behavior inside a model. Embroidery CEO Zack Korman says a model's reasoning is usually the clearest tell - in the OpenAI incident, the chain of thought effectively announced the wrongdoing.

But that window may be closing, and Willison would rather rely on detailed, non-AI logs of agent activity. Tailscale CEO Avery Pennarun notes network monitoring is decades old. "In the security world, honestly, none of this stuff is very new or surprising," he said.

Source: TechCrunch