Anthropic and OpenAI Researchers Sound Alarm Over AI Self-Improvement Risks

Concerns about AI systems and their potential to harm humanity gained viral attention this week after Evan Hubinger, an alignment lead at Anthropic, said on X that he believes there is more than a 10% chance AI could kill all humans within the next decade. His comments followed a colleague's resignation over safety fears.

In a reply to his own post, Hubinger clarified his specific worry: superintelligence arising from recursive self-improvement, or RSI, which he said is happening faster than expected. RSI refers to AI helping improve the process of building new models, potentially leading to spiralling capability as better systems build better systems. The concern is that if AI takes control of how new models are trained, the humans who built those systems could lose control.

Both OpenAI and Anthropic have said in recent months that this autonomous model improvement is accelerating faster than they thought. Anthropic posted on X in June that its internal data shows Claude is accelerating AI development, calling it a possible path to RSI. The company said in an August blog post that its engineers now ship eight times as much code per quarter as they did between 2021 and 2025.

Vincent Conitzer, a computer science professor at Carnegie Mellon University, said AI is already at a level where it can introduce new ideas, making it hard to predict when the process might drastically accelerate capabilities.

On Saturday, OpenAI Chief Scientist Jakub Pachocki wrote in a company blog post that he was concerned no one was prepared for the consequences of a continued rapid rise in machine intelligence. He said systems in the next few years are likely to represent further capability jumps and increasingly drive their own development.

Warnings about RSI flooded social media this week from researchers at both labs following Jacob Coxon's resignation. Jasmine Wang, an OpenAI researcher working on alignment, said it is hard to overstate how dangerous speeding towards RSI is. Anna Wang, who works on AGI safety and alignment at Anthropic, said there is not yet a viable scientific plan to solve risks from recursively self-improving AI.

Anthropic's blog post laid out three possible scenarios. In one, progress at the frontier stalls and AI capabilities are widely diffused, which Anthropic said it does not believe is likely. A second possibility is that AI labs continue to make gains with humans in control, which Anthropic called likely. A third scenario could see AI systems become capable of full recursive self-improvement, with humans playing a substantially diminished role in their development. Anthropic said how the alignment problem gets solved, or not, in this future is what it is least certain about.

Source: CNBC