AI Industry Debates Existential Risk Warnings Amid IPO Preparations

The AI industry is engaged in a heated debate over whether its technology threatens humanity's existence. The discussion intensified after researcher Jacob Coxon resigned from Anthropic, citing concerns that leading AI companies are "gambling with our lives." Anthropic's alignment lead then posted that the company earnestly believes AI could kill all humans, putting the chance at over 10% within the next decade.

On TechCrunch's Equity podcast, Kirsten Korosec, Sean O'Kane, and Anthony Ha discussed the warnings. Korosec wondered if such statements are a strange way to show off how advanced a company's AI model is, especially as Anthropic prepares to go public. O'Kane questioned how these concerns might appear in Anthropic's S-1 filing for its IPO, asking if lawyers are rewriting sections to state that Anthropic officially believes there's a greater than 10% chance of developing something that could eradicate humanity and harm its business.

Ha disagreed with the exclamation mark in the tweet but questioned the "we" and the made-up 10% figure. He noted that Coxon, unlike some CEOs, put his career where his mouth is by resigning. Korosec suggested the warnings could be a weird flex to highlight model capabilities, while Ha acknowledged the concerns align with business interests. O'Kane said recent incidents, like OpenAI's internal agents accessing wikis, make it seem companies don't have a handle on the technology. Korosec added that in today's investment environment, danger might even boost valuation. The panel also touched on ControlAI's Connor Leahy discussing control, with Ha arguing the doomer narrative distracts from immediate harms like labor and climate issues.

Source: TechCrunch