Anthropic Filing Puts AI's Existential Risks Alongside Its Business Case

Anthropic has filed the prospectus for its initial public offering, exposing a wide range of previously private information. Beyond the striking revenue and even larger expenditures, the filing's largest section is devoted to "risk factors" - the things that could derail the company's plans, or in this case, humanity itself.

The prospectus gives 94 pages to these risks, making clear where Anthropic sees danger coming from and how its researchers are racing toward it to win the AI arms race. "Sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight and security measures, pose catastrophic or existential risks to humanity," the filing states, citing critics, regulators and concerns from inside Anthropic. It is the first of three appearances of the phrase "existential risks."

Anthropic ties that outcome to a specific process: recursive self-improvement, in which humans hand over AI research and development to other AI models. The doomsday scenario is one where humans are no longer in the loop at all. "Risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models," the filing says. The prospectus repeatedly warns that RSI could produce models that outpace the company's safeguards before anyone notices.

It also gets specific about what might follow. "As AI systems become more autonomous and capable of taking consequential actions with less human oversight," the prospectus says, "there is a risk that such systems could, whether due to misalignment with human intentions, errors in objective specification, or self-preserving behaviors, take harmful actions independently and at scale without human intervention." Anthropic also flags risks from models directing physical equipment, where errors or misuse "could result in bodily injury, property damage, damage to costly equipment or experimental materials, or the release of hazardous substances."

Yet the same process is framed as a competitive necessity. In a section on "core advantages," the prospectus calls the recursive dynamic "closing the loop" and says the company believes "it is one of the most significant structural advantages in frontier AI." Failure to close the loop, Anthropic says, could mean "falling progressively further behind as competitors continuously use AI to further enhance their own development capabilities." The systems driving existential concern are the same ones Anthropic sees as critical to its success.

Source: Wired