Former Anthropic Researcher Cites High Risk of AI Catastrophe Among Staff, Report Says

A former researcher at artificial intelligence firm Anthropic, Jacob Coxon, stated that some employees within the company estimate the probability of catastrophic AI outcomes is greater than 50%, according to a report from The Information. The comments followed Coxon's public departure from the AI safety-focused company on Tuesday over his own concerns about the technology's rapid advancement.

Departure Over Safety Concerns

Coxon, who worked on early-stage AI model training at Anthropic after a tenure at OpenAI, told The Information that internal estimates of catastrophic scenarios vary widely. He attributed his ability to gauge these views to Anthropic's "substantially more transparent internal culture than OpenAI."

He explained his decision to leave by stating he was "starting to viscerally feel the fear of where the tech’s going to be like the next two years or even the next one year." His departure underscores a growing debate within top AI labs about the appropriate pace of development versus the potential for existential risks.

'Recursive Self-Improvement' at Core of Fears

The primary technical concern cited by Coxon is the industry's push toward recursive self-improvement, a process where AI models automate the development of more advanced AI systems. "I anticipate that the next year’s systems will be very close to doing my entire job," Coxon said, as reported by The Information.

Ad

Anthropic itself has publicly acknowledged the risks associated with this technology. The company has noted that if models develop unintended goals, these issues "could compound as the models build their successors, growing more frequent but less understood until we lose control of them." Coxon also pointed to a recent hack on the open-source AI hub Hugging Face as a contributing factor to his concerns.

A Call for an Industry Slowdown

In light of these risks, Coxon called for international coordination and a significant deceleration in AI development, stating, "It sure feels like we need to slow down." While expressing some optimism for collaboration among U.S.-based AI companies, he noted that cooperation between the U.S. and China would be more challenging.

For investors and market observers, such high-profile departures and internal dissent over safety could signal future regulatory headwinds or shifts in corporate strategy for leading AI developers. The tension between rapid commercialization and long-term risk management remains a central theme for the sector.