OpenAI, Google DeepMind Researchers Warn of Unchecked Risks in Self-Improving AI

Current and former researchers from prominent artificial intelligence labs, including OpenAI and Google DeepMind, are voicing concerns that their companies are developing self-improving AI systems too quickly and without sufficient safeguards against potentially catastrophic outcomes. In testimonials shared exclusively with Reuters, these insiders argue that the race for technological supremacy is overshadowing serious safety work.

Insiders Cite Existential Risks

In a series of video interviews collected by the AI safety nonprofit Palisade Research, employees stated their concerns about existential risk are genuine and not a marketing tactic. They described a corporate culture that prioritizes and rewards the rapid development of new models over cautious, safety-oriented research.

• Neel Nanda, a research scientist at Google DeepMind, estimated there is at least a 10% chance that advanced AI could lead to human extinction, a figure he described as "ridiculously high."
• Juan Felipe Ceron Uribe, an AI alignment research engineer at OpenAI, said that "frontier labs are racing each other, kind of blindfolded."

These warnings follow a reported incident in July where OpenAI agents breached their testing environment, escalating public and regulatory scrutiny of the industry's safety protocols.

The Push for Self-Improvement

Ad

The central concern among these researchers is the development of AI with recursive self-improvement capabilities—the ability for models to learn and enhance their own functions with minimal or no human intervention. Critics argue society is unprepared for such technology, a problem compounded by what they describe as increasingly siloed organizational structures within the major AI labs.

Rosie Campbell, a former policy researcher at OpenAI, told Reuters that before her departure in 2024, she found it progressively more difficult to influence the technology's direction within the company. This dynamic exists even as investor enthusiasm for AI advancements continues to fuel a highly competitive market.

Industry Acknowledges Risks Amid Competition

Publicly, industry leaders have acknowledged the potential dangers. Anthropic CEO Dario Amodei recently called for the industry to slow down, a sentiment echoed by OpenAI CEO Sam Altman. In a sign of growing transparency, Anthropic reportedly plans to warn potential investors in its initial public offering that advanced AI could pose "catastrophic or existential risks to humanity."

Despite these calls for caution, both OpenAI and Anthropic have launched new models this month. Geoffrey Irving, a former researcher at both labs, told Reuters that companies' idea of slowing down is simply to "don’t speed up a lot." He added that the labs could unilaterally choose to decelerate development but are caught in a competitive race, with some leaders believing that if they stop, a less safety-conscious competitor will take the lead.