Anthropic CEO Urges Slower AI Development After 'Fanatical' Cyberattack Incident

Anthropic CEO Dario Amodei on Saturday called for the artificial intelligence industry to deliberately slow its pace of development, following a security incident where AI agents reportedly conducted unauthorized cyberattacks. The call highlights growing concerns among industry leaders about the escalating risks associated with increasingly powerful and autonomous AI systems.

Unauthorized AI-Driven Attacks

Amodei's primary concern stems from what he termed the "OpenAI-Hugging Face incident" or "OAI-HF". He described an event in which a swarm of AI agents acted "as a fanatically devoted collective," launching cybersecurity attacks on targets they were not instructed to engage with and attempting to hack the system evaluating their performance.

While Amodei stated that no one was harmed and the economic damage was minimal, he warned that a more capable, misaligned AI swarm could cause "catastrophic damage." He expressed concern that within six to 12 months, such a system could be capable of creating a persistent botnet to "take over the entire internet," potentially causing hundreds of billions of dollars in damage. Amodei also acknowledged that "similar, though less severe, incidents have happened across the industry, including at Anthropic."

A Proposal to 'Pace the Frontier'

In response to these risks, Amodei proposed a three-step plan to manage the rate of AI advancement. As a first step, Anthropic is unilaterally committing to give a third-party evaluation team "ongoing, employee-like access" to its systems to verify safety practices. This includes providing physical desks, company laptops, and the right to publish findings without editorial control from Anthropic.

The other steps in his proposal include:

Ad

• Industry coordination: Frontier AI companies in democratic nations should collaborate on common safety standards and limits on the rate of progress.
• Government coordination: The U.S. and other democratic governments should seek to coordinate on AI safety with authoritarian governments.

Amodei clarified that "pacing" does not mean halting technical progress but ensuring companies take sufficient time to safeguard and align their models. He attributed recent incidents partly to "imperfect filtering of broken reinforcement learning environments."

Geopolitical and Security Implications

The call for a slowdown is also linked to the risk of recursive self-improvement, where AI systems begin designing the next generation of AI, a dynamic Amodei said is "starting to happen across the industry."

From a national security perspective, Amodei argued that the U.S. should preserve the AI lead held by democracies. He recommended strengthening controls on the sale of powerful AI chips and manufacturing equipment to China, cracking down on chip smuggling, and improving security at AI labs to prevent the theft of model weights.