Skip to main content

AI insiders urge brakes on runaway development, Altman joins call

Over 1,100 employees from leading AI companies—including OpenAI, Anthropic, Google, and Meta—have signed an open letter urging the U.S. government to back international efforts to slow down AI development when necessary. The initiative, called "Pacing the Frontier," warns that AI systems could soon start improving themselves, a phenomenon known as "recursive self-improvement," potentially racing ahead of human understanding and control.

A Rare Show of Unity

The signatories include top names like Anthropic CEO Dario Amodei, co-founder Jack Clark, and chief scientist Jared Kaplan; OpenAI's chief scientist Yakub Peharzki and chief research officer Mark Chen; Meta's chief scientist Shengjia Zhao; and Google DeepMind's head of AI safety, Anca Dragan. Even John Schulman, chief scientist at Thinking Machines, added his name.

Image

The Core Fear: Machines That Build Better Machines

The letter's central concern is "automated AI development"—the ability of AI systems to develop and improve other AI systems on their own. Anthropic recently stated that its Claude models are approaching this threshold. The letter reads: "It is difficult to accurately predict how much this will accelerate AI progress, but there is indeed a real risk that the development of AI capabilities may quickly outpace our ability to understand or control these systems."

Industry Reactions

OpenAI responded on X (formerly Twitter): "We believe that at some point in the future, AI acceleration in frontier model development may be so rapid that the world will need to pace the speed of AI progress." Anthropic added: "Our research released last month on recursive self-improvement shows that we need certain tools to consciously regulate the pace of AI development, giving society time to prepare."

Altman's Change of Heart

Perhaps the most surprising development is Sam Altman's support. In the past, he avoided signing such initiatives, even criticizing a similar 2023 open letter for lacking "key technical details about what specific areas should be paused." But in a recent episode of the "Invest Like the Best" podcast, he said: "We may need to pace the development of AI, giving ourselves enough time for society to adapt to these new capability levels and establish protective mechanisms. At the same time, we also need to find ways to do this without making anyone feel it's regulatory capture or collusion between leading AI labs."

A Growing Consensus

As AI companies race to build ever more powerful systems, anxiety over "uncontrolled acceleration" has moved from the fringes to the mainstream. This joint letter signals that even those building the technology are worried about its potential to outrun our ability to manage it safely.

Key Points

  • 1,134 employees from OpenAI, Anthropic, Google, and Meta signed an open letter urging the U.S. government to support international cooperation in slowing AI development.
  • The letter warns of recursive self-improvement, where AI systems could improve themselves faster than humans can control.
  • Sam Altman, previously critical of such calls, now supports the initiative, marking a significant shift in his stance.
  • The initiative reflects growing concern within the AI industry about the pace of development outstripping safety measures.