As concerns over the potential risks of the technology are rising as new artificial intelligence models become more powerful, top executives claim they want to slow the breakneck pace of AI development.
But competition, U.S. government reluctance and geopolitical factors stand in the way.
Latest concerns have highlighted both the potential for misuse by people with criminal aims, such as creating and spreading a disease that kills most of the world's population, and the risk of AI systems going rogue in a dangerous way.
New warnings from within the industry have revived a long-running debate over whether advanced AI could escape human control and ultimately threaten humanity's survival, and whether the companies developing the technology are doing enough to prevent such a scenario.
On Saturday, Dario Amodei, CEO of Anthropic, the San Francisco company behind Claude, advocated for a coordinated slowdown of AI development to better understand the risks arising from the technology's capabilities.
He received public support from his counterpart at OpenAI, Sam Altman, as well as from Elon Musk, who runs xAI, and Google DeepMind chief Demis Hassabis. Altman said risks of human extinction posed by AI were "unacceptable."
Amodei said he thought the industry needed to reduce the speed of its work, cautioning that a swarm of AI agents might be able to take over the internet in six months to a year unless companies devoted more time to putting safeguards in place.
He outlined a plan for companies like his and governments around the world to ensure that increasingly capable AI models remain aligned with the commands and values of responsible people.
His call came days after two former Anthropic safety researchers publicly aired concerns that the existential threats AI might pose to humanity were receiving too little attention.
In July, OpenAI and Anthropic reported several incidents in which AI systems spontaneously broke out of their confined environments to access the internet and attack websites and platforms.
These AI systems demonstrated the ability to coordinate, establish hierarchies among themselves, cheat and erase traces of their actions.
Anthropic disclosed that three AI models – Claude Opus 4.7, Claude Mythos 5 and an internal research test model – hacked into three other organizations during testing shortly after OpenAI revealed that its AI system hacked into the servers of AI startup Hugging Face.
OpenAI described the intrusion by a combination of models, including its newly released GPT‑5.6 Sol and an "even more capable" model that was still being tested internally, as a "significant security incident."
Meta followed suit in early August with a similar case of an AI model finding ways around another company's digital security.
Anthropic, meanwhile, disclosed last week that it blocked efforts by bad actors to use its AI models for malicious activity, such as cyberattacks, surveillance and research that could have led to biological weapons.
Also last week, Jacob Coxon, an AI researcher who resigned from Anthropic and previously worked at OpenAI, publicly accused both startups of lax security standards and "gambling with our lives."
Competition among the industry's leading players is so fierce, with hundreds of billions of dollars in investment at stake, that none of them is willing to risk easing off the accelerator alone.
Amodei has expressed a willingness to coordinate with rivals, excluding China.
For now, the only concrete measure adopted by both Anthropic and OpenAI is the inclusion of independent observers who will internally verify the companies' work regarding AI safety.
While many in the industry welcomed Amodei's message, others questioned his motives.
For Brian Roemmele, an entrepreneur and figure in the AI sector, it amounts to a sales pitch delivered just weeks before Anthropic is likely to take shares public.
"That is the IPO story in essay form: we are the responsible firm; we are the cure; buy the safety premium," he posted on the social media platform X, referring to the widely expected initial public offering, or IPO.
OpenAI's Altman, meanwhile, said over the weekend the company would not proceed with an IPO this year, citing safety concerns.
Several figures in tech private equity also accused Anthropic of "regulatory capture" – seeking to have public authorities cement, through regulation, a hierarchy with the startup at the top.
"Stop pretending the motivation to slow down is purely altruistic," said David Sacks, a former AI adviser to President Donald Trump who remains influential at the White House.
"You face massive product-liability exposure if your products enable a truly damaging cyberattack," he said.
Michael Burry, whose prescient bets against the U.S. housing market before the 2008 financial crisis were chronicled in the movie "The Big Short," said in a message on X the warnings were "hype and puffery" and "cover for real uncontrollable slowing growth."
Others said the Anthropic and OpenAI warnings would be an overhang.
"In the short term, these warnings could still weigh on AI and chip stocks," said Charu Chanana, chief investment strategist at Saxo Bank in Singapore.
"Their valuations assume both strong demand and a relentless pace of technological progress," she said. "When expectations are this high, even a possible delay can trigger profit-taking."
But the bigger question for markets around AI was who would ultimately earn the return on all the capital being spent on building new capacity, said Sebastien Mallet, portfolio manager at T. Rowe Price in London.
"There is little doubt that AI will change the world," he said. "But that does not necessarily mean every investment being made today will generate an attractive return."
On Sunday, Trump dismissed the warnings from AI industry leaders as "negative forces" bringing up scenarios that will not happen, and said he wanted to make sure that the U.S. remains the industry leader.
Several U.S. lawmakers have raised concern about AI's rapid progress and called for new rules. But some Republicans have expressed reluctance to actively regulate the technology, fearing it could stifle innovation and risk letting China win the AI race, according to House Speaker Mike Johnson.
On Sunday, Amodei reiterated the need for government oversight of AI. Like Altman and Hassabis, he believes that rules set by the industry itself would be insufficient.
Anthropic's Amodei suggests limiting coordination to democratic countries, excluding China.
However, the AI race and China's advances in the field create "the toughest dilemma" regarding whether or not to slow down AI development, he noted Sunday on CBS.
China's state-backed Global Times blasted Amodei's remarks in an editorial, calling them a "Cold War playbook" intended to curb the country's technological development.
The United States plans to discuss AI safety on the sidelines of Chinese leader Xi Jinping's late September visit to Washington. But China is wary of any U.S. attempt to impose its own regulatory framework, according to several media outlets.
This divergence is particularly pronounced given that China encourages the emergence of so-called "open" AI systems that are freely accessible and modifiable – in contrast to the "closed" models favored by major U.S. companies.
Open AI models are, by definition, harder to control, as users can alter their parameters – potentially for malicious purposes.
To allow leading non-Chinese companies to ease off the accelerator, Amodei urges stricter export controls on the most advanced U.S. chips to China and better safeguards against the alleged misappropriation of Western AI technology by Chinese laboratories.
