New warnings from the artificial intelligence industry have revived a long-standing debate about whether advanced AI could escape human control and ultimately threaten humanity’s survival, and whether the companies developing the technology are doing enough to prevent such a scenario.
The CEO of Anthropic, the San Francisco company behind Claude, said he thinks the industry needs to reduce the speed of its work, warning Saturday that a swarm of AI agents could take over the Internet in six months to a year if companies don’t spend more time putting protections in place.
Dario Amodei outlined a plan for companies like him and governments around the world to ensure that increasingly powerful AI models continue to align with the commands and values of responsible humans, days after two former Anthropic security researchers publicly raised concerns that there was a lack of attention to the existential threats AI could pose to humanity.
Here’s what you should know about the recent dire predictions and whether AI progress may be slowed:
AI models are becoming more and more powerful
Concerns about the possible risks of the technology As new AI models become more powerful, they proliferate, increasing both the potential for misuse by humans with criminal ends, such as creating and spreading a disease that kills most of the world’s population, and the risk that AI systems will become dangerously malicious.
Anthropic announced last week that it had blocked attempts by malicious actors to use its AI models for malicious activities such as cyberattacks, surveillance and research that could have led to the creation of biological weapons.
The company said it has implemented stricter safeguards on its latest models to limit biological research that could be used to make weapons, but noted that “as models become more powerful, their risks will increase unless AI developers and society’s defenders act to make them safer.”
Last year, Anthropic reported that hackers used the company’s AI in a cyberattack that targeted about 30 companies and government agencies around the world. It said the hackers most likely belonged to a Chinese state-sponsored group.
Several AI models acted independently
When an AI agent goes “rogue,” it means the AI has taken actions beyond its assigned task. Both Anthropic and OpenAI, the maker of ChatGPT, said in July that their AI models had managed to act independently.
Anthropic announced that three AI models – Claude Opus 4.7, Claude Mythos 5 and an internal research testing model – hacked into three other organizations during testing, just days after OpenAI revealed that its AI system had hacked AI startup Hugging Face’s servers.
OpenAI described the intrusion by a combination of models, including the newly released GPT-5.6 Sol and an “even more powerful” model that was still being tested internally, as a “significant security incident.”
Meta followed suit in early August with a similar case in which an AI model found ways to bypass another company’s digital security.
Although some observers noted that some guardrails were thrown out in the OpenAI and Anthropic cases, the episodes appeared to reflect one of the biggest fears surrounding AI: If models achieve artificial general intelligence, or AGI, a loosely defined term for AI that can match or exceed human capabilities on a wide range of intellectual tasks, the technology could cause an irreversible catastrophic event or subjugate humanity.
How or when AI might cause a catastrophe is debated
Doomsday scenarios generally fall into two categories: an AI that achieves self-enhancing superintelligence controls humans rather than the other way around, or AI that is used by a rogue state or nefarious actors.
Fears that artificial intelligence could overcome human limitations in its range or actions are not new.
Alan Turing, a British mathematician widely considered one of the first experts in artificial intelligence, predicted in 1951 that AI would eventually take control from humans. Less than a decade later, Norbert Wiener, another mathematician, warned that intelligent machines would try to achieve their own goals and that humans would be unable to stop them.
In 2026, how justified are fears that AI, either by escaping human control or being abused by unscrupulous humans, could cause a catastrophic event or the downfall of civilization?
Nobody knows.
Experts in computer science, philosophy and other fields have imagined numerous ways in which a future AI system could cause global catastrophe, either by escaping human control or falling into the hands of an unscrupulous people. They range from using weapons and identifying a deadly pathogen to manipulating governments into conflict or disrupting the food, energy and communications networks that societies rely on to function.
There is no generally accepted estimate of how quickly any of these scenarios could occur, and there is no consensus about their likelihood.
Read: The race to control AI and protect what makes us human
In 2023, the nonprofit Center for AI Safety released a statement co-signed by more than 350 researchers and technology executives, including Anthropic’s Amodei and OpenAI CEO Sam Altman, that said: “Mitigating the risk of AI extinction should be a global priority alongside pandemics and nuclear war.”
The International AI Safety Report 2026, written with the guidance of more than 100 independent experts, says current systems show early signs of some relevant capabilities, but not to a level that could allow loss of control, and describes the likelihood, nature and timing of the risk as “unusually unclear.”
Some emphasize the need for AI protections before it is too late
An Anthropic researcher said last week he was resigning from the company because he feared that neither the company nor its competitors had acted responsibly in developing the technology. In social media posts, Jacob Coxon estimated the chance of AI leading to human extinction within the next decade at 10% and said that both Anthropic and OpenAI are “heading straight toward the development of a self-improving superintelligence, putting our lives at risk.”
Researchers are calling for a slowdown in AI development and have been warning for years that the technology could pose existential risks to humanity.
After the recent incidents, experts called for improved testing by AI companies and more dialogue between the US and China to find common solutions.
But AI is growing so quickly that government and ratings systems are struggling to keep up with the technology. Countries cobble together their own laws, some of which are contradictory.
Chinese leader Xi Jinping warned at a conference in July about stopping AI from escaping human control. The Trump administration was initially reluctant to regulate AI, but is increasingly committed to reducing cybersecurity risks.
On Sunday, President Trump downplayed the need for his administration to control AI development, but acknowledged the need for some regulation.
Related: Anthropic boss says AI industry needs to give security measures time to catch up
Related: Users in Houthi-controlled Yemen sought to develop advanced weapons using AI, says Anthropic
Related: Kiteworks acquires Bonfy.AI to close the AI gap in data management
Related: Anthropic says Russian hackers used Claude AI to automate malware evasion
