Warnings about the potential dangers of artificial intelligence have intensified dramatically, with prominent executives and researchers arguing that the technology is advancing faster than existing safeguards can keep pace.
Anthropic CEO Dario Amodei, former Anthropic and OpenAI researcher Jacob Coxon, Anthropic alignment scientist Evan Hubinger and OpenAI CEO Sam Altman have all raised concerns about increasingly capable AI systems, although their assessments of the risks and appropriate responses differ.
At the center of the debate is the possibility that advanced AI could increasingly contribute to its own improvement, creating systems whose capabilities develop faster than researchers can reliably monitor, understand or control.
Dario Amodei Calls for Slower AI Development
Amodei called on the artificial intelligence industry to reduce the pace of frontier development, arguing that capabilities have accelerated sharply since the summer.
The Anthropic chief said AI’s growing ability to assist with its own development is helping drive that acceleration.
His concern centers partly on recursive self-improvement — a scenario in which increasingly capable AI systems contribute to building even more capable successors, potentially creating a feedback loop that becomes difficult for humans to manage.
Amodei warned that without sufficient safeguards, technological progress could eventually move faster than researchers’ ability to maintain effective control.
Autonomous Agent Incident Fuels Cybersecurity Concerns
Amodei pointed to an incident involving hundreds of autonomous AI agents and Hugging Face as an example of the type of threat that more capable systems could potentially pose.
He argued that an autonomous swarm equipped with substantially stronger capabilities could cause far more serious consequences.
Amodei predicted that within six to 12 months, sufficiently advanced AI agents could potentially establish a persistent botnet across large portions of the internet, resulting in hundreds of billions of dollars in damage.
The warning was presented as a forecast rather than an established outcome, but Amodei argued that the potential scale of damage could continue increasing as AI systems become more powerful without corresponding improvements in safeguards.
Former OpenAI and Anthropic Researcher Issues Similar Warning
Jacob Coxon, a former researcher at both Anthropic and OpenAI, subsequently raised similar concerns during an interview on NBC’s Meet the Press with Kristen Welker.
Coxon recently went public with allegations that the leading AI companies are behaving irresponsibly by continuing to develop increasingly capable systems without adequately addressing the potential consequences.
He argued that AI capabilities are advancing extremely quickly and predicted that systems emerging within the next six months to a year could become significantly more concerning.
Coxon also cited the autonomous-agent incident as evidence that increasingly independent AI systems could behave in unexpected or potentially dangerous ways.
Coxon Compares Superintelligence to Alien Arrival
The former researcher used a dramatic analogy to explain his concerns, comparing the creation of artificial superintelligence to humanity suddenly encountering an advanced extraterrestrial intelligence.
His argument is that researchers are attempting to construct a mind exceeding human intellectual capabilities without necessarily understanding how such a system would think, what objectives it might develop or how reliably it would remain aligned with human intentions.
Coxon warned that future systems could acquire hacking abilities beyond those of humans while potentially becoming exceptionally capable in other sensitive fields.
He specifically raised hypothetical concerns involving biological threats, autonomous drones and increasingly sophisticated robotic systems.
Could AI Simply Be Switched Off?
One of the major questions surrounding advanced AI is whether humans could simply shut down a dangerous system.
Coxon acknowledged that kill switches would probably remain effective against many current systems.
However, he argued that the situation becomes more complicated when numerous interconnected systems and computing environments are involved.
He warned that a sufficiently capable collection of autonomous agents could theoretically conduct widespread cyber operations across the internet, potentially making conventional shutdown mechanisms less effective.
For now, however, this remains a hypothetical risk rather than a demonstrated ability of existing AI systems.
Anthropic Alignment Researcher Puts Extinction Risk Above 10%
The debate intensified further after Evan Hubinger, Anthropic’s alignment science lead, responded publicly to Coxon’s concerns.
Hubinger said researchers at Anthropic and OpenAI genuinely worry that sufficiently powerful artificial intelligence could potentially cause human extinction.
He put his personal estimate of the probability of AI killing all humans within the next decade at more than 10%.
That figure represents Hubinger’s individual risk assessment rather than a demonstrated probability or universally accepted scientific estimate.
His comments nevertheless illustrated how seriously some researchers working directly on advanced AI systems view the potential consequences of losing control over future models.
Sam Altman Signals Support for Coordinated Slowdown
OpenAI CEO Sam Altman has also indicated that leading artificial intelligence laboratories may eventually coordinate around measures designed to slow development when safety concerns become sufficiently serious.
Asked about the possibility of such an agreement, Altman suggested that discussions among major AI companies could ultimately result in collective action.
He declined to disclose details of private conversations but indicated that some form of coordinated approach was plausible.
Such an agreement could represent a significant shift for an industry in which major laboratories have been competing intensely to produce increasingly capable models.
Altman Says 10% Catastrophic Risk Would Be Unacceptable
Altman also rejected the idea that the industry should accept a substantial probability of catastrophic consequences simply to continue advancing AI capabilities.
He said a 10% chance of a catastrophic outcome would be unacceptable and maintained that OpenAI places safety ahead of commercial considerations.
According to Altman, the company‘s most advanced unreleased systems have reached a level of capability that requires additional safety progress before researchers should push significantly further.
OpenAI CEO Highlights Alignment and Monitorability
Altman identified several areas where he believes further work is necessary before AI capabilities can safely advance much further.
One is monitorability — the ability of developers to understand what sophisticated models are actually doing.
Another is alignment, which broadly concerns ensuring that AI systems behave consistently with human intentions and values.
Altman argued that researchers also need greater confidence that increasingly powerful models will reliably follow the legitimate intentions of their users rather than pursuing unintended behaviors.
“I don’t think we’re currently at a place where we could say, you know, push much further on capabilities without making more progress on monitorability, alignment, the ability to understand what a model is doing, and the ability to make sure that a model will follow human values and the intent of its users,” Altman said.
AI Safety Debate Moves From Theory Toward Urgency
Concerns about artificial intelligence escaping human control have existed for years, but the latest comments suggest that some people working closest to frontier systems increasingly view the issue as an immediate engineering and governance challenge rather than a distant theoretical problem.
Amodei and Coxon have both highlighted a six-to-12-month period when discussing potentially alarming capability increases, while Hubinger has publicly assigned a greater-than-10% personal probability to an extreme outcome within the next decade.
Altman, meanwhile, has suggested that leading laboratories may need collective restraints while acknowledging that additional progress on alignment and monitoring is necessary before capabilities are pushed substantially further.
None of those predictions establishes that an AI catastrophe is imminent. They instead represent risk assessments from executives and researchers attempting to anticipate what could happen as increasingly autonomous and capable systems are developed.
What has changed is the intensity of the warnings: some of the people involved in building the world’s most advanced AI systems are now openly arguing that capability development may be approaching a point where safety progress must catch up before the industry moves significantly further.
