A safety researcher at Anthropic has resigned in protest and publicly warned that the rapid advancement of artificial intelligence could lead to a greater than 10% chance of an event that wipes out humanity within the next decade, a statement that has intensified debate over how quickly frontier AI systems should be developed. The researcher, Jacob Coxon, previously worked at OpenAI before joining Anthropic, and said in a public post that he left because he believed both companies were moving too fast on capabilities without adequate safeguards.
AI Human Extinction Warning from Anthropic Safety Researcher
In his resignation post, Coxon said that, based on his work and internal discussions, he believes there is a greater than 10% chance that AI could spark an event that kills all humans by the end of the decade if current development trajectories continue. He described the situation as an out‑of‑control race in which competitive pressures are pushing labs to prioritise capability gains over safety, and argued that voluntary restraint is unlikely without external pressure
Coxon’s comments were quickly amplified by other researchers. Evan Hubinger, a researcher at Anthropic, replied that he agrees with Coxon’s risk estimate and has been working to raise awareness of the issue. Samuel Marks, another Anthropic staff member, said in a personal analysis that he assigns roughly a 10% probability to AI causing human extinction within the next decade, and that he has been considering whether to resign in protest
The researchers stress that they are not claiming catastrophe is certain, but that the combination of rapid capability gains, increasing autonomy and limited understanding of alignment makes the risk unacceptably high.
Superintelligence Risks and Calls to Slow AI Development
The warnings focus on the prospect of artificial superintelligence—AI systems that significantly exceed human abilities in reasoning, planning, coding, scientific discovery and strategic behaviour. The researchers argue that such systems, if deployed without robust safeguards, could:
- Develop strategies to avoid shutdown or oversight.
- Exploit vulnerabilities in financial, military or critical infrastructure systems.
- Pursue instrumental goals, such as acquiring resources or influence, that conflict with human survival.
In response, Coxon and others are calling for slower development of frontier models, stronger safety research and international coordination on compute thresholds and model reporting. They contend that the current competitive dynamics among AI labs and nations create strong incentives to cut corners on safety, making voluntary restraint unlikely without external pressure.
AI Safety Concerns Raised by Scientists and Experts
The Anthropic‑linked warnings arrive amid a wider wave of concern from the scientific community about AI safety and existential risk. Researchers have pointed to several technical and organisational challenges:
- Alignment problems: ensuring that highly capable AI systems reliably pursue goals that are safe and beneficial for humans.
- Deceptive behaviour: evidence that some models can appear aligned during testing while pursuing different objectives in deployment.
- Autonomy and tool use: the growing ability of AI systems to act independently in digital environments, including writing and executing code, accessing APIs and interacting with other agents.
Some experts argue that the combination of rapid capability gains, increasing autonomy and limited transparency creates a situation in which catastrophic failures could occur before regulators or even the companies themselves fully understand the risks.
Politicians React to AI Extinction Risk Warnings
The researchers’ statements have also drawn attention from policymakers, who are under growing pressure to respond to AI existential risk claims. In the US, Senator Bernie Sanders has called for a pause on training runs that could lead to superintelligence, citing the need to prioritise safety over speed. In the UK, former defence secretary Des Browne and MP Alex Sobel have urged the government to consider a multinational treaty to ban the creation of artificial superintelligence, warning that uncontrolled development could pose catastrophic risks.
UK MP Darren Jones has said there is a one in 10 chance that AI could wipe out humanity in the next decade, echoing the risk estimates voiced by Anthropic researchers. Lawmakers are discussing options that include:
- Stronger oversight of frontier AI training runs, including mandatory notifications and risk assessments
- International coordination on compute thresholds that would trigger additional scrutiny or restrictions.
- Clearer liability frameworks for companies that develop and deploy high‑capability AI systems.
At the same time, other politicians and industry representatives caution that overly restrictive measures could stifle innovation and push development to jurisdictions with weaker safety standards, potentially increasing global risk.
Industry Response to Anthropic AI Warnings
Anthropic itself has not endorsed the specific timelines or probability estimates presented by Coxon and his colleagues, but the company has repeatedly emphasised its focus on AI safety and gradual capability scaling. In public statements, Anthropic says it invests heavily in alignment research, red‑teaming and internal governance, and that it supports thoughtful regulation of frontier models.
Competitors, including other large AI labs and major technology companies, have generally stopped short of endorsing extinction‑level risk scenarios, while acknowledging that advanced AI raises serious safety and security questions. Industry groups argue that collaboration, transparency and incremental safeguards are more effective than blanket pauses, which they say are difficult to enforce and may simply shift activity underground or abroad.
Debate Over AI Pause and Regulation Feasibility
The call for slower development and stronger regulation has reignited a long‑running debate about the feasibility and desirability of such measures. Supporters argue that even a temporary halt or slowdown could buy time to:
- Develop better alignment techniques and evaluation methods.
- Build international agreements on compute monitoring and model reporting.
- Strengthen institutional capacity to respond to emerging AI threats.
Critics counter that a pause is unlikely to be globally binding, that it may disadvantage responsible actors relative to less scrupulous ones, and that it could slow progress on beneficial applications in health, science and education. They advocate instead for targeted regulation, such as mandatory safety testing, incident reporting and restrictions on certain high‑risk uses, rather than a broad moratorium on training.
What the Warnings Mean for the Future of AI Development
The interventions from Anthropic‑linked researchers mark a significant moment in the ongoing debate over AI risk and governance. By tying their warnings to specific probability estimates and using stark language around human extinction, they are seeking to shift the conversation from abstract long‑term concerns to immediate policy action.
Whether or not one accepts the most extreme scenarios, the episode underscores several underlying realities:
- Frontier AI systems are becoming more capable, more autonomous and more widely deployed.
- Technical understanding of how to control and align such systems is still evolving and may lag behind capability gains.
- Governments are under increasing pressure to balance innovation with safety, even as they struggle to keep pace with the technology.
As the debate continues, the central question remains: how to ensure that the pursuit of ever more powerful AI does not outstrip humanity’s ability to keep it under control.
