A former researcher at OpenAI and Anthropic, Jacob Coxon, has warned that the rapid development of advanced artificial intelligence could eventually produce systems more capable than humans and create an unprecedented threat to humanity.
Coxon resigned from Anthropic this week, saying the leading AI companies are moving too quickly towards what he described as self-improving superintelligence without having adequate safeguards in place. He previously worked on GPT-4o at OpenAI.
In his resignation statement, Coxon argued that researchers developing advanced AI systems believe the technology could become capable of causing catastrophic harm, potentially including the destruction of humanity, by the end of the decade.
He said the industry’s competitive race to develop increasingly powerful systems was creating pressure to move ahead before researchers had developed reliable methods for keeping future AI systems under human control.
The central concern raised by Coxon and other AI-safety researchers is the possibility of artificial superintelligence systems that substantially outperform humans across a wide range of intellectual tasks.
Researchers are particularly concerned about systems capable of improving their own capabilities. If such systems became significantly more capable than their creators while pursuing objectives that were not properly aligned with human interests, they could become difficult to control.
Evan Hubinger, an alignment science lead at Anthropic, publicly backed Coxon’s concerns, saying there is not yet a proven solution to the problem of aligning future superintelligent systems with human goals. Hubinger estimated that he personally puts the risk of AI causing human extinction at more than 10 per cent over the next decade.
The warnings are not limited to Anthropic. Marcus Williams, a member of OpenAI’s safety oversight team, said in a social-media post that without regulation or a coordinated slowdown among AI laboratories, human extinction could become a serious risk in the coming years.
The latest warnings come as OpenAI, Anthropic and other technology companies compete to develop increasingly capable AI models and autonomous systems.
OpenAI chief scientist Jakub Pachocki has also called for caution around the development of increasingly powerful AI, while Paul Christiano, a prominent AI-safety researcher who recently joined OpenAI, has warned that building superintelligence without stronger alignment could result in a permanent loss of human control.
The debate centres on how quickly AI capabilities should advance compared with the development of safety measures, oversight and regulation.
Supporters of continued rapid development argue that increasingly capable AI could produce major advances in science, medicine, productivity and other fields. Safety researchers, however, say the potential benefits do not eliminate the need to address risks associated with systems that could eventually operate with considerable autonomy.
The prospect of AI causing human extinction remains highly contested.
Some researchers and technology figures argue that predictions of an AI-driven catastrophe are speculative and that current systems remain far from the level of autonomous superintelligence described by AI “doomers”.
The latest debate has also drawn criticism from Elon Musk, who questioned whether Coxon’s resignation and the subsequent wave of warnings were part of a coordinated effort to influence public opinion and regulation. Coxon rejected the suggestion, maintaining that his concerns were genuine.
Other experts have focused less on hypothetical superintelligence and more on risks that are already emerging, including AI-generated disinformation, cyberattacks and the potential misuse of AI in biological research.
Anthropic itself recently disclosed that it had disrupted attempts to misuse its AI systems for activities involving biological and conventional weapons, underscoring concerns about the immediate risks posed by increasingly capable AI tools.
Coxon’s resignation has renewed calls for AI companies and governments to establish stronger safety mechanisms before highly autonomous systems become significantly more powerful.
The challenge is particularly difficult because researchers have yet to establish a universally accepted method for guaranteeing that a future superintelligent system will reliably follow human intentions.
For now, the debate remains largely about risk assessment and preparedness rather than evidence that AI is already capable of overtaking humanity.
But as AI systems become more capable and increasingly able to perform complex tasks with limited human supervision, researchers such as Coxon argue that safety measures must develop alongside the technology rather than after it.
