Will AI kill humans shortly? Why Anthropic's resignation has the world worried

Artificial Intelligence (AI) taking away human jobs has been one of the biggest fears surrounding the technology. But a more disturbing question is now back in focus: Could AI eventually kill humans?

The debate has resurfaced after AI researcher Jacob Coxon, who has worked at OpenAI and Anthropic, warned that the industry is racing towards self-improving superintelligence without adequate safeguards.

In a series of posts on X, Coxon said, “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

“Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing,” he wrote.

Researchers warn

His warning was echoed by Evan Hubinger, Anthropic’s Alignment Science Lead, who said the possibility of AI killing all humans was being taken seriously within the industry.

Also read: Anthropic warns AI could one day build its own successors, challenging human control

“Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Hubinger wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Hubinger’s comments are particularly significant because he works on AI alignment, the effort to ensure that advanced systems follow human values and remain under human control.

Beyond software

The concern is no longer about a technology that exists only inside computers. AI has already found applications across sectors, from medical research to defence.

“The reality is that AI is not just software, but a capability which now has a foothold right from medical science to defence,” Avinash M, an industry expert working on building an AI agent, told The Federal.

Also read: 'Build ethics into AI from day one, or risk paying for it later'

He explained, “AI-enabled defence systems and AI-controlled drones are a reality and are being used in recent conflicts across the world. AI is also actively used in medical research.”

Central concern

The central concern is not that an advanced AI system would necessarily hate humans. It is that a highly capable system could pursue a goal in ways that humans cannot predict or control.

The fear is that AI could eventually rewrite its own code, improve its capabilities and become more powerful faster than humans can understand or control it.

In July, more than 1,300 employees of frontier AI companies signed a statement warning of a “real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems”. They called on the US government to support an international effort to develop tools that could deliberately pace the development of advanced AI.

Also read: Most firms still at first stage of AI adoption: Hexaware Global CTO

Anthropic has also acknowledged that the speed of AI development could leave society with little time to prepare.

Why are companies still building it?

This raises another common question: If researchers believe AI could pose an existential threat, why are they continuing to develop it?

Coxon said that while some people at OpenAI had not fully understood the civilisational stakes, researchers at Anthropic were aware of the risks but felt trapped in a race. “They are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk,” he wrote.

Former Google DeepMind employee Alex Turner also backed the warning. “Many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that,” he said.

For years, leading technology figures, including Bill Gates, have warned about the risks posed by advanced AI. But the latest comments suggest that concern is no longer limited to outside critics. It is increasingly coming from the people building these systems themselves.