Why AI researchers are worrying about the potential for machine-controlled catastrophe

In June 2026, Rishub Jain resigned from Google DeepMind, citing concerns over AI surpassing human control. He warned that recursive self-improvement could lead to catastrophic outcomes if humans remain sidelined. Jacob Coxon, who left Anthropic, echoed these fears, warning of a race toward self-improving superintelligence. A senior Anthropic leader stated AI could kill all humans with over 10% probability within a decade. Nate Soares, coauthor of 'If Anybody Builds It, Everybody Dies,' argued that recursive self-improvement creates a feedback loop, though no lab has achieved full autonomy. Startups like Recursive Intelligence pursue the concept, while major firms issue warnings. Soares noted aligning AI with human values grows more complex as models advance. Daniel Kokotajlo highlighted that fears about recursive self-improvement predate recent events, alongside rising public concerns over AI’s impact on jobs and trust. Jain launched Sampura Research to develop human-in-the-loop techniques for AI alignment, suggesting collaboration could mitigate risks.