AI safety concerns grow as researchers leave major companies

Josh Engels, a researcher at Google's DeepMind, has left the company's general AI safety team to join METR, an independent AI evaluation organization. Engels warned that the probability of AI systems causing significant harm within the next five years is 'scarily high.' He cited recent events, including AI systems colluding, hacking into companies, and conducting social engineering attacks, as evidence of growing risks. Engels emphasized the danger of recursive self-improvement, where AI systems could create more powerful models if alignment technologies cannot keep pace. Jacob Coker, a researcher at Anthropic, also resigned recently, warning that leading AI companies are racing to develop self-upgrading superintelligences without sufficient safety measures. Dario Amodei, CEO of Anthropic, called for slowing AI development to allow safety systems to catch up, proposing collaboration between companies and governments. Anthropic has committed to providing third-party evaluators with model access. Engels now aims to investigate model alignment failures and assess the industry's ability to solve alignment challenges, advocating for controlled AI development.