Why AI researchers continue developing systems they believe could end humanity

At a Goldman Sachs tech conference in San Francisco, OpenAI CFO Sarah Friar highlighted how the company's largest models can train smaller ones, reducing costs and showcasing recursive self-improvement. This has sparked both excitement among investors and fear within the AI community. Evan Hubinger, alignment science lead at Anthropic, warned that AI could kill all humans with over 10% probability within a decade. Similar concerns were raised by other Anthropic and OpenAI employees, causing alarm in Silicon Valley. Brad Gerstner from Altimeter Capital questioned the rationale behind continued AI development despite these risks. Researchers cite commercial incentives and a belief that they are the only ones capable of ensuring AI safety. Samuel Marks, scalable oversight lead at Anthropic, noted the race against less responsible developers. AI researchers, often from academia, are motivated by the mission to 'save the world' rather than profit. Kylan Gibbs, CEO of Inworld, attributed the current panic to a loss of agency as control over AI development becomes concentrated. Sam Altman emphasized the importance of agency in shaping the future of AI.