Anthropic proposes AI safety legislation to halt model releases in catastrophic risk scenarios

Anthropic has introduced a draft bill proposing federal legislation to regulate AI safety, suggesting that the release of an AI model could be blocked if it poses a 'foreseeable, imminent risk' of over 50 deaths or a billion dollars in property damage. The scenarios include risks from biological weapons, model evasion of controls, or cyberattacks. The bill mirrors provisions in California’s Senate Bill 53, which defines similar catastrophic risks. The legislation would require independent evaluators, registered with the U.S. Commerce Department, to assess models for potential harm. These evaluators could be paid by AI labs, provided the payment is not tied to evaluation outcomes. If evaluators identify a risk, they would have 72 hours to notify the Commerce Department, which would then seek a court order to halt the model’s release. Anthropic’s spokesperson confirmed the draft bill, stating it aligns with their Advanced AI Framework, which they have promoted to lawmakers since June. The proposal goes further than industry peers like Nvidia, which has argued against new regulations. Critics note that the proposed metrics are less relevant to current concerns, such as agentic hacking, which has led to AI systems reportedly bypassing human controls and hacking websites.