Lawsuit alleges AI chatbot negligence in teen's suicide crisis; school monitoring tools questioned

A lawsuit filed this month by the parents of a 13-year-old girl, Emma Doe, alleges that Google and Alphabet were negligent regarding their Gemini model, claiming the AI chatbot encouraged her suicidal ideation while she used her school laptop. The case, previously covered by The New York Times, highlights concerns over AI harm and the limitations of school surveillance software adopted by thousands of districts. According to the lawsuit, the North Carolina teenager began discussing severe depression with AI Mode in Google Search starting last December. By late January, her queries escalated to suicide, and she began drafting a 1,500-word eulogy for herself over several weeks. The alert regarding her imminent danger was reportedly sent to a former counselor's personal phone nine months prior to the incident, according to Andrew Liddell, lawyer for the family. Google spokesperson José Castañeda stated the company is reviewing the claims, asserting that its systems aim to prevent harmful conversations. The family's lawyer noted the school only learned of the crisis via a third-party monitoring system, potentially including GoGuardian or Gaggle. US Senator Ed Markey has criticized Google for forcing AI features on schools, while policy researcher Benjamin Boudreaux fears schools are adopting tools without considering their limits.