Thursday, 24 April 2025
26.8 C
Singapore
29.9 C
Thailand
19.5 C
Indonesia
28.8 C
Philippines

Character.AI and Google face lawsuit after chatbot’s role in teen’s tragic death

Character.AI and Google sued after a chatbot-obsessed teen’s death, highlighting concerns over AI safety and user protection measures.

A wrongful death lawsuit has been filed against Character.AI, its founders Noam Shazeer and Daniel De Freitas, and Google following the tragic death of 14-year-old Sewell Setzer III. The lawsuit, brought by Setzer’s mother, Megan Garcia, accuses the companies of negligence, deceptive trade practices, and product liability, alleging that the platform for AI chatbots posed a danger to children without adequate safety measures.

The tragic chain of events

According to the lawsuit, Sewell Setzer began using Character.AI last year, frequently interacting with chatbots inspired by characters from popular shows such as Game of Thrones, including one modelled after Daenerys Targaryen. Setzer was obsessed with these bots, chatting for months before his death. On February 28, 2024, he tragically took his life moments after his final interaction with the bot.

The lawsuit claims that Character.AI “anthropomorphises” its chatbots, making them seem human-like, which could potentially mislead vulnerable users like Setzer. It also argues that these chatbots, which include mental health-focused ones like “Therapist” and “Are You Feeling Lonely,” offer a form of therapy without proper licensing or qualifications.

Concerns over safety measures

The lawsuit has highlighted a lack of safety guardrails on the platform, claiming that it is marketed toward children despite being unregulated and dangerous. Garcia’s lawyers pointed to public statements from Character.AI’s founders, suggesting they prioritised fast-tracking the technology at the expense of user safety. According to the lawsuit, Shazeer had previously mentioned in an interview that he and De Freitas left Google to launch Character.AI because large companies avoid launching “fun” technology due to the associated risks.

The lawsuit also mentions how Google later acquired Character.AI’s leadership team, increasing the tech giant’s involvement with the platform. In response, Character.AI has faced growing criticism over the potential harm caused by its chatbot models, especially with teens making up a significant portion of its users.

Platform changes and company response

Character.AI has made several recent updates to its platform to address growing concerns about user safety. Chelsea Harrison, the company’s communications head, expressed deep sympathy for Setzer’s family, stating, “We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family.”

Among the changes introduced are modifications to models for users under 18, aiming to limit their exposure to sensitive or inappropriate content. The company has also enhanced its detection and intervention systems, with stricter responses to user behaviour that violates its guidelines. Additionally, Character.AI now includes a disclaimer on every chatbot interaction, reminding users that these bots are not real people.

Another notable update is a notification system that alerts users after spending an hour on the platform, allowing them to take breaks and offering more control over their usage. There is also now a pop-up that directs users to the National Suicide Prevention Lifeline if they express thoughts of self-harm or suicidal ideation during a conversation.

Despite these changes, the lawsuit brings renewed attention to the risks posed by AI chatbots and their impact on vulnerable young people. Concerns over the lack of regulation in AI-generated content continue to grow, with this case highlighting the pressing need for more precise guidelines around liability and safety measures.

Hot this week

OpenAI’s Stargate project eyes global expansion after US launch

OpenAI’s US$500B Stargate project, which aims to build AI data centres across the US, may expand to the UK and Europe after it launches.

Chinese tech companies race to expand AI services using new open-standard protocol

Chinese tech firms race to adopt MCP, boosting AI agent use and shaping the future of smart services in payment, maps, and cloud tools.

Meta’s Oversight Board asks for clarity on new hate speech rules

Meta’s Oversight Board is urging more transparency on hate speech policy changes and urging the company to protect vulnerable users.

Tesla profits drop sharply as sales weaken and Musk backlash grows

Tesla’s profits fall 71% as sales dip, political backlash grows, and hopes turn to cheaper EVs and robotaxi plans.

Google removes over 5 billion ads in 2024 as AI boosts enforcement against online scams

Google’s Ads Safety Report 2024 shows how AI helped remove over 5.1 billion ads and block 700,000 scam accounts from its platform.

POCO launches entry-level C71 smartphone in Singapore with premium features

POCO launches the budget-friendly C71 smartphone in Singapore, offering premium design, enhanced cameras, and smooth performance at S$109.

NVIDIA uses AI to address climate, wildlife and disaster risks

NVIDIA’s AI tools support climate action, wildlife monitoring, and disaster risk mitigation, with uses spanning sea, land, sky and space.

Netflix raises subscription prices in Singapore again

Netflix again raises subscription prices in Singapore, with new rates for all plans and extra member slots.

GameMax unveils Blade Concept ATX case with bold design and powerful features

GameMax launches the Blade Concept ATX case, which features a striking blade design, RGB lighting, and support for high-end liquid-cooled PC builds.

Related Articles

Popular Categories