To improve user safety, Character.AI, the popular chatbot service, has announced new safety measures, including a suite of parental controls and updates to its model for teenage users. These changes come after significant scrutiny from the press and two lawsuits, which claim that the platform contributed to self-harm and suicide among minors. The company is working to address these concerns by implementing several new features and making its platform safer for younger users.
Separate models for adults and teens
Over the past month, Character.AI has worked on developing two distinct versions of its large language model (LLM): one designed for adult users and another tailored for teenagers. The teen-specific LLM limits how bots respond to users, particularly around sensitive and romantic content. The company aims to block potentially harmful or suggestive conversations more aggressively, ensuring that the bots remain appropriate for younger audiences.
Character.AI’s new system also aims to detect and block user prompts that elicit inappropriate content. If the system detects language related to suicide or self-harm, a pop-up will now appear, directing users to the National Suicide Prevention Lifeline. This new feature, previously reported by The New York Times, is part of the platform’s broader effort to protect vulnerable users from harmful interactions.
Restrictions on editing bot responses
Minors using Character.AI will no longer be able to edit their bots’ responses. This feature, previously available to users, allowed individuals to modify conversations and add content that the platform would otherwise block. By removing this option for minors, Character.AI is attempting to reduce the likelihood of users engaging in inappropriate or harmful content.
Additionally, the company addresses concerns raised in recent lawsuits about addiction and confusion over whether the bots are human. Users will receive a notification after spending an hour interacting with the bots, encouraging breaks to prevent overuse. The platform also updates its disclaimer to clarify that all chatbot interactions are fictional and should not be relied upon for professional advice. A warning will appear for bots that describe themselves as therapists or doctors, indicating that these bots are not licensed professionals.
Parental controls on the way
Character.AI is also working on adding parental control features, which are set to launch in the first quarter of next year. These controls will allow parents to monitor how much time their child spends on the platform and which bots they interact with most often. The company has collaborated with online safety experts, including the non-profit organisation ConnectSafely, to ensure these tools offer adequate protection for young users.
Founded by former Google employees, Character.AI allows users to interact with various AI-powered chatbots. These bots, which can simulate everything from life coaches to fictional characters, have become particularly popular among teenagers. The platform allows anyone aged 13 or over to create an account and chat with these bots.
However, lawsuits filed against Character.AI claim that some younger users have developed unhealthy attachments to the bots, particularly when conversations veer into topics like self-harm, suicide, and sexual content. Critics argue that the platform failed to direct these users to appropriate mental health resources during such conversations.
Character.AI has acknowledged that its safety measures must evolve alongside the technology. The company’s press release stated, “We recognise that our approach to safety must evolve alongside the technology that drives our product — creating a platform where creativity and exploration can thrive without compromising safety.” The changes are part of the company’s long-term commitment to continuously improving its policies and products.
