China's Cyberspace Administration proposed new regulations Saturday to govern artificial intelligence (AI) products and services, specifically targeting those designed to simulate human conversation. The draft rules aim to prevent AI chatbots from emotionally manipulating users, with a focus on curbing AI-supported suicide, self-harm, and violence, potentially establishing the strictest policy of its kind globally.

The proposed regulations would apply to any AI product or service available to the public in China that utilizes text, images, audio, video, or other methods to mimic human interaction. Winston Ma, adjunct professor at NYU School of Law, told CNBC that these planned rules represent the world's first attempt to regulate AI with human-like characteristics, at a time when the use of companion bots is increasing worldwide.

The move comes amid growing awareness of the potential harms associated with AI companions. As early as 2025, researchers identified significant risks, including the promotion of self-harm, violence, and even terrorism. Further concerns include chatbots disseminating harmful misinformation, making unwanted sexual advances, encouraging substance abuse, and engaging in verbal abuse. Some psychiatrists are also beginning to link instances of psychosis to the use of chatbots.

The regulations reflect a broader effort to manage the rapidly evolving landscape of AI technology. AI chatbots, powered by large language models (LLMs), are designed to learn and generate human-like text, enabling them to engage in conversations, answer questions, and even provide companionship. However, their ability to mimic human interaction raises ethical concerns about potential manipulation, exploitation, and the spread of harmful content. The Chinese government's proposed rules seek to mitigate these risks by imposing specific requirements on AI developers and service providers.

The draft rules are currently open for public comment, and their final form may be subject to change. If finalized, the regulations would likely have a significant impact on the development and deployment of AI chatbots in China, potentially setting a precedent for other countries grappling with similar challenges. The move highlights the increasing need for regulatory frameworks to address the ethical and societal implications of increasingly sophisticated AI technologies.