AI Regulation In China: Proposed Measures to Prevent Emotional Manipulation By Algorithms

China has introduced a new regulatory proposal aimed at artificial intelligence services that simulate human personality and interaction. The initiative, announced by the Cyberspace Administration of China (CAC), seeks to prevent emotional manipulation by AI systems that interact with users through text, images, audio, and video. A public consultation period of one month has been opened to gather feedback on the proposal.

The draft regulation targets AI products and services designed to engage users on an emotional level, particularly conversational systems capable of maintaining continuous interactions and forming perceived emotional bonds. According to the authorities, these capabilities raise concerns about the potential for harmful influence on users’ mental and emotional well-being.

Among the practices the proposal aims to curb are the encouragement of suicide or self-harm, the promotion of gambling-related content, and other forms of emotional manipulation that could negatively affect mental health. Regulators argue that conversational AI systems, if left unchecked, may exert undue psychological influence, especially on vulnerable individuals.

The proposal outlines several specific obligations for AI service providers. Chatbots would be prohibited from encouraging suicide, self-harm, verbal abuse, or emotionally manipulative behavior. If a user shows signs of suicidal intent, companies would be required to transfer the interaction to a human operator and immediately contact a guardian or a person close to the user.

Additional restrictions include bans on generating gambling-related, obscene, or violent content. For minors, the use of AI as an emotional companion would require authorization from a legal guardian, along with enforced access limits. Platforms must also be capable of identifying underage users, even if age is not explicitly disclosed, and apply child-safe settings by default in cases of uncertainty.

The proposal also introduces usage reminders after two consecutive hours of interaction and mandates security assessments for chatbots with more than one million registered users or over 100,000 monthly active users. By emphasizing emotional and psychological risks, the initiative highlights China’s growing focus on the societal impact of advanced AI systems.

Filed in General >Robots. Read more about AI (Artificial Intelligence) and China.