It will regulate the content of our conversations, but also the bonds we forge
Among the many dangers we often hear about in coverage of AI, some of the most concerning involve cases where a chatbot encourages a person to commit suicide or aids in the commission of a violent offence. Other, less extreme cases of harm involve people developing unhealthy psychological relationships with chatbots, relying on them for erroneous medical or legal advice.
Bill C-34, Canada’s new Safe Social Media Act , tabled last week, imposes a new set of duties on the AI companies behind ChatGPT, Claude, and Gemini that may push them to do more to prevent these various harms. While most of the coverage of the bill has so far focused on the social media ban for those under 16, another important aspect is that it represents Canada’s first attempt to regulate the behaviour of conversational AI systems themselves.
The bill will seek to minimize harmful outputs, but more broadly, it will regulate the kinds of relationships these systems are allowed to form with users.
C-34 imposes a general duty on AI operators to “act responsibly” by complying with more specific duties in the act and with orders made by the Digital Safety Commission, a new body the bill will create. Much of the detail is left for future regulations.
One of these duties is to implement measures that are “adequate to mitigate the risk that the service will communicate harmful content to a user of the service.” ‘Harmful content’ is a defined term that includes non-consensual intimate images, content that induces a child to harm themselves, foments hatred, or incites violence. Regulations can set out additional measures an operator must take. The bill treats AI here much like social media platforms, but with an obligation to ensure the content doesn’t appear, rather than removing it once it does.
A further duty arises in cases where a user expresses suicidal thoughts, an intention to harm themselves, or to inflict “death or serious bodily harm.” The chatbot must “immediately interrupt its interaction with the user” and direct them to “crisis intervention services that are appropriate to the situation” and “available at the moment when the user is directed towards them.” The operator must also “permit the user to interact with a human being.”
It’s not clear what that last point would entail — a hotline number or a button popping up that would open a live audio or video link with staff at OpenAI or Anthropic, or with dedicated mental health professionals hired for this purpose?
Regardless of the practical specifics, the duty to immediately terminate the conversation and connect a person with “appropriate” and “available” services might help avoid the pattern seen in several reported cases, where lengthy chats culminated in encouragement to self-harm.
Chatbot operators must also implement measures to curb three other kinds of harmful behaviour.
One pertains to the risk arising from “posing as a human being in a manner likely to lead a user of the service to mistake it for a human being or otherwise being deceptive about being an artificial intelligence system.” It’s hard to imagine what this would entail aside from frequent reminders that “I am only a chatbot.” Even so, messages to this effect wouldn’t dispel the illusion of communicating with a human that we often have with AI.
Chatbots should also avoid posing as “a medical, legal or other licensed professional and giving advice based on that deception that could reasonably be expected to be relied on by a user of the service.” This one is harder still to imagine. Even with the usual disclaimers, all the big frontier models answer our legal and medical queries with such confidence that it seems implausible they won’t still be crossing this line much of the time.
A further and welcome provision requires measures to avoid the use of “manipulative engagement techniques to encourage a user of the service to form or maintain an emotional attachment to the service in a way that may encourage the user to withdraw socially or disconnect from reality.” This might include being less sycophantic, but it doesn’t preclude techniques to get you to stay on the platform longer. The concern here is emotional attachment and social withdrawal, rather than extended or excessive use.
An AI operator must also mitigate the risk of a chatbot encouraging self-harm, suicide, serious bodily harm or death, by ensuring it thwarts any attempt to steer conversation in this direction. Can this be done? Are even the most powerful chatbots not unavoidably susceptible to being jailbroken or having their safeguards circumvented by clever prompting?
Maybe so, but the bill signals an expectation that AI providers will have a duty to make this difficult. The onus will be on the operator to show that it wasn’t easy to get its chatbot to generate harmful content or be steered in this direction. And the failure to meet this duty will mean a failure to act responsibly, which carries significant monetary penalties under the act.
The act also imposes obligations on AI companies to publicly disclose the measures they’re taking to meet their duties under the act, and come up with ways for users to flag harmful content, or report failures of the system to address harm.
The focus here, then, is not just on content but on the relationship between users and AI platforms, on how AI should behave when dealing with users, with an emphasis on the avoidance of emotional dependency and manipulation.
How AI companies will meet these duties is not clear, and many of the measures may be subtle or even hard to notice. But they may prove to be as important as the parts of the bill targeting social media.
Given how much time we now spend relating to chatbots, the aim of regulating the relationships they can form with us, and not just the content they generate, is a laudable one. ■
{To receive new posts by email, follow me on Substack .}