Age verification and police reporting obligations on AI will come with a cost
The summer break has given us more time to assess the flurry of bills tabled in Parliament in the final weeks of June relating to privacy, AI, and the internet. One of them, the Safe Social Media Act, in Bill C-34 , has been much in the news in recent weeks, with questions about how the ban on children under 16 holding social media accounts will work and whether it’s constitutional.
I’ve weighed in on whether a ban would infringe young people’s freedom of expression under the Charter, and how the bill’s various obligations on AI providers might change how we interact with chatbots.
Here I address another question people have been asking: how will Bill C-34 affect our privacy? Can age verification be made effective without forcing all of us to divulge and put at risk sensitive information? And would obligations on AI providers to notify police about certain chats make AI less private?
In both cases, we can look to other countries where similar measures have been imposed for evidence of how they affect privacy. We don’t have to speculate.
In short, we can’t have effective age verification or police reporting in AI without harming rights to privacy and expression. The promise of safer social media and AI in C-34 will either have to be broken or bought at too high a cost.
Why age verification can only come at a serious cost to privacy
Bill C-34 requires social media and pornography sites to verify the age of their users holding accounts in the one case and accessing content in the other. Both have to implement measures that are (a) “effective” in ensuring persons under 18 for pornography and under 16 for social media do not access content or hold accounts (respectively), but also
(b) do not involve the collection or use of personal information other than for age-verification or age-estimation purposes;
(c) provide for the destruction of personal information collected for age-verification or age-estimation purposes once the verification or estimation is completed;
(d) provide for the protection of personal information that is collected for age-verification or age-estimation purposes until that information is destroyed. [These are found in sections 22 and 27 of the bill]
On a straight reading of this, we could have effective measures that limit the impact on privacy by meeting the conditions in (b) to (d).
But even before we look at the evidence from abroad, a closer look at these conditions reveals a clear conflict between (a) and (b)-(d).
We can’t know who is under 16 without checking everyone’s age. To be fully effective, we would have to submit to facial recognition technology and provide copies of government-issued ID. And verification would need to be continuous, since young people take various steps to get around these measures, including VPN’s, fake accounts, and fake documents.
Even if social media and porn sites hold the data only briefly, they would still have to collect and store enormous amounts of sensitive records, highly attractive to hackers.
And while these provisions require data to be destroyed once verification is complete, this can be easily defeated in practice. Operators will need appeal machinery to handle people wrongly flagged as underage, and that will mean retaining copious amounts of data for longer. This is precisely how Discord’s data breach came about.
The information collected here should also be used only for age verification, but if platforms outsource to third-party services, their terms could permit repurposing collected data in ways not caught by Canada’s regulations. Reddit, Discord, and Grindr, for example, have all used providers that reserved the right to reuse age-verification data for advertising or marketing.
Looking abroad, Australia adopted a similar “reasonable steps” approach to age verification, and paired the demand for effectiveness with a duty to destroy data and a bar on compelling government ID. Yet its own technology trial concluded that no single method was suited to every case and none was guaranteed to work. From the outset, children have evaded the ban through VPNs and age-estimation tools misreading their faces, with more than 80 percent of under-16s in Australia still on social media three months into the ban.
The UK’s experience with age verification under its Online Safety Act is instructive on data breaches, with the Discord case becoming something of a pattern. The Tea app has exposed user IDs and selfies, and researchers were able to compromise a verification vendor that Reddit and Discord had used. The ICO has fined Reddit £14 million and is using penalties to drive platforms toward using facial estimation tools and a national digital ID. But after Ofcom ,which enforces the Online Safety Act, began to enforce age verification obligations, VPN downloads climbed more than 1,400 percent, raising questions about effectiveness.
Why AI reporting obligations will compromise privacy and cast a chill on expression
Bill C-24’s second privacy question concerns its reporting duty. Section 17 of the bill requires the new Digital Safety and Data Protection Commission of Canada to issue guidelines on when an operator of a regulated service, an AI provider among them, should notify the RCMP of “content that gives rise to reasonable grounds to suspect that there is a risk that an individual will commit an act that would cause death or serious bodily harm to another individual.”
Here too we can look abroad rather than speculate as to where this might lead.
The closest analogue is Article 18 of the EU’s Digital Services Act , which is worded in a similar way. It requires a hosting provider that becomes aware of information suggesting the possibility of “a criminal offence” threatening a person’s life or safety, whether it has occurred or is likely to, to promptly inform law enforcement. (This may capture self-harm, unlike C-34, which is confined to harm to other people — though how Article 18 treats self-harm content in practice is not settled; see the note below.)
Article 18 has proved hard to apply because it doesn’t define the offences to which it applies. The European Commission ran a consultation to clarify how it operates, after industry submissions flagged that the broad wording (‘a criminal offence’) forces providers to assess criminality across many legal systems and to notify quickly, which invites over-reporting and misreporting.
Can Canada’s new Commission avoid this in guidelines to come?
Germany imposed more stringent obligations, which raised further issues. A 2021 amendment to its Network Enforcement Act required large platforms to report suspected content to the federal police, along with identifying data including the user’s name and IP address. The scheme would have led to police collecting large amounts of data on people never found to have done anything unlawful, on a provider’s suspicion alone. Google and Meta among others challenged it, a Cologne court granted them interim relief in 2022, and the duty never took effect as designed .
The United States has an older law useful to point to here. Under its PROTECT Our Children Act of 2008 (§ 2258A ), internet service providers must report apparent child sexual abuse material to a central clearinghouse, which forwards it to police. In United States v. Ackerman , the US Court of Appeals (10th Cir. 2016) held that the clearinghouse acted as a government agent for Fourth Amendment purposes. Courts have declined to treat the reporting platform the same way, because the statute requires it to report what it finds and does not compel it to search.
This might become an important distinction in Canada under section 8 of the Charter (the right to be secure against unreasonable search or seizure). Section 17 of C-34 speaks only of when an operator should notify, which is softer than an explicit command to monitor. If the guidelines mandate a more prescriptive approach and reward proactive detection, an argument might be made that AI providers act as agents of the state in collecting and turning over information, thus engaging section 8.
But one feature sets section 17 apart. The duties for child abuse material concern content that is already illegal. Section 17, like Article 18 in the EU, applies to content believed to signal a risk of future violence, a vague threshold that captures a wide swath of lawful if alarming speech.
Depending on how Canada’s guidelines are worded, a more prescriptive approach to reporting could also have a chilling effect by inducing people to self-censor when chatting with an AI service caught by the bill.
It’s too early to point to empirical evidence supporting this proposition. But it seems to be a likely consequence of reporting guidelines that encourage reporting based on a mere risk of violence — not on the basis that people will fear that specific things they say will trigger a police report, but the ever present possibility that whatever they do say can be caught and disclosed to police.
Much will turn on the guidelines and regulations still to come. But the trade-off between safety and privacy and expression seems unavoidable. ■
{To receive new posts by email, follow me on Substack .}