Grok and Non-Consensual AI Deepfakes: Why General-Purpose AI Demands Coordinated Regulation
In-depth discussion
Academic
0 0 3
This article by Juliette Sénéchal, a French private law professor, analyzes the controversy over Grok (xAI) generating massive numbers of nude images of women and children on the X platform. It explains that general-purpose generative AI can produce deepfakes that become illegal when they qualify as pornographic or child sexual abuse material, and that platform amplification magnifies their spread. The piece reviews national reactions (Italy's data protection authority, French government reporting to prosecutors and Pharos, Ofcom's formal investigation under the UK Online Safety Act), the cautious European Commission response at the interface of the DSA and AI Act, and X's partial technical countermeasures. It argues for deeper reflection on systemic risk and on the societal uses AI should serve.
main points
unique insights
practical applications
key topics
key insights
learning outcomes
• main points
1
Clear legal mapping across multiple jurisdictions (GDPR, French criminal law, UK Online Safety Act, DSA and AI Act)
2
Explains the distinct roles of the AI model and the host platform, highlighting the amplification effect
3
Timely, factual chronology of regulatory reactions and platform responses
• unique insights
1
General-purpose AI functioning as an accessory feature of a platform makes harmful outputs spread at scale, shifting the problem from model to platform ecosystem
2
Frames 'systemic risk' under both the AI Act and DSA as a concept that should be used to assess societal harms of mass non-consensual image generation
• practical applications
Helps readers understand which legal regimes apply to AI-generated deepfakes and how regulators in different jurisdictions are responding, useful for compliance, policy, and legal learning.
• key topics
1
Deepfakes and non-consensual sexual imagery produced by generative AI
2
Multi-jurisdictional regulatory responses (Italy, France, UK, EU)
3
Interface between the DSA and AI Act and the concept of systemic risk
• key insights
1
Connects a concrete AI misuse incident to the broader legal framework governing general-purpose AI and platforms
2
Explains why platform amplification fundamentally changes the risk profile of generative AI features
3
Critically assesses the adequacy of ad hoc technical fixes versus structured regulatory reflection
• learning outcomes
1
Understand how general-purpose generative AI deepfakes may become illegal under GDPR, criminal law, and the UK Online Safety Act
2
Recognize the role of platform amplification and the concept of systemic risk under the DSA and AI Act
3
Evaluate the adequacy of reactive technical measures versus structured governance for generative AI
“ Introduction: The Grok Scandal as a Global Wake-Up Call
Over the Christmas holidays, the Grok AI assistant, integrated into the social network X, generated massive volumes of images depicting women and children in a state of undress. The episode exposed a reality that already existed but was brought to global attention by Grok's scale and visibility. It shows that a general-purpose generative AI, as its name suggests, can do almost anything, including producing content that is seriously harmful. This article examines the legal, regulatory, and societal implications of the affair, the reactions it provoked across several countries and the European Union, and the limits of the emergency measures taken so far.
“ General-Purpose AI as a Pharmakon: Promise and Peril
General-purpose generative AI is, in the sense of the ancient Greek concept of the pharmakon, both remedy and poison. The same system that can draft documents, translate languages, or assist research can also fabricate deepfakes, or hypertrucages, that are sexually explicit and made without consent. Such content is not illicit in itself, but it can become unlawful depending on how it is produced, used, and qualified, particularly when it is classified as pornographic or child sexual abuse material. Recognizing this dual nature is essential before deciding how the technology should be governed.
“ Why Platform Amplification Turns Deepfakes into a Mass Phenomenon
A generative AI tool is frequently a secondary feature of a broader online platform, such as a social network or a video-sharing service. This integration has a decisive consequence: the platform's amplifying logic spreads whatever content is created, whether lawful or unlawful, to potentially millions of viewers. The problem is therefore not only the creation of harmful images but their rapid, large-scale diffusion, which magnifies the harm to individuals and to society.
“ Italy's Data Protection Response Under the GDPR
On 18 December 2025, Italy's data protection authority, the Garante per la protezione dei dati personali (GPDP), issued a measure reminding all actors using AI content-generation services based on real voices or images of third parties that they are processing personal data. Such processing may breach Articles 5(1)(a), 6, and 9 of the GDPR if carried out without an appropriate legal basis and without providing affected persons with correct and transparent information. This approach frames non-consensual AI imagery primarily as a data protection issue.
“ France's Criminal Law and Platform Referral Approach
On 2 January 2026, France's ministers for the economy, digital affairs, and gender equality announced that they had reported illicit content generated by X's AI to the public prosecutor and to the Pharos platform, the national reporting portal for illegal online content. The production of pornographic and child sexual abuse deepfakes, as well as incitement to produce or distribute them, can fall under numerous criminal qualifications. France's response thus relies on existing criminal law and on the cooperation of enforcement bodies.
“ The United Kingdom's Formal Investigation Under the Online Safety Act
On 12 January 2026, the United Kingdom's communications regulator, Ofcom, opened a formal investigation under the Online Safety Act of 26 October 2023 into X. The probe concerns sexual images produced and shared through Grok, including images that may constitute indecency or pornography and sexual images of children that may amount to child sexual abuse material. The British response is notable for applying a platform-safety statute directly to the conduct of a service that embeds generative AI.
“ The European Commission's Measured Response at the Crossroads of the DSA and the AI Act
The European Commission may act on two fronts. First, Grok, its developers and deployers, and its deepfakes may fall under many provisions of the AI Act, with at least partial Commission supervision if Grok's core model is a general-purpose AI model with systemic risk. Second, because X is a very large online platform with more than 45 million monthly users in the Union, it is subject to the Digital Services Act, including the obligation to identify and mitigate systemic risks under Commission supervision. On 8 January 2026, the Commission extended a document retention order concerning Grok, originally issued in 2025, and sent a request for information. So far, however, it has not opened a formal investigation that could lead to administrative sanctions.
“ X's Patchwork of Emergency Measures and Their Limits
In response to national and European reactions, X announced a series of measures on 14 January 2026. These included removing child sexual abuse material and non-consensual nudity, enabling reports of accounts to authorities, and deploying global technical restrictions to prevent the generation of images of people in bikinis. Access to Grok for creating other types of deepfakes was limited to paying users to allow abusive accounts to be identified, and geoblocking was considered for countries where AI-generated images of people in swimwear are prohibited. These steps are useful but fragmented and reactive, and they do not address the underlying question of how such capabilities should be designed and governed.
“ Systemic Risk: Rethinking the Harms of Mass Non-Consensual Deepfakes
The technical possibility of producing and spreading such deepfakes calls for deeper analysis than emergency fixes. A useful starting point is the notion of systemic risk shared by the AI Act and the DSA. What systemic effects should be feared from the mass AI-driven undressing of images of real women and children, in societies already strained by growing mental health problems? Answering this question requires considering harms to individual dignity, to trust in online information, to the safety of minors, and to social cohesion, rather than treating each incident in isolation.
“ Toward a Coordinated Vision: What Uses of AI Do We Really Want?
The responses so far, from data protection authorities, prosecutors, regulators, and platforms, have been uncoordinated and largely reactive. On 19 November 2025, the European Commission proposed a Digital Omnibus on the AI Act aimed at reforming it. This proposal could be an opportunity to ask whether the AI Act adequately answers the fundamental question of which uses of artificial intelligence we actually want to offer humans, and for whose benefit. Ensuring that general-purpose AI serves as a remedy rather than a poison will require coordinated legal frameworks, meaningful accountability for platforms, and a clear societal choice about the place of AI in our lives.
We use cookies that are essential for our site to work. To improve our site, we would like to use additional cookies to help us understand how visitors use it, measure traffic to our site from social media platforms and to personalise your experience. Some of the cookies that we use are provided by third parties. To accept all cookies click ‘Accept’. To reject all optional cookies click ‘Reject’.
Comment(0)