On 13 December 2024, the UK Information Commissioner’s Office (ICO) published the report of outcomes from its consultation on generative AI (genAI). The report sets out key themes that emerged from responses to the ICO’s five-part genAI consultation series that launched in January 2024, which covered:
- The relevant lawful basis for using web scraping to train genAI models.
- Purpose limitation in the context of genAI.
- Accuracy of genAI training data and outputs.
- Engineering General Data Protection Regulation (GDPR) rights into genAI models.
- Allocating GDPR “controller” status across the genAI supply chain.
The report does not signal any drastic changes in direction from the ICO, but does indicate progress from the office in acknowledging and responding to feedback on its regulatory guidance. Below, we analyse takeaway points from the outcomes report.
I. Organizations Are Expected To Demonstrate How Their Transparency Measures Work
Transparency is a priority for the ICO throughout the outcomes report. The ICO assesses that despite widespread genAI development, individuals do not necessarily fully understand genAI processing. The office notes that “common [industry] practice does not equate to meeting people’s reasonable expectations” for transparency.
To address this, the ICO expects genAI platforms to provide users with transparency information that is “meaningful rather than a token gesture.” This means organizations will need to test whether transparency measures are effective. For example, the ICO expects organizations to test whether individuals are, in fact, aware that the organization is using those individuals’ data to train AI models, rather than assuming that publishing a (never-to-be-read) privacy notice is enough.
II. The Standard To Demonstrate 'Legitimate Interests' Is High
As the European Data Protection Board’s (EDPB’s) recent guidance on legal bases for web scraping does, the ICO reiterates that the long-standing three-prong test for legitimate interest applies to AI development. But meeting the three-step test presents a high bar in practice:
- In relation to the “purpose” limb: Like the EDPB opinion, the ICO notes that purposes must be specifically defined — e.g., developing a specific model for a specific purpose. Poorly defined purposes such as “training AI models” are unlikely to pass muster.
- In relation to the “necessity” limb: The ICO emphasizes that companies relying on legitimate interests to undertake web scraping will need to demonstrate that other methods of data collection, such as licensing data from third parties, were not a feasible way to achieve the desired outcomes (for example, because the model trained on a small amount of third-party data would not be as accurate as a model trained on a broader set of web-scraped data). This is often difficult to demonstrate because knowing in advance how much more effective a model trained on a large data set would be than a model trained on a smaller data set is not necessarily possible.
- In relation to the “balancing” limb: The ICO requires companies to take a broad approach to assessing the harms that their training may cause and not just focus on privacy harms. As an example, the ICO notes that developing AI image generation models may cause fashion models to lose their jobs; the ICO expects organizations to balance this risk within their legitimate-interest assessment even though it is not a traditional “privacy” risk. This is a broader approach to defining “harm” than most organizations’ legitimate-interest assessments have taken.
The ICO also expects companies to calculate and monitor the alleged benefits flowing from AI models as part of the “balancing” assessment. For example, if an organization claims that a new AI model will improve users’ experience, the organization should conduct surveys or other tracking to prove that users’ experience did actually improve.
III. The GenAI State of the Art Remains a Work in Progress
The ICO emphasizes repeatedly that the genAI technology landscape is advancing quickly, and the ICO is therefore open to engaging with industry to understand how the state of the art is evolving. For example, similar to the EDPB’s recent opinion on AI model training, the ICO refers to the concept of “machine unlearning” to remove data from AI models, but unlike the EDPB, the ICO acknowledges that “machine unlearning” — while theoretically attractive from a data protection perspective — has not (yet) proven to be technologically feasible in practice. The ICO further acknowledges that many mooted AI safeguards remain theoretical concepts.
IV. The ICO Expects GenAI Developers To Embed Data Subject Rights Into Their Models (Without Providing a Mechanism To Do So)
The report states the ICO is “increasingly concerned” that genAI developers and deployers do not provide sufficient mechanisms to respond to data subject rights. Interestingly, and analogously to the EDPB’s recent opinion on AI models, the ICO highlights that output filters “may not be sufficient” to implement data deletion requests because the filters do not actually remove the data from the model. This is a significant stance, as there are currently few realistic mechanisms other than output filters to implement data deletion requests.
The ICO’s position (which aligns with other regulators’ positions on this topic) leaves AI developers and deployers in the difficult position of being told that existing compliance approaches are insufficient, without being given any alternative.
More ICO Guidance To Come
The ICO references the development of further guidance on a range of topics, including purposes limitation and data accuracy, along with a rewrite of its 2020 guidance on AI and data protection (though the ICO says it will wait to issue such guidance until the UK Parliament approves the Data Use and Access Bill).
What To Do Now
The ICO’s genAI consultation positions and its outcomes report suggest that the ICO has high standards for organisations’ protection of individuals’ data and for the documentation requirements for businesses to demonstrate they have achieved this goal. Organizations developing or deploying genAI should therefore review their existing documentation, such as legitimate-interests assessments, transparency notices and data subject rights flows, against the ICO’s guidance, and continue to do so as further guidance emerges.
The ICO’s report also signals a willingness to engage with, and listen to, industry regarding genAI issues — a willingness often not shared by other GDPR regulators. Organizations, particularly those operating at the forefront of genAI development or deployment, may want to consider proactive engagement with the ICO regarding new genAI technologies in order to help influence the ICO’s regulatory posture.
This memorandum is provided by Skadden, Arps, Slate, Meagher & Flom LLP and its affiliates for educational and informational purposes only and is not intended and should not be construed as legal advice. This memorandum is considered advertising under applicable state laws.