---
id: SAFE-PRIV-001
title: "Privacy and Data Protection in AI"
alternative_names:
  - "AI and Data Protection Law"
  - "Privacy-Preserving Machine Learning"
  - "GDPR and AI"
category: "Safety & Governance → Ethics and Law → Privacy"
volume: VIII
chapter: 47
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-18
version: "0.2.0"
prerequisites:
  - "SAFE-ETHICS-001"
related_entries:
  - "SAFE-SEC-001"
  - "SAFE-IP-001"
  - "GOV-LAW-001"
  - "GOV-EU-001"
  - "TECH-DATA-001"
  - "TECH-LLM-001"
---

# Privacy and Data Protection in AI

> **Scope and legal notice.** This is an encyclopedia entry, not legal advice. Legal statements are dated to 2026-10-18 and are limited to the jurisdictions named. The primary focus is the European Union's General Data Protection Regulation (GDPR); other jurisdictions are outside this draft's scope. Laws and enforcement positions change, and several matters below are still being litigated.

## 1. Concise Definition
**Privacy in AI** concerns how AI systems collect, learn from, retain and reveal information about individuals. **Data protection law** is the legal regime governing the processing of personal data; in the EU, the GDPR defines *personal data* as "any information relating to an identified or identifiable natural person" and *processing* as "any operation or set of operations which is performed on personal data" (Regulation (EU) 2016/679, Art. 4(1)-(2)).

## 2. Overview & Significance
Machine-learning systems interact with personal data at several stages: collection and curation of training sets, training (which may encode information in model parameters), deployment (inputs, logs, outputs) and downstream decisions about people. Two technical facts drive much of the debate. First, models can memorise and reveal training data (Carlini et al., 2021; Nasr et al., 2023). Second, a trained model can leak information even when the training data are never released (Shokri et al., 2016; Fredrikson et al., 2015). These facts raise the legal question of when a model itself contains personal data, which authorities and courts are only beginning to address.

Related topics are treated elsewhere: attacks that exploit leakage in [SAFE-SEC-001], copyright in training data in [SAFE-IP-001], bias and fairness in [SAFE-ETHICS-001], and EU AI-specific regulation in [GOV-EU-001].

## 3. Historical Development
Dwork et al. (2006) introduced differential privacy as a formal definition of privacy for data analysis; Dwork and Roth (2014) consolidated the theory. Fredrikson et al. (2015) showed model-inversion attacks that exploit confidence values, and Shokri et al. (2016) formalised membership inference against black-box models. McMahan et al. (2017; arXiv 2016) proposed federated learning, and Abadi et al. (2016) showed how to train deep networks with differential privacy. Bourtoule et al. (arXiv 2019; published 2021) introduced SISA training to make "machine unlearning" cheaper. With large language models, Carlini et al. (2021) extracted training sequences from GPT-2, Carlini et al. (2022) quantified how memorisation scales, and Lee et al. (2021) showed that deduplicating training data reduces verbatim emission.

On the legal side, the GDPR has applied since 25 May 2018 (Art. 99). The Italian authority's decision on OpenAI (order No. 755 of 2 November 2024, announced 20 December 2024) and its annulment by the Court of Rome in 2026 are discussed in Section 6. The European Data Protection Board (EDPB) adopted Opinion 28/2024 on AI models and personal data in December 2024; the Opinion's cover states "Adopted on 17 December 2024", while the EDPB document page lists 18 December 2024 (EDPB, 2024).

## 4. Technical / Conceptual Explanation

### 4.1 How models leak information
- **Memorisation and extraction.** Carlini et al. (2021) recovered hundreds of verbatim sequences from GPT-2, including (public) names, phone numbers and email addresses, even for sequences that appeared in only one training document, and found larger models more vulnerable. Nasr et al. (2023) report extracting gigabytes of training data from open-source, semi-open and closed models such as ChatGPT; for the aligned ChatGPT they developed a divergence attack, and they conclude that current alignment techniques do not eliminate memorisation. Carlini et al. (2022) describe three log-linear relationships: memorisation grows significantly with model capacity, example duplication and prompt context length; they add that generalising across model families is more complicated. Lee et al. (2021) report that over 1% of unprompted output of models trained on the datasets they studied was copied verbatim from training data, and that deduplication allowed them to train models that emit memorised text ten times less frequently.
- **Membership inference.** Given a record and black-box access to a model, decide whether the record was in the training set (Shokri et al., 2016).
- **Model inversion.** Reconstruct input features from model outputs (Fredrikson et al., 2015).

### 4.2 Differential privacy
A randomised algorithm $\mathcal M$ is $(\varepsilon,\delta)$-differentially private if for all datasets $D, D'$ differing in one record and all output sets $S$,
$$\Pr[\mathcal M(D)\in S]\ \le\ e^{\varepsilon}\,\Pr[\mathcal M(D')\in S]+\delta.$$
Smaller $\varepsilon$ means stronger protection and, generally, lower accuracy. Abadi et al. (2016) report training deep neural networks with non-convex objectives under differential privacy "under a modest privacy budget, and at a manageable cost" in software complexity, training efficiency and model quality (abstract). Differential privacy bounds what can be inferred about any single record; it does not prevent all harms from the model's use.

### 4.3 Federated learning and unlearning
Federated learning trains a shared model from decentralised data without centralising raw data (McMahan et al., 2017); it reduces data movement but does not by itself guarantee that updates reveal nothing. Machine unlearning aims to remove the influence of a data point from a trained model. Bourtoule et al. (2019) state that having models unlearn is "notoriously difficult" and propose SISA training to limit each point's influence; they report speed-ups over retraining from scratch of 4.63x (Purchase) and 2.45x (SVHN) for simple tasks and 1.36x for ImageNet. Whether approximate unlearning of a deployed large language model satisfies a legal erasure request is unsettled: no court ruling on the point was found in the sources reviewed. EDPB Opinion 28/2024 (para. 102(d), 107(b)) lists accepting regurgitation or memorisation claims, assessing "relevant unlearning techniques", and "post-training techniques that attempt to remove or suppress personal data" as possible mitigating measures to be assessed case by case; it does not hold that unlearning satisfies Art. 17. No regulator decision treating unlearning as Art. 17 compliance was located (searched narrowly; absence is not proof).

## 5. Types, Components & Classifications
| Risk or measure | Description | Source |
| :--- | :--- | :--- |
| Training-data extraction | Verbatim output of training examples | Carlini et al. (2021); Nasr et al. (2023) |
| Membership inference | Detect whether a record was used in training | Shokri et al. (2016) |
| Model inversion | Recover sensitive attributes from outputs | Fredrikson et al. (2015) |
| Deduplication | Reduce memorisation by removing repeats | Lee et al. (2021) |
| Differential privacy | Formal bound on per-record influence | Dwork & Roth (2014); Abadi et al. (2016) |
| Federated learning | Decentralised training | McMahan et al. (2017) |
| Machine unlearning | Remove influence of specific data | Bourtoule et al. (2019) |

## 6. Applications & Concrete Examples
**EU legal framework (as of 2026-10-18; jurisdiction: EU/EEA).**
- *Automated decisions.* GDPR Art. 22(1) gives data subjects the right not to be subject to "a decision based solely on automated processing, including profiling" that produces legal effects or similarly significantly affects them, subject to exceptions in Art. 22(2) (contract, authorising law with safeguards, explicit consent). In *SCHUFA Holding (Scoring)*, Case C-634/21, judgment of 7 December 2023, the Court of Justice of the EU (First Chamber) ruled that Art. 22(1) "must be interpreted as meaning that the automated establishment, by a credit information agency, of a probability value based on personal data relating to a person and concerning his or her ability to meet payment commitments in the future constitutes 'automated individual decision-making' within the meaning of that provision, where a third party, to which that probability value is transmitted, draws strongly on that probability value to establish, implement or terminate a contractual relationship with that person" (operative part, read on EUR-Lex).
- *EDPB Opinion 28/2024* (cover: "Adopted on 17 December 2024"; requested by the Irish supervisory authority on 4 September 2024 under Art. 64(2) GDPR) addresses AI models. From the Opinion text: whether a model is anonymous is for supervisory authorities to assess case by case, because models trained with personal data "cannot, in all cases, be considered anonymous"; for anonymity both the likelihood of direct (including probabilistic) extraction of personal data of training-data subjects and the likelihood of obtaining such data from queries "should be insignificant", taking into account "all the means reasonably likely to be used". The Opinion recalls that legitimate interest (Art. 6(1)(f)) is assessed by a three-step test (legitimate interest, necessity, balancing against data subjects' interests and rights), with attention to data subjects' reasonable expectations. For unlawfully processed development data it distinguishes three scenarios: (1) personal data retained in the model and processed by the same controller, where the effect on later processing is assessed case by case; (2) retained and processed by another controller, who should assess whether the model was developed lawfully as part of its accountability duties; (3) the model is anonymised before deployment, in which case the GDPR would not apply to the model's subsequent operation if it is demonstrated that it entails no processing of personal data. An opinion of the Board is not itself binding law; it guides national authorities.
- *Enforcement on a generative AI provider.* On 20 December 2024 the Italian data protection authority (Garante) announced a EUR 15 million fine on OpenAI (order No. 755 of 2 November 2024), citing failure to notify a March 2023 data breach, processing of user data for training without an adequate legal basis, breach of transparency duties and lack of age-verification mechanisms, and ordering a six-month institutional communication campaign. The Garante's own page states the decision was later temporarily removed from its website after the Court of Rome, in judgment No. 4153/2026 published 18 March 2026, upheld OpenAI's challenge (Garante, 2024). The judgment text (R.G. 4785/2025, decided 18 March 2026) shows that the court annulled the order on its first ground: OpenAI Ireland Ltd, recognised as OpenAI's single EU establishment from 15 February 2024, brought the one-stop-shop mechanism (Arts. 60-61 GDPR) into play while the proceeding was still open, so the Garante lacked competence to adopt the November 2024 order; the remaining grounds were treated as absorbed and the court did not rule on the alleged GDPR violations (Tribunale di Roma, 2026). Whether the Garante appealed was not checked.
- *AI-specific law.* The AI Act (Regulation (EU) 2024/1689) operates alongside the GDPR. Its application dates were amended in 2026 (see [GOV-EU-001]); this article does not restate them.

**Technical practice.** Deduplication, differential privacy and access controls are research and engineering measures; which providers use which measures is generally not publicly documented and is not asserted here.

## 7. Advantages, Limitations & Unresolved Questions
- **Is a model personal data?** The EDPB position is case by case (EDPB, 2024). Whether parameters that can in some cases emit personal data are themselves personal data is not resolved by a court ruling in the sources reviewed (the CJEU has not been found to have ruled on the point; searched narrowly).
- **Erasure and unlearning.** Exact removal from a trained model generally requires retraining; approximate unlearning methods lack agreed verification standards (Bourtoule et al., 2019). Legal sufficiency is unsettled.
- **Legal basis for web-scale training.** Legitimate interest has been recognised by the EDPB as potentially available but requires a demonstrated necessity and balancing (EDPB, 2024). The Garante's 2024 order found no adequate legal basis for OpenAI's training; that order was annulled on competence grounds without a ruling on the merits (Tribunale di Roma, 2026). Other national authorities' conclusions were not surveyed here.
- **Cross-border competence.** The Rome judgment illustrates that which authority has competence (one-stop-shop) can be decisive, independent of the merits; the court held that creation of a single EU establishment during a pending proceeding moved competence to the lead authority.
- **Privacy versus fairness.** Auditing for bias may require sensitive attributes (see [SAFE-ETHICS-001]).
- **Accuracy cost of formal privacy.** Differential privacy reduces utility as $\varepsilon$ shrinks (Abadi et al., 2016).
- **Reform.** The Commission's Digital Omnibus proposal amending Regulation (EU) 2016/679 and other acts (interinstitutional file 2025/0360 (COD), transmitted 20 November 2025) was still in the legislative process: a Council Secretariat note of 22 June 2026 (doc. 10729/26) sought a mandate for negotiations with the European Parliament. Secondary reports from September-October 2026 say neither Parliament nor Council had yet fixed a final position, and no adoption was found. [VERIFICATION NEEDED: status at 2026-10-18 against the Legislative Observatory and Council press releases; only secondary reports after 22 June 2026.]

## 8. Current State of Knowledge
As of 2026-10-18:
- That language models can memorise and emit training data, including personal information, is **Well Supported** (Carlini et al., 2021; Nasr et al., 2023; Carlini et al., 2022).
- That differential privacy provides a formal bound on per-record leakage is **Established**; its practical cost for large models is **Preliminary**.
- GDPR Art. 22 is in force; the CJEU held in SCHUFA (C-634/21) that a credit-agency score can itself be an automated decision where a third party draws strongly on it; the extent of its application to other AI outputs is not settled by that ruling.
- Whether models are personal data, whether unlearning satisfies erasure, and the lawful basis for large-scale scraping are **Contested** or unresolved.
- The one court outcome reviewed here (Garante v. OpenAI) annulled the fine on competence grounds, leaving the merits undecided (Tribunale di Roma, 2026).

## 9. Future Trajectory & Strategic Outlook
This section contains no registered forecasts. Observed open questions:
- **Near-term (0-3 years).** Pending court and regulator decisions on model status, legal basis and one-stop-shop competence.
- **Medium-term (3-10 years).** Technical standards for verifying anonymity or unlearning could shape what regulators accept; none is established.
- **Long-term (10+ years).** Persistent personal agents with long-term memory raise new data-protection questions (see [TECH-LLM-001]).

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[SAFE-ETHICS-001] AI Ethics, Bias and Fairness](./SAFE-ETHICS-001_ai_ethics_bias_and_fairness.md): The ethical framing of privacy among AI principles.
- **Related Methods & Technologies**:
  - [[SAFE-SEC-001] AI Security](./SAFE-SEC-001_ai_security_prompt_injection_poisoning_extraction.md): Extraction and inference attacks.
  - [[TECH-DATA-001] Training Data, Contamination and Provenance](../vol03_technical/TECH-DATA-001_training_data_contamination_and_provenance.md): Data provenance.
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): Memorisation behaviour.
  - [[SAFE-IP-001] Copyright and Intellectual Property](./SAFE-IP-001_copyright_and_intellectual_property.md): Parallel legal questions about training data.
- **Advanced Implications & Future Trajectories**:
  - [[GOV-LAW-001] Global AI Governance & Regulation](./GOV-LAW-001_ai_governance.md): Overview of regulation.
  - [[GOV-EU-001] The EU AI Act](./GOV-EU-001_the_eu_ai_act.md): AI-specific EU law and application dates.

## 11. References & Foundational Literature
1. European Union. (2016). *Regulation (EU) 2016/679 (General Data Protection Regulation)*. OJ L 119, 4.5.2016. https://eur-lex.europa.eu/eli/reg/2016/679/oj. [Arts. 4(1), 4(2) and 22 quoted; wording and OJ L 119, 4.5.2016 reference re-read in the EUR-Lex Official Journal text on 2026-10-18; Art. 99 (application from 25 May 2018) confirmed.]
2. Court of Justice of the European Union. (2023). *SCHUFA Holding (Scoring)*, Case C-634/21, judgment of 7 December 2023. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62021CJ0634. [Operative part read on EUR-Lex: credit-agency probability value as Art. 22(1) automated individual decision-making where a third party draws strongly on it.]
3. European Data Protection Board. (2024). *Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models*, adopted 18 December 2024. https://www.edpb.europa.eu/our-work-tools/our-documents/opinion-board-art-64/opinion-282024-certain-data-protection-aspects_en. [Full PDF text (edpb_opinion_202428_ai-models_en.pdf) read on 2026-10-18: cover "Adopted on 17 December 2024"; EDPB document page lists 18 December 2024; executive summary, paras 43 and 102-107 checked.]
4. Garante per la protezione dei dati personali. (2024, 20 December). *Press release on closing the investigation into OpenAI's ChatGPT* (EUR 15 million fine), with 2026 note on Court of Rome judgment No. 4153/2026. https://www.garanteprivacy.it/home/docweb/-/docweb-display/print/10085432. [Primary source (read 2026-10-18) for the fine, grounds, six-month campaign, order No. 755 of 2 November 2024 and the court note.]
5. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., & Raffel, C. (2021). *Extracting Training Data from Large Language Models*. arXiv:2012.07805. [Verbatim extraction from GPT-2; 12 authors confirmed on arXiv API.]
6. Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., et al. (2023). *Scalable Extraction of Training Data from (Production) Language Models*. arXiv:2311.17035. [Divergence attack; alignment does not eliminate memorisation; 10 authors.]
7. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022). *Quantifying Memorization Across Neural Language Models*. arXiv:2202.07646. [Log-linear memorisation relationships.]
8. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021). *Deduplicating Training Data Makes Language Models Better*. arXiv:2107.06499. [Deduplication reduces memorised output tenfold in their setting.]
9. Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2016). *Membership Inference Attacks against Machine Learning Models*. arXiv:1610.05820. [Membership inference formulation.]
10. Fredrikson, M., Jha, S., & Ristenpart, T. (2015). *Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures*. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, 1322-1333. https://doi.org/10.1145/2810103.2813677. [Model inversion.]
11. Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006). *Calibrating Noise to Sensitivity in Private Data Analysis*. Theory of Cryptography (TCC 2006), LNCS, 265-284. https://doi.org/10.1007/11681878_14. [Introduces differential privacy.]
12. Dwork, C., & Roth, A. (2014). *The Algorithmic Foundations of Differential Privacy*. Foundations and Trends in Theoretical Computer Science, 9(3-4), 211-487. https://doi.org/10.1561/0400000042. [Consolidated theory of differential privacy.]
13. Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016). *Deep Learning with Differential Privacy*. arXiv:1607.00133. [Differentially private training of neural networks.]
14. McMahan, H. B., Moore, E., Ramage, D., Hampson, S., & Agüera y Arcas, B. (2017). *Communication-Efficient Learning of Deep Networks from Decentralized Data*. arXiv:1602.05629. [Federated learning.]
15. Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie, D., & Papernot, N. (2019). *Machine Unlearning*. arXiv:1912.03817. [SISA training for cheaper unlearning.]

16. Tribunale ordinario di Roma, Sezione diritti della persona e immigrazione. (2026). *OpenAI OpCo, LLC v. Garante per la protezione dei dati personali*, R.G. n. 4785/2025, judgment decided 18 March 2026 (cited by the Garante as No. 4153/2026). Copy of the judgment text hosted at https://dei.web.uniroma1.it/sites/default/files/allegati/2026-05/Trib_Roma_OpenAI_Garante_2026.pdf. [One-stop-shop competence ground; other grounds absorbed. The hosting site is a university department page, not the court; the Garante page confirms judgment number and date.]

Note: arXiv entries are cited as preprints. Final venues, confirmed via Crossref where a DOI exists: Abadi et al. (2016), ACM CCS 2016, pp. 308-318, https://doi.org/10.1145/2976749.2978318. Others (Carlini 2021, Shokri 2016, McMahan 2017, Bourtoule 2019) have later proceedings versions that were not verified here; arXiv IDs, titles, authors and first-posting years were checked on the arXiv API (all 8 correct).

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| LLMs memorise and can emit personal data | Well Supported | Carlini et al. (2021); Nasr et al. (2023) | High |
| Memorisation scales with capacity, duplication, context | Well Supported | Carlini et al. (2022) | Moderate-High |
| Deduplication reduces verbatim emission | Well Supported | Lee et al. (2021) | Moderate-High |
| Differential privacy gives a per-record bound | Established | Dwork et al. (2006); Dwork & Roth (2014) | High |
| GDPR Art. 22 text | Established (law in force) | Ref 1 | High |
| SCHUFA: score can be an Art. 22 decision | Established (CJEU, operative part read) | Ref 2 | High |
| EDPB position on model anonymity and legitimate interest | Established (Opinion text read) | Ref 3 | High |
| Garante fine and later annulment (competence ground) | Established (Garante page and judgment text) | Refs 4, 16 | High |
| Models are / are not personal data; unlearning satisfies erasure | Contested / unresolved | Refs 3, 15 | Low |
| Digital Omnibus GDPR proposal still pending | Well Supported (Council note of 22 June 2026; later status secondary) | Council doc. 10729/26 | Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.
- **v0.2.0** (2026-10-18): Fact-check audit (docs/audits/SAFE-PRIV-001_audit.md). Corrected EDPB adoption date (17 Dec 2024 per Opinion cover) and anonymity wording ("insignificant", not "very unlikely"); replaced secondary summaries with primary text for SCHUFA, EDPB Opinion and Court of Rome judgment (grounds now stated); added Ref 16; resolved four verification markers; Digital Omnibus status updated.