Elamur Research⌕
☰
home / papers / 2026 / Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

Paper · 2026 · Marton Szep, Jorge Marin Ruiz, Georgios Kaissis et al.

Summary

The paper examines a privacy risk that has received little attention. When an LLM is fine-tuned on sensitive records, it may memorize personal data (PII) that appears only in the inputs, even though the training targets never require it. The authors formalize this input-only memorization under a black-box threat model. They measure it with a True-Prefix Attack (TPA): the model receives the real text preceding a name and is checked for whether it completes the name. Experiments use one synthetic multilingual corpus and two private German clinical corpora, covering classification, information extraction and generation tasks. The authors vary language, PII frequency, task, prefix length and model size. They then compare four defenses: differential privacy, UnDial unlearning, a UnDial-based regularizer, and DPO. Post-training defenses (DPO, UnDial) give the most consistent privacy–utility balance. DP sometimes cuts leakage sharply but is unstable and sensitive to hyperparameters. No method removes leakage completely. This matters for anyone fine-tuning on health or other personal records before deployment.

Key points

  • Datasets:
    • GretelAI-Financial: synthetic, about 30k samples in 7 languages, 52 classification labels.
    • Pathology reports: 2,553 German documents from 2020–2023, used for 5-field JSON extraction.
    • Discharge Summaries: 26,306 German reports from 1996–2014, used for follow-up plan generation.
  • Setup:
    • Baseline is Llama 3.2 1B with QLoRA (r = 8); models up to 12B parameters are studied.
    • DP is applied through Opacus with ε between 2 and 8 and δ = 10⁻⁵.
  • Evaluation:
    • Extraction is measured with greedy decoding, with 32 samples at temperature 1, and with cross-memorization.
    • TPA is treated as an approximate lower bound compared with more elaborate attacks.
  • DPO data: Masked PII spans serve as preferred responses and the original text as rejected. This yields 1,489 training pairs for GretelAI and 5,636 for DS.
  • Effect of scale and PII type: DPO becomes notably more effective on larger models. On real data, fine-tuning can reduce extraction of structured PII such as dates, but unstructured identifiers like names remain at risk.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🏥TUM KlinikumMunich · Germany3 works · 234 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer and Daniel Rueckert. 2026. Unintended Memorization of Sensitive Information in Fine-Tuned Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.304

@inproceedings{szep2026unintended,
  title = {{Unintended Memorization of Sensitive Information in Fine-Tuned Language Models}},
  author = {Marton Szep, Jorge Marin Ruiz, Georgios Kaissis, Paulina Seidl, Rüdiger {von Eisenhart-Rothe}, Florian Hinterwimmer and Daniel Rueckert},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.304},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.304}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Representations. find
  2. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. find
  3. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium, pages 2633–2650. Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. 2024. UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models. arXiv preprin… find
  4. Cynthia Dwork. 2006. Differential Privacy. In Automata, Languages and Programming, pages 1–12, Berlin, Heidelberg. Springer. find
  5. Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2025. Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning. arXiv preprint. ArXiv:2410.07163 [cs]. — [arXiv:2410.07163](https://arxiv.org/abs/2410.07163) find
  6. Vitaly Feldman and Chiyuan Zhang. 2020. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. Advances in Neural Information Processing Systems, 33:2881–2891. find
  7. Ellen R. Girden. 1992. ANOVA: Repeated Measures. 84. SAGE. find
  8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  9. Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, and Yossi Matias. 2021. Learning and Evaluating a Differentially Private Pre-trained Language Model. In Findings of the Association for Computationa… find
  10. Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. 2022. Membership Inference Attacks on Machine Learning: A Survey. ACM Comput. Surv., 54(11s):235:1–235:37. find
  11. Bogdan Kulynych, Juan Felipe Gomez, Georgios Kaissis, Jamie Hayes, Borja Balle, Flavio du Pin Calmon, and Jean Louis Raisaro. 2025. Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy. arXiv preprint. ArXiv:2507.06969 [cs]. — [arXiv:2507.06969](https://arxiv.org/abs/2507.06969) find
  12. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large Language Models Can Be Strong Differentially Private Learners. In International Conference on Learning Representations. find
  13. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing Leakage of Personally Identifiable Information in Language Models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 346–363. ISSN: 2375-1207. find
  14. John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. 2025. How much do language models memorize? arXiv preprint. ArXiv:2505.24832 [cs]. — [arXiv:2505.24832](https://arxiv.org/abs/2505.24832) find
  15. Krishna Kanth Nakka, Ahmed Frikha, Ricardo Mendes, Xue Jiang, and Xuebing Zhou. 2024a. PII-Compass: Guiding LLM training data extraction prompts towards the target PII via grounding. In Proceedings of the Fifth Workshop on Privacy in Natural Language Processing, pages 63–73, Bangkok, Thailand. Assoc… find
  16. Krishna Kanth Nakka, Ahmed Frikha, Ricardo Mendes, Xue Jiang, and Xuebing Zhou. 2024b. PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs. arXiv preprint. ArXiv:2410.06704 [cs]. — [arXiv:2410.06704](https://arxiv.org/abs/2410.06704) find
  17. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable Extraction of Training Data from (Production) Language Models. arXiv preprint. ArXiv:2311.17035 [cs].
  18. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems, 36:53728– 53741.
  19. Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. 2024. MUSE: Machine Unlearning Six-Way Evaluation for Language Models. In International Conference on Learning Representations. find
  20. Marton Szep, Daniel Rueckert, Rüdiger von Eisenhart-Rothe, and Florian Hinterwimmer. 2024. A Practical Guide to Fine-tuning Language Models with Limited Data. arXiv preprint. ArXiv:2411.09539 [cs]. — [arXiv:2411.09539](https://arxiv.org/abs/2411.09539) find
  21. John W. Tukey. 1949. Comparing Individual Means in the Analysis of Variance. Biometrics, 5(2):99–114. Publisher: [Wiley, International Biometric Society]. find
  22. Alex Watson, Yev Meyer, Maarten Van Segbroeck, Matthew Grossman, Sami Torbey, Piotr Mlocek, and Johnny Greco. 2024. Synthetic-PII-Financial-Documents-North-America: A synthetic dataset for training language models to label and detect pii in domain specific formats. find
  23. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  24. Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. 2022. Opacus: User-Friendly Differential Privacy Library in PyTorch. arXiv preprint. ArXiv:2109.12298 [cs]… — [arXiv:2109.12298](https://arxiv.org/abs/2109.12298) find
  25. Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2021. Differentially Private Fine-tuning of Language Models. In International Conference on Learning Representatio… find
  26. Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, and Dawei Yin. 2024. Exploring Memorization in Fine-tuned Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa… find
  27. Effect of downstream task on PII memorization. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/szep-2026-unintended-memorization-sensitive-information-fine/