Elamur Research⌕
☰
home / papers / 2023 / Analyzing Leakage of Personally Identifiable Information in Language Models

Analyzing Leakage of Personally Identifiable Information in Language Models

Paper · 2023 · Nils Lukas, Ahmed Salem, Robert Sim et al.

Summary

Fine-tuned language models can memorize and expose personal data. Many practitioners assume that NER-based scrubbing or differential privacy removes this risk, but that assumption has rarely been tested. Lukas et al. define three levels of PII leakage through black-box API access as formal security games. In extraction, the attacker knows nothing about the training data. In reconstruction, the attacker holds a sentence with the PII masked out. In inference, the attacker also has a list of candidate values.

The authors build an attack for each game and run them against GPT-2 models of several sizes. The models are fine-tuned on court cases (ECHR), corporate mail (Enron) and health-facility reviews (Yelp-Health), either undefended, scrubbed, trained with DP, or both. The reconstruction attack uses text on both sides of the mask, helped by a public masked LM, and recovers up to ten times more PII than earlier methods. Sentence-level DP sharply reduces leakage but still lets about 3% of PII sequences through. Scrubbing costs roughly as much utility as DP. The resulting metrics let practitioners tune the privacy/utility balance of a whole pipeline.

Key points

  • Datasets used for fine-tuning:
    • ECHR: about 118k records, 23.75% of them containing person names.
    • Enron: about 139k records, 81.45% with names.
    • Yelp-Health: about 79k records, 54.55% with names.
  • PII is tagged with the Flair model ner-english-ontonotes-large across 21 entity classes, such as person, gpe, org, date, phone number and email address.
  • Extraction success is measured with precision and recall over unique PII sequences. Reconstruction and inference are scored by top-1 accuracy on a randomly chosen masked PII.
  • The extraction attacks achieve roughly twice the precision of prior work, even against DP-trained models. With DP (ε = 8), about 3% of PII sequences still leak.
  • Prior work cited in the paper shows clinical NER recall can drop to 80% for some PII types (care unit numbers, versus 97% for names), so scrubbing leaves residual PII. Code is released at github.com/microsoft/analysing_pii_leakage.

Interesting ideas

  • I1. Tiered attacker knowledge. The paper grades threats by how much the attacker knows: nothing, the surrounding context, or the context plus a candidate list. This gives a reusable template for privacy audits of any text model, including models trained on Ukrainian or other low-resource corpora.
  • I2. Candidate ranking by model likelihood. Each candidate is inserted into the masked slot and the target LM's probability of the full sentence is scored. This turns a generative model into a probe of what it has memorized, which is cheap to apply wherever only API access is available.
  • I3. Filling extra masks left to right. When a query contains several masked PII, a public masked LM fills them one at a time, keeping the sentence fluent before scoring. The trick is handy whenever off-the-shelf fill-mask tools cannot fill multiple slots jointly.
  • I4. Evaluating defenses together. Scrubbing and DP are compared alone and combined, with both leakage and perplexity reported side by side. This end-to-end view suggests designing a lighter scrubber that relies on DP for the rest, instead of over-scrubbing and losing utility.
  • I5. Separate metrics for separate threats. Extraction is scored with precision and recall, while reconstruction and inference use top-1 accuracy. Keeping these apart prevents one aggregate "memorization" number from hiding which attack actually succeeds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations🎓University of WaterlooWaterloo · Canada80 works · 15,030 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz and Santiago Zanella-Beguelin. 2023. Analyzing Leakage of Personally Identifiable Information in Language Models. arXiv:2302.00539.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.00539 · arXiv 2302.00539

@misc{lukas2023analyzing,
  title = {{Analyzing Leakage of Personally Identifiable Information in Language Models}},
  author = {Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz and Santiago Zanella-Beguelin},
  year = {2023},
  doi = {10.48550/arxiv.2302.00539},
  eprint = {2302.00539},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.00539}
}

References

70 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in 23rd ACM SIGSAC Conference on Computer and Communications Security, CCS. ACM, 2016, pp. 308–318. find
  2. A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, and R. Vollgraf, “FLAIR: An easy-to-use framework for state-of-the-art NLP,” in 2019 Conference of the North American Chapter of the ACL (demonstrations). ACL, 2019, pp. 54–59.
  3. Apple, “Differential privacy,” Available online at https://www.apple.com/privacy/docs/Differential Privacy Overview.pdf, retrieved 28 Nov 2022. ↗
  4. S. Bird, E. Klein, and E. Loper, Natural language processing with Python: analyzing text with the Natural Language Toolkit. O’Reilly Media, 2009. find
  5. H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr, “What does it mean for a language model to preserve privacy?” in 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT. ACM, 2022, p. 2280–2292. find
  6. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” in Advances in neural information processing systems (NeurIPS), vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901.
  7. P. Budzianowski and I. Vulić, “Hello, it’s GPT-2 – How can I help you? towards the use of pretrained language models for task-oriented dialogue systems,” arXiv preprint arXiv:1907.05774 [cs.CL], 2019. find
  8. N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, “The secret sharer: Evaluating and testing unintended memorization in neural networks,” in 28th USENIX Security Symposium (USENIX Security 19), 2019, pp. 267– 284.
  9. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in 30th USENIX Security Symposium. USENIX Association, 2021, pp. 2633–2650.
  10. N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “Membership inference attacks from first principles,” in 43rd IEEE Symposium on Security and Privacy (S&P). IEEE, 2022, pp. 1897–1914.
  11. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” arXiv preprint arXiv:2202.07646 [cs.CL], 2022. find
  12. I. Chalkidis, I. Androutsopoulos, and N. Aletras, “Neural legal judgment prediction in English,” in 57th Annual Meeting of the Association for Computational Linguistics, ACL. ACL, 2019, pp. 4317–4323.
  13. M. X. Chen, B. N. Lee, G. Bansal, Y. Cao, S. Zhang, J. Lu, J. Tsay, Y. Wang, A. M. Dai, Z. Chen et al., “Gmail smart compose: Real-time assisted writing,” in 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 2287–2295. find
  14. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. ACL, 2019, pp. 4171–4186.
  15. C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, “Our data, ourselves: Privacy via distributed noise generation,” in Advances in Cryptology – EURO-CRYPT 2006. Springer, 2006, pp. 486–503. find
  16. C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of cryptography conference, TCC. Springer, 2006, pp. 265–284. find
  17. European Union, “Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation),”… find
  18. A. Fan, M. Lewis, and Y. Dauphin, “Hierarchical neural story generation,” in 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 2018, pp. 889–898.
  19. W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022. find
  20. V. Feldman, “Does learning require memorization? a short tale about a long tail,” in 52nd Annual ACM SIGACT Symposium on Theory of Computing. ACM, 2020, pp. 954–959. find
  21. P. Golle, “Revisiting the uniqueness of simple demographics in the us population,” in 5th ACM Workshop on Privacy in Electronic Society. ACM, 2006, pp. 77–80. find
  22. A. Hoang, A. Bosselut, A. Celikyilmaz, and Y. Choi, “Efficient adaptation of pretrained transformers for abstractive summarization,” arXiv preprint arXiv:1906.00138 [cs.CL], 2019. — [arXiv:1906.00138](https://arxiv.org/abs/1906.00138) find
  23. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optim… — Training Compute-Optimal Large Language Models
  24. M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength natural language processing in Python,” 2020. find
  25. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, and R. Weischedel, “OntoNotes: The 90% solution,” in Human Language Technology Conference of the NAACL. ACL, 2006, pp. 57–60. find
  26. J. Huang, H. Shao, and K. C.-C. Chang, “Are large pre-trained language models leaking your personal information?” in Findings of the Association for Computational Linguistics, EMNLP. ACL, 2022, pp. 2038–2047. find
  27. T. Humphries, S. Oya, L. Tulloch, M. Rafuse, I. Goldberg, U. Hengartner, and F. Kerschbaum, “Investigating membership inference attacks under data dependencies,” arXiv preprint arXiv:2010.12112 [cs.CR], 2020. — [arXiv:2010.12112](https://arxiv.org/abs/2010.12112) find
  28. H. A. Inan, O. Ramadan, L. Wutschitz, D. Jones, V. Rühle, J. Withers, and R. Sim, “Training data leakage analysis in language models,” arXiv preprint arXiv:2101.05405v2 [cs.CR], 2021. — [arXiv:2101.05405](https://arxiv.org/abs/2101.05405) find
  29. D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini, “Preventing verbatim memorization in language models gives a false sense of privacy,” arXiv preprint arXiv:2210.17546 [cs.LG], 2022. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) find
  30. A. Jagannatha, B. P. S. Rawat, and H. Yu, “Membership inference attack susceptibility of clinical language models,” arXiv preprint arXiv:2104.08305 [cs.CL], 2021. — [arXiv:2104.08305](https://arxiv.org/abs/2104.08305) find
  31. M. Jagielski, O. Thakkar, F. Tramèr, D. Ippolito, K. Lee, N. Carlini, E. Wallace, S. Song, A. Thakurta, N. Papernot et al., “Measuring forgetting of memorized training examples,” arXiv preprint arXiv:2207.00099 [cs.LG], 2022. — [arXiv:2207.00099](https://arxiv.org/abs/2207.00099) find
  32. B. Jayaraman, E. Ghosh, M. Chase, S. Roy, H. Inan, W. Dai, and D. Evans, “Combing for credentials: Active pattern extraction from smart reply,” arXiv preprint arXiv:2203.07618 [cs.CR], 2022. — [arXiv:2203.07618](https://arxiv.org/abs/2203.07618) find
  33. N. Kandpal, E. Wallace, and C. Raffel, “Deduplicating training data mitigates privacy risks in language models,” in 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 10 697–10 707.
  34. B. Klimt and Y. Yang, “Introducing the Enron corpus,” in International Conference on Email and Anti-Spam, CEAS, vol. 45, 2004, pp. 92–96. find
  35. G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural architectures for named entity recognition,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. ACL, 2016, pp. 260–270.
  36. J. Lee, T. Le, J. Chen, and D. Lee, “Do language models plagiarize?” arXiv preprint arXiv:2203.07618 [cs.CL], 2022. — [arXiv:2203.07618](https://arxiv.org/abs/2203.07618) find
  37. E. Lehman, S. Jain, K. Pichotta, Y. Goldberg, and B. C. Wallace, “Does BERT pretrained on clinical notes reveal sensitive data?” arXiv preprint arXiv:2104.07762 [cs.CL], 2021. find
  38. X. Li, F. Tramer, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,” in 10th International Conference on Learning Representations (ICLR). OpenReview.net, 2022. find
  39. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692 [cs.CL], 2019.
  40. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR. OpenReview.net, 2019. find
  41. N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin, “Analyzing leakage of personally identifiable information in language models,” in 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023.
  42. R. T. McCoy, P. Smolensky, T. Linzen, J. Gao, and A. Celikyilmaz, “How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN,” arXiv preprint arXiv:2111.09509 [cs.CL], 2021.
  43. B. McMahan and A. Thakurta, “Federated learning with formal differential privacy guarantees,” Google, Tech. Rep., 2022. [Online]. Available: https://ai.googleblog.com/2022/02/federated-learning-with-formal.html ↗
  44. N. Miao, H. Zhou, L. Mou, R. Yan, and L. Li, “CGMH: Constrained sentence generation by metropolis-hastings sampling,” in AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 6834–6842. find
  45. F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri, “Quantifying privacy risks of masked language models using membership inference attacks,” arXiv preprint arXiv:2203.03929 [cs.LG], 2022.
  46. F. Mireshghallah, A. Uniyal, T. Wang, D. Evans, and T. Berg-Kirkpatrick, “Memorization in NLP fine-tuning methods,” arXiv preprint arXiv:2205.12506 [cs.CL], 2022. — [arXiv:2205.12506](https://arxiv.org/abs/2205.12506) find
  47. D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes, vol. 30, no. 1, pp. 3–26, 2007. find
  48. M. Nadeem, A. Bethke, and S. Reddy, “StereoSet: Measuring stereotypical bias in pretrained language models,” in 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP. ACL, 2021, pp. 5356–5371.
  49. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al., “WebGPT: Browser-assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332 [cs.CL], 2021.
  50. R. Parikh, C. Dupuy, and R. Gupta, “Canary extraction in natural language understanding models,” in 60th Annual Meeting of the Association for Computational Linguistics, ACL. ACL, 2022, pp. 552–560. find
  51. I. Pilán, P. Lison, L. Øvrelid, A. Papadopoulou, D. Sánchez, and M. Batet, “The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization,” Computational Linguistics, vol. 48, no. 4, pp. 1053–1101, 2022.
  52. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners.” [Online]. Available: https://cdn.openai.com/better-language-models/language models are unsupervised multitask learners.pdf ↗
  53. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu et al., “Exploring the limits of transfer learning with a unified text-to-text transformer.” J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020.
  54. L. Rocher, J. M. Hendrickx, and Y.-A. De Montjoye, “Estimating the success of re-identifications in incomplete datasets using generative models,” Nature communications, vol. 10, no. 1, pp. 1–9, 2019. find
  55. P. Samarati and L. Sweeney, “Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression,” Computer Science Laboratory, SRI International, Tech. Rep., 1998. [Online]. Available: http://www.csl.sri.com/ papers/sritr-98-04/ ↗
  56. R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” arXiv preprint arXiv:1508.07909 [cs.CL], 2015.
  57. W. Shi, A. Cui, E. Li, R. Jia, and Z. Yu, “Selective differential privacy for language modeling,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACACL-HLT. ACL, 2022, pp. 2848–2859. find
  58. R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 37th IEEE symposium on security and privacy (S&P). IEEE, 2017, pp. 3–18.
  59. P. Stock, I. Shilov, I. Mironov, and A. Sablayrolles, “Defending against reconstruction attacks with Rényi differential privacy,” arXiv preprint arXiv:2202.07623 [cs.LG], 2022. — [arXiv:2202.07623](https://arxiv.org/abs/2202.07623) find
  60. K. Tirumala, A. H. Markosyan, L. Zettlemoyer, and A. Aghajanyan, “Memorization without overfitting: Analyzing the training dynamics of large language models,” in Advances in neural information processing systems (NeurIPS), vol. 35. Curran Associates, Inc., 2022, pp. 38 274–38 290.
  61. T. Vakili and H. Dalianis, “Are clinical BERT models privacy preserving? The difficulty of extracting patient-condition associations.” in AAAI 2021 Fall Symposium on Human Partnership with Medical AI: Design, Operationalization, and Ethics (AAAI-HUMAN). CEUR-WS.org, 2021. find
  62. T. Vakili, A. Lamproudis, A. Henriksson, and H. Dalianis, “Downstream task performance of BERT models pre-trained using automatically de-identified clinical data,” in 13th Language Resources and Evaluation Conference, LREC. ACL, 2022, pp. 4245–4252. find
  63. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems (NeurIPS), vol. 30, p. 6000–6010, 2017.
  64. L. Wutschitz, H. A. Inan, and A. Manoel, “dp-transformers: Training transformer models with differential privacy.” [Online]. Available: https://www.microsoft.com/en-us/research/project/dp-transformers ↗
  65. S. Yeom, I. Giacomelli, A. Menaged, M. Fredrikson, and S. Jha, “Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning,” Journal of Computer Security, vol. 28, no. 1, pp. 35–70, 2020. find
  66. A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao et al., “Opacus: User-friendly differential privacy library in PyTorch,” arXiv preprint arXiv:2109.12298 [cs.LG], 2021. — [arXiv:2109.12298](https://arxiv.org/abs/2109.12298) find
  67. D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, S. Yekhanin, and H. Zhang, “Differentially private fine-tuning of language models,” in 10th International Conference on Learning Representations (ICLR). OpenReview.net, 2022. find
  68. S. Zanella-Béguelin, L. Wutschitz, S. Tople, V. Rühle, A. Paverd, O. Ohrimenko, B. Köpf, and M. Brockschmidt, “Analyzing information leakage of updates to natural language models,” in 27th ACM SIGSAC Conference on Computer and Communications Security, 2020, pp. 363–375. find
  69. R. Zhang, S. Hidano, and F. Koushanfar, “Text revealer: Private text reconstruction via model inversion attacks against transformers,” arXiv preprint arXiv:2209.10505 [cs.CL], 2022. — [arXiv:2209.10505](https://arxiv.org/abs/2209.10505) find
  70. X. Zhao, L. Li, and Y.-X. Wang, “Provably confidential language modelling,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACACL-HLT. ACL, 2022, pp. 943–955. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lukas-2023-analyzing-leakage-personally-identifiable-information/