Elamur Research⌕
☰
home / papers / 2022 / Are Large Pre-Trained Language Models Leaking Your Personal Information?

Are Large Pre-Trained Language Models Leaking Your Personal Information?

Paper · 2022 · Jie Huang, Hanyin Shao and Kevin Chen-Chuan Chang

Summary

The paper asks whether large pre-trained language models (PLMs) can expose personal data. It separates two abilities. Memorization means reproducing a piece of information when given the text that preceded it in training. Association means linking information to its owner, so an attacker who knows only a name can retrieve it. The authors use email addresses as the test case. They build 3238 (name, email) pairs from the Enron Corpus, which is part of the Pile, and discard addresses on the easily guessable Enron domain. They then probe three GPT-Neo models (125M, 1.3B, 2.7B parameters) with greedy decoding in three settings:

  • original training context,
  • zero-shot prompts built around a person's name,
  • few-shot prompts containing true name–email examples.

Memorization does occur, and it increases with model size and context length. Association, however, is very weak. Name-based prompts almost never yield the correct address unless the domain is known. Even then, the models perform below a simple rule-based guesser. The authors conclude that targeted extraction risk is low but not negligible. They suggest defenses such as deduplication, DP-SGD training and output filtering.

Key points

  • Data: 3238 (name, email) pairs from the Enron Corpus (over 600,000 emails), mapped to owners via the UC Berkeley Enron Database. Only non-Enron domains appearing at least 3 times are kept.
  • Memorization: feeding 50, 100 or 200 preceding tokens recovers up to 8.80% of addresses, achieved by GPT-Neo 2.7B with 200 tokens. The 125M model stays around 1%.
  • Association, unknown domain: zero-shot and few-shot accuracy stays close to zero. The best prompt imitates an email-header pattern ("Original Message … From: {name} [mailto:"), and with it GPT-Neo 2.7B gets only 40 addresses right.
  • Association, known domain: 5-shot GPT-Neo 2.7B reaches 37.06%, still below the rule-based baseline at 46.85%. Addresses that follow common name patterns are the most vulnerable.
  • Code and data: published at github.com/jeffhj/LM_PersonalInfoLeak.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States228 works · 74,590 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSociology and Political Science · 630 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Jie Huang, Hanyin Shao and Kevin Chen-Chuan Chang. 2022. Are Large Pre-Trained Language Models Leaking Your Personal Information?. arXiv:2205.12628.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.12628 · arXiv 2205.12628

@misc{huang2022are,
  title = {{Are Large Pre-Trained Language Models Leaking Your Personal Information?}},
  author = {Jie Huang, Hanyin Shao and Kevin Chen-Chuan Chang},
  year = {2022},
  doi = {10.48550/arxiv.2205.12628},
  eprint = {2205.12628},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.12628}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318. find
  2. Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differentially private bert. arXiv preprint arXiv:2108.01624. find
  3. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. find
  4. Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7456– 7463.
  5. Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What does it mean for a language model to preserve privacy? arXiv preprint arXiv:2202.05520. — [arXiv:2202.05520](https://arxiv.org/abs/2202.05520) find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  7. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646. find
  8. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633– 2650.
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  10. Cynthia Dwork. 2008. Differential privacy: A survey of results. In International conference on theory and applications of models of computation, pages 1–19. Springer. find
  11. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer. find
  12. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  13. Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al. 2021. Learning and evaluating a differentially private pre-trained language model. In Findings of the Association for Computational Linguisti… find
  14. Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342.
  15. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-factr: Multilingual factual knowledge retrieval from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959. find
  16. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  17. Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In Proceedings of the 15th European Conference on Machine Learning, ECML’04, page 217–226, Berlin, Heidelberg. Springer-Verlag. find
  18. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  19. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C Wallace. 2021. Does bert pre-trained on clinical notes reveal sensitive data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages… find
  20. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners. In International Conference on Learning Representations. find
  21. Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pages 1314–1331. IEEE. find
  22. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  23. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897.
  24. Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. 2021. Selective differential privacy for language modeling. arXiv preprint arXiv:2108.12944. find
  25. Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. 2021. Understanding unintended memorization in language models under federated learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pages 1–10. find
  26. Thomas Vakili and Hercules Dalianis. 2021. Are clinical bert models privacy preserving? the difficulty of extracting patient-condition associations. find
  27. Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967. find
  28. Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. 2021. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500. — [arXiv:2110.06500](https://arxiv.org/abs/2110.06500) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2022-large-pre-trained-language-models/