Elamur Research⌕
☰
home / papers / 2022 / Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks

Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks

Paper · 2022 · Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal et al.

Summary

Earlier studies suggested that masked language models (MLMs) leak little of their training data. The authors argue this conclusion came from weak attacks that only threshold the target model's loss. Such a loss partly reflects how intrinsically hard a sample is, so it is a poor membership signal. They propose a likelihood ratio membership inference attack that compares how a sample scores under the target MLM and under a reference MLM trained on separate data from the same population. MLMs do not directly provide sequence probabilities, so the authors treat them as energy-based models (following Goyal et al., 2022). The energy is estimated from MLM predictions over randomly sampled 15% masking patterns, and the intractable normalizers cancel out as a constant. They test the attack on clinical BERT-style models trained on sensitive medical notes (MIMIC-III, i2b2). On ClinicalBERT-Base, AUC rises from 0.66 with the loss-based baseline to 0.90. The work shows that MLMs memorize substantially, which supports applying defenses such as differentially private training.

Key points

  • Attack performance on ClinicalBERT-Base:
    • AUC rises from 0.66 to 0.90.
    • At a 10% false positive rate, recall is 79.2%, compared with 15.6% for the baseline.
    • At a 1% false positive rate, the attack is roughly 51× stronger than prior work.
  • Test statistic: the score is the energy difference E(s; θ) − E(s; θ_R) between the target and reference models. For efficiency, it is approximated with K random masking patterns rather than all possible combinations.
  • Additional analyses: the paper studies how model size, sample length and the choice of reference model affect attack success.
  • Which samples are exposed: samples containing many non-alphanumeric symbols, such as punctuation, are more vulnerable. A logistic regression that uses digit count, sequence length, non-alphanumeric count and rare-token features predicts leaked samples with about 72.1% precision and 61.3% recall on test data.
  • Data and code access: the MIMIC-III and i2b2 data are available only by request. The code is shared only with researchers who can show authorized access to these datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations🎓University of California San DiegoSan Diego · United States34 works · 51,167 citations🎓Toyota Technological Institute at ChicagoChicago · United States29 works · 10,273 citations🎓University of VirginiaCharlottesville · United States18 works · 8,786 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Social Sciences · 744 works Sociology and Political Science · 221 works

🧭 Topics 1 Language models & pretraining · 1,343 works

Cite

Cite as Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick and Reza Shokri. 2022. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.570

@inproceedings{mireshghallah2022quantifying,
  title = {{Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks}},
  author = {Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick and Reza Shokri},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.570},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.570}
}

References

44 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318. find
  2. Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2 Access can be requested through https://mimic.mit.edu/docs/gettingstarted/ and https://portal.dbmi.hms.harvard.edu/ projects/n2c2-nlp/ 3 Data Usage Agreements (DUA) are available in https://physionet.org/con… — [arXiv:2202.05520](https://arxiv.org/abs/2202.05520) ↗
  3. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2021a. Membership inference attacks from first principles. find
  4. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. find
  5. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284. our card
  6. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021b. Extracting training data from large language models. our card
  7. Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. 2011. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3). find
  8. Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association, 24(3):596–606. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. our card
  10. Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. 2022. Exposing the implicit energy networks behind masked language models via metropolis– hastings. In International Conference on Learning Representations. find
  11. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23. our card
  12. Sorami Hisamoto, Matt Post, and Kevin Duh. 2020. Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System? Transactions of the Association for Computational Linguistics, 8:49–63. find
  13. Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu. 2021. Membership inference attack susceptibility of clinical language models. find
  14. Bargav Jayaraman, Lingxiao Wang, Katherine Knip-meyer, Quanquan Gu, and David Evans. 2021. Revisiting membership inference under realistic assumptions. find
  15. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539. our card
  16. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. our card
  17. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. 2021. Does BERT pretrained on clinical notes reveal sensitive data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 946… find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  19. Yunhui Long, Vincent Bindschaedler, and Carl A. Gunter. 2017. Towards measuring membership privacy. ArXiv, abs/1712.09136. — [arXiv:1712.09136](https://arxiv.org/abs/1712.09136) find
  20. Saeed Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. 2021. Membership inference on word embedding and beyond. arXiv preprint arXiv:2106.11384. — [arXiv:2106.11384](https://arxiv.org/abs/2106.11384) find
  21. Fatemehsadat Mireshghallah, Huseyin Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg-Kirkpatrick, and Robert Sim. 2021. Privacy regularization: Joint privacy-utility optimization in languagemodels. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computati… find
  22. Fatemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. 2020. Privacy in deep learning: A survey. arXiv preprint arXiv:2004.12254. — [arXiv:2004.12254](https://arxiv.org/abs/2004.12254) find
  23. Sasi Kumar Murakonda and Reza Shokri. 2020. Ml privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. In Workshop on Hot Topics in Privacy Enhancing Technologies (HotPETs). find
  24. Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. 2021. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligence and Statistics, pages 2287–2295. PMLR. find
  25. Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2020. Kart: Privacy leakage framework of language models pre-trained with clinical records. find
  26. Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pages 739–753. IEEE. find
  27. Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. 2021. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 866–882. IEEE. find
  28. Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pages 58–65. our card
  29. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. our card
  30. Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. 2020. Updates-Leak: Data set inference and reconstruction attacks in online learning. In 29th USENIX Security Symposium (USENIX Security 20), pages 1291–1308. USENIX Association. find
  31. Ahmed Salem, Yang Zhang, Mathias Humbert, Mario Fritz, and Michael Backes. 2018. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. ArXiv, abs/1806.01246. — [arXiv:1806.01246](https://arxiv.org/abs/1806.01246) find
  32. Virat Shejwalkar, Huseyin A Inan, Amir Houmansadr, and Robert Sim. 2021. Membership inference attacks against nlp classification models. In NeurIPS 2021 Workshop Privacy in Machine Learning. find
  33. Reza Shokri. 2022. Auditing data privacy for machine learning. Santa Clara, CA. USENIX Association. on Challenges in Natural Language Processing for Clinical Data. find
  34. Thomas Vakili and Hercules Dalianis. 2021. Are clinical bert models privacy preserving? the difficulty of extracting patient-condition associations. In Proceedings of the AAAI 2021 Fall Symposium on Human Partnership with Medical AI : Design, Operationalization, and Ethics (AAAI-HUMAN 2021), number… find
  35. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. our card
  36. Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. arXiv preprint arXiv:2006.08097. find
  37. Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2021. Enhanced membership inference attacks against machine learning models. arXiv preprint arXiv:2111.09679. — [arXiv:2111.09679](https://arxiv.org/abs/2111.09679) find
  38. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE. find
  39. Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. 2020. Analyzing Information Leakage of Updates to Natural Language Models, pages 363–375. Association for Computing Machinery, New York, NY, USA. find
  40. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  41. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE. our card
  42. Congzheng Song and Ananth Raghunathan. 2020. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 377–390. find
  43. Congzheng Song and Vitaly Shmatikov. 2018. The natural auditor: How to tell if someone used your words to train their model. ArXiv, abs/1811.00513. — [arXiv:1811.00513](https://arxiv.org/abs/1811.00513) find
  44. Amber Stubbs and Özlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus. Journal of Biomedical Informatics, 58:S20–S29. Supplement: Proceedings of the 2014 i2b2/UTHealth Shared-Tasks and Workshop *Table 8: Summary of our attack’s notation… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mireshghallah-2022-quantifying-privacy-risks-masked-language/