Elamur Research⌕
☰
home / papers / 2023 / Membership Inference Attacks against Language Models via Neighbourhood Comparison

Membership Inference Attacks against Language Models via Neighbourhood Comparison

Paper · 2023 · Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin et al.

Summary

The paper examines membership inference attacks (MIAs) on language models, which try to determine whether a specific text was part of a model's training set. The simple LOSS approach thresholds the model's loss and produces many false positives, because some texts are inherently easy to predict. Reference-based Likelihood Ratio Attacks (LiRAs) correct for this by comparing against reference models, but those models must be trained on data resembling the target's training distribution. The authors show that this is a strong and often unrealistic assumption. LiRA performance degrades sharply when reference data comes from other sources in the same domain. As an alternative, they propose a neighbourhood attack that needs no extra data. A masked language model generates minimally altered variants of a sample through word substitutions. The sample's loss is then compared with the average loss of these neighbours under the target model. This method beats reference-free attacks and realistic LiRAs, and it rivals LiRAs that have ideal data access. The authors conclude that privacy threat models for deployed language models need rethinking.

Key points

  • Access setting: The attack assumes grey-box access, meaning loss or confidence scores only, with no weights or gradients.
  • Decision rule: A sample is classified as a member if its loss, minus the mean loss of its n neighbours, falls below a threshold γ.
  • Neighbour generation: Neighbours come from BERT-style MLM lexical substitution, following Zhou et al. (2019). Each candidate's probability is renormalised to exclude the original token, so that substitutions preserve syntax and meaning.
  • Comparison with LiRAs: The attack outperforms LiRAs that use realistic, imperfect reference data by up to 100%. It remains competitive with LiRAs given perfect knowledge of the training distribution.
  • Limitations and release: The neighbour-generation method is specific to text, and the baseline attacks had to be reimplemented by the authors. The code is not public and is available to researchers on request.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓RWTH Aachen UniversityAachen · Germany18 works · 4,791 citations🎓ETH ZurichZurich · Switzerland36 works · 4,534 citations

Also on the byline: Intelligent Systems Research.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan and Taylor Berg-Kirkpatrick. 2023. Membership Inference Attacks against Language Models via Neighbourhood Comparison. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.719

@inproceedings{mattern2023membership,
  title = {{Membership Inference Attacks against Language Models via Neighbourhood Comparison}},
  author = {Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan and Taylor Berg-Kirkpatrick},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.719},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.719}
}

References

48 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, page 308–318, New York, NY, USA. Association for Com… find
  2. Raef Bassily, Adam Smith, and Abhradeep Thakurta. 2014. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464–473. find
  3. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, A. Terzis, and Florian Tramèr. 2021a. Membership inference attacks from first principles. 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. find
  4. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022a. Quantifying memorization across neural language models. find
  5. Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. 2022b. The privacy onion effect: Memorization is relative. In Advances in Neural Information Processing Systems. find
  6. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021b. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21…
  7. Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. 2023. On the possibilities of ai-generated text detection. find
  8. Dingfan Chen, Ning Yu, and Mario Fritz. 2022. Relaxloss: Defending membership inference attacks without losing utility. In International Conference on Learning Representations. find
  9. Christopher A. Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. 2021. Label-only membership inference attacks. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1964–1974. PMLR. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  11. Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography, pages 265–284, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  12. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. find
  13. Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009. find
  14. Sorami Hisamoto, Matt Post, and Kevin Duh. 2020. Membership inference attacks on sequence-to-sequence models: Is my data in your machine translation system? Transactions of the Association for Computational Linguistics, 8:49–63. find
  15. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  16. Bargav Jayaraman, Lingxiao Wang, David E. Evans, and Quanquan Gu. 2020. Revisiting membership inference under realistic assumptions. Proceedings on Privacy Enhancing Technologies, 2021:348 – 368. find
  17. Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. 2019. Memguard: Defending against black-box membership inference attacks via adversarial examples. Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. find
  18. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning.
  19. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense.
  20. Klas Leino and Matt Fredrikson. 2020. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In Proceedings of the 29th USENIX Conference on Security Symposium, SEC’20, USA. USENIX Association. find
  21. Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners. In International Conference on Learning Representations. find
  22. Zheng Li and Yang Zhang. 2021. Membership leakage in label-only exposures. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS ’21, page 880–895, New York, NY, USA. Association for Computing Machinery. find
  23. Yiyong Liu, Zhengyu Zhao, Michael Backes, and Yang Zhang. 2022. Membership Inference Attacks by Exploiting Loss Trajectory. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2085–2098. ACM. find
  24. Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A. Gunter, and Kai Chen. 2018. Understanding membership inferences on well-generalized learning models. ArXiv, abs/1802.04889. — [arXiv:1802.04889](https://arxiv.org/abs/1802.04889) find
  25. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations.
  26. R. Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2016. Membership inference attacks against machine learning models. 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18.
  27. Fatemehsadat Mireshghallah, Arturs Backurs, Huseyin A Inan, Lukas Wutschitz, and Janardhan Kulkarni. Differentially private model compression. In Advances in Neural Information Processing Systems. find
  28. Congzheng Song and Ananth Raghunathan. 2020. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, CCS ’20, page 377–390, New York, NY, USA. Association for Computing Machinery. find
  29. Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. 2022a. Quantifying privacy risks of masked language models using membership inference attacks.
  30. Fatemehsadat Mireshghallah, Huseyin Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg-Kirkpatrick, and Robert Sim. 2021. Privacy regularization: Joint privacy-utility optimization in LanguageModels. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computati… find
  31. Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. 2013. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pages 245–248. find
  32. Shuang Song and David Marn. 2020. Introducing a new privacy testing library in tensorflow. find
  33. Florian Tramèr, Gautam Kamath, and Nicholas Carlini. 2022. Considerations for differentially private learning with large-scale public pretraining. find
  34. Fatemehsadat Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick. 2023. Smaller language models are better black-box machine-generated text detectors. arXiv preprint arXiv:2305.09859. — [arXiv:2305.09859](https://arxiv.org/abs/2305.09859) find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  36. Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022b. Memorization in nlp fine-tuning methods. find
  37. Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. 2022. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations. find
  38. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature.
  39. Sasi Kumar Murakonda and R. Shokri. 2020. Ml privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. ArXiv, abs/2007.09339. — [arXiv:2007.09339](https://arxiv.org/abs/2007.09339) find
  40. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and…
  41. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho…
  42. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  43. Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2022. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS ’22, page 3093–3106, New York, NY, USA. A… find
  44. Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. 2019. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning. find
  45. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268–282. *Figure 3: Attack Performance of reference attacks w.r.t validation PPL of… find
  46. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2021. Counterfactual memorization in neural language models. find
  47. Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019. BERT-based lexical substitution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3368– 3373, Florence, Italy. Association for Computational Linguistics. find
  48. ACL 2023 Responsible NLP Checklist find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mattern-2023-membership-inference-attacks-against-language/