Elamur Research⌕
☰
home / papers / 2021 / Membership Inference Attacks From First Principles

Membership Inference Attacks From First Principles

Paper · 2021 · Nicholas Carlini, Steve Chien, Milad Nasr et al.

Summary

Membership inference attacks ask whether a specific example was used to train a model. They are the most common way to empirically audit the privacy of machine learning systems. Carlini et al. argue that the usual yardsticks, balanced accuracy and ROC-AUC averaged over all thresholds, reveal little about whether an attacker can confidently identify even a few individual members. They propose a different test: an attack's true-positive rate at very low false-positive rates (around 0.1% or below), shown on log-scale ROC curves. By that test, most earlier attacks perform poorly.

The authors then design the Likelihood Ratio Attack (LiRA). LiRA trains shadow models that either include or exclude the target example. It fits Gaussian distributions to the logit-scaled confidences in each case and scores membership with a likelihood ratio. A cheaper offline variant uses only models trained without the example.

LiRA was tested on well-regularized models for CIFAR-10, CIFAR-100, WikiText-103, Texas and Purchase. At low FPR it is roughly ten times stronger than prior work, and it still wins on average-case metrics. The result suggests that privacy audits built on weaker attacks may seriously underestimate leakage.

Key points

  • Proposed metric: TPR at low FPR (e.g., ≤0.1%) rather than balanced accuracy or AUC. Aggregate scores can disagree with this metric. The Yeom et al. attack reaches 59.5% accuracy yet fails at low FPR. The Long et al. attack scores only 53.5% accuracy but performs better in that regime.
  • On a non-overfit CIFAR-10 model with 92% test accuracy, LiRA reaches 63.7% balanced accuracy, compared with roughly 53–60% for earlier attacks. It improves low-FPR success by about 10×.
  • LiRA combines per-example difficulty calibration, logit scaling, a Gaussian likelihood test and multiple augmented queries. Ablations on CIFAR-100, WikiText-103, Texas and Purchase show that logit scaling and the Gaussian test help only when applied together.
  • There are two variants: online (in and out shadow models) and offline (out models only). Further experiments examine attackers whose shadow models use a different architecture (CNN, WRN28 sizes), optimizer (SGD, momentum, Adam) or augmentation than the target.
  • Code is released in the tensorflow/privacy repository (research/mi_lira_2021).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations🎓University of Massachusetts AmherstAmherst Center · United States114 works · 68,731 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis and Florian Tramer. 2021. Membership Inference Attacks From First Principles. arXiv:2112.03570.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2112.03570 · arXiv 2112.03570

@misc{carlini2021membership,
  title = {{Membership Inference Attacks From First Principles}},
  author = {Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis and Florian Tramer},
  year = {2021},
  doi = {10.48550/arxiv.2112.03570},
  eprint = {2112.03570},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.03570}
}

References

73 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, page 308–318. ACM, 2016. find
  2. Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 123–132, 2021. find
  3. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, 2019.
  4. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
  5. Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, et al. Gmail smart compose: Real-time assisted writing. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2287–2295, 2019. find
  6. Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International Conference on Machine Learning, pages 1964–1974. PMLR, 2021. find
  7. Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data, 2018. find
  8. Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. CINIC-10 is not Imagenet or CIFAR-10. arXiv preprint arXiv:1810.03505, 2018. — [arXiv:1810.03505](https://arxiv.org/abs/1810.03505) find
  9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. find
  10. Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9 (3-4):211–407, 2014. find
  11. Cynthia Dwork, Adam Smith, Thomas Steinke, Jonathan Ullman, and Salil Vadhan. Robust traceability from trace amounts. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, pages 650–669. IEEE, 2015. find
  12. Cynthia Dwork, Adam Smith, Thomas Steinke, and Jonathan Ullman. Exposed! a survey of attacks on private data. Annual Review of Statistics and Its Application, 4:61–84, 2017. find
  13. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639):115–118, 2017. find
  14. Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954–959, 2020. find
  15. Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprint arXiv:2008.03703, 2020.
  16. Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1322–1333, 2015. find
  17. Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pages 619–633, 2018. find
  18. Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. LOGAN: Membership inference attacks against generative models. In Proceedings on Privacy Enhancing Technologies (PoPETs), pages 133–152. De Gruyter, 2019. find
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
  20. Xinlei He, Jinyuan Jia, Michael Backes, Neil Zhenqiang Gong, and Yang Zhang. Stealing links from graph neural networks. In 30th USENIX Security Symposium (USENIX Security 21), 2021. find
  21. Grant Ho, Aashish Sharma, Mobin Javed, Vern Paxson, and David Wagner. Detecting credential spearphishing in enterprise settings. In 26th USENIX Security Symposium (USENIX Security 17), pages 469–485, 2017. find
  22. Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLoS g… find
  23. Robert A Jacobs. Increased rates of convergence through learning rate adaptation. Neural networks, 1(4):295–307, 1988. find
  24. Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? arXiv preprint arXiv:2006.07709, 2020. — [arXiv:2006.07709](https://arxiv.org/abs/2006.07709) find
  25. Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu. Revisiting membership inference under realistic assumptions. In Proceedings on Privacy Enhancing Technologies (PoPETs), 2021. find
  26. Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Memguard: Defending against black-box membership inference attacks via adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 259–274, 2019. find
  27. Alex Kantchelian, Michael Carl Tschantz, Sadia Afroz, Brad Miller, Vaishaal Shankar, Rekha Bachwani, Anthony D Joseph, and J Doug Tygar. Better malware ground truth: Techniques for weighting anti-virus vendor labels. In Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security, pag… find
  28. Zico Kolter and Marcus A Maloof. Learning to detect and classify malicious executables in the wild. Journal of Machine Learning Research, 7(12), 2006. find
  29. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. find
  30. Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950–957, 1992. find
  31. Aleksandar Lazarevic, Levent Ertoz, Vipin Kumar, Aysel Ozgur, and Jaideep Srivastava. A comparative study of anomaly detection schemes in network intrusion detection. In Proceedings of the 2003 SIAM international conference on data mining, pages 25–36. SIAM, 2003. find
  32. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  33. Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. arXiv preprint arXiv:1906.11798, 2019. — [arXiv:1906.11798](https://arxiv.org/abs/1906.11798) find
  34. Zheng Li and Yang Zhang. Membership leakage in label-only exposures. arXiv preprint arXiv:2007.15528, 2020. — [arXiv:2007.15528](https://arxiv.org/abs/2007.15528) find
  35. Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. ML-Doctor: Holistic risk assessment of inference attacks against machine learning models. arXiv preprint arXiv:2102.02551, 2021. — [arXiv:2102.02551](https://arxiv.org/abs/2102.02551) find
  36. Yunhui Long, Vincent Bindschaedler, and Carl A Gunter. Towards measuring membership privacy. arXiv preprint arXiv:1712.09136, 2017. — [arXiv:1712.09136](https://arxiv.org/abs/1712.09136) find
  37. Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 521–534. IEEE, 2020. find
  38. Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. find
  39. Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP), pages 691–706. IEEE, 2019. find
  40. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  41. Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. Spam filtering with naive Bayes–which naive Bayes? In CEAS, volume 17, pages 28–69, 2006. find
  42. Sasi Kumar Murakonda and Reza Shokri. ML Privacy Meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. arXiv preprint arXiv:2007.09339, 2020. — [arXiv:2007.09339](https://arxiv.org/abs/2007.09339) find
  43. Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. Ultimate power of inference attacks: Privacy risks of learning high-dimensional graphical models. arXiv e-prints, pages arXiv–1905, 2019. find
  44. Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligence and Statistics, pages 2287–2295. PMLR, 2021. find
  45. Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 634–646, 2018. find
  46. Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pages 739–753. IEEE, 2019. find
  47. Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adversary instantiation: Lower bounds for differentially private machine learning. arXiv preprint arXiv:2101.04535, 2021. — [arXiv:2101.04535](https://arxiv.org/abs/2101.04535) find
  48. Jerzy Neyman and Egon Sharpe Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London., 231(694-706): 289–337, 1933. find
  49. Patrick Pantel and Dekang Lin. SpamCop: A spam classification & organization program. In Proceedings of AAAI-98 Workshop on Learning for Text Categorization, pages 95–98, 1998. find
  50. Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with PATE. arXiv preprint arXiv:1802.08908, 2018. — [arXiv:1802.08908](https://arxiv.org/abs/1802.08908) find
  51. Huy Phan. huyvnphan/pytorch cifar10, January 2021. URL https://doi.org/10.5281/zenodo.4431043. — [doi:10.5281/zenodo.4431043](https://doi.org/10.5281/zenodo.4431043) ↗
  52. Apostolos Pyrgelis, Carmela Troncoso, and Emiliano De Cristofaro. Knock knock, who’s there? membership inference on aggregate location data. arXiv preprint arXiv:1708.06145, 2017. — [arXiv:1708.06145](https://arxiv.org/abs/1708.06145) find
  53. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019. find
  54. Shadi Rahimian, Tribhuvanesh Orekondy, and Mario Fritz. Sampling attacks: Amplification of membership inference attacks by repeated queries. arXiv preprint arXiv:2009.00395, 2020. — [arXiv:2009.00395](https://arxiv.org/abs/2009.00395) find
  55. Md Atiqur Rahman, Tanzila Rahman, Robert Laganière, Noman Mohammed, and Yang Wang. Membership inference attack against differentially private deep learning model. Trans. Data Priv., 11(1):61–79, 2018. find
  56. Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pages 5558–5567. PMLR, 2019. find
  57. Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and data independent membership inference attacks and defenses on machine learning models, 2018. find
  58. Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. Updates-leak: Data set inference and reconstruction attacks in online learning. In 29th USENIX Security Symposium (USENIX Security 20), pages 1291–1308, 2020. find
  59. Sriram Sankararaman, Guillaume Obozinski, Michael I Jordan, and Eran Halperin. Genomic privacy and limits of individual detection in a pool. Nature genetics, 41(9):965–967, 2009. find
  60. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. arXiv preprint arXiv:1610.05820, 2016.
  61. Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), 2021. find
  62. Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 241–257, 2019. find
  63. Shuang Song and David Marn. Introducing a new privacy testing library in tensorflow. https://blog.tensorflow.org/2020/06/introducing-new-privacy-testing-library.html, 2020. ↗
  64. Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pages 245–248. IEEE, 2013. find
  65. Thomas Steinke and Jonathan Ullman. The pitfalls of average-case differential privacy. DifferentialPrivacy.org, 07 2020. https://differentialprivacy.org/average-case-dp/. ↗
  66. Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Towards demystifying membership inference attacks. arXiv preprint arXiv:1807.09173, 2018. — [arXiv:1807.09173](https://arxiv.org/abs/1807.09173) find
  67. David A Van Dyk and Xiao-Li Meng. The art of data augmentation. Journal of Computational and Graphical Statistics, 10 (1):1–50, 2001. find
  68. Lauren Watson, Chuan Guo, Graham Cormode, and Alex Sablayrolles. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440, 2021. — [arXiv:2111.08440](https://arxiv.org/abs/2111.08440) find
  69. Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. Enhanced membership inference attacks against machine learning models. arXiv preprint arXiv:2111.09679, 2021. — [arXiv:2111.09679](https://arxiv.org/abs/2111.09679) find
  70. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268–282. IEEE, 2018.
  71. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. — [arXiv:1605.07146](https://arxiv.org/abs/1605.07146) find
  72. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107–115, 2021. find
  73. Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13001–13008, 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/carlini-2021-membership-inference-attacks-first-principles/