Elamur Research
☰
home / papers / 2016 / Membership Inference Attacks against Machine Learning Models

Membership Inference Attacks against Machine Learning Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1610.05820

Summary

The paper asks whether a trained classifier reveals which individual records were in its training set. The attacker can only query the model as a black box and see its prediction vectors. The authors treat this as a supervised learning task. They build several "shadow models" that imitate the target and whose training data they control. The shadow models' outputs on known members and non-members then serve as labeled examples for an attack classifier. Shadow training data comes from one of three sources: records synthesized by querying the target, population-level statistics, or a noisy copy of the real training data. Targets include neural networks and models built with Amazon ML and Google Prediction API, whose algorithms and architectures are not disclosed to users. The evaluation draws on data covering pictures, shopping transactions, location histories, and hospital admissions. The attacks work in most settings, and the degree of leakage tracks how much the model overfits. This matters because cloud ML platforms let anyone train and publish models on sensitive personal data, with little warning or control over overfitting.

Key points

  • On retail-purchase models trained on 10,000 records with default settings, median membership-inference accuracy reached about 94% against Google Prediction API and 74% against Amazon ML.
  • With no prior knowledge of the data distribution and fully synthetic shadow data, accuracy against Google-trained models was still around 90%.
  • On the Texas hospital discharge dataset, accuracy exceeded 70%, a serious concern for publicly queryable medical models.
  • The authors compare four defenses: returning only the top k classes, coarsening probability precision, raising output entropy, and regularization. They link leakage directly to overfitting.
  • The method does not depend on any particular model type or dataset. The statistics-based and noisy-data variants need only a single query per candidate record.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

222 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 208 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1610.05820 · arXiv 1610.05820

@misc{shokri2016membership,
  title = {{Membership Inference Attacks against Machine Learning Models}},
  author = {Shokri, Reza and Stronati, Marco and Song, Congzheng and Shmatikov, Vitaly},
  year = {2016},
  doi = {10.48550/arxiv.1610.05820},
  eprint = {1610.05820},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1610.05820}
}

References

38 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in CCS, 2016. find
  2. G. Ateniese, L. V. Mancini, A. Spognardi, A. Villani, D. Vitali, and G. Felici, “Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers,” International Journal of Security and Networks, vol. 10, no. 3, pp. 137–150, 2015. find
  3. M. Backes, P. Berrang, M. Humbert, and P. Manoharan, “Membership privacy in MicroRNA-based studies,” in CCS, 2016. find
  4. M. Barni, P. Failla, R. Lazzeretti, A. Sadeghi, and T. Schneider, “Privacy-preserving ECG classification with branching programs and neural networks,” Trans. Info. Forensics and Security, vol. 6, no. 2, pp. 452– 468, 2011. find
  5. R. Bassily, A. Smith, and A. Thakurta, “Private empirical risk minimization: Efficient algorithms and tight error bounds,” in FOCS, 2014. find
  6. J. Bos, K. Lauter, and M. Naehrig, “Private predictive analysis on encrypted medical data,” J. Biomed. Informatics, vol. 50, pp. 234–243, 2014. find
  7. J. Calandrino, A. Kilzer, A. Narayanan, E. Felten, and V. Shmatikov, ““You might also like:” Privacy risks of collaborative filtering,” in S&P, 2011. find
  8. K. Chaudhuri and C. Monteleoni, “Privacy-preserving logistic regression,” in NIPS, 2009. find
  9. K. Chaudhuri, C. Monteleoni, and A. Sarwate, “Differentially private empirical risk minimization,” JMLR, vol. 12, pp. 1069–1109, 2011. find
  10. I. Diakonikolas, M. Hardt, and L. Schmidt, “Differentially private learning of structured discrete distributions,” in NIPS, 2015. find
  11. W. Du, Y. Han, and S. Chen, “Privacy-preserving multivariate statistical analysis: Linear regression and classification.” in SDM, 2004. find
  12. C. Dwork, “Differential privacy,” in Encyclopedia of Cryptography and Security. Springer, 2011, pp. 338–340. find
  13. C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in TCC, 2006. find
  14. C. Dwork and M. Naor, “On the difficulties of disclosure prevention in statistical databases or the case for differential privacy,” J. Privacy and Confidentiality, vol. 2, no. 1, pp. 93–107, 2010. find
  15. C. Dwork, A. Smith, T. Steinke, J. Ullman, and S. Vadhan, “Robust traceability from trace amounts,” in FOCS, 2015. find
  16. M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in CCS, 2015. find
  17. M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, “Privacy in pharmacogenetics: An end-to-end case study of personalized Warfarin dosing,” in USENIX Security, 2014. find
  18. M. Hardt, B. Recht, and Y. Singer, “Train faster, generalize better: Stability of stochastic gradient descent,” in ICML, 2016. find
  19. T. Hastie, R. Tibshirani, J. Friedman, and J. Franklin, “The elements of statistical learning: Data mining, inference and prediction,” The Mathematical Intelligencer, vol. 27, no. 2, pp. 83–85, 2005. find
  20. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv:1503.02531, 2015. find our card
  21. N. Homer, S. Szelinger, M. Redman, D. Duggan, W. Tembe, J. Muehling, J. V. Pearson, D. A. Stephan, S. F. Nelson, and D. W. Craig, “Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays,” PLoS Genetics, vol. 4, no. 8, 2008. find
  22. G. Jagannathan and R. Wright, “Privacy-preserving distributed k-means clustering over arbitrarily partitioned data,” in KDD, 2005. find
  23. P. Jain, V. Kulkarni, A. Thakurta, and O. Williams, “To drop or not to drop: Robustness, consistency and differential privacy properties of dropout,” arXiv:1503.02031, 2015. — [arXiv:1503.02031](https://arxiv.org/abs/1503.02031) find
  24. A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, University of Toronto, 2009. find
  25. M. J. Kusner, J. R. Gardner, R. Garnett, and K. Q. Weinberger, “Differentially private Bayesian optimization,” in ICML, 2015. find
  26. Y. Lindell and B. Pinkas, “Privacy preserving data mining,” in CRYPTO, 2000. find
  27. F. McSherry, “Statistical inference considered harmful,” https://github.com/frankmcsherry/blog/blob/master/posts/2016-06-14.md, 2016. ↗
  28. B. Rubinstein, P. Bartlett, L. Huang, and N. Taft, “Learning in a large function space: Privacy-preserving mechanisms for SVM learning,” J. Privacy and Confidentiality, vol. 4, no. 1, p. 4, 2012. find
  29. S. Sankararaman, G. Obozinski, M. I. Jordan, and E. Halperin, “Genomic privacy and limits of individual detection in a pool,” Nature Genetics, vol. 41, no. 9, pp. 965–967, 2009. find
  30. R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in CCS, 2015. find
  31. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” JMLR, vol. 15, no. 1, pp. 1929–1958, 2014. find
  32. F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction APIs,” in USENIX Security, 2016. find
  33. J. Vaidya, M. Kantarcıoğlu, and C. Clifton, “Privacy-preserving Naive Bayes classification,” VLDB, vol. 17, no. 4, pp. 879–898, 2008. find
  34. M. Wainwright, M. Jordan, and J. Duchi, “Privacy aware learning,” in NIPS, 2012. find
  35. P. Xie, M. Bilenko, T. Finley, R. Gilad-Bachrach, K. Lauter, and M. Naehrig, “Crypto-nets: Neural networks over encrypted data,” arXiv:1412.6181, 2014. — [arXiv:1412.6181](https://arxiv.org/abs/1412.6181) find
  36. D. Yang, D. Zhang, and B. Qu, “Participatory cultural mapping based on collective behavior data in location-based social networks,” ACM TIST, vol. 7, no. 3, p. 30, 2016. find
  37. J. Zhang, Z. Zhang, X. Xiao, Y. Yang, and M. Winslett, “Functional mechanism: Regression analysis under differential privacy,” VLDB, vol. 5, no. 11, pp. 1364–1375, 2012. find
  38. J. Zhu and T. Hastie, “Kernel logistic regression and the import vector machine,” in NIPS, 2001. find