Elamur Research⌕
☰
home / papers / 2017 / Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Paper · 2017 · Samuel Yeom, Irene Giacomelli, Matt Fredrikson et al.

Summary

Models trained on sensitive records can expose facts about the people in their training sets. Before this work, the causes of that leakage were explained mostly through scattered anecdotes. Yeom, Giacomelli, Fredrikson and Jha study two black-box threats. Membership inference asks whether a given record was used for training. Attribute inference tries to recover a hidden feature of a training record. The authors connect both threats to overfitting, measured as generalization error, and to the influence of the target feature on the model's output. They define new advantage measures that compare an attacker's success on training points with its success on points drawn from the general population. They derive analytic bounds and test them on real datasets by varying regularization and model structure, including deep CNNs. Overfitting proves sufficient for membership leakage but not necessary, because a subverted training algorithm can remain stable and still encode exact membership. Attribute risk depends on how overfitting and influence balance each other. The results clarify when ordinary models put training data at risk. They also show that good generalization alone does not guarantee privacy.

Key points

  • Membership advantage grows with generalization error and is directly proportional to it in some settings (Theorem 2). A simple threshold attack on CNNs comes close to the shadow-model attack of Shokri et al. while using much less computation and background knowledge.
  • Theorem 4 describes a colluding pair of training algorithm and adversary. The algorithm stays nearly stable yet reveals exact membership at little cost to accuracy. The authors implemented this on deep CNNs (Section 6.4) and relate it to algorithm substitution attacks from cryptography.
  • For attribute inference, influence is decisive. As the target feature's influence rises, the attacker's extra advantage on training data over the population disappears, however large the generalization error is. The authors argue that the earlier, simulator-based notion of advantage (Definition 6) does not isolate the risk to training data.
  • The authors build reductions in both directions. Any attacker with attribute advantage can be turned into one with membership advantage (Theorem 6), though it remains unknown whether the reverse holds. A reduction-based attribute attack clearly outperforms the attack of Fredrikson et al. when generalization error is high.
  • The work comes from Carnegie Mellon University and the University of Wisconsin–Madison. The full version (arXiv:1709.01604) accompanies the CSF 2018 paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of Wisconsin–MadisonMadison · United States42 works · 48,486 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Machine learning · 51 works

Cite

Cite as Samuel Yeom, Irene Giacomelli, Matt Fredrikson and Somesh Jha. 2017. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. arXiv:1709.01604.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.01604 · arXiv 1709.01604

@misc{yeom2017privacy,
  title = {{Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting}},
  author = {Samuel Yeom, Irene Giacomelli, Matt Fredrikson and Somesh Jha},
  year = {2017},
  doi = {10.48550/arxiv.1709.01604},
  eprint = {1709.01604},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.01604}
}

References

54 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Ateniese, L. V. Mancini, A. Spognardi, A. Villani, D. Vitali, and G. Felici, “Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers,” International Journal of Security and Networks, vol. 10, pp. 137–150, Sept. 2015. find
  2. G. Cormode, “Personal privacy vs population privacy: Learning to attack anonymization,” in KDD, 2011. find
  3. M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in ACM Conference on Computer and Communications Security (CCS), 2015. find
  4. M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, “Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing,” in USENIX Security Symposium, pp. 17–32, 2014. find
  5. X. Wu, M. Fredrikson, S. Jha, and J. F. Naughton, “A methodology for formalizing model-inversion attacks,” in 2016 IEEE Computer Security Foundations Symposium (CSF), 2016. find
  6. N. Li, W. Qardaji, D. Su, Y. Wu, and W. Yang, “Membership privacy: A unifying framework for privacy definitions,” in Proceedings of ACM CCS, 2013. find
  7. R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE Symposium on Security and Privacy (Oakland), pp. 3–18, 2017.
  8. X. Wu, M. Fredrikson, W. Wu, S. Jha, and J. F. Naughton, “Revisiting Differentially Private Regression: Lessons From Learning Theory and their Consequences,” CoRR, vol. abs/1512.06388, 2015. — [arXiv:1512.06388](https://arxiv.org/abs/1512.06388) find
  9. J. Brickell and V. Shmatikov, “The cost of privacy: destruction of data-mining utility in anonymized data publishing,” in KDD, 2008. find
  10. J. Lei, “Differentially private m-estimators,” in NIPS, 2011. find
  11. J. Zhang, Z. Zhang, X. Xiao, Y. Yang, and M. Winslett, “Functional mechanism: regression analysis under differential privacy,” in VLDB, 2012. find
  12. A. G. Thakurta and A. Smith, “Differentially private feature selection via stability arguments, and the robustness of the lasso,” in Proceedings of the 26th Annual Conference on Learning Theory, vol. 30 of Proceedings of Machine Learning Research, pp. 819–850, PMLR, 12–14 Jun 2013. find
  13. C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. L. Roth, “Preserving statistical validity in adaptive data analysis,” in Proceedings of the Forty-seventh Annual ACM Symposium on Theory of Computing, STOC ’15, (New York, NY, USA), pp. 117–126, ACM, 2015. find
  14. C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth, “Generalization in adaptive data analysis and holdout reuse,” in Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS’15, (Cambridge, MA, USA), pp. 2350–2358, MIT Press, 2015. find
  15. Y.-X. Wang, J. Lei, and S. E. Fienberg, “Learning with differential privacy: Stability, learnability and the sufficiency and necessity of ERM principle,” Journal of Machine Learning Research, vol. 17, no. 183, pp. 1–40, 2016. find
  16. R. Bassily, A. Smith, and A. Thakurta, “Private empirical risk minimization: Efficient algorithms and tight error bounds,” in In 55th IEEE Annual Symposium on Foundations of Computer Science (FOCS), 2014. find
  17. K. Chaudhuri, C. Monteleoni, and A. D. Sarwate, “Differentially private empirical risk minimization,” Journal of Machine Learning Research, 2011. find
  18. R. Bassily, K. Nissim, A. Smith, T. Steinke, U. Stemmer, and J. Ullman, “Algorithmic stability for adaptive data analysis,” in Proceedings of the 48th Annual ACM Symposium on Theory of Computing, pp. 1046–1059, 2016. find
  19. Y.-X. Wang, J. Lei, and S. E. Fienberg, “On-average KL-privacy and its equivalence to generalization for max-entropy mechanisms,” in International Conference on Privacy in Statistical Databases, pp. 121– 134, 2016. find
  20. R. O’Donnell, Analysis of Boolean Functions. Cambridge University Press, 2014. find
  21. N. Homer, S. Szelinger, M. Redman, D. Duggan, W. Tembe, J. Muehling, J. V. Pearson, D. A. Stephan, S. F. Nelson, and D. W. Craig, “Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays,” PLoS Genetics, vol. 4, no. 8, 2008. find
  22. S. Sankararaman, G. Obozinski, M. I. Jordan, and E. Halperin, “Genomic privacy and limits of individual detection in a pool,” Nature Genetics, vol. 41, no. 9, pp. 965–967, 2009. find
  23. M. Bellare, K. G. Paterson, and P. Rogaway, “Security of symmetric encryption against mass surveillance,” in Advances in Cryptology - CRYPTO 2014 - 34th Annual Cryptology Conference, Santa Barbara, CA, USA, August 17-21, 2014, Proceedings, Part I, pp. 1–19, 2014. find
  24. C. Song, T. Ristenpart, and V. Shmatikov, “Machine learning models that remember too much,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017, pp. 587–601, 2017. find
  25. K. P. Murphy, Machine Learning: A Probabilistic Perspective. The MIT Press, 2012. find
  26. C. Dwork, “Differential privacy,” in ICALP, Springer, 2006. find
  27. S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan, “Learnability, stability and uniform convergence,” Jouranal of Machine Learing. Research, vol. 11, Dec. 2010. find
  28. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” CoRR, vol. abs/1611.03530, 2016. find
  29. I. Giacomelli, R. F. Olimid, and S. Ranellucci, “Security of linear secret-sharing schemes against mass surveillance,” in Cryptology and Network Security - 14th International Conference, CANS 2015, Marrakesh, Morocco, December 10-12, 2015, Proceedings, pp. 43–58, 2015. find
  30. G. Ateniese, B. Magri, and D. Venturi, “Subversion-resilient signature schemes,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, October 12-6, 2015, pp. 364–375, 2015. find
  31. M. Bellare, J. Jaeger, and D. Kane, “Mass-surveillance without the state: Strongly undetectable algorithm-substitution attacks,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, October 12-6, 2015, pp. 1431–1440, 2015. find
  32. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, p… find
  33. O. Bousquet and A. Elisseeff, “Stability and generalization,” Journal of Machine Learning Research, vol. 2, pp. 499–526, 2002. find
  34. T. E. Scheetz, K.-Y. A. Kim, R. E. Swiderski, A. R. Philp, T. A. Braun, K. L. Knudtson, A. M. Dorrance, G. F. DiBona, J. Huang, T. L. Casavant, V. C. Sheffield, and E. M. Stone, “Regulation of gene expression in the mammalian eye and its relevance to eye disease,” Proceedings of the National Academy… find
  35. International Warfarin Pharmacogenetics Consortium, “Estimation of the warfarin dose with clinical and pharmacogenetic data,” New England Journal of Medicine, vol. 360, no. 8, pp. 753–764, 2009. find
  36. Netflix, “Netflix prize.” http://netflixprize.com, 2006. ↗
  37. F. Chollet, “Keras: Deep learning library for Theano and TensorFlow.” https://keras.io, 2017. ↗
  38. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. deeplearningbook.org. find
  39. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” CoRR, vol. abs/1409.1556, 2014.
  40. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference for Learning Representations (ICLR), 2015.
  41. Y. LeCun, C. Cortes, and C. Burges, “The MNIST database of handwritten digits.” http://yann.lecun.com/exdb/mnist/, 1998. ↗
  42. A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009. find
  43. B. Neuberg, “Personal photos model.” https://github.com/BradNeuberg/personal-photos-model, 2017. ↗
  44. P. Krähenbühl, C. Doersch, J. Donahue, and T. Darrell, “Data-dependent initializations of convolutional neural networks,” CoRR, vol. abs/1511.06856, 2015. find
  45. T. Komarova, D. Nekipelov, and E. Yakovlev, “Estimation of treatment effects from combined data: Identification versus data security,” in Economic Analysis of the Digital Economy, pp. 279–308, University of Chicago Press, 2015. find
  46. R. Wang, Y. F. Li, X. Wang, H. Tang, and X. Zhou, “Learning your identity and disease from research papers: information leaks in genome wide association studies,” in CCS, 2009. find
  47. K. El Emam, E. Jonker, L. Arbuckle, and B. Malin, “A systematic review of re-identification attacks on health data,” PLOS ONE, vol. 6, pp. 1–12, 12 2011. find
  48. M. Gymrek, A. L. McGuire, D. Golan, E. Halperin, and Y. Erlich, “Identifying personal genomes by surname inference,” Science, vol. 339, no. 6117, pp. 321–324, 2013. MIT Press, 2016. http://www. find
  49. S. S. Shringarpure and C. D. Bustamante, “Privacy risks from genomic data-sharing beacons,” The American Journal of Human Genetics, vol. 97, pp. 631–646, May 2015. find
  50. J. A. Calandrino, A. Kilzer, A. Narayanan, E. W. Felten, and V. Shmatikov, ““You might also like:” Privacy risks of collaborative filtering,” in Proceedings of the 2011 IEEE Symposium on Security and Privacy (Oakland), 2011. find
  51. I. Dinur and K. Nissim, “Revealing information while preserving privacy,” in PODS, 2003. find
  52. C. Dwork, F. McSherry, and K. Talwar, “The price of privacy and the limits of LP decoding,” in STOC, 2007. find
  53. S. P. Kasiviswanathan, M. Rudelson, A. Smith, and J. Ullman, “The price of privately releasing contingency tables and the spectra of random matrices with correlated rows,” in STOC, 2010. find
  54. S. P. Kasiviswanathan, M. Rudelson, and A. Smith, “The power of linear reconstruction attacks,” in SODA, 2013. Training Set Error (Netflix, Ridge regression with λ = 5) 0.2 Experimental Gaussian 0.15 frequency 0.1 0.05 0 −1.5 −1 −0.5 0 error 0.5 1.5 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yeom-2017-privacy-risk-machine-learning-analyzing/