Elamur Research
☰
home / papers / 2018 / The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1802.08232

Summary

Generative sequence models, such as language models behind auto-completion, are often trained on private text. They can retain rare or one-off fragments, like a card number, that have nothing to do with their task, and later reveal them. Carlini et al. offer a way to quantify this risk. They plant randomly generated "canary" sequences into the training data, varying how many times each appears. They then check how much more likely the trained model finds a planted canary than similar random sequences that were never inserted. This perplexity-based gap is called exposure. To confirm that high exposure means real danger, they build an extraction algorithm guided by exposure. It recovers secrets using only the model's output probabilities, and it succeeds in pulling credit card numbers out of a language model built on the Enron email data. Their experiments show that memorization appears early in training rather than only through overtraining. Dropout and early stopping do not prevent it. Only differentially private training removes it, at some cost in accuracy. The test is cheap to run and already supports privacy regression testing for Google's Smart Compose.

Key points

  • Exposure metric: Canaries are inserted into training data a controlled number of times. Exposure compares a canary's perplexity against non-inserted random sequences of the same form. The authors say it takes only a few dozen lines of code to implement.
  • Hyperparameters matter: Two word-level language models trained to nearly identical accuracy with different hyperparameters (A vs. B) memorized very differently. Under setting A, a canary that appeared 9 times could be extracted.
  • Real extraction: Credit card numbers were pulled from a model trained on the Enron emails. The attacker only had black-box access to the model's output probabilities (logits).
  • Defenses: Memorization occurs even when secrets are rare and the model is far smaller than its corpus. Early stopping and dropout fail to stop it, while differentially private SGD eliminates it with some utility loss.
  • Scope:
    • The models studied are LSTMs and qRNNs.
    • The method is used in Google's Smart Compose, which is trained on millions of users' emails.
    • The authors note that the extraction attack assumes the attacker knows the context and format of the secret.
    • Extending the approach to non-generative models, such as image classifiers, is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

163 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 149 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1802.08232 · arXiv 1802.08232

@misc{carlini2018secret,
  title = {{The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks}},
  author = {Carlini, Nicholas and Liu, Chang and Erlingsson, Ulfar and Kos, Jernej and Song, Dawn},
  year = {2018},
  doi = {10.48550/arxiv.1802.08232},
  eprint = {1802.08232},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.08232}
}

References

57 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM CCS, 2016. find
  2. Giuseppe Ateniese, Luigi V Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International Journal of Security and Networks, 2015. find
  3. D Bahdanau, K Cho, and Y Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015. find our card
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. JMLR, 2003. find
  5. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. arXiv preprint arXiv:1611.01576, 2016. — [arXiv:1611.01576](https://arxiv.org/abs/1611.01576) find
  6. Lord Castleton. Review: Amazon’s ‘Patriot’ is the best show of the year. 2017. Pa-jiba. http://www.pajiba.com/tv\_reviews/review-amazons-patriot-is-the-best-show-of\\ -the-year.php. ↗
  7. Kamalika Chaudhuri and Claire Monteleoni. Privacy-preserving logistic regression. In NIPS, 2009. find
  8. Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, et al. Gmail smart compose: Real-time assisted writing. arXiv preprint arXiv:1906.00080, 2019. — [arXiv:1906.00080](https://arxiv.org/abs/1906.00080) find
  9. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS Workshop, 2014. find
  10. Joseph Conrad. The Secret Sharer. EBook \#220. Project Gutenberg, 2009. Originally published in Harper’s Magazine, 1910. find
  11. T Cormen, C Leiserson, R Rivest, and C Stein. Introduction to Algorithms. MIT Press, 2009. find
  12. TensorFlow Developers. chine translation tutorial. tensorflow/nmt, 2017. Tensorflow neural ma-https://github.com/ find
  13. Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems. ACM, 2003. find
  14. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011. find
  15. C Dwork, F McSherry, K Nissim, and A Smith. Calibrating noise to sensitivity in private data analysis. In TCC, volume 3876, 2006. find
  16. Cynthia Dwork. Differential privacy: A survey of results. In Intl. Conf. on Theory and Applications of Models of Computation, 2008. find
  17. Úlfar Erlingsson, Vitaly Feldman, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Abhradeep Thakurta. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2468–2479. SIAM, 201… find
  18. Vitaly Feldman, Ilya Mironov, Kunal Talwar, and Abhradeep Thakurta. Privacy amplification by iteration. In IEEE FOCS, 2018. find
  19. Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM CCS, 2015. find
  20. Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study of personalized Warfarin dosing. In USENIX Security Symposium, pages 17–32, 2014. find
  21. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016. find
  22. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. find
  23. Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. LOGAN: Evaluating privacy leakage of generative models using generative adversarial networks. PETS, 2018. find
  24. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  25. Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741, 2017. — [arXiv:1705.08741](https://arxiv.org/abs/1705.08741) find
  26. I Hubara, M Courbariaux, D Soudry, R El-Yaniv, and Y Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint arXiv:1609.07061, 2016. find
  27. Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In USENIX Security Symposium, 2019. find
  28. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. find
  29. Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In NIPS, pages 950–957, 1992. find
  30. Yunhui Long, Vincent Bindschaedler, and Carl A Gunter. Towards measuring membership privacy. arXiv preprint 1712.09136, 2017. — [arXiv:1712.09136](https://arxiv.org/abs/1712.09136) find
  31. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2):313–330, 1993. find our card
  32. Frank J Massey Jr. The Kolmogorov-Smirnov test for goodness of fit. Journal of the American statistical Association, 46(253), 1951. find
  33. Joseph Menn. Amazon posts a tell-all of buying lists. 1999. Los Angeles Times. https://www.latimes.com/archives/la-xpm-1999-aug-26-fi-3760-story.html. ↗
  34. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182, 2017. find our card
  35. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv:1803.08240, 2018. find
  36. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, page 3, 2010. find
  37. Randall Munroe. Predictive models. https://xkcd. com/2169/, 2019. find
  38. Milad Nasr, Reza Shokri, and Amir Houmansadr. Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 634–646. ACM, 2018. find
  39. Seong Joon Oh, Max Augustin, Mario Fritz, and Bernt Schiele. Towards reverse-engineering black-box neural networks. In ICLR, 2018. find
  40. A O’hagan and Tom Leonard. Bayes estimation subject to uncertainty about parameter constraints. Biometrika, 63(1), 1976. find
  41. Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy-preserving deep learning: Revisited and enhanced. In International Conference on Applications and Techniques in Information Security, pages 100–110. Springer, 2017. find
  42. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, 2017. find our card
  43. Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint 1703.00810, 2017. — [arXiv:1703.00810](https://arxiv.org/abs/1703.00810) find
  44. Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In ACM CCS, 2017. find
  45. Congzheng Song and Vitaly Shmatikov. The natural auditor: How to tell if someone used your words to train their model. arXiv preprint arXiv:1811.00513, 2018. — [arXiv:1811.00513](https://arxiv.org/abs/1811.00513) find
  46. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958, 2014. find
  47. Igor V. Tetko, David J. Livingstone, and Alexander I. Luik. Neural network studies. 1. Comparison of overfitting and overtraining. Journal of Chemical Information and Computer Sciences, 35(5):826–833, 1995. find
  48. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31, 2012. find
  49. Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction APIs. In USENIX Security Symposium, pages 601–618, 2016. find
  50. Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Towards demystifying membership inference attacks. arXiv preprint arXiv:1807.09173, 2018. — [arXiv:1807.09173](https://arxiv.org/abs/1807.09173) find
  51. Binghui Wang and Neil Zhenqiang Gong. Stealing hyperparameters in machine learning. arXiv preprint arXiv:1802.05351, 2018. — [arXiv:1802.05351](https://arxiv.org/abs/1802.05351) find
  52. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  53. Yuanshun Yao, Bimal Viswanath, Jenna Cryan, Haitao Zheng, and Ben Y Zhao. Automated crowdturfing attacks and defenses in online review systems. ACM CCS, 2017. find
  54. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268–282. IEEE, 2018. find
  55. Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for ImageNet training. arXiv preprint arXiv:1708.03888, 2017. — [arXiv:1708.03888](https://arxiv.org/abs/1708.03888) find
  56. Matthew D Zeiler. ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012. find our card
  57. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017. find