Elamur Research
☰
home / papers / 2022 / Deduplicating Training Data Mitigates Privacy Risks in Language Models

Deduplicating Training Data Mitigates Privacy Risks in Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2202.06539

Summary

Large language models can leak their training data. In the attack of Carlini et al. (2021b), an adversary samples large amounts of text from a model and ranks the outputs with membership inference scores to find likely training sequences. Kandpal, Wallace and Raffel ask how much of this attack's success comes from repeated content in web-scraped corpora. They adapt the suffix-array exact-matching tools of Lee et al. (2021) to count how often generated sequences occur in OpenWebText and C4. They then study Transformer LMs from 117M to 1.5B parameters, including a pair of 1.5B models trained on original versus sequence-level deduplicated C4. Three findings stand out:

  • The rate at which a model reproduces a sequence grows faster than linearly with how often that sequence appears in training.
  • Membership inference is barely better than guessing for sequences that appear only once.
  • Deduplication sharply reduces leakage.

The work positions deduplication as an inexpensive privacy safeguard. It also argues that attack evaluations should treat duplication as a confounding factor.

Key points

  • Superlinear regeneration: a sequence duplicated 10 times is regenerated roughly 1000× more often than a unique one. This trend holds under standard, top-k and temperature sampling and for match lengths from 100 to 700 characters.
  • Detection depends on duplication: the three membership inference scores tested were Reference Model (GPT-2 small), zlib and Lowercase. They reach AUROC up to 0.90 on heavily duplicated sequences but perform near chance on singletons.
  • Effect of deduplication: the model trained on deduplicated C4 emits about 20× less training data, and two of the three scoring methods become less effective against it.
  • Corpora: OpenWebText (39GB) and C4 (750GB) are deduplicated only at the document level. Both still contain millions of 400-character sequences repeated 10+ times, and some sequences recur tens of thousands of times.
  • Models: the study uses Mistral 117M and 345M, the 1.5B model of West et al. (2021) (all trained on OpenWebText), and the 1.5B C4 / deduplicated-C4 models from Lee et al. (2021). The paper was published at ICML 2022.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 111 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2202.06539 · arXiv 2202.06539

@misc{kandpal2022deduplicating,
  title = {{Deduplicating Training Data Mitigates Privacy Risks in Language Models}},
  author = {Kandpal, Nikhil and Wallace, Eric and Raffel, Colin},
  year = {2022},
  doi = {10.48550/arxiv.2202.06539},
  eprint = {2202.06539},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2202.06539}
}

References

42 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, G., Bun, M., Feldman, V., Smith, A., and Talwar, K. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing. ACM, 2021. find
  2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… find
  3. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2019. find our card
  4. Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. arXiv preprint arXiv:2112.03570, 2021a. find
  5. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C. Extracting training data from large language models. In USENIX Security Symposium, 2021b. find our card
  6. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. find
  7. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. find
  8. Coavoux, M., Narayan, S., and Cohen, S. B. Privacy-preserving neural representations of text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. find
  9. Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In TCC, 2006. find
  10. Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In ACL, 2018. find
  11. Feldman, V. and Zhang, C. What neural networks memorize and why: Discovering the long tail via influence estimation. In NeurIPS, 2020. find
  12. Fredrikson, M., Jha, S., and Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM CCS, 2015. find
  13. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. find
  14. Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Open-WebText corpus, 2019. find
  15. Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Henighan, T., Hume, T., Johnston, S., Mann, B., Olah, C., Olsson, C., Amodei, D., Joseph, N., Kaplan, J., and McCandlish, S. Scaling laws and interpretability of learning from repeated data.… find
  16. Hidano, S., Murakami, T., Katsumata, S., Kiyomoto, S., and Hanaoka, G. Model inversion attacks for prediction systems: Without knowledge of non-sensitive attributes. In 2017 15th Annual Conference on Privacy, Security and Trust (PST), 2017. find
  17. Inan, H. A., Ramadan, O., Wutschitz, L., Jones, D., Rühle, V., Withers, J., and Sim, R. Training data leakage analysis in language models. In Privacy Preserving Machine Learning Workshop, 2021. find
  18. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. find
  19. Lehman, E., Jain, S., Pichotta, K., Goldberg, Y., and Wallace, B. Does BERT pretrained on clinical notes reveal sensitive data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021. find
  20. Li, X., Tramèr, F., Liang, P., and Hashimoto, T. Large language models can be strong differentially private learners. In ICLR, 2022. find
  21. Li, Y., Baldwin, T., and Cohn, T. Towards robust and privacy-preserving text representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018. find
  22. Long, Y., Bindschaedler, V., Wang, L., Bu, D., Wang, X., Tang, H., Gunter, C. A., and Chen, K. Understanding membership inferences on well-generalized learning models. arXiv preprint arXiv:1802.04889, 2018. — [arXiv:1802.04889](https://arxiv.org/abs/1802.04889) find
  23. McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., and Celikyilmaz, A. How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN. arXiv preprint arXiv:2111.09509, 2021. find
  24. Mireshghallah, F., Inan, H., Hasegawa, M., Rühle, V., Berg-Kirkpatrick, T., and Sim, R. Privacy regularization: Joint privacy-utility optimization in LanguageModels. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  25. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC… find our card
  26. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. find
  27. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR, 2020. find
  28. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv preprint arXiv:1806.00451, 2018. — [arXiv:1806.00451](https://arxiv.org/abs/1806.00451) find
  29. Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. find
  30. Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. In IEEE S&P, 2017. find our card
  31. Song, C. and Raghunathan, A. Information Leakage in Embedding Models, pp. 377–390. Association for Computing Machinery, 2020. find
  32. Song, C. and Shmatikov, V. Auditing data provenance in text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery I& Data Mining. Association for Computing Machinery, 2019. find
  33. Van den Burg, G. and Williams, C. On memorization in probabilistic deep generative models. In NeurIPS, 2021. find
  34. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In NIPS, 2017. find
  35. Watson, L., Guo, C., Cormode, G., and Sablayrolles, A. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440, 2021. — [arXiv:2111.08440](https://arxiv.org/abs/2111.08440) find
  36. West, P., Lu, X., Holtzman, A., Bhagavatula, C., Hwang, J., and Choi, Y. Reflective decoding: Beyond unidirectional generation with off-the-shelf language models. In ACL, 2021. find
  37. Yang, Z., Zhang, J., Chang, E.-C., and Liang, Z. Neural network inversion in adversarial setting via background knowledge alignment. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, 2019. find
  38. Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE CSF, 2018. find
  39. Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500, 2021. — [arXiv:2110.06500](https://arxiv.org/abs/2110.06500) find
  40. Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramèr, F., and Carlini, N. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938, 2021. — [arXiv:2112.12938](https://arxiv.org/abs/2112.12938) find
  41. Zhao, X., Li, L., and Wang, Y.-X. Provably confidential language modelling. arXiv preprint arXiv:2205.01863, 2022. find
  42. Ziegler, A. A first look at rote learning in GitHub Copilot suggestions, June 2021. find