Elamur Research⌕
☰
home / papers / 2023 / Privacy Issues in Large Language Models: A Survey

Privacy Issues in Large Language Models: A Survey

Paper · 2023 · Seth Neel and Peter Chang

Summary

This survey maps the technical research on privacy risks in Large Language Models. It deliberately leaves legal and policy scholarship aside, though it uses regulation to frame the problems: the US White House Executive Order on AI, the FTC's July 2023 investigation of OpenAI, and the GDPR "Right to Erasure."

The review is organized around five themes:

  • how and why LLMs memorize training data;
  • attacks that exploit memorization (membership inference, training data extraction, attribute inference);
  • privacy-preserving training through differential privacy and federated learning;
  • algorithmic responses to copyright concerns;
  • machine unlearning, meaning removing specific data from a model after training.

The authors conclude that LLMs clearly memorize large amounts of data, yet current attacks on them succeed less often than attacks on discriminative models. DP fine-tuning looks promising, but private pre-training remains essentially open. Unlearning methods are mostly heuristic and hard to verify.

The survey is useful as an entry point for teams training or fine-tuning models on personal or copyrighted text. A companion repository lists the covered papers and code.

Key points

  • The paper list and available code are at github.com/safr-ml-lab/survey-llm, which the authors plan to keep updated.
  • The survey identifies five risk factors for memorization: model capacity, dataset size, data duplication, prompt length and type, and the point during training when data is seen.
  • Chen and Yang [2023] adapt SCRUB to T5 by retraining only about 0.5% of parameters. They test on IMDB and SAMSum, forgetting 0.5%, 1% and 10% of the data.
  • Hayes et al. [2024] built U-LiRA, a per-example membership attack. Against PaLM 2 fine-tuned on LIMA and unlearned with NegGrad or GradDesc, it identified the unlearned model in over 80% of cases.
  • Zanella-Béguelin et al. [2020] show that model updates, including unlearning, can themselves leak membership information. This happens when an attacker can compare model versions before and after the update.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvard UniversityCambridge · United States137 works · 45,850 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 worksSociology and Political Science · 630 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Seth Neel and Peter Chang. 2023. Privacy Issues in Large Language Models: A Survey. arXiv:2312.06717.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.06717 · arXiv 2312.06717

@misc{neel2023privacy,
  title = {{Privacy Issues in Large Language Models: A Survey}},
  author = {Seth Neel and Peter Chang},
  year = {2023},
  doi = {10.48550/arxiv.2312.06717},
  eprint = {2312.06717},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.06717}
}

References

147 references, 41 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demsz… — On the Opportunities and Risks of Foundation Models
  2. Oct 2023. URL https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/ executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/. ↗
  3. Nicola Lucchi. Chatgpt: A case study on copyright challenges for generative artificial intelligence systems. European Journal of Risk Regulation, page 1–23, 2023. doi:10.1017/err.2023.59. — [doi:10.1017/err.2023.59](https://doi.org/10.1017/err.2023.59) find
  4. Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security Review, 34(2):304–313, 2018. ISSN 0267-3649. doi:https://doi.org/10.1016/j.clsr.2017.08.007. URL https://www.sciencedirect.com/sc… — [doi:10.1016/j.clsr.2017.08.007](https://doi.org/10.1016/j.clsr.2017.08.007) ↗
  5. Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning, 2022. find
  6. Antonio Gulli, 2004. URL http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html. ↗
  7. Eva Sharma, Chen Li, and Lu Wang. Bigpatent: A large-scale dataset for abstractive and coherent summarization, 2019.
  8. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, 2015.
  9. W. N. Francis and H. Kucera. Brown corpus manual. Technical report, Department of Linguistics, Brown University, Providence, Rhode Island, US, 1979. URL http://icame.uib.no/brown/bcm.html. ↗
  10. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend, 2015.
  11. Common Crawl. Common crawl, 2008. URL https://commoncrawl.org/. ↗
  12. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021. find
  13. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer, 2021.
  14. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Br…
  15. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc., 2022.… — Pointer Sentinel Mixture Models
  16. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia, editors, Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada, August 2017. Asso…
  17. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks, 2019.
  18. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2020. URL https://…
  19. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2023a. find
  20. Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, page 954–959, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369794. doi:10.1145/3357713.3384290. U… — [doi:10.1145/3357713.3384290](https://doi.org/10.1145/3357713.3384290) find
  21. Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021, page 123–132, New York, NY, USA, 2021. Association for… — [doi:10.1145/3406325.3451131](https://doi.org/10.1145/3406325.3451131) find
  22. Griffin Davis. Chatgpt allegedly trained on copyrighted books!, May 2023a. URL https://www.techtimes.com/articles/291210/20230505/ chatgpt-allegedly-trained-copyrighted-books-new-study-claims-memorized-harry.htm. ↗
  23. Melissa Heikkila. What does gpt-3 “know” about me?, Sep 2022a. URL https://www.technologyreview.com/ 2022/08/31/1058800/what-does-gpt-3-know-about-me/. ↗
  24. SCRT Labs. The secret behind large language models: Memorization over understanding?, Apr 2023. URL https://www.linkedin.com/pulse/ secret-behind-large-language-models-memorization-over-understanding/. ↗
  25. Cecilia Kang and Cade Metz. F.t.c. opens investigation into chatgpt maker over technology’s potential harms, Jul 2023. URL https://www.nytimes.com/2023/07/13/technology/chatgpt-investigation-ftc-openai. html. ↗
  26. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find
  27. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy, 2023. find
  28. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models, 2022.
  29. Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models, 2022.
  30. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/23…
  31. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023.
  32. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models, 2021. find
  33. Zhenhong Zhou, Jiuyang Xiang, Chaomeng Chen, and Sen Su. Quantifying and analyzing entity-level memorization in large language models, 2023b. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editor… find
  34. May 2021. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, L…
  35. CRFM Stanford. Mistral, 2021. URL https://github.com/stanford-crfm/mistral. Peter West, Ximing Lu, Ari Holtzman, Chandra Bhagavatula, Jena D. Hwang, and Yejin Choi. Reflective decoding: Beyond unidirectional generation with off-the-shelf language models. In Proceedings of the 59th Annual Meeting of… ↗
  36. Julien Simon. Large language models: A new moore’s law?, 2021. URL https://huggingface.co/blog/ large-language-models. ↗
  37. Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models, 2023a.
  38. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and… find
  39. Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile, 2022. find
  40. R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven, 2021.
  41. Eugene Kharitonov, Marco Baroni, and Dieuwke Hupkes. How bpe affects memorization in transformers, 2021. find
  42. Vitaly Feldman, Ilya Mironov, Kunal Talwar, and Abhradeep Thakurta. Privacy amplification by iteration. In Mikkel Thorup, editor, 59th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2018, Paris, France, October 7-9, 2018, pages 521–532. find
  43. IEEE Computer Society, 2018. doi:10.1109/FOCS.2018.00056. URL https://doi.org/10.1109/FOCS.2018.00056. — [doi:10.1109/focs.2018.00056](https://doi.org/10.1109/focs.2018.00056) ↗
  44. Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Provably confidential language modelling, 2022. find
  45. H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data, 2016. find
  46. Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning, 2020. find
  47. Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H. Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data, 2020. find
  48. Ting-Yun Chang, Jesse Thomason, and Robin Jia. Do localization methods actually localize memorized data in llms?, 2023. find
  49. Vikas Raunak and Arul Menezes. Finding memo: Extractive memorization in constrained sequence generation tasks, 2022. find
  50. Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership inference attacks on machine learning: A survey. ACM Comput. Surv., 54(11s), sep 2022. ISSN 0360-0300. doi:10.1145/3523273. URL https://doi.org/10.1145/3523273. — [doi:10.1145/3523273](https://doi.org/10.1145/3523273) ↗
  51. Bo Liu, Ming Ding, Sina Shaham, Wenny Rahayu, Farhad Farokhi, and Zihuai Lin. When machine learning meets privacy: A survey and outlook. ACM Comput. Surv., 54(2), mar 2021. ISSN 0360-0300. doi:10.1145/3436755. URL https://doi.org/10.1145/3436755. — [doi:10.1145/3436755](https://doi.org/10.1145/3436755) ↗
  52. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting, 2018.
  53. Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11330–11343, To…
  54. Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. Quantifying the privacy risks of learning high-dimensional graphical models, 2021. find
  55. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models, 2017.
  56. Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A. Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 521–534, 2020. doi:10.1109/EuroSP48549.2… — [doi:10.1109/eurosp48549.2020.00040](https://doi.org/10.1109/eurosp48549.2020.00040) find
  57. Alexandre Sablayrolles, Matthijs Douze, Yann Ollivier, Cordelia Schmid, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference, 2019. find
  58. Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models, 2020. find
  59. Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles, 2022.
  60. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does bert pretrained on clinical notes reveal sensitive data?, 2021. Congzheng Song and Ananth Raghunathan. Information leakage in embedding models, 2020. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.…
  61. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA, June 2019. Association for Comp… — [doi:10.18653/v1/w19-1909](https://doi.org/10.18653/v1/w19-1909)
  62. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find
  63. Springer, 2006. Zachary Izzo, Jinsung Yoon, Sercan O. Arik, and James Zou. Provable membership inference privacy, 2022. Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Pro… — [doi:10.1109/globalsip.2013.6736861](https://doi.org/10.1109/globalsip.2013.6736861) find
  64. Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson. Tempered sigmoid activations for deep learning with differential privacy. CoRR, abs/2007.14191, 2020. URL https://arxiv.org/abs/2007. 14191. Jacob Poushter. Smartphone ownership and internet usage continues to clim… ↗
  65. Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, and Yossi Matias. Learning and evaluating a differentially private pre-trained language model. In Findings of the Association for Computational Linguistic… find
  66. Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556, 06 2011. ISSN 1067-5027. doi:10.1136/amiajnl-2011-000203. URL https://doi.org/10.1136… — [doi:10.1136/amiajnl-2011-000203](https://doi.org/10.1136/amiajnl-2011-000203) find
  67. Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. Selective differential privacy for language modeling, 2022. find
  68. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units, 2016.
  69. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109–11…
  70. Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training. In Proceedings of the Second Workshop on Privacy in NLP, pages 39–45, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.privatenlp-1.5. URL https… find
  71. Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners, 2022. find
  72. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation, 2017.
  73. Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping, 2020. find
  74. Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models, 2022. find
  75. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Bruss…
  76. Christophe Dupuy, Radhika Arava, Rahul Gupta, and Anna Rumshisky. An efficient dp-sgd mechanism for large scale nlp models, 2022. find
  77. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  78. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  79. Yiming Zhang and Daphne Ippolito. Prompts should not be seen as secrets: Systematically measuring prompt extraction attack success, 2023. find
  80. Xinyu Tang, Richard Shin, Huseyin A. Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with differentially private few-shot generation, 2023. find
  81. Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models, 2023.
  82. Antonio Ginart, Laurens van der Maaten, James Zou, and Chuan Guo. Submix: Practical private prediction for large-scale language models, 2022. find
  83. Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with pate, 2018. find
  84. Jimit Majmudar, Christophe Dupuy, Charith Peris, Sami Smaili, Rahul Gupta, and Richard Zemel. Differentially private decoding in large language models, 2022. find
  85. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. find
  86. H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models, 2018. find
  87. Rami Al-Rfou, Marc Pickett, Javier Snaider, Yun hsuan Sung, Brian Strope, and Ray Kurzweil. Conversational contextual cues: The case of personalization and history for response ranking, 2016. find
  88. Haşim Sak, Andrew Senior, and Françoise Beaufays. Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition, 2014. find
  89. Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately?, 2010. find
  90. Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork. Natural language understanding with privacy-preserving bert. In Proceedings of the 30th ACM International Conference on Information; Knowledge Management, CIKM ’21. ACM, October 2021. doi:10.1145/3459637.3482281. URL… — [doi:10.1145/3459637.3482281](https://doi.org/10.1145/3459637.3482281) find
  91. Minxin Du, Xiang Yue, Sherman S. M. Chow, and Huan Sun. Sanitizing sentence embeddings (and labels) for local differential privacy. Proceedings of the ACM Web Conference 2023, 2023a. URL https://api.semanticscholar.org/CorpusID:258333822. ↗
  92. Mário Alvim, Konstantinos Chatzikokolakis, Catuscia Palamidessi, and Anna Pazii. Invited paper: Local differential privacy on metric spaces: Optimizing the trade-off with utility. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 262–267, 2018. doi:10.1109/CSF.2018.00026. — [doi:10.1109/csf.2018.00026](https://doi.org/10.1109/csf.2018.00026) find
  93. Benjamin Weggenmann and Florian Kerschbaum. Differential privacy for directional data. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS ’21, page 1205–1222, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384544. doi:10.1145/3460… — [doi:10.1145/3460120.3484734](https://doi.org/10.1145/3460120.3484734) find
  94. Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey F. Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics, 2017. find
  95. Minxin Du, Xiang Yue, Sherman S. M. Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS ’23. find
  96. ACM, November 2023b. doi:10.1145/3576915.3616592. URL http://dx.doi.org/10.1145/3576915.3616592. — [doi:10.1145/3576915.3616592](https://doi.org/10.1145/3576915.3616592) ↗
  97. Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising, 2018. find
  98. Thee Chanyaswad, Alex Dytso, H. Vincent Poor, and Prateek Mittal. Mvg mechanism: Differential privacy under matrix-valued query. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS ’18. find
  99. ACM, October 2018. doi:10.1145/3243734.3243750. URL http://dx.doi.org/10.1145/ 3243734.3243750. — [doi:10.1145/3243734.3243750](https://doi.org/10.1145/3243734.3243750) find
  100. Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization, 2021. find
  101. Yijia Xiao, Yiqiao Jin, Yushi Bai, Yue Wu, Xianjun Yang, Xiao Luo, Wenchao Yu, Xujiang Zhao, Yanchi Liu, Haifeng Chen, Wei Wang, and Wei Cheng. Large language models can be good privacy protection learners, 2023. find
  102. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
  103. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
  104. Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K. Bressem. Medalpaca – an open-source collection of medical conversational ai models and training data, 2023. — MedAlpaca — An Open-Source Collection of Medical Conversational AI Models and Training Data
  105. LeapBeyond. Scrubadub: A python library for cleaning sensitive data from text. https://github.com/LeapBeyond/ scrubadub, 2023. GitHub Repository. ↗
  106. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013. ↗
  107. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020. find
  108. James Vincent. The scary truth about ai copyright is nobody knows what will happen next, Nov 2022. URL https://www.theverge.com/23444685/ generative-ai-copyright-infringement-legal-fair-use-training-data. ↗
  109. Melissa Heikkila. This artist is dominating ai-generated art. and he’s not happy about it., Nov 2022b. URL https://www.technologyreview.com/2022/09/16/1059598/ this-artist-is-dominating-ai-generated-art-and-hes-not-happy-about-it/?truid=& utm_source=the_algorithm&utm_medium=email&utm_campaign=the_al… ↗
  110. Jack Bandy and Nicholas Vincent. Addressing "documentation debt" in machine learning research: A retrospective datasheet for bookcorpus, 2021. find
  111. Wes Davis. Sarah silverman is suing openai and meta for copyright infringement, Jul 2023b. URL https://www.theverge.com/2023/7/9/23788741/ sarah-silverman-openai-meta-chatgpt-llama-copyright-infringement-chatbots-artificial-intelligence-ai? ICID. ↗
  112. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models, 2023.
  113. Daniel E. Rose. It might be cheaper to pay them: Artificial intelligence, copyright, and the hollywood writers’ strike -copyright - united states, May 2023a. URL https://www.mondaq.com/unitedstates/copyright/1314270/ it-might-be-cheaper-to-pay-them-artificial-intelligence-copyright-and-the-hollywood… ↗
  114. Ryan Broderick. Ai can’t replace humans yet - but if the wga writers don’t win, it might not matter, May 2023. URL https://www.polygon.com/23742770/ai-writers-strike-chat-gpt-explained. ↗
  115. Sylvie Delacroix. Data rivers: Re-balancing the data ecosystem that makes generative ai possible, Apr 2023. URL https://ssrn.com/abstract=4388928. ↗
  116. Nov 2021. URL https://harvardlawreview.org/print/vol-135/google-llc-v-oracle-america-inc/. ↗
  117. Benjamin L. W. Sobel. Artificial intelligence’s fair use crisis. The Columbia Journal of Law and the Arts, 41(1): 45–97, Dec. 2017. doi:10.7916/jla.v41i1.2036. URL https://journals.library.columbia.edu/index.php/ lawandarts/article/view/2036. — [doi:10.7916/jla.v41i1.2036](https://doi.org/10.7916/jla.v41i1.2036) ↗
  118. Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang. Foundation models and fair use, 2023. find
  119. Mark A Lemley and Bryan Casey. fair-learning/. Fair learning, Mar 2021. find
  120. URL https://texaslawreview.org/Matthew Sag. The new legal landscape for text mining and machine learning. SSRN Electronic Journal, 66:291–365, 2019. doi:10.2139/ssrn.3331606. — [doi:10.2139/ssrn.3331606](https://doi.org/10.2139/ssrn.3331606) ↗
  121. Kalin Hristov. Artificial intelligence and the copyright dilemma, May 2017. URL https://ssrn.com/abstract= 2976428. ↗
  122. US Copyright. Copyright registration guidance: Works containing material generated by artificial intelligence 2023, 2023. find
  123. Daniel E. Rose. Dabus dares to dream: A look at stephen thaler’s patent puzzle, Jan 2023b. URL https://www.vklaw.com/ImagineThatIPLawBlog/dabus-dares-to-dream-a-look-at-stephen-thalers-patent-puzzle. ↗
  124. Dan L Burk. Algorithmic fair use. The University of Chicago Law Review, 86(2):283, 2019. 499 U.S. 340 US Supreme Court. Feist publications, inc. v. rural telephone service company, inc., 1991. URL https://www.law.cornell.edu/supremecourt/text/499/340. ↗
  125. Amanda Levendowski. How copyright law can fix artificial intelligence’s implicit bias problem. Washington Law Review, 93, 2017. find
  126. Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable copyright protection for generative models, 2023. Niva Elkin-Koren, Uri Hacohen, Roi Livni, and Shay Moran. Can copyright be reduced to privacy?, 2023. Sarah Scheffler, Eran Tromer, and Mayank Varia. Formalizing human ingenuity: A quantitative frame… find
  127. Routledge, 2005. Brooke Auxier, Lee Rainie, Monica Anderson, Andrew Perrin, Madhu Kumar, and Erica Turner. Americans and privacy: Concerned, confused and feeling lack of control over their personal information, Nov 2019. URL https://www.pewresearch.org/internet/2019/11/15/ americans-and-privacy-conc… — [doi:10.1007/s00521-018-3523-0](https://doi.org/10.1007/s00521-018-3523-0) ↗
  128. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389… find
  129. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. CoRR, abs/2003.02960, 2020. URL https://arxiv.org/ abs/2003.02960. — [arXiv:2003.02960](https://arxiv.org/abs/2003.02960) ↗
  130. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. CoRR, abs/1911.04933, 2019. URL http://arxiv.org/abs/1911.04933. — [arXiv:1911.04933](https://arxiv.org/abs/1911.04933) ↗
  131. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer, 2023. nostalgebraist. Interpreting gpt: The logit lens, 2020. URL https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. Accessed: April 29, 2024. ↗
  132. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5… find
  133. Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense, 2023.
  134. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… — Training language models to follow instructions with human feedback
  135. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models, 2023. find
  136. Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491–6506, Online and Punta C… find
  137. Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. Privacy adhering machine un-learning in nlp, 2022. find
  138. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on… find
  139. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642,…
  140. Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference, 2018.
  141. Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks, 2022. find
  142. Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning, 2023.
  143. Aly Kassem, Omar Mahmoud, and Sherif Saad. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce… find
  144. Matt Post. A call for clarity in reporting BLEU scores. In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Tu…
  145. Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners, 2023.
  146. BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît… find
  147. Curran Associates, Inc., 2019. URL — SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization https://proceedings.neurips.cc/paper_files/paper/2019/file/cb79f8fa58b91d3af6c9c991f63962d3-Paper.pdf. Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms, 2023. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neel-2023-privacy-issues-large-language-models/