Elamur Research⌕
☰
home / papers / 2023 / Copyright Violations and Large Language Models

Copyright Violations and Large Language Models

Paper · 2023 · Antonia Karamolegkou, Jiaang Li, Li Zhou et al.

Summary

The paper asks whether large language models can reproduce copyrighted text word for word. This matters because fair use generally covers extracting information from protected works, not redistributing them. The authors test six model families. For the open models (OPT, Pythia, LLaMA, Falcon/Falcon-Instruct), they use prefix probing: the model receives the first 50 tokens of a passage and its continuation is checked. For the closed, instruction-tuned systems (GPT-3.5, Claude), they use direct probing, asking for passages such as the first page of a named title. The test material is bestselling novels and LeetCode problem descriptions. Overlap with the original is measured as LCS length in words, with Levenshtein distance reported in the appendix. Based on common quotation practice, 50 words serves as the threshold of concern. Verbatim recall rises roughly linearly with model size, and the largest models reproduce long book fragments and entire LeetCode descriptions. Memorized content correlates with popularity signals. The authors draw no legal conclusions. Instead, they provide a method and conservative empirical evidence for debates about copyright and LLMs.

Key points

  • Data: bestsellers published between 1930 and 2010 (full list in Table 1) and the first 1,826 LeetCode problems, whose Terms of Use forbid redistribution.
  • Scale effect: verbatim memorization grows nearly linearly with parameter count across OPT, Pythia, LLaMA and Falcon. GPT-3.5 and Claude show the strongest reproduction.
  • Popularity link for books: GPT-3.5 memorization correlates with number of reviews (r = 0.68) and number of editions (r = 0.76).
  • Popularity link for LeetCode: the paper reports strong correlations with engagement indicators such as discussions, submissions and companies (|r| between 0.82 and 0.95).
  • Hidden memorization: GPT-3.5 sometimes confabulates passages, as with The Girl with the Dragon Tattoo, yet supplies the real text when asked more specifically. This suggests memorization can be "unlocked" and that the reported numbers understate it. Code: github.com/coastalcph/CopyrightLLMs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark78 works · 14,228 citations🎓University of Electronic Science and Technology of ChinaChengdu · China8 works · 1,738 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksBusiness, Management and Accounting · 4 worksMarketing · 1 work

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 1,343 works

Cite

Cite as Antonia Karamolegkou, Jiaang Li, Li Zhou and Anders Søgaard. 2023. Copyright Violations and Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.458

@inproceedings{karamolegkou2023copyright,
  title = {{Copyright Violations and Large Language Models}},
  author = {Antonia Karamolegkou, Jiaang Li, Li Zhou and Anders Søgaard},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.458},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.458}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization : Quantifying data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1325–1335, Online. Asso… find
  2. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. find
  3. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  4. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  6. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. OpenAI Blog. find
  7. Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. 2023. Controlling the extraction of memorized data from large language models via prompt-tuning. find
  8. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306… find
  9. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  10. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. web crawl data. In Proceedi…
  11. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramer, and Nicholas Carlini. 2023. Counterfactual memorization in neural language models. find
  12. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trai…
  13. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/karamolegkou-2023-copyright-violations-large-language-models/