Elamur Research⌕
☰
home / papers / 2023 / How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN

How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN

Paper · 2023 · R. Thomas McCoy, Paul Smolensky, Tal Linzen et al.

Summary

When researchers judge a language model by the text it produces, they usually assume the output reflects learned linguistic knowledge. That assumption fails if the model simply reproduces passages from its training data. To address this, the authors built RAVEN (RAting VErbal Novelty), a set of analyses that compare generated text against the training corpus. RAVEN checks n-grams of every size and syntactic structures such as dependency arcs and whole-sentence parses. They ran it on an LSTM, a Transformer and Transformer-XL trained on Wikitext-103, and on all four GPT-2 sizes trained on WebText. They varied decoding methods and prompt lengths, and used human-written continuations from each test set as the reference point. For small units, models produced fewer novel items than humans did. For larger structures, they matched or exceeded human novelty, yet occasionally reproduced very long training passages. A manual study of GPT-2 found mostly well-formed new words and sentences, but frequent semantic errors. The work shows that novelty must be verified directly before generated text can count as evidence of generalization.

Key points

  • The study tested 12 generation methods and 4 prompt lengths. The Wikitext-103 models share a 103-million-word training set, which allows a controlled comparison of architectures.
  • For bigrams, human baseline text contained between 1.5 and 3.2 times more unseen items than model output. Beyond 5-grams, the models became more novel than humans.
  • GPT-2 sometimes reproduced training passages exceeding 1,000 words. This "supercopying" was linked mainly to repetition within the training data.
  • Transformer-XL produced a sentence structure absent from training in 83% of its sentences. GPT-2 invented words such as Swissified and IKEA-ness, but also overgeneralized, e.g., 752th.
  • GPT-2 showed evidence of both compositional and analogical generalization, but handled acronym structure and form–meaning relations poorly. The code is available at github.com/tommccoy1/raven.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations🏢MetaMenlo Park · United States158 works · 211,459 citations🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations🎓New York UniversityNew York · United States136 works · 86,390 citations🎓Princeton UniversityPrinceton · United States61 works · 36,471 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 3 Language models & pretraining · 1,343 works Evaluation & benchmarks · 685 works Text generation · 145 works

Cite

Cite as R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao and Asli Celikyilmaz. 2023. How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00567

@article{mccoy2023how,
  title = {{How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN}},
  author = {R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao and Asli Celikyilmaz},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00567},
  url = {https://doi.org/10.1162/tacl_a_00567}
}

References

63 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. 2022. Towards tracing knowledge in language models back to the training data. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2429–2446, Abu Dhabi, United Arab Emira… find
  2. Adam Albright and Bruce Hayes. 2003. Rules vs. analogy in English past tenses: A computational/ experimental study. Cognition, 90(2):119–161. https://doi.org/10.1016/S0010 -0277(03)00146-X, PubMed: 14599751 — [doi:10.1016/s0010](https://doi.org/10.1016/s0010) ↗
  3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21. https://doi.org/10.1145 /3442188.3445922… our card
  4. We thank OpenAI for providing access to the WebText dataset. For helpful comments and discussion, we are grateful to Suhas Arehalli, Saadia Gabriel, Coleman Haley, Yichen Jiang, Nebojsa Jojic, Najoung Kim, Géraldine Legendre, Grusha Prasad, Eric Rosen, Sebastian Schuster, Paul Soulos, Shiyue Zhang,… — [doi:10.1080/00437956](https://doi.org/10.1080/00437956) find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, SandhiniAgarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… our card
  6. Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mi… our card
  7. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. Association for Comput… find
  8. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. find
  9. Ishita Dasgupta, Erin Grant, and Tom Griffiths. 2022. Distinguishing rule and exemplar-based generalization in learning systems. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4816–4830. PMLR. find
  10. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium, pages 267–284. our card
  11. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium, pages 2633–2650. our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  13. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799v2. find
  14. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… our card
  15. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. Association for Computationa… find
  16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. https://doi.org/10 .1162/neco.1997.9.8.1735, PubMed: 9377276 find
  17. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  18. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. 2022. Measuring causal effects of data statistics on language model’s ‘factual’ predictions. arXiv preprint arXiv: 2207.14251v1. https://doi.org/10.4… — [arXiv:2207.14251](https://arxiv.org/abs/2207.14251) find
  19. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… find
  20. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65–170. https://doi.org/10.1613/jair.5477 ↗
  21. Robert F. Hadley. 1994. Systematicity in connectionist language learning. Mind & Language, 9(3):247–272. https://doi.org/10.1111/j.1468-0017.1994.tb00225.x Coleman Haley. 2020. This is a BERT. Now there are several of them. Can they generalize to novel words? In Proceedings of the Third BlackboxNLP… ↗
  22. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757–795. https://doi.org/10.1613/jair.1.11674 ↗
  23. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10697–10707. PMLR. our card
  24. Xiaochuang Han and Yulia Tsvetkov. 2022. ORCA: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. arXiv preprint arXiv:2205.12600v1. — [arXiv:2205.12600](https://arxiv.org/abs/2205.12600) find
  25. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361v1. https://doi.org/10 .48550/arXiv.2001.08361 our card
  26. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan-Nitish Shirish Keskar, Bryan McCann, L… our card
  27. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring compositional generalization: A comprehensive method on… find
  28. Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.750 ↗
  29. Stan A. Kuczaj II. 1977. The acquisition of regular and irregular past tense forms. Journal of Verbal Learning and Verbal Behavior, 16(5):589–600. https://doi.org/10.1016 /S0022-5371(77)80021-2 find
  30. Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2873–2882. find
  31. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. find
  32. Benjamin LeBrun, Alessandro Sordoni, and Timothy J. O’Donnell. 2022. Evaluating distributional distortion in neural language modeling. In International Conference on Learning Representations. find
  33. Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105. Association for Computational Linguistics. https://doi.org/10 .18653/… find
  34. Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. 2022a. Do language models plagiarize? arXiv preprint arXiv:2203.07618v2. https://doi.org/10.1145/3543507 .3583199 — [doi:10.1145/3543507](https://doi.org/10.1145/3543507) ↗
  35. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686. Association for Computational Linguistics. https://doi.org/10 .18653/v1/P18-1249 our card
  36. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022b. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  37. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  38. Yafu Li, Yongjing Yin, Yulong Chen, and Yue Zhang. 2021. On compositional generalization of neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  39. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Gary Marcus.… find
  40. Gary Marcus, Steven Pinker, Michael Ullman, Michelle Hollander, T. John Rosen, Fei Xu, and Harald Clahsen. 1992. Overregularization in language acquisition. Monographs of the Society for Research in Child Development, i–178. https://doi.org/10.2307/1166115 phological patterns. Language and Cognitive… — [doi:10.2307/1166115](https://doi.org/10.2307/1166115) ↗
  41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
  42. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140. https://doi.org/10.1162/tacl_a_00304 ↗ our card
  43. Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline-conditioned generation with dynamic plot state tracking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4274–4295. Association for Computational Lingu… find
  44. Clara Meister and Ryan Cotterell. 2021. Language model evaluation beyond perplexity. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5328–5339. Associa… find
  45. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lon… find
  46. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations. our card
  47. Richard Montague. 1970. Universal grammar. Theoria, 36(3):373–398. https://doi.org/10.1111/j.1755-2567.1970.tb00434.x Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. 2019. Enhancing the Transformer with explicit relational encoding for math pro… — [doi:10.1111/j.1755-2567.1970.tb00434.x](https://doi.org/10.1111/j.1755-2567.1970.tb00434.x) ↗
  48. Andrew Mutton, Mark Dras, Stephen Wan, and Robert Dale. 2007. GLEU: Automatic evaluation of sentence-level fluency. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 344–351. Association for Computational Linguistics. find
  49. Timothy J. O’Donnell. 2015. Productivity and Reuse in Language: A Theory of Linguistic Computation and Storage. MIT Press. https://doi.org/10.7551/mitpress/9780262028844 .001.0001 — [doi:10.7551/mitpress/9780262028844](https://doi.org/10.7551/mitpress/9780262028844) ↗
  50. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725. Association for Computational Linguistics. https:/… our card
  51. Ludovica Pannitto and Aurélie Herbelot. 2020. Recurrent babbling: Evaluating the acquisition of grammar from limited input data. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 165–176. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2… find
  52. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18. IEEE. our card
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Sandeep Prasada and Steven Pinker. 1993. Generalisation of regular and irregular mor-Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pa… our card
  54. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in Transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computation… our card
  55. Takateru Yamakoshi, Thomas Griffiths, and Robert Hawkins. 2022. Probing BERT’s priors with serial reproduction chains. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3977–3992, Dublin, Ireland. Association for Computational Linguistics. https://doi.org/10 .18653/v1/202… find
  56. Charles Yang. 2016. The Price of Linguistic Productivity: How Children Learn to Break the Rules of Language. MIT Press. https://doi.org/10.7551/mitpress/9780262035323 .001.0001 — [doi:10.7551/mitpress/9780262035323](https://doi.org/10.7551/mitpress/9780262035323) ↗
  57. Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open pre-trained Transformer language models. arXiv preprint arXiv:2205.01068v4. https://doi.org/10.48550/arXiv.2205.01068 ↗ our card
  58. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. find
  59. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021b. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume… find
  60. Yu Zhang, Zhenghua Li, and Min Zhang. 2020b. Efficient second-order TreeCRF for neural dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3295–3305. Association for Computational Linguistics. our card
  61. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2021a. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115. https://doi.org/10.1145/3446776 — [doi:10.1145/3446776](https://doi.org/10.1145/3446776) ↗
  62. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097–1100. https://doi .org/10.1145/3209978.3… our card
  63. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Albert Ziegler. 2021. A first look at rote learning in GitHub Copilot suggestions. GitHu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/