Elamur Research⌕
☰
home / papers / 2023 / Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4

Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4

Summary

Large language models such as ChatGPT and GPT-4 are increasingly used in cultural analytics, but their training data is undisclosed. This makes it hard to know whether evaluation texts were already seen during training. The authors run a "data archaeology" to estimate which works of fiction these models know. They use a membership inference probe called name cloze: the model must recover a single masked character name in a short passage that contains no other named entities.

The probe covers 571 English-language novels published between 1749 and 2020, drawn from LitBank, Pulitzer nominees, bestsellers, works by Black authors and other sources. The results show that OpenAI models, GPT-4 in particular, have memorized many in-copyright books. Memorization tracks how often a book's passages appear on the web. Downstream tasks also go better on memorized titles. The authors conclude that closed models undermine measurement validity in literary research, and they advocate using open models with publicly documented training data.

Key points

  • Evaluation set: BookNLP extracted passages of 40–60 tokens, each with exactly one person name and no other entities. The authors sampled 100 passages per book. Always guessing the most frequent name ("Mary") scores 0.6%, and a human annotator scored 0%.
  • What is memorized: The most memorized books are science fiction/fantasy, popular public-domain classics and bestsellers. Science fiction/fantasy accounts for 12 of the top 20 memorized in-copyright titles, including Harry Potter, 1984 and Dune.
  • Link to web frequency: Memorization levels align with how often a book appears in Google, Bing and C4 results. Web prevalence therefore offers a rough indicator of contamination risk.
  • Effect on downstream tasks: The models perform better on memorized books when predicting a book's first publication year and estimating narrative time duration.
  • BERT finding: BookCorpus contains copyrighted novels such as Fifty Shades of Grey and Outlander, and BERT has memorized them. Code and data are available at github.com/bamman-group/gpt4-books.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.453

@inproceedings{chang2023speak,
  title = {{Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4}},
  author = {Chang, Kent K. and Cramer, Mackenzie and Soni, Sandeep and Bamman, David},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.453},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.453}
}

References

66 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bamman, Michelle Carney, Jon Gillick, Cody Hennesy, and Vijitha Sridhar. 2017. Estimating the date of first publication in a large-scale digital library. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1–10. IEEE. find
  2. David M. Berry. 2011. The computational turn: Thinking about the digital humanities. Culture Machine, 12. find
  3. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373. find
  4. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Co… find
  5. Pierre Bourdieu. 1987. Distinction: A social critique of the judgement of taste. Harvard university press. find
  6. Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. 2023. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188. — [arXiv:2301.13188](https://arxiv.org/abs/2301.13188) find
  7. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646. find
  8. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6. find our card
  9. Mariona Coll Ardanuy, Federico Nanni, Kaspar Beelen, Kasra Hosseini, Ruth Ahnert, Jon Lawrence, Katherine McDonough, Giorgia Tolfo, Daniel CS Wilson, and Barbara McGillivray. 2020. Living machines: A study of atypical animacy. In Proceedings of the 28th International Conference on Computational Ling… find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  11. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… find
  12. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… find
  13. M Elam. 2023. Poetry Will Not Optimize, or What Is Literature to AI? American literature; a journal of literary history, criticism and bibliography. find
  14. Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization : Quantifying data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1325–1335, Online. Asso… find
  15. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. 2022. Measuring causal effects of data statistics on language model’s ‘factual’ predictions. arXiv preprint arXiv:2207.14251. — [arXiv:2207.14251](https://arxiv.org/abs/2207.14251) find
  16. Katherine Elkins and Jon Chun. 2020. Can GPT-3 pass a writer’s Turing test? Journal of cultural analytics, 5(2). find
  17. Elizabeth F. Evans and Matthew Wilkens. 2018. Nation, ethnicity, and the geography of British fiction, 1880– 1940. Cultural Analytics. find
  18. Kathleen Fitzpatrick. 2012. The humanities, done digitally. Debates in the digital humanities, pages 12–15. find
  19. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. find
  20. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86– 92. find
  21. Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. find
  22. Lauren M E Goodlad and Wai Chee Dimock. 2021. AI and the Human. PMLA, 136(2):317–319. find
  23. John Guillory. 1993. Cultural capital the problem of literary canon formation. University of Chicago Press, Chicago. find
  24. Suchin Gururangan, Dallas Card, Sarah K Drier, Emily K Gade, Leroy Z Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A Smith. 2022. Whose language counts as high quality? Measuring language ideologies in text data selection. arXiv preprint arXiv:2201.10474. find
  25. Gary Gutting. 1989. Michel Foucault’s Archaeology of Scientific Reason: Science and the History of Reason. Cambridge University Press. find
  26. Sil Hamilton and Andrew Piper. 2022. The COVID that wasn’t: Counterfactual journalism using GPT. In Proceedings of the 6th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 83–93, Gyeongju, Republic of Korea. International Con… find
  27. N. Katherine Hayles. 2012. How We Think: Transforming Power and Digital Technologies. In David M Berry, editor, Understanding Digital Humanities, pages 42–66. Palgrave Macmillan UK, London. find
  28. Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. 2023. Foundation models and fair use. arXiv preprint arXiv:2303.15715. — [arXiv:2303.15715](https://arxiv.org/abs/2303.15715) find
  29. Leah Henrickson and Albert Meroño-Peñuela. 2022. The Hermeneutics of Computer-Generated Texts. Configurations, 30(2):115–139. find
  30. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks principle: Reading children’s books with explicit memory representations. ICLR. find our card
  31. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628. find
  32. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. 2022. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) find
  33. FM Jong, Henning Rode, and Djoerd Hiemstra. 2005. Temporal language models for the disclosure of historical text. Proceedings of the XVIth International Conference of the Association for History and Computing (AHC 2005). find
  34. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022a. Large language models struggle to learn long-tail knowledge. arXiv preprint arXiv:2211.08411. find our card
  35. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022b. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pages 10697–10707. PMLR. find our card
  36. Folgert Karsdorp and Lauren Fonteyn. 2019. Cultural entrenchment of folktales is encoded in language. Palgrave Communications, 5(1):25. find
  37. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. find
  38. Abhimanu Kumar, Matthew Lease, and Jason Baldridge. 2011. Supervised language modeling for temporal resolution of texts. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 2069–2072. find
  39. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  40. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online. Associ… find
  41. Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, Virtual. Association for Computational Linguistics. find our card
  42. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland. Association for Computational Linguistics. find
  43. Marc Marone and Benjamin Van Durme. 2023. Data portraits: Recording foundation model training data. find
  44. Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022. An empirical analysis of memorization in fine-tuned autoregressive language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816–1826, Abu… find
  45. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235, Austin, Texas. Association for Computational Lingui… find
  46. Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 298–311, Online and Punta Cana, Dominican Republic. Association for Computational Lingui… find
  47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. find
  48. Stephen Ramsay and Geoffrey Rockwell. 2012. Developing things: Notes toward an epistemology of building in the digital humanities. Debates in the digital humanities, pages 75–84. find
  49. Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 840–854, Abu Dhabi, United Arab Emirates. Association for Computational Li… find our card
  50. Nicolas Ruth, Andreas Niekler, and Manuel Burghardt. 2022. Peeking Inside the DH Toolbox – Detection and Classification of Software Tools in DH Publications. Proceedings of the Computational Humanities Research Conference 2022, 1613:0073. find
  51. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  52. Henning Schmidgen, Bernhard Dotzler, and Benno Stein. 2023. From the archive to the computer: Michel Foucault and the digital humanities. Journal of cultural analytics, 7(4). find
  53. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE. find our card
  54. Arthur Spirling. 2023. Why open-source generative AI models are an ethical way forward for science. Nature, 616(7957):413–413. find
  55. Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: Automated extraction of character roles without training data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47–56, Seattle, United States. Association for Computation… find
  56. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p… find
  57. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274–38290. find our card
  58. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  59. Ted Underwood. 2018. Why literary time is measured in minutes. ELH, 85(2). find
  60. Ted Underwood. 2023. Using GPT-4 to measure the passage of time in fiction. https://tedunderwood.com/2023/03/19/using-gpt-4-to-measure-the-passage-of-time-in-fiction/. ↗
  61. Melanie Walsh and Maria Antoniak. 2021. The Goodreads “Classics”: A computational study of readers, Amazon, and crowdsourced amateur criticism. Journal of Cultural Analytics, 6(2). find
  62. Michael Yoder, Sopan Khosla, Qinlan Shen, Aakanksha Naik, Huiming Jin, Hariharan Muralidharan, and Carolyn Rosé. 2021. FanfictionNLP: A text processing pipeline for fanfiction. In Proceedings of the Third Workshop on Narrative Understanding, pages 13–23, Virtual. Association for Computational Lingui… find
  63. Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2021. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938. — [arXiv:2112.12938](https://arxiv.org/abs/2112.12938) find
  64. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. find
  65. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… find our card
  66. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large language models transform computational social science? arXiv submission 4840038. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/