Elamur Research
☰
home / papers / 2023 / The Curse of Recursion: Training on Generated Data Makes Models Forget

The Curse of Recursion: Training on Generated Data Makes Models Forget

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2305.17493

Summary

Generative models are increasingly filling the web with text and images, so future models will likely be trained partly on their predecessors' output. The authors study what happens when this loop runs over many generations. They call the resulting degradation "model collapse." Each new model learns a slightly distorted copy of the data. Rare events at the edges of the distribution disappear first. Eventually the learned distribution narrows into a low-variance shape that barely resembles the original.

They trace the effect to two causes: finite-sample statistical error, which is the main driver, and limited model expressiveness, a secondary one. For simple Gaussian settings they derive risk bounds based on Wasserstein distance. They then show the effect empirically in Gaussian Mixture Models, Variational Autoencoders and fine-tuned OPT-125m models on wikitext2. Even in the fine-tuning setting, early collapse is clearly visible, and keeping some original data softens it.

The finding matters for anyone building web-crawled corpora. Authentic human-written data, and the ability to trace where content came from, become scarce and valuable resources.

Key points

  • The paper separates two error sources. The primary one is statistical approximation error from finite sampling. Even with 10^7 samples, the estimated mean of a 1D Gaussian still deviates noticeably from the true value. The secondary one is functional approximation error from imperfect function approximators.
  • The authors distinguish two stages. In early collapse, the tails of the distribution are lost. In late collapse, modes merge and the distribution converges to something with very small variance.
  • In the LLM experiments, OPT-125m is fine-tuned on wikitext2, and each generation is trained on text produced by the previous one, up to generation 9. Outputs drift into off-topic, repetitive text.
  • The first model, trained on real data, was used to score later outputs. Later generations produce more sequences that model finds likely, but also a long tail of sequences it would never generate. Preserving 10% of the original data limits the performance loss to a minor one.
  • Adding a repetition penalty of 2.0 does not prevent collapse. Perplexity roughly doubles, and degradation is at least as strong. The authors infer a "first mover advantage" and call for community-wide tracking of content provenance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

101 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 98 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.17493 · arXiv 2305.17493

@misc{shumailov2023curse,
  title = {{The Curse of Recursion: Training on Generated Data Makes Models Forget}},
  author = {Shumailov, Ilia and Shumaylov, Zakhar and Zhao, Yiren and Gal, Yarin and Papernot, Nicolas and Anderson, Ross},
  year = {2023},
  doi = {10.48550/arxiv.2305.17493},
  eprint = {2305.17493},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.17493}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139–154, 2018. find
  2. Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254–11263, 2019. find
  3. Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012. — [arXiv:1206.6389](https://arxiv.org/abs/1206.6389) find
  4. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… find our card
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  6. Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.09667, 2021. https://googleblog.blogspot.com/2011/02/finding-more-high-quality-sites-in.html https://www.technologyreview.com/2010/07/26/26327/the-search-engine-backlash-against-content-mil… — [arXiv:2106.09667](https://arxiv.org/abs/2106.09667) ↗
  7. Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149, 2023. — [arXiv:2302.10149](https://arxiv.org/abs/2302.10149) find
  8. W. G. Cochran. The distribution of quadratic forms in a normal system, with applications to the analysis of covariance. Mathematical Proceedings of the Cambridge Philosophical Society, 30(2):178–191, 1934. doi: 10.1017/S0305004100016595. — [doi:10.1017/s0305004100016595](https://doi.org/10.1017/s0305004100016595) find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  10. Adrian Fischer, Robert E. Gaunt, and Andrey Sarantsev. The variance-gamma distribution: A review, 2023. find
  11. Matthias Gelbrich. On a formula for the l2 wasserstein metric between measures on euclidean and hilbert spaces. Mathematische Nachrichten, 147(1):185–203, 1990. find
  12. Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems,… find
  13. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. find
  14. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation, 2019. find our card
  15. Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. find
  16. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526,… find
  17. Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017. find
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  19. Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, 2015. find
  20. OpenAI. Gpt-4 technical report, 2023. find our card
  21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 20… find our card
  22. Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009. find
  23. Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. Sponge examples: Energy-latency attacks on neural networks. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 212–231. find
  24. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. Release strategies and the social impacts of language models, 2019. find our card
  25. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019. find our card
  26. Nassim Nicholas Taleb. Black swans and the domains of statistics. The American Statistician, 61(3):198–200, 2007. find
  27. Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019. find
  28. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. find