Elamur Research⌕
☰
home / papers / 2024 / AI models collapse when trained on recursively generated data

AI models collapse when trained on recursively generated data

Paper · 2024 · Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao et al.

Summary

The paper examines what happens to generative models once their own outputs make up much of the web data used to train their successors. The authors name the resulting degradation "model collapse." When each generation learns only from samples produced by the previous one, rare events in the original distribution vanish first (early collapse). Eventually the model drifts toward a narrow, low-variance distribution that barely resembles the real data (late collapse). They attribute this to three compounding error sources: finite-sample statistical error, limited functional expressivity, and imperfections of the learning procedure. The argument rests on analytical models (a discrete distribution with exact fitting and a multidimensional Gaussian) and on experiments with GMMs, VAEs on MNIST, and the OPT-125m language model trained generation after generation on wikitext2-derived data. Later generations show rising perplexity and begin producing text the original model would essentially never generate. The work matters because it implies that authentic human-written data, and records of real human interaction with systems, will grow more valuable as LLM-generated content spreads online.

Key points

  • Recursive training is formalized as mixing data from the original distribution, the previous generation, and the newest model. The analytical cases use only the newest model's samples.
  • In the discrete case without fitting error, any state with probability at most 1/M (M = sample size) is expected to appear less than once per sample. Such states are gradually lost, and the support shrinks.
  • Language-model experiments use OPT-125m evaluated on the wikitext2 test set, with generations up to 9 and five independent runs. They compare two settings: no original data kept over five epochs, and 10% original data kept over ten epochs.
  • Perplexity histograms, scored by the model trained on real data, show two effects across generations. Typical outputs become more "probable," while a long tail of implausible samples emerges.
  • The phenomenon is shown to be distinct from catastrophic forgetting and data poisoning. Code is released on Zenodo (doi 10.5281/zenodo.10866595).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations🎓University of TorontoToronto · Canada110 works · 179,062 citations🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🎓Imperial College LondonLondon · United Kingdom50 works · 13,154 citations🔬Vector InstituteToronto · Canada37 works · 5,703 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

Cite

Cite as Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal. 2024. AI models collapse when trained on recursively generated data. Nature.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s41586-024-07566-y

@inproceedings{shumailov2024ai,
  title = {{AI models collapse when trained on recursively generated data}},
  author = {Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal},
  booktitle = {Nature},
  year = {2024},
  doi = {10.1038/s41586-024-07566-y},
  url = {https://doi.org/10.1038/s41586-024-07566-y}
}

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shumailov-2024-ai-models-collapse-trained-recursively/