Elamur Research⌕
☰
home / papers / 2019 / The Curious Case of Neural Text Degeneration

The Curious Case of Neural Text Degeneration

Paper · 2019 · Ari Holtzman, Jan Buys, Li Du et al.

Summary

The paper seeks the best way to decode text from a neural language model in open-ended generation, where the model must freely continue a given context, such as a story. The authors point out a paradox. Methods that search for the most likely continuation, including beam search, produce bland, generic text that gets stuck in repetitions. Sampling from the full distribution, by contrast, produces incoherent language that is barely related to the context. The authors attribute the latter to the "unreliable tail" of the distribution: tens of thousands of low-probability tokens collectively receive too large a share of the probability mass. As a solution, they propose Nucleus Sampling (top-p). At each step, the method takes the smallest set of tokens whose cumulative probability reaches the threshold p and samples only from that set. As a result, the candidate pool expands and contracts dynamically. Generations from GPT-2 Large are compared with human-written WebText in terms of perplexity, vocabulary diversity, Self-BLEU, and tendency toward repetition, as well as with the combined HUSE score. Nucleus Sampling proved to be the best: its text is both high-quality and as diverse as human text. The method has become one of the default generation settings in modern language models.

Key points

  • Beam search (b=32) with GPT-2 Large falls into repetitive loops. With b ≥ 64, the GPT-2 Large (774M parameters) and XL (1542M) models mostly stop generating immediately after the context.
  • The probability "nucleus" typically contains anywhere from one to roughly a thousand tokens. A fixed k in top-k therefore fits poorly across all steps.
  • Text obtained by maximization or top-k is too predictable: its perplexity is lower than that of human text. Text from pure sampling, conversely, is too improbable. For top-k to match humans in vocabulary and Self-BLEU, a large k is required, which in turn yields unstable likelihood and incoherent text. The parameter p, by contrast, can be tuned so that perplexity matches that of human text.
  • According to HUSE (Hashimoto et al., 2019), which combines human and statistical evaluation, Nucleus Sampling ranked first. The examples use p=0.95.
  • The code and all generations are published at github.com/ari-holtzman/degen.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,762 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations🎓University of Cape TownRondebosch · South Africa4 works · 2,689 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Natural language processing · 131 works

Cite

Cite as Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. arXiv:1904.09751.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.09751 · arXiv 1904.09751

@misc{holtzman2019curious,
  title = {{The Curious Case of Neural Text Degeneration}},
  author = {Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi},
  year = {2019},
  doi = {10.48550/arxiv.1904.09751},
  eprint = {1904.09751},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.09751}
}

References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9(1):147–169, 1985. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Proceedings of the 2015 International Conference on Learning Representations, 2015.
  3. Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. In Critiquing and Correcting Trends in Machine Learning: NeurIPS 2018 Workshop, 2018. URL http://arxiv.org/abs/1811.02549. — [arXiv:1811.02549](https://arxiv.org/abs/1811.02549) ↗
  4. Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. Recurrent neural networks as weighted language recognizers. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long P… find
  5. Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. Neural text generation in stories using entity representations as context. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2… find
  6. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 933–941, 2017.
  7. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 889–898, 2018.
  8. Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94–104, 2017.
  9. H Paul Grice. Logic and conversation. In P Cole and J L Morgan (eds.), Speech Acts, volume 3 of Syntax and Semantics, pp. 41–58. Academic Press, 1975. find
  10. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
  11. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the Association for Computational Linguistics, 2018.
  12. Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pp. 28–39, 2017. find
  13. Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. Importance of search and evaluation strategies in neural dialogue modeling. International Conference on Natural Language Generation, 2019.
  14. Jiwei Li, Will Monroe, and Dan Jurafsky. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562, 2016a. find
  15. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192–1202, 2016b.
  16. Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421, 2015.
  17. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280–290, 2016.
  18. Chris Pal, Charles Sutton, and Andrew McCallum. Sparse forward-backward using minimum divergence beams for fast training of conditional random fields. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 5, May 2006. find
  19. Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pp. 43–49, New Orleans, Louisiana, June 2018. doi: 10.18653/v1/W18-1505. find
  20. Steven T Piantadosi. Zipfs word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112–1130, 2014. find
  21. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/ language_understanding_paper.pdf. Unpublished manuscript. ↗
  22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, February 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language_ models_are_unsupervised_multitask_learners.pdf. Unpublished manuscript. ↗
  23. Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of gans for language generation. arXiv preprint arXiv:1806.04936, 2018. — [arXiv:1806.04936](https://arxiv.org/abs/1806.04936) find
  24. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pp. 6830–6841, 2017.
  25. Felix Stahlberg and Bill Byrne. On nmt search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3347–3353, 2019.
  26. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language modeling. In Thirteenth annual conference of the international speech communication association, 2012. find
  27. Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. Evaluating text gans as language models. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2241–2247, 2019… find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.
  29. Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. find
  30. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  31. Sam Wiseman, Stuart Shieber, and Alexander Rush. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2253–2263, Copenhagen, Denmark, September 2017.
  32. Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Diversity-promoting gan: A cross-entropy based generative adversarial network for diversified text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3940–3949, Brussels, Belgium, oct 2018. find
  33. Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, 2017. find
  34. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/