Elamur Research
☰
home / papers / 2019 / Neural Text Generation with Unlikelihood Training

Neural Text Generation with Unlikelihood Training

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1908.04319

Summary

Language models trained with standard maximum likelihood tend to produce dull text that repeats itself. They overuse frequent words and loop on the same phrases, especially with greedy or beam search. Earlier fixes such as top-k sampling, nucleus sampling and n-gram beam blocking only change how text is decoded. The model's probabilities stay the same. The authors argue that the likelihood objective itself causes much of this degeneration. They propose unlikelihood training, which keeps the usual likelihood term for the correct tokens and adds a penalty that lowers the probability of chosen negative candidates. At the token level, the negatives are tokens that already appeared in the context. At the sequence level, the model is fine-tuned on its own decoded continuations, and tokens inside repeated n-grams are penalized. They tested this on a Transformer language model trained on wikitext-103 using sequence completion. Repetition fell sharply, vocabulary became more varied and perplexity stayed close to the baseline. In human evaluations, beam search with the new model was preferred over the usual decoding-time fixes. The work matters because it fixes the model's training rather than patching the decoder.

Key points

  • The paper describes two problems with likelihood training: it pays little attention to which token ends up ranked first, and it trains only next-token prediction, not whole generated sequences.
  • With greedy decoding, the baseline Transformer repeated about 43% of n-grams in its continuations, compared with 0.5% in human text. Its next-token predictions copied a word from the previous 128 words 62% of the time, compared with 49% for human text.
  • The methods are token-level (L_UL-token), sequence-level (L_UL-seq) and the two combined. Perplexity stays roughly unchanged (about 24.6 for the MLE baseline versus about 25–26.6 for the unlikelihood variants).
  • In crowdworker evaluations, L_UL-token+seq won 82% of comparisons against the MLE baseline, 59% against nucleus sampling and 60% against beam blocking. Expert judges preferred it over nucleus sampling 83% of the time and over beam blocking 74% of the time.
  • Sequence-level fine-tuning also helps when the penalized tokens are chosen at random (10% of generated tokens), so it does not depend on explicitly detecting repetition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

271 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 252 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.04319 · arXiv 1908.04319

@misc{welleck2019neural,
  title = {{Neural Text Generation with Unlikelihood Training}},
  author = {Welleck, Sean and Kulikov, Ilia and Roller, Stephen and Dinan, Emily and Cho, Kyunghyun and Weston, Jason},
  year = {2019},
  doi = {10.48550/arxiv.1908.04319},
  eprint = {1908.04319},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.04319}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In International Conference on Learning Representations. find
  2. Yejin Choi. 2018. The missing representation in neural (language) models. 3rd Workshop on Representation Learning for NLP (RepL4NLP). find
  3. Michael Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002). Association for Computational Linguistics. find our card
  4. Hal Daumé, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine learning, 75(3):297–325. find
  5. Adji B Dieng, Kyunghyun Cho, David M Blei, and Yann LeCun. 2018. Learning with reflective likelihoods. find
  6. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098. find our card
  7. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2017. Classical structured prediction losses for sequence to sequence learning. arXiv preprint arXiv:1711.04956. find
  8. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. find
  9. Tianxing He and James Glass. 2019. Negative training for neural dialogue response generation. arXiv preprint arXiv:1903.02134. find
  10. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638–1649. Association for Computa… find our card
  11. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. find
  12. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. find our card
  13. Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. 2018. Importance of a search strategy in neural dialogue modelling. arXiv preprint arXiv:1811.00907. — [arXiv:1811.00907](https://arxiv.org/abs/1811.00907) find
  14. Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. 2006. A tutorial on energy-based learning. Predicting structured data. find
  15. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562. — [arXiv:1611.08562](https://arxiv.org/abs/1611.08562) find
  16. Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087. — [arXiv:1909.03087](https://arxiv.org/abs/1909.03087) find
  17. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. find our card
  18. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  19. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304. find our card
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  21. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732. find our card
  22. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. find
  23. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… find our card
  24. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015. Minimum risk training for neural machine translation. arXiv preprint arXiv:1512.02433. find our card
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  26. Jesse Vig. 2018. Deconstructing bert: Distilling 6 patterns from 100 million parameters. Medium. find
  27. Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  28. Jason Weston, Emily Dinan, and Alexander H Miller. 2018. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint arXiv:1808.04776. find
  29. Lantao Yu, Weinan Zhang, Jun Wang, and Yingrui Yu. 2016. Seqgan: Sequence generative adversarial nets with policy gradient. ArXiv, abs/1609.05473. — [arXiv:1609.05473](https://arxiv.org/abs/1609.05473) find
  30. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbou… find our card
  31. Prefix GPT2-greedy Lyrically the song has excerpts of different languages including French , Spanish , German , Italian , Portuguese , Spanish , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese , Portuguese ,… find