Elamur Research
☰
home / papers / 2019 / Representation Degeneration Problem in Training Natural Language Generation Models

Representation Degeneration Problem in Training Natural Language Generation Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/1907.12009

Summary

The paper identifies a flaw in neural text generators trained by maximum likelihood with weight tying, where the input embedding matrix is shared with the softmax output layer. In such models, especially on large corpora, most word vectors collapse into a narrow cone, and nearly all pairwise cosine similarities are positive. This weakens their semantic expressiveness and limits the softmax's ability to separate classes with a large margin. The authors call this the representation degeneration problem. They show it with rank-2 SVD projections of a base Transformer trained on WMT 2014 English-German. For contrast, Word2Vec vectors and MNIST classifier weights spread evenly around the origin. Their theory links the problem to rare words. When the convex hull of hidden states excludes the origin, which is likely under layer normalization, embeddings of unseen words drift without bound in a shared direction. Low-frequency words also end up close together. As a fix, they add a regularizer that penalizes cosine similarity between word embeddings, which widens the cone. This improves perplexity on WikiText-2 and BLEU on WMT 2014, showing that embedding geometry affects generation quality.

Key points

  • Diagnostic setup:
    • Baseline was the official base Transformer (27.3 BLEU on WMT 2014 En-De).
    • An LSTM-based model showed the same clustering.
    • Comparison points were Word2Vec embeddings and the softmax weights of a two-layer CNN trained on MNIST.
  • Frequency analysis: Over 90% of words in WMT 2014 En-De have a frequency below 10e-4, and even "is" accounts for only about 1% of tokens.
  • Theory:
    • A direction that is negative for every hidden state exists exactly when the convex hull of those states does not contain the origin.
    • The authors argue that layer normalization before the last layer makes this condition almost always hold.
  • Method: The cosine-similarity regularizer can be rewritten as the squared norm of the sum of normalized embeddings, so it is computed in Θ(N) time for vocabulary size N.
  • Results:
    • Perplexity improves by 2.0 points on WikiText-2.
    • BLEU improves by 1.08 on WMT 2014 English→German and by 0.93 on German→English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1907.12009 · arXiv 1907.12009

@misc{gao2019representation,
  title = {{Representation Degeneration Problem in Training Natural Language Generation Models}},
  author = {Gao, Jun and He, Di and Tan, Xu and Qin, Tao and Wang, Liwei and Liu, Tie-Yan},
  year = {2019},
  doi = {10.48550/arxiv.1907.12009},
  eprint = {1907.12009},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1907.12009}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations, 2015. find our card
  3. Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. Massive exploration of neural machine translation architectures. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1442–1451. Association for Computational Linguistics, 2017. doi: 10.18653/v1/D17-11… find
  4. Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning, pp. 933–941, 2017. find our card
  5. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pp. 1243– 1252, 2017. find our card
  6. Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Frage: frequency-agnostic word representation. In Advances in Neural Information Processing Systems, pp. 1341–1352, 2018. find
  7. Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. In Proceedings of the International Conference on Learning Representations, 2017. find
  8. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In Proceedings of the International Conference on Learning Representations, 2017. find
  9. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  10. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence, pp. 2741–2749, 2016. find our card
  11. Philipp Koehn, Marcello Federico, Wade Shen, Nicola Bertoldi, Ondrej Bojar, Chris Callison-Burch, Brooke Cowan, Chris Dyer, Hieu Hoang, Richard Zens, et al. Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding. In Final Report of the 200… find
  12. Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In International Conference on Machine Learning, pp. 507–516, 2016. find
  13. Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873–4907, 2017. find
  14. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6297– 6308, 2017. find our card
  15. Jorma Kaarlo Merikoski. On the trace and the sum of elements of a matrix. Linear algebra and its applications, 60:177–185, 1984. find
  16. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In Proceedings of the International Conference on Learning Representations, 2017. find our card
  17. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm language models. In Proceedings of the International Conference on Learning Representations, 2018. find our card
  18. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. find
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pp. 3111–3119, 2013. find
  20. Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: simple and effective postprocessing for word representations. In Proceedings of the International Conference on Learning Representations, 2018. find
  21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. Association for Computational Linguistics, 2002. find our card
  22. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014. find
  23. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157–163. Association for Computational Linguistics, 2017. URL http://aclweb.org… ↗ our card
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725. Association for Computational Linguistics, 2016. find
  25. Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pp.… find our card
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 6000–6010, 2017. find
  27. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416, 2018. URL http://arxiv.org/abs/… ↗ our card
  28. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  29. Yingce Xia, Jiang Bian, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Dual inference for machine learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pp. 3112–3118, 2017a. find
  30. Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, and Tie-Yan Liu. Dual supervised learning. In International Conference on Machine Learning, pp. 3789–3798, 2017b. find
  31. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: a high-rank rnn language model. In Proceedings of the International Conference on Learning Representations, 2018. find
  32. Zhao Yue, Zhao Deli, Wan Shaohua, and Zhang Bo. Softmax supervision with isotropic normalization, 2018. URL https://openreview.net/forum?id=SyXNErg0W. ↗