Elamur Research
☰
home / papers / 2017 / Using the Output Embedding to Improve Language Models

Using the Output Embedding to Improve Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/E17-2025.pdf

Summary

Neural language models usually have two word-representation matrices. One embeds the input word. The other sits at the top of the network and turns hidden states into scores over the vocabulary. Normally only the input matrix is treated as a word embedding. Press and Wolf show that the top matrix is also a meaningful embedding. In LSTM language models it even beats the input matrix on word-similarity benchmarks such as Simlex999 and MEN. Their gradient analysis explains why: when the two matrices share weights, the shared matrix trains mostly like the output matrix. For models trained without dropout, they add a regularized projection just before the output layer. They tested on PTB, text8, IMDB and BBC. Weight tying lowered perplexity both with and without dropout, and adding the projection set a new best result for small non-dropout models. In attention-based NMT, tying the decoder embeddings, or the encoder and decoder embeddings over a shared BPE vocabulary, roughly halved the parameter count without hurting BLEU. Weight tying has since become a standard technique in language models.

Key points

  • On the PTB test set, the large dropout LSTM of Zaremba et al. improved from 78.4 to 74.3 perplexity with weight tying (WT), and the model shrank from 66M to 51M parameters. A recurrent highway network (RHN) with Bayesian dropout and WT reached 66.0.
  • For the small model without dropout, PTB test perplexity fell from 114.5 to 100.9 when WT was combined with projection regularization (PR). PR uses an H×H matrix with an L2 penalty, λ = 0.15. PR gave no gain for the large models, which already use dropout.
  • Tying makes the word2vec skip-gram embeddings worse, which agrees with Goldberg and Levy. In NNLMs, the tied embedding resembles the untied output embedding more than the input one.
  • In the NMT experiments (DL4MT; WMT14 EN→FR, WMT15 EN→DE), up to 90% (EN–FR) and 85% (EN–DE) of BPE subwords appear in both languages. Tying the decoder embeddings cut about 28% of parameters, and three-way weight tying (TWWT) cut about 52%. BLEU stayed comparable in both cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

295 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 274 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1608.05859

@inproceedings{press2017using,
  title = {{Using the Output Embedding to Improve Language Models}},
  author = {Press, Ofir and Wolf, Lior},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  year = {2017},
  eprint = {1608.05859},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1608.05859}
}

References

42 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Çaglar Gülçehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. 2016. Noisy activation functions. arXiv preprint arXiv:1603.00391. — [arXiv:1603.00391](https://arxiv.org/abs/1603.00391) find
  2. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362, Edinburgh, Scotland, UK., July. Association for Computational Linguistics. find our card
  3. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1406– 1414. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  5. Felix Hill, Roi Reichart, and Anna Korhonen. 2016. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics. find our card
  6. Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 278–289. Association for Computational Linguistics. find
  7. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  8. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137–1155, March. find
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780, November. find
  10. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. J. Artif. Intell. Res.(JAIR), 49(1-47). find
  11. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462. find
  12. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  13. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709, Seattle, Washington, USA, October. Association for Computational Linguistics. find
  14. Thang Luong, Richard Socher, and Christopher Manning, 2013. Proceedings of the Seventeenth Conference on Computational Natural Language Learning, chapter Better Word Representations with Recursive Neural Networks for Morphology, pages 104–113. Association for Computational Linguistics. find our card
  15. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. find
  16. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML-16). find
  17. L. Andrew Maas, E. Raymond Daly, T. Peter Pham, Dan Huang, Y. Andrew Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150. Association for… find our card
  18. Yarin Gal. 2015. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. arXiv preprint arXiv:1512.05287. find our card
  19. Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722. — [arXiv:1402.3722](https://arxiv.org/abs/1402.3722) find
  20. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):313–330, June. find our card
  21. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  22. Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pages 1045–104… find
  23. Derek Greene and Pádraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proc. 23rd International Conference on Machine learning (ICML’06), pages 377–384. ACM Press. find
  24. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In Interspeech, pages 194–197, Portland, OR, USA, September. find
  25. Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5528–5531. IEEE. find
  26. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  27. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  28. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329. find our card
  29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. find
  30. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  31. Shiliang Zhang, Hui Jiang, Mingbin Xu, Junfeng Hou, and Li-Rong Dai. 2015. A fixed-size encoding method for variable-length sequences with its application to neural network language models. arXiv preprint arXiv:1505.01504. — [arXiv:1505.01504](https://arxiv.org/abs/1505.01504) find
  32. Tomas Mikolov, Armand Joulin, Sumit Chopra, Michaël Mathieu, and Marc’Aurelio Ranzato. 2014. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753. — [arXiv:1412.7753](https://arxiv.org/abs/1412.7753) find
  33. Julian G. Zilly, Rupesh Kumar Srivastava, Jan Koutnı́k, and Jürgen Schmidhuber. 2016. Recurrent highway networks. arXiv preprint arXiv:1607.03474. find
  34. Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. 2016. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137. — [arXiv:1602.01137](https://arxiv.org/abs/1602.01137) find
  35. Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1992–1997. Association for Computational Linguistics. find
  36. Andriy Mnih and Geoffrey E Hinton. 2009. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pages 1081–1088. find
  37. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems, pages 2265–2273. find
  38. Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206.6426. find
  39. Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2013. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026. find
  40. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891. find our card
  41. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL. find
  42. Nitish Srivastava. 2013. Improving Neural Networks with Dropout. Master’s thesis, University of Toronto, Toronto, Canada, January. find