Elamur Research⌕
☰
home / papers / 2016 / Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Paper · 2016 · Hakan Inan, Khashayar Khosravi and Richard Socher

Summary

Standard recurrent language models treat next-word prediction as ordinary classification against one-hot targets. Two consequences follow. The model ignores the fact that words already form a meaningful similarity space. It also learns separate input and output word representations, even though both describe the same vocabulary. Inan, Khosravi and Socher address this with a new training objective. Alongside the usual cross-entropy, they add a KL-divergence term that pulls the model's temperature-scaled prediction toward a soft target. That target is built from embedding similarity between the gold word and every other word. A theoretical analysis, under idealized conditions such as high temperature, zero training loss and equal embedding and hidden sizes, shows that this term pushes the output projection into the subspace spanned by the input embeddings. This motivates explicitly reusing the embedding matrix, transposed, as the output classifier, with no output bias. Experiments with two-layer LSTMs on Penn Treebank and Wikitext-2 report that this beats conventional training. The authors claim state-of-the-art Penn Treebank results while sharply cutting parameter counts.

Key points

  • Augmented loss (AL): the total loss is cross-entropy plus α times KL(ỹ ‖ ŷ). The soft target ỹ is a softmax over inner products between the target word's embedding and all embeddings, divided by a temperature τ.
  • Weight tying (reuse of embeddings): the output matrix W is set equal to Lᵀ, removing one vocabulary-sized matrix. For large networks, most of the gain came from tying alone.
  • Setup:
    • Two-layer LSTMs in small, medium and large sizes.
    • Backpropagation unrolled for 35 steps.
    • Gal (2015)-style variational dropout. The PTB models used rates of 0.7 (small), 0.5 (medium) and 0.35 (large), while the Wikitext-2 models used 0.8 (small) and 0.6 (medium).
  • AL hyperparameters: τ = 20 and α = γτ, with γ of 0.5–0.8 for PTB and 1.0–1.5 for Wikitext-2. Results were robust to moderate changes in either value.
  • Analysis tool: the appendix defines a subspace distance, the average squared sine of the principal angles, used to compare the spans of W and Lᵀ. The supplied excerpt omits the actual perplexity figures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Hakan Inan, Khashayar Khosravi and Richard Socher. 2016. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. arXiv:1611.01462.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.01462 · arXiv 1611.01462

@misc{inan2016tying,
  title = {{Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling}},
  author = {Hakan Inan, Khashayar Khosravi and Richard Socher},
  year = {2016},
  doi = {10.48550/arxiv.1611.01462},
  eprint = {1611.01462},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.01462}
}

References

27 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. 2001. URL http://www.iro.umontreal.ca/˜lisa/pointeurs/nips00_lm.ps. ↗
  2. ◦ Ake Björck and Gene H Golub. Numerical methods for computing angles between linear subspaces. Mathematics of computation, 27(123):579–594, 1973. find
  3. Wei-Chen Cheng, Stanley Kok, Hoai Vu Pham, Hai Leong Chieu, and Kian Ming Adam Chai. Language modeling with sum-product networks. 2014. find
  4. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
  5. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine translation with a shared attention mechanism. arXiv preprint arXiv:1601.01073, 2016.
  6. Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learning with a wasserstein loss. In Advances in Neural Information Processing Systems, pp. 2053– 2061, 2015. find
  7. Yarin Gal. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287, 2015.
  8. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  9. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  10. Hakan Inan and Khashayar Khosravi. Improved learning through augmenting the loss. Stanford CS 224D: Deep Learning for Natural Language Processing, Spring 2016, 2016. find
  11. Kazuki Irie, Zoltán Tüske, Tamer Alkhouli, Ralf Schlüter, and Hermann Ney. Lstm, gru, highway and a bit of attention: an empirical overview for language modeling in speech recognition. Interspeech, San Francisco, CA, USA, 2016. find
  12. Camille Jordan. Essai sur la géométrie à n dimensions. Bulletin de la Société mathématique de France, 3:103–174, 1875. find
  13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. arXiv preprint arXiv:1508.06615, 2015.
  14. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330, 1993.
  15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  16. Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, pp. 3, 2010. find
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013.
  18. Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning, pp. 641–648. ACM, 2007. find
  19. Ramesh Nallapati, Bowen Zhou, Çaglar Gulçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence rnns and beyond. 2016.
  20. Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. CoRR, abs/1312.6026, 2013a.
  21. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. ICML (3), 28:1310–1318, 2013b. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pp. 1532–43, 2014.
  23. Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016.
  24. Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685, 2015.
  25. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. Citeseer, 2013.
  26. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.
  27. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnı́k, and Jürgen Schmidhuber. Recurrent highway networks. arXiv preprint arXiv:1607.03474, 2016.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/