Elamur Research⌕
☰
home / papers / 2013 / Distributed Representations of Words and Phrases and their Compositionality

Distributed Representations of Words and Phrases and their Compositionality

Summary

The paper extends the Skip-gram model, which learns vector representations of words by predicting neighboring words in a text. The authors, from Google, propose several extensions that both speed up training and improve the quality of the vectors. The first is subsampling of frequent words: a word is discarded with a probability that increases with its frequency. This accelerates training by a factor of 2–10 and yields better vectors for rare words. The second is Negative sampling (NEG), a simplified variant of NCE. Here the model uses logistic regression to learn to distinguish the true context word from several random "noise" words. The third is a simple statistical method for finding fixed phrases (for example, "Air Canada") and training them as separate tokens. To evaluate such vectors, the authors created a new analogy test that contains both words and phrases. A model trained on roughly 30 billion words produces noticeably better nearest neighbors than previously published representations, while training in a small fraction of their time. It is also shown that simple vector addition often yields a meaningful result. The code was released under the name word2vec, and the method became a standard tool in NLP.

Key points

  • A noise distribution for NEG and NCE in the form of unigram frequencies raised to the power 3/4 works well. The number of negative samples k should be 5–20 for small corpora and 2–5 for large ones.
  • The subsampling threshold is t ≈ 10⁻⁵. For 300-dimensional models on the analogy test, NEG-15 trained for 97 min without subsampling and 36 min with it, while accuracy stayed the same at 61%. NEG-5 with subsampling reached 60% in 14 min.
  • Hierarchical softmax uses a binary Huffman tree in which frequent words are assigned short codes. On the analogy test for single words it lags behind NEG: 47% without subsampling and 55% with it. For phrases, the best combination turned out to be hierarchical softmax with subsampling.
  • An optimized single-machine implementation processes more than 100 billion words per day. A Skip-Phrase model of dimensionality 1000 was trained in one day, whereas the models of Collobert, Turian and Mnih required from a week to two months.
  • Vectors compose additively, for example: "Russia" + "river" ≈ "Volga River", "Germany" + "capital" ≈ "Berlin". The described techniques are also applicable to the CBOW model; the code is available at code.google.com/p/word2vec.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1310.4546 · arXiv 1310.4546

@misc{mikolov2013distributed,
  title = {{Distributed Representations of Words and Phrases and their Compositionality}},
  author = {Mikolov, Tomas and Sutskever, Ilya and Chen, Kai and Corrado, Greg and Dean, Jeffrey},
  year = {2013},
  doi = {10.48550/arxiv.1310.4546},
  eprint = {1310.4546},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1310.4546}
}

References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155, 2003. find our card
  2. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM, 2008. find
  3. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML, 513–520, 2011. find
  4. Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research, 13:307–361, 2012. find
  5. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. Extensions of recurrent neural network language model. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pages 5528–5531. IEEE, 2011. find
  6. Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukas Burget and Jan Cernocky. Strategies for Training Large Scale Neural Network Language Models. In Proc. Automatic Speech Recognition and Understanding, 2011. find
  7. Tomas Mikolov. Statistical Language Models Based on Neural Networks. PhD thesis, PhD Thesis, Brno University of Technology, 2012. find
  8. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. ICLR Workshop, 2013. find our card
  9. Tomas Mikolov, Wen-tau Yih and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013. find our card
  10. Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. Advances in neural information processing systems, 21:1081–1088, 2009. find
  11. Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206.6426, 2012. find
  12. Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics, pages 246–252, 2005. find
  13. David E Rumelhart, Geoffrey E Hintont, and Ronald J Williams. Learning representations by backpropagating errors. Nature, 323(6088):533–536, 1986. find
  14. Holger Schwenk. Continuous space language models. Computer Speech and Language, vol. 21, 2007. find
  15. Richard Socher, Cliff C. Lin, Andrew Y. Ng, and Christopher D. Manning. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 26th International Conference on Machine Learning (ICML), volume 2, 2011. find
  16. Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. Semantic Compositionality Through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2012. find our card
  17. Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics, 2010. find our card
  18. Peter D. Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. In Journal of Artificial Intelligence Research, 37:141-188, 2010. find our card
  19. Peter D. Turney. Distributional semantics beyond words: Supervised learning of analogy and paraphrase. In Transactions of the Association for Computational Linguistics (TACL), 353–366, 2013. find
  20. Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In Proceedings of the Twenty-Second international joint conference on Artificial Intelligence-Volume Volume Three, pages 2764–2770. AAAI Press, 2011. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/