Elamur Research⌕
☰
home / papers / 2013 / Efficient Estimation of Word Representations in Vector Space

Efficient Estimation of Word Representations in Vector Space

Summary

Most NLP systems of that time treated words as separate indices in a vocabulary, so similarity between words was not taken into account at all. Existing neural network models of word vectors were too slow to be trained on billions of words. The authors from Google propose two simplified log-linear architectures. CBOW reconstructs the central word from the averaged vectors of its surrounding words, while Skip-gram does the opposite: given a word, it predicts its neighbors. Neither model has a nonlinear hidden layer. The output is computed via hierarchical softmax over a Huffman tree, and training is distributed in the DistBelief framework with Adagrad. For evaluation, the authors created a new test set of semantic and syntactic analogies. These are solved with simple vector arithmetic, for example Paris − France + Italy ≈ Rome. The main result: high-quality vectors from a corpus of 1.6 billion words can be obtained in less than a day. On analogies they are more accurate than previous neural network approaches, while requiring far less computation. This work laid the foundation of word2vec, and word embeddings became a basic component of almost all subsequent NLP.

Key points

  • CBOW takes a context of 4 preceding and 4 following words and ignores word order. The training complexity is Q = N×D + D×log₂(V). Skip-gram predicts the words surrounding the current one.
  • Hierarchical softmax over a Huffman tree gives short codes to frequent words. For a vocabulary of 1 million words, this is roughly twice as fast as with a balanced binary tree.
  • The best vectors in the examples were produced by the Skip-gram model: 300 dimensions, trained on 783 million words. If a relation is specified not by a single example but by the average of ten, test accuracy increases by about 10 percentage points.
  • On the Microsoft Sentence Completion Challenge, Skip-gram alone does not outperform LSA. But combined with RNNLM it achieves a new record of 58.9% (the previous one was 55.4%).
  • After publication, the authors released multithreaded C++ code that processes billions of words per hour. They also published more than 1.4 million vectors of named entities trained on more than 100 billion words.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1301.3781 · arXiv 1301.3781

@misc{mikolov2013efficient,
  title = {{Efficient Estimation of Word Representations in Vector Space}},
  author = {Mikolov, Tomas and Chen, Kai and Corrado, Greg and Dean, Jeffrey},
  year = {2013},
  doi = {10.48550/arxiv.1301.3781},
  eprint = {1301.3781},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1301.3781}
}

References

32 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bengio, R. Ducharme, P. Vincent. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003. find our card
  2. Y. Bengio, Y. LeCun. Scaling learning algorithms towards AI. In: Large-Scale Kernel Machines, MIT Press, 2007. find
  3. T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. Large language models in machine translation. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, 2007. find our card
  4. R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In International Conference on Machine Learning, ICML, 2008. find
  5. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493-2537, 2011. find our card
  6. J. Dean, G.S. Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le, M.Z. Mao, M.A. Ranzato, A. Senior, P. Tucker, K. Yang, A. Y. Ng., Large Scale Distributed Deep Networks, NIPS, 2012. find
  7. J.C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011. find
  8. J. Elman. Finding Structure in Time. Cognitive Science, 14, 179-211, 1990. find
  9. Eric H. Huang, R. Socher, C. D. Manning and Andrew Y. Ng. Improving Word Representations via Global Context and Multiple Word Prototypes. In: Proc. Association for Computational Linguistics, 2012. find our card
  10. G.E. Hinton, J.L. McClelland, D.E. Rumelhart. Distributed representations. In: Parallel distributed processing: Explorations in the microstructure of cognition. Volume 1: Foundations, MIT Press, 1986. find
  11. D.A. Jurgens, S.M. Mohammad, P.D. Turney, K.J. Holyoak. Semeval-2012 task 2: Measuring degrees of relational similarity. In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), 2012. find
  12. A.L. Maas, R.E. Daly, P.T. Pham, D. Huang, A.Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of ACL, 2011. find our card
  13. T. Mikolov. Language Modeling for Speech Recognition in Czech, Masters thesis, Brno University of Technology, 2007. find
  14. T. Mikolov, J. Kopecký, L. Burget, O. Glembek and J. Černocký. Neural network based language models for higly inflective languages, In: Proc. ICASSP 2009. find
  15. T. Mikolov, M. Karafiát, L. Burget, J. Černocký, S. Khudanpur. Recurrent neural network based language model, In: Proceedings of Interspeech, 2010. find
  16. T. Mikolov, S. Kombrink, L. Burget, J. Černocký, S. Khudanpur. Extensions of recurrent neural network language model, In: Proceedings of ICASSP 2011. find
  17. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, J. Černocký. Empirical Evaluation and Combination of Advanced Language Modeling Techniques, In: Proceedings of Interspeech, 2011. The code is available at https://code.google.com/p/word2vec/ ↗
  18. T. Mikolov, A. Deoras, D. Povey, L. Burget, J. Černocký. Strategies for Training Large Scale Neural Network Language Models, In: Proc. Automatic Speech Recognition and Understanding, 2011. find
  19. T. Mikolov. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012. find
  20. T. Mikolov, W.T. Yih, G. Zweig. Linguistic Regularities in Continuous Space Word Representations. NAACL HLT 2013. find our card
  21. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed Representations of Words and Phrases and their Compositionality. Accepted to NIPS 2013. find our card
  22. A. Mnih, G. Hinton. Three new graphical models for statistical language modelling. ICML, 2007. find
  23. A. Mnih, G. Hinton. A Scalable Hierarchical Distributed Language Model. Advances in Neural Information Processing Systems 21, MIT Press, 2009. find
  24. A. Mnih, Y.W. Teh. A fast and simple algorithm for training neural probabilistic language models. ICML, 2012. find
  25. F. Morin, Y. Bengio. Hierarchical Probabilistic Neural Network Language Model. AISTATS, 2005. find
  26. D. E. Rumelhart, G. E. Hinton, R. J. Williams. Learning internal representations by backpropagating errors. Nature, 323:533.536, 1986. find
  27. H. Schwenk. Continuous space language models. Computer Speech and Language, vol. 21, 2007. find
  28. R. Socher, E.H. Huang, J. Pennington, A.Y. Ng, and C.D. Manning. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS, 2011. find
  29. J. Turian, L. Ratinov, Y. Bengio. Word Representations: A Simple and General Method for Semi-Supervised Learning. In: Proc. Association for Computational Linguistics, 2010. find our card
  30. P. D. Turney. Measuring Semantic Similarity by Latent Relational Analysis. In: Proc. International Joint Conference on Artificial Intelligence, 2005. find
  31. A. Zhila, W.T. Yih, C. Meek, G. Zweig, T. Mikolov. Combining Heterogeneous Models for Measuring Relational Similarity. NAACL HLT 2013. find
  32. G. Zweig, C.J.C. Burges. The Microsoft Research Sentence Completion Challenge, Microsoft Research Technical Report MSR-TR-2011-129, 2011. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/