Elamur Research⌕
☰
home / papers / 2014 / GloVe: Global Vectors for Word Representation

GloVe: Global Vectors for Word Representation

Summary

The paper explains why word vector representations capture semantic relationships as linear relations. The authors also propose a model that combines the strengths of two approaches. The first approach factorizes a global word co-occurrence matrix (like LSA). It makes good use of corpus statistics but performs poorly on analogies. The second learns from local context windows (like skip-gram and CBOW). It handles analogies better but does not take global counts into account. The authors show that meaning is best captured by ratios of word co-occurrence probabilities rather than by the probabilities themselves. From this they derive the GloVe model, a weighted least-squares regression. The dot product of a word vector and a context vector approximates the logarithm of the number of their co-occurrences. Training covers only the nonzero cells of the matrix, and a weighting function reduces the influence of rare and overly frequent pairs. GloVe achieves 75 % accuracy on the analogy task and outperforms other models on word similarity tests and NER. Under identical conditions it performs better than word2vec and reaches a better result faster. The code and pretrained vectors have been made publicly available.

Key points

  • The core idea is illustrated by the pair ice/steam. For the probe word solid, the probability ratio is approximately 8.9, and for gas it is about 0.085. For the neutral water and fashion it is close to 1.
  • The comparison with word2vec uses a corpus of 6 billion tokens (Wikipedia 2014 + Gigaword 5), a vocabulary of 400 000 words, 300-dimensional vectors, and a symmetric window of size 10.
  • On the NER task with 50-dimensional vectors, GloVe obtains F1 of 93.2 (dev), 88.3 (test), 82.9 (ACE), and 82.2 (MUC7). This is better than CBOW, HPCA, CW, and the SVD variants.
  • Beyond approximately 200 dimensions, accuracy increases only slightly. Syntactic analogies are better captured by small asymmetric windows, and semantic ones by wider windows.
  • The code and trained vectors are available at nlp.stanford.edu/projects/glove.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1162

@inproceedings{pennington2014glove,
  title = {{GloVe: Global Vectors for Word Representation}},
  author = {Pennington, Jeffrey and Socher, Richard and Manning, Christopher},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1162},
  url = {https://doi.org/10.3115/v1/d14-1162}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom M. Apostol. 1976. Introduction to Analytic Number Theory. Introduction to Analytic Number Theory. find
  2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL. find our card
  3. Yoshua Bengio. 2009. Learning deep architectures for AI. Foundations and Trends in Machine Learning. find
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR, 3:1137–1155. find our card
  5. John A. Bullinaria and Joseph P. Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39(3):510–526. find
  6. Dan C. Ciresan, Alessandro Giusti, Luca M. Gambardella, and Jürgen Schmidhuber. 2012. Deep neural networks segment neuronal membranes in electron microscopy images. In NIPS, pages 2852–2860. find
  7. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167. find
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. JMLR, 12:2493–2537. find our card
  9. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41. find
  10. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12. find
  11. Lev Finkelstein, Evgenly Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406–414. ACM. find
  12. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving Word Representations via Global Context and Multiple Word Prototypes. In ACL. find our card
  13. Rémi Lebret and Ronan Collobert. 2014. Word embeddings through Hellinger PCA. In EACL. find
  14. Omer Levy, Yoav Goldberg, and Israel Ramat-Gan. 2014. Linguistic regularities in sparse and explicit word representations. CoNLL-2014. find our card
  15. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instrumentation, and Computers, 28:203–208. find
  16. Minh-Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. CoNLL-2013. find our card
  17. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In ICLR Workshop Papers. find our card
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. find our card
  19. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In HLT-NAACL. find our card
  20. George A. Miller and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and cognitive processes, 6(1):1–28. find
  21. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In NIPS. find
  22. Douglas L. T. Rohde, Laura M. Gonnerman, and David C. Plaut. 2006. An improved model of semantic similarity based on lexical co-occurence. Communications of the ACM, 8:627–633. find
  23. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  24. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM Computing Surveys, 34:1–47. find
  25. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing With Compositional Vector Grammars. In ACL. find
  26. Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. 2003. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the SIGIR Conference on Research and Development in Informaion Retrieval. find
  27. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In CoNLL-2003. find our card
  28. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394. find our card
  29. Mengqiu Wang and Christopher D. Manning. 2013. Effect of non-linear deep architecture in sequence labeling. In Proceedings of the 6th International Joint Conference on Natural Language Processing (IJCNLP). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/