Elamur Research
☰
home / papers / 2014 / Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P14-1023.pdf

Summary

Distributional semantic models (DSMs) represent the meaning of a word as a vector built from the contexts in which that word occurs. The authors compare two approaches to building such vectors. The classical ("count") approach counts word co-occurrences and then reweights and compresses these counts. The newer ("predict", or embeddings) approach trains vectors so that they predict a word's surrounding context as well as possible. Until then, there had been almost no systematic comparison of these approaches. All models are built on the same English corpus. The count vectors were obtained with DISSECT, and the predict vectors were trained with word2vec. Each approach was tested across many parameter settings on standard lexical-semantic benchmarks, including word relatedness rating and concept clustering. Two off-the-shelf models, dm and cw, were also evaluated. The authors expected the count models to win, or at least a mixed picture. Instead, the predict vectors won convincingly almost everywhere, and on some benchmarks they came close to the best known results. As a result, the paper became a strong empirical argument for moving to neural vector representations in semantics.

Key points

  • Corpus: about 2.8 billion tokens, combining ukWaC, the English Wikipedia and the British National Corpus. The target and context vocabulary is restricted to the 300K most frequent words.
  • Count models (36 variants, DISSECT): context windows of 2 and 5 words, PPMI or LMI weighting. The vectors were used either in full or compressed via SVD or NMF to 200–500 dimensions. The best configuration used PMI without any compression, i.e. all 300K dimensions. Among the compression methods, SVD outperformed NMF.
  • Predict models (48 variants, word2vec CBOW): windows of 2 and 5, 200–500 dimensions. The comparison covered hierarchical softmax versus negative sampling with k = 5 or 10, as well as training without subsampling and with subsampling at t = 1e-5. The best model has only 400 dimensions and uses negative sampling (k = 10) and subsampling of frequent words.
  • Benchmarks: among them rg (65 noun pairs), WordSim353, and the ap (402 concepts, 21 categories) and esslli (44 concepts, 6 categories) clustering datasets.
  • Off-the-shelf models: the word2vec vectors outperformed even the syntactically enriched count model dm (Distributional Memory). The off-the-shelf Collobert & Weston vectors (cw, 100 dimensions) performed poorly. In other words, predict models vary considerably in quality, and a comparison against cw alone would have led to the opposite conclusion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

314 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 308 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-1023

@inproceedings{baroni2014don,
  title = {{Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors}},
  author = {Baroni, Marco and Dinu, Georgiana and Kruszewski, Germán},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-1023},
  url = {https://doi.org/10.3115/v1/p14-1023}
}

References

47 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasça, and Aitor Soroa. 2009. find
  2. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of HLT-NAACL, pages 19–27, Boulder, CO. find
  3. Abdulrahman Almuhareb. 2006. Attributes in Lexical Acquisition. Phd thesis, University of Essex. find
  4. Marco Baroni and Alessandro Lenci. 2010. Distributional Memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
  5. Marco Baroni, Stefan Evert, and Alessandro Lenci, editors. 2008. Bridging the Gap between Semantic Theory and Computational Simulations: Proceedings of the ESSLLI Workshop on Distributional Lexical Semantic. FOLLI, Hamburg. find
  6. Marco Baroni, Eduard Barbu, Brian Murphy, and Massimo Poesio. 2010. Strudel: A distributional semantic model based on properties and types. Cognitive Science, 34(2):222–254. find
  7. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  8. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of EMNLP, pages 546–556, Jeju Island, Korea. find
  9. David Blei, Andrew Ng, and Michael Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  10. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2013. Multimodal distributional semantics. Journal of Artificial Intelligence Research. In press; http://clic.cimec.unitn.it/marco/publications/mmds-jair.pdf. ↗
  11. John Bullinaria and Joseph Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39:510–526. find
  12. John Bullinaria and Joseph Levy. 2012. Extracting semantic representations from word co-occurrence statistics: Stop-lists, stemming and SVD. Behavior Research Methods, 44:890–907. find
  13. Yanqing Chen, Bryan Perozzi, Rami Al-Rfou’, and Steven Skiena. 2013. The expressive power of word embeddings. In Proceedings of the ICML Workshop on Deep Learning for Audio, Speech and Language Processing, Atlanta, GA. Published online: https://sites.google.com/site/deeplearningicml2013/accepted\_pap… ↗
  14. Stephen Clark. 2013. Vector space models of lexical meaning. In Shalom Lappin and Chris Fox, editors, Handbook of Contemporary Semantics, 2nd ed. Blackwell, Malden, MA. In press; http://www.cl.cam.ac.uk/˜sc609/pubs/sem\_handbook.pdf. ↗
  15. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167, Helsinki, Finland. find
  16. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  17. Katrin Erk. 2012. Vector space models of word meaning and phrase meaning: A survey. Language and Linguistics Compass, 6(10):635–653. find
  18. Stefan Evert. 2005. The Statistics of Word Cooccurrences. Ph.D dissertation, Stuttgart University. find
  19. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  20. Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A deep visual-semantic embedding model. In Proceedings of NIPS, pages 2121–2129, Lake Tahoe, Nevada. find
  21. Gene Golub and Charles Van Loan. 1996. Matrix Computations (3rd ed.). JHU Press, Baltimore, MD. find
  22. Tom Griffiths, Mark Steyvers, and Josh Tenenbaum. 2007. Topics in semantic representation. Psychological Review, 114:211–244. find
  23. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of KDD, pages 1406–1414. find
  24. Samer Hassan and Rada Mihalcea. 2011. Semantic relatedness using salient semantic analysis. In Proceedings of AAAI, pages 884–889, San Francisco, CA. find
  25. Amaç Herdağdelen and Marco Baroni. 2009. Bag-Pack: A general framework to represent semantic relations. In Proceedings of GEMS, pages 33–40, Athens, Greece. find
  26. Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873–882, Jeju Island, Korea. find
  27. George Karypis. 2003. CLUTO: A clustering toolkit. Technical Report 02-017, University of Minnesota Department of Computer Science. find
  28. Sophia Katrenko and Pieter Adriaans. 2008. Qualia structures and their impact on the concrete noun categorization task. In Proceedings of the ESS-LLI Workshop on Distributional Lexical Semantics, pages 17–24, Hamburg, Germany. find
  29. Thomas Landauer and Susan Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211– 240. find
  30. Daniel Lee and Sebastian Seung. 2000. Algorithms for Non-negative Matrix Factorization. In Proceedings of NIPS, pages 556–562. find
  31. Chih-Jen Lin. 2007. Projected gradient methods for Nonnegative Matrix Factorization. Neural Computation, 19(10):2756–2779. find
  32. Ken McRae, Michael Spivey-Knowlton, and Michael Tanenhaus. 1998. Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38:283–312. find
  33. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. http://arxiv.org/abs/1301.3781/. ↗
  34. Tomas Mikolov, Quoc Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for Machine Translation. http://arxiv.org/abs/1309.4168. ↗
  35. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013c. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111–3119, Lake Tahoe, Nevada. find
  36. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013d. Linguistic regularities in continuous space word representations. In Proceedings of NAACL, pages 746–751, Atlanta, Georgia. find
  37. George Miller and Walter Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
  38. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  39. Ulrike Padó. 2007. The Integration of Syntax and Semantic Plausibility in a Wide-Coverage Model of Sentence Processing. Dissertation, Saarland University, Saarbrücken. find
  40. Klaus Rothenhäusler and Hinrich Schütze. 2009. Unsupervised classification with dependency based word spaces. In Proceedings of GEMS, pages 17– 24, Athens, Greece. find
  41. Herbert Rubenstein and John Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  42. Magnus Sahlgren. 2006. The Word-Space Model. Ph.D dissertation, Stockholm University. find
  43. Richard Socher, Brody Huval, Christopher Manning, and Andrew Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP, pages 1201–1211, Jeju Island, Korea. find
  44. Richard Socher, Milind Ganjoo, Christopher Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Proceedings of NIPS, pages 935–943, Lake Tahoe, Nevada. find
  45. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394, Uppsala, Sweden. find our card
  46. Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
  47. Peter Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533– 585. find