Elamur Research⌕
☰
home / papers / 2015 / Improving Distributional Similarity with Lessons Learned from Word Embeddings

Improving Distributional Similarity with Lessons Learned from Word Embeddings

Summary

The authors examine whether neural word embeddings (SGNS from word2vec, GloVe) really outperform classical count-based distributional approaches when evaluated on tasks of similarity and analogy between words. Baroni et al. (2014) had previously claimed such an advantage. The researchers show that much of the gain does not come from the algorithms themselves. It comes from accompanying engineering choices and hyperparameters: a dynamic context window, subsampling of frequent words, the number of negative samples, smoothing of the context distribution, and adding context vectors. These settings are transferred to PPMI and SVD. In particular, the authors propose a smoothed variant of PMI. Four methods (PPMI, SVD, SGNS, GloVe) are compared in controlled experiments on several datasets. After identical tuning, the differences between the methods are mostly local or statistically insignificant, and no approach wins everywhere. Often, changing a single hyperparameter yields more than switching to a different algorithm or a larger corpus. The work is important as a methodological lesson: a fair comparison must account for hyperparameters and preprocessing.

Key points

  • Theoretical basis: SGNS implicitly factorizes the word–context PMI matrix shifted by log k (Levy & Goldberg, 2014c). That is, SGNS relies on the same information as the classical methods.
  • Context distribution smoothing (cds = 0.75), borrowed from word2vec, is transferred to PMI. It yields on average more than 3 points per task. For example, on the MSR analogies, PPMI accuracy rose from .443 to .535.
  • Evaluation datasets: WordSim (Similarity/Relatedness), MEN (Bruni et al.), M. Turk (Radinsky et al.), Rare Words (Luong et al.), SimLex (Hill et al.), as well as the Google and MSR analogies.
  • Practical recommendations:
    • always use cds = 0.75;
    • for SVD, use symmetric variants instead of the classical eig = 1;
    • SGNS is a robust baseline: the fastest to train and the most economical in memory and disk usage;
    • for SGNS, it is better to use more negative samples;
    • for SGNS and GloVe, the w + c variant is worth trying.
  • The code is released as hyperwords (bitbucket.org/omerlevy/hyperwords). Publication: TACL, vol. 3, 2015.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00134

@misc{levy2015improving,
  title = {{Improving Distributional Similarity with Lessons Learned from Word Embeddings}},
  author = {Levy, Omer and Goldberg, Yoav and Dagan, Ido},
  year = {2015},
  doi = {10.1162/tacl\_a\_00134},
  url = {https://doi.org/10.1162/tacl_a_00134}
}

References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the As… find our card
  2. Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
  3. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Dont count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 238–247, Bal… find our card
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find our card
  5. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 136–145, Jeju Island, Korea, July. Association for Computational Lingui… find our card
  6. John A Bullinaria and Joseph P Levy. 2007. Extracting semantic representations from word co-occurrence statistics: a computational study. Behavior Research Methods, 39(3):510–526. find
  7. John A Bullinaria and Joseph P Levy. 2012. Extracting semantic representations from word co-occurrence statistics: Stop-lists, stemming, and SVD. Behavior Research Methods, 44(3):890–907. find
  8. John Caron. 2001. Experiments with LSA scoring: optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157–169. find
  9. Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29. find our card
  10. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. find
  11. Scott C. Deerwester, Susan T. Dumais, Thomas K. Landauer, George W. Furnas, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. JASIS, 41(6):391–407. find
  12. C Eckart and G Young. 1936. The approximation of one matrix by another of lower rank. Psychometrika, 1:211–218. find
  13. Roi Reichart Felix Hill and Anna Korhonen. 2014. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. arXiv preprint arXiv:1408.3456. find our card
  14. Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008. Introducing and evaluating ukwac, a very large web-derived corpus of English. In Proceedings of the 4th Web as Corpus Workshop (WAC-4), pages 47–54. find
  15. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  16. Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722. — [arXiv:1402.3722](https://arxiv.org/abs/1402.3722) find
  17. Zellig Harris. 1954. Distributional structure. Word, 10(23):146–162. find
  18. Omer Levy and Yoav Goldberg. 2014a. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302–308, Baltimore, Maryland. find
  19. Omer Levy and Yoav Goldberg. 2014b. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171–180, Baltimore, Maryland. find our card
  20. Omer Levy and Yoav Goldberg. 2014c. Neural word embeddings as implicit matrix factorization. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2177–2185. find
  21. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 104–113, Sofia, Bulgaria, August. Association for Computationa… find our card
  22. Oren Melamud, Ido Dagan, Jacob Goldberger, Idan Szpektor, and Deniz Yuret. 2014. Probabilistic modeling of joint-context in distributional similarity. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 181–190, Baltimore, Maryland, June. Association for Com… find
  23. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR). find our card
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find our card
  25. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751. find our card
  26. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  27. Patrick Pantel and Dekang Lin. 2002. Discovering word senses from text. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 613–619. ACM. find
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics. find our card
  29. Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: Computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337–346. ACM. find
  30. Magnus Sahlgren. 2006. The Word-Space Model. Ph.D. thesis, Stockholm University. find
  31. Peter D. Turney and Michael L. Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. Transactions on Information Systems, 21(4):315–346. find
  32. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141–188. find our card
  33. Peter D. Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533– 585. find
  34. Torsten Zesch, Christof Müller, and Iryna Gurevych. 2008. Using wiktionary for computing semantic relatedness. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI’08, pages 861–866. AAAI Press. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/levy-2015-improving-distributional-similarity-lessons-learned/