Elamur Research⌕
☰
home / papers / 2016 / Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

Summary

Word similarity benchmarks are the most common intrinsic test for word embeddings. Models are scored by the Spearman correlation between cosine similarities of word pairs and human ratings. They are cheap and fast, but the authors argue they are unreliable. This workshop paper is a position and survey piece rather than an experimental study. It lists the main weaknesses of this evaluation style and collects the remedies proposed in earlier work:

  • unclear and subjective definitions of "similarity"
  • a mismatch with task-trained embeddings
  • no fixed train/dev/test splits, which encourages overfitting
  • weak correlation with downstream results
  • missing significance testing
  • frequency and hubness effects in cosine space
  • polysemy ignored in the gold ratings

The authors note that word analogy evaluation reduces to combinations of word similarities, so the same criticisms apply to it. They conclude that similarity benchmarks can lead researchers to wrong conclusions. Until better intrinsic methods exist, they recommend comparing embeddings on the downstream task of interest. This matters because much embedding research ranks models mainly by these small benchmarks.

Key points

  • The surveyed benchmarks are RG (65 pairs), MC (30), WS-353, YP-130, MTurk-287, MTurk-771, MEN (3000), RW (2034), Verb (144) and SimLex (999). Their average size is only about 781 pairs, which makes further splitting unstable.
  • Similarity is often mixed up with relatedness. In WS-353, cup–coffee gets a higher rating than car–train, and MEN has similar anomalies. SimLex and the similarity/relatedness split of WS-353 by Agirre et al. try to address this.
  • On significance testing, Shalaby and Zadrozny applied Steiger's test, but it needs the full model rankings. Rastogi et al. proposed MRDS, the smallest gap in correlation needed for a result to count as significant, and it can be computed from the correlation values alone. They also showed that many reported gains on small datasets were not significant.
  • Schnabel et al. found a power law between a word's frequency rank and the average rank of its 1000 nearest neighbours (exponent ≈ 0.17). As a result, words of similar frequency look artificially similar under cosine.
  • Most benchmarks assume one sense per word; for example, WS-353 rates bank–money 8.5/10. SCWS addresses this by rating word pairs inside their contexts.

Interesting ideas

  • I1. Cross-benchmark tuning: tune hyperparameters on one benchmark and report results on all the others, following Faruqui and Dyer. This gives held-out evaluation even when no official splits exist, and works for any field with many small test sets.
  • I2. Selecting for breadth: pick the configuration with the best average across many tasks, or the one that beats baselines on most tasks (Lu et al.). This favours generalisation over leaderboard tuning on a single dataset.
  • I3. Significance thresholds per dataset: publish, for each dataset size, the minimum correlation gap that counts as significant (the MRDS idea). Readers can then judge claimed improvements without access to the raw model outputs.
  • I4. Checking for frequency confounds: before trusting nearest neighbours or similarity scores, test whether neighbours mostly share frequency rank rather than meaning. This check is useful for any embedding-based retrieval, not only for benchmarks.
  • I5. Grading embeddings by their training objective: do not judge embeddings learned for one task, such as POS tagging, by a semantic yardstick. Choose evaluation that matches what the vectors were trained to encode.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

86 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2506

@inproceedings{faruqui2016problems,
  title = {{Problems With Evaluation of Word Embeddings Using Word Similarity Tasks}},
  author = {Faruqui, Manaal and Tsvetkov, Yulia and Rastogi, Pushpendre and Dyer, Chris},
  booktitle = {Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP},
  year = {2016},
  doi = {10.18653/v1/w16-2506},
  url = {https://doi.org/10.18653/v1/w16-2506}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In Proc. of NAACL. find our card
  2. Omer Levy and Yoav Goldberg. 2014a. Linguistic regularities in sparse and explicit word representations. In Proc. of CoNLL. find our card
  3. Omer Levy and Yoav Goldberg. 2014b. Neural word embedding as implicit matrix factorization. In Proc. of NIPS. find
  4. Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In Proc. of EMNLP. find
  5. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. TACL, 3:211–225. find
  6. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in nlp. In Proc. of EMNLP. find
  7. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proc. of ACL. find our card
  8. Ang Lu, Weiran Wang, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Deep multilingual correlation for improved word embeddings. In Proc. of NAACL. find
  9. Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proc. of ACL. find
  10. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proc. of CoNLL. find our card
  11. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proc. of ICML. find
  12. Christopher D. Manning and Hinrich Schütze. 1999. Foundations of statistical natural language processing. MIT Press, Cambridge, MA, USA. find
  13. Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2014. Improving zero-shot learning by mitigating the hubness problem. arXiv preprint arXiv:1412.6568. — [arXiv:1412.6568](https://arxiv.org/abs/1412.6568) find
  14. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  15. Manaal Faruqui and Chris Dyer. 2014a. Community evaluation and exchange of word vectors at word-vectors.org. In Proc. of ACL: System Demo. find
  16. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proc. of NAACL. find our card
  17. Manaal Faruqui and Chris Dyer. 2014b. Improving vector space word representations using multilingual correlation. In Proc. of EACL. find our card
  18. George Miller and Walter Charles. 1991. Contextual correlates of semantic similarity. In Language and Cognitive Processes, pages 1–28. find
  19. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1). find
  20. Jerome L. Myers and Arnold D. Well. 1995. Research Design & Statistical Analysis. Routledge. find
  21. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proc. of EMNLP. find our card
  22. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proc. of SIGKDD. find
  23. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proc. of EMNLP. find
  24. Zellig Harris. 1954. Distributional structure. Word, 10(23):146–162. find
  25. Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: computing word relatedness using temporal semantic analysis. In Proc. of WWW. find
  26. Felix Hill, Roi Reichart, and Anna Korhonen. 2014. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. CoRR, abs/1408.3456. find our card
  27. Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović. 2010. Hubs in space: Popular nearest neighbors in high-dimensional data. The Journal of Machine Learning Research, 11:2487–2531. find
  28. Pushpendre Rastogi, Benjamin Van Durme, and Raman Arora. 2015. Multiview lsa: Representation learning via generalized cca. In Proc. of NAACL. find
  29. Joseph Reisinger and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In Proc. of NAACL. find our card
  30. Sascha Rothe and Hinrich Schütze. 2015. Autoextend: Extending word embeddings to embeddings for synsets and lexemes. In Proc. of ACL. find
  31. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Commun. ACM, 8(10). find
  32. Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proc. of EMNLP. find our card
  33. Walid Shalaby and Wlodek Zadrozny. 2015. Measuring semantic relatedness using mined semantic analysis. CoRR, abs/1512.03465. — [arXiv:1512.03465](https://arxiv.org/abs/1512.03465) find
  34. Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Héctor Martínez Alonso. 2014. What’s in a p-value in nlp? In Proc. of CoNLL. find
  35. James H Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological bulletin, 87(2):245. find
  36. Karl Stratos, Michael Collins, and Daniel Hsu. 2015. Model-based word embeddings from decompositions of count matrices. In Proc. of ACL. find
  37. Nenad Tomašev, Miloš Radovanovic, Dunja Mladenic, and Mirjana Ivanovic. 2011. A probabilistic approach to nearest-neighbor classification: Naive hubness bayesian knn. In Proc. of CIKM. find
  38. Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In Proc. of EMNLP. find
  39. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proc. of ACL. find our card
  40. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning : Vector space models of semantics. JAIR, pages 141–188. find
  41. Dongqiang Yang and David M. W. Powers. 2006. Verb similarity on the taxonomy of wordnet. In 3rd International WordNet Conference. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/faruqui-2016-problems-evaluation-word-embeddings-word/