Elamur Research⌕
☰
home / papers / 2015 / Evaluation methods for unsupervised word embeddings

Evaluation methods for unsupervised word embeddings

Summary

Schnabel and colleagues investigate how word embeddings should be judged. The common practice relies on borrowed benchmark inventories and single aggregate scores, which implicitly assumes there is one "best" embedding. The authors train six unsupervised models (CBOW, C&W, GloVe, Hellinger PCA, TSCCA, Sparse Random Projections) under matched conditions: a 2008 Wikipedia dump, 50 dimensions and a shared vocabulary. They then compare the models on three kinds of criteria: relatedness, neighborhood coherence and downstream performance.

The rankings of the models shift depending on the criterion. Embeddings should therefore be compared for a specific purpose, and extrinsic tasks are not a reliable proxy for general quality. The paper also introduces comparative intrinsic evaluation, in which crowd workers choose between nearest neighbors proposed by different models for a given query. These crowdsourced preferences correlate strongly with offline similarity benchmarks.

The authors release a query inventory calibrated for word frequency, part of speech and abstractness, along with all collected judgments. Finally, they show that embeddings carry substantial word-frequency information, and that this distorts cosine similarity.

Key points

  • All six models were trained on a Wikipedia snapshot from 2008-03-01, chosen to match the pretrained C&W data. The text was processed with the Stanford tokenizer, lower-cased, and had digits replaced by zeros. Every model used D = 50, and nearest-neighbor experiments used the 103,647-word intersection of the models' vocabularies.
  • On 14 absolute intrinsic datasets (relatedness, analogy, categorization and selectional preference, following Baroni et al. 2014):
    • CBOW came out on top in 10 cases but did not lead on selectional preferences.
    • Random projection was the weakest model in 13 cases, with Hellinger PCA next.
    • TSCCA did disproportionately poorly on analogies.
  • A logistic regression classifier separated frequent from rare words using length-normalized vectors, with thresholds from 100 to 50,000. Every embedding beat chance, and GloVe and TSCCA reached close to 100% accuracy for thresholds up to 1000.
  • For C&W, a word's nearest-neighbor rank follows a power law in its corpus frequency rank, roughly nn-rank ∼ 1000 · corpus-rank^0.17. This was measured over the 1000 nearest neighbors of WordSim-353 words.
  • The new query inventory and all user judgments were made public on the authors' Cornell page.

Interesting ideas

  • I1. Comparative evaluation instead of absolute scores: Annotators are shown candidate neighbors from several models for the same query and simply pick the best one. This avoids having to rank unrelated pairs against each other, and choice questions are quick and cheap to crowdsource. The setup could work for comparing any retrieval or representation systems.
  • I2. Designing query inventories deliberately: Test words are sampled to cover a range of frequencies, parts of speech and levels of abstractness, rather than reused from older resources. This makes weak spots such as rare words visible, which matters for any benchmark built for a new language like Ukrainian.
  • I3. Probing for nuisance signals: A simple linear classifier checks whether the vectors encode word frequency. This kind of diagnostic can uncover unintended information in any learned representation before trusting its similarity scores.
  • I4. Being skeptical of raw cosine similarity: Since frequency contaminates neighbor rankings, the authors suggest treating frequency as an explicit factor in experiments and possibly learning a custom similarity metric.
  • I5. Adapting word intrusion to embeddings: The paper borrows the topic-model intrusion test and recasts it as a coherence check on nearest-neighbor sets. This gives a human-interpretable test for models whose dimensions themselves carry no meaning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

129 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 125 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1036

@inproceedings{schnabel2015evaluation,
  title = {{Evaluation methods for unsupervised word embeddings}},
  author = {Schnabel, Tobias and Labutov, Igor and Mimno, David and Joachims, Thorsten},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1036},
  url = {https://doi.org/10.18653/v1/d15-1036}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas and Dan Klein. 2014. How much do word embeddings encode about syntax? In ACL: Short Papers, pages 822–827. find
  2. Javed Aslam and Mark Montague. 2001. Models for metasearch. In SIGIR, pages 276–284. find
  3. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL, pages 238–247. find our card
  4. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. JAIR, 49:1–47. find
  5. James McKeen Cattell. 1886. The time taken up by cerebral operations. Mind,, (42):220–242. find
  6. Jonathan Chang, Jordan Boyd-Graber, Sean Gerrish, Chong Wang, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. In NIPS, pages 288–296. find our card
  7. Yanqing Chen, Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2013. The expressive power of word embeddings. arXiv preprint: 1408.3456. find our card
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JLMR, 12:2493–2537. find our card
  9. Paramveer S. Dhillon, Jordan Rodu, Dean P. Foster, and Lyle H. Ungar. 2012. Two step CCA: A new spectral method for estimating vector models of words. In ICML, pages 1551–1558. find
  10. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLIN-EAR: A library for large linear classification. JLMR, 9:1871–1874. find
  11. Manaal Faruqui and Chris Dyer. 2014. Community evaluation and exchange of word vectors at word-vectors.org. In ACL: System Demonstrations. find
  12. Lev Finkelstein, Ehud Rivlin Zach Solan Gadi Wolfman Evgeniy Gabrilovich, Yossi Matias, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. TOIS, 20(1):116–131, January. find
  13. Bin Gao, Jiang Bian, and Tie-Yan Liu. 2014. WordRep: A benchmark for research on learning word representations. ICML Workshop on Knowledge-Powered Deep Learning for Text Mining. find
  14. Rémi Lebret and Ronan Collobert. 2014. Word embeddings through Hellinger PCA. In EACL, pages 482–490. find
  15. Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In NIPS, pages 2177–2185. find
  16. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015a. Improving distributional similarity with lessons learned from word embeddings. TACL. find
  17. Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015b. Do supervised distributional methods really learn lexical inference relations? In NAACL. find our card
  18. Ping Li, Trevor J Hastie, and Kenneth W Church. 2006. Very sparse random projections. In KDD, pages 287–296. find
  19. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In HLT-ACL, pages 142–150. find our card
  20. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL: System Demonstrations, pages 55–60. find our card
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. find
  22. Tomas Mikolov, Wen-Tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746– 751. find our card
  23. George A Miller. 1995. WordNet: a lexical database for english. Communications of the ACM, 38(11):39–41. find our card
  24. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In HLT-NAACL, pages 100–108. find our card
  25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP. find
  26. Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In EMNLP. find
  27. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In ACL, pages 384– 394. find our card
  28. Peter D. Turney. 2008. A uniform approach to analogies, synonyms, antonyms, and associations. In COLING, pages 905–912. find
  29. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In ACL, pages 947–953. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schnabel-2015-evaluation-methods-unsupervised-word-embeddings/