Elamur Research⌕
☰
home / papers / 2014 / Linguistic Regularities in Sparse and Explicit Word Representations

Linguistic Regularities in Sparse and Explicit Word Representations

Summary

The paper asks whether the analogy-solving ability of neural word embeddings, popularized by examples like king − man + woman ≈ queen, comes from the embedding process itself or from the co-occurrence statistics underneath it. The authors show that Mikolov et al.'s additive method (3COSADD) on unit-normalized vectors is algebraically the same as choosing a word that is similar to b and a* and dissimilar to a. This turns relational similarity into a combination of three ordinary word similarities. That view motivates a modified objective, 3COSMUL, which reports state-of-the-art results on two recent word-analogy datasets. The authors compare skip-gram embeddings from word2vec with a sparse, explicit PPMI word–context representation. Both are built from the same English Wikipedia corpus and use the same window, so the comparison is fair. The explicit vectors recover a similar amount of relational structure. The authors conclude that neural embeddings mainly preserve patterns already present in the co-occurrence matrix, and that explicit representations are a strong baseline for this kind of research.

Key points

  • The corpus is English Wikipedia article text, chosen so the work can be reproduced. It was lowercased, deduplicated, and stripped of sentences under 5 tokens, which left roughly 77.5 million sentences totaling around 1.5 billion tokens.
  • The vocabulary contains 189,533 words, each occurring at least 100 times.
  • Contexts come from a window of 2 tokens on each side, with each relative position treated as a separate context.
  • The explicit vectors use PPMI weights. They are very sparse: the mean number of non-zero dimensions is 1595 and the median is 415.
  • The neural baseline is word2vec skip-gram with 600 dimensions, 15 negative samples (NEG-15) and subsampling at 10⁻⁵.
  • Evaluation covers the SEMEVAL dataset (79 relations from SemEval 2012 Task 2) cast as analogy questions. Two methods are compared: 3COSADD and PAIRDIRECTION, which compares the directions of offset vectors.
  • The authors note that Mikolov et al. silently used PAIRDIRECTION for the semantic SemEval analogies and 3COSADD for the syntactic ones, which they confirmed with the original authors.

Interesting ideas

  • I1. Rewrite a puzzling method as a combination of simple scores. Expanding vector arithmetic into "near b, near a*, far from a" makes the method easy to explain and shows how to improve it. A similar rewrite could help analyze other heuristics built on embedding arithmetic.
  • I2. Use multiplication instead of addition when combining similarities. Multiplying and dividing the three similarities stops one large term from dominating the sum. This is a general trick for any ranking built from several similarity scores.
  • I3. Hold everything constant except the representation. Explicit and neural vectors are built from the same corpus, window and vocabulary, so differences in results reflect the representation itself. This is a good template for honest baseline comparisons, for example when testing Ukrainian embeddings against a PPMI baseline.
  • I4. Read relations from interpretable features. Multiplying two sparse vectors element by element (e.g., woman and queen) gives an "aspect vector" whose top contexts show what the two words share. This is a cheap way to inspect what a relation looks like in the data.
  • I5. Expect PMI to favor rare contexts. PMI pushes rare names and places to the top of feature lists. Reordering the features by frequency gives a more intuitive picture, which is worth remembering whenever PMI-weighted features are inspected or used.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

162 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 157 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-1618

@inproceedings{levy2014linguistic,
  title = {{Linguistic Regularities in Sparse and Explicit Word Representations}},
  author = {Levy, Omer and Goldberg, Yoav},
  booktitle = {Proceedings of the Eighteenth Conference on Computational Natural Language Learning},
  year = {2014},
  doi = {10.3115/v1/w14-1618},
  url = {https://doi.org/10.3115/v1/w14-1618}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. In Proc. of CoNLL 2013. find our card
  2. Marco Baroni and Alessandro Lenci. 2009. One distributional memory, many semantic spaces. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 1–8, Athens, Greece, March. Association for Computational Linguistics. find
  3. Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
  4. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Dont count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, Marylan… find our card
  5. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  6. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 546–556, Jeju Island, Korea, July. Associ… find
  7. John A. Bullinaria and Joseph P. Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39(3):510–526. find
  8. Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22–29. find our card
  9. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. find
  10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. find our card
  11. Ido Dagan, Fernando Pereira, and Lillian Lee. 1994. Similarity-based estimation of word cooccurrence probabilities. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 272–278. Association for Computational Linguistics. find
  12. David A Jurgens, Peter D Turney, Saif M Mohammad, and Keith J Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, pages 356–364. Association for Computational Linguistics. find
  13. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  14. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, Maryland, USA, June. Association for Computational Linguistics. find
  15. Dekang Lin and Patrick Pantel. 2001. Dirt: discovery of inference rules from text. In KDD, pages 323– 328. find
  16. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 2, ACL ’98, pages 768–774, Stroudsburg, PA, USA. Association for C… find
  17. Tomas Mikolov, Stefan Kombrink, Lukas Burget, JH Cernocky, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pages 5528–5531. IEEE. find
  18. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. find
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceeding… find
  20. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia,… find
  21. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1439. find
  22. Andriy Mnih and Geoffrey E Hinton. 2008. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems, pages 1081–1088. find
  23. Yoshiki Niwa and Yoshihiko Nitta. 1994. Co-occurrence vectors from corpora vs. distance vectors from dictionaries. In Proceedings of the 15th conference on Computational linguistics-Volume 1, pages 304–309. Association for Computational Linguistics. find
  24. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  25. Fernando Pereira, Naftali Tishby, and Lillian Lee. 1993. Distributional clustering of english words. In Proceedings of the 31st annual meeting on Association for Computational Linguistics, pages 183–190. Association for Computational Linguistics. find
  26. Magnus Sahlgren. 2006. The Word-Space Model: Using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces. Ph.D. thesis, Stockholm. find
  27. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. Association for Computat… find our card
  28. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384–394. Association for Computational Linguistics. find our card
  29. Peter D. Turney and Michael L. Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60(1-3):251–278. find
  30. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141–188. find
  31. Peter D. Turney. 2001. Mining the web for synonyms: Pmi-ir versus lsa on toefl. In Proceedings of the 12th European Conference on Machine Learning, pages 491–502. Springer-Verlag. find
  32. Peter D. Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. find
  33. Peter D. Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533– 585. find
  34. Peter D. Turney. 2013. Distributional semantics beyond words: Supervised learning of analogy and paraphrase. CoRR, abs/1310.5042. find
  35. Alisa Zhila, Wen-tau Yih, Christopher Meek, Geoffrey Zweig, and Tomas Mikolov. 2013. Combining heterogeneous models for measuring relational similarity. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/levy-2014-linguistic-regularities-sparse-explicit-word/