Elamur Research⌕
☰
home / papers / 2014 / Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

Summary

Standard word embeddings such as Skip-gram assign one vector to each word type. A polysemous word like plant then ends up as a blend of its meanings, and through the triangle inequality, unrelated words tied to different senses (e.g., pollen and refinery) get pulled toward each other. The authors extend Skip-gram with negative sampling into Multiple-sense Skip-gram (MSSG). Each word keeps a global vector plus several sense vectors, each with its own context cluster center. During online training, a token's context is summarized as the average of its neighbours' global vectors. The nearest cluster picks the sense, and only that sense vector is updated. Sense discrimination and embedding learning therefore happen jointly, not as a separate clustering step beforehand. A non-parametric variant, NP-MSSG, opens new senses on the fly, so the number of senses varies per word. The models set a new state of the art on the in-context similarity benchmark SCWS. Training on nearly 1 billion tokens takes under 6 hours on one machine, about 27x faster than the earlier multi-sense approach of Huang et al. (2012).

Key points

  • On SCWS (Spearman ρ×100), MSSG-300d scores 69.3 with avgSimC, versus 65.7 for Huang et al. (2012). NP-MSSG-300d reaches 69.1 and is best on globalSim, avgSim and localSim.
  • On WordSim353, MSSG 300d with avgSim gets 70.9, slightly above Skip-gram-300d (70.4) and well above Huang et al. (64.2).
  • On the Google Analogy task, MSSG and NP-MSSG reach 64% accuracy, compared with 12% for Huang et al. and 67% for single-sense Skip-gram.
  • Multiple senses were learned for the same ~6,000 words used by Huang et al. When this is extended to the top 30,000 frequent words (300d, 3 senses), SCWS avgSimC is still 69.2.
  • Global vectors, sense vectors, cluster centers and training code are publicly released. By comparison, Huang et al.'s method needed about a week of computation for a similar setup.

Interesting ideas

  • I1. Joint clustering and learning: the sense of each token is chosen with the model's current parameters, so the clustering improves as the vectors improve. This avoids a costly two-stage pipeline and could apply to any task where latent categories and representations are learned together.
  • I2. Distance-triggered cluster creation: following facility location (Meyerson, 2001), a new sense is created with probability proportional to how far a context is from the nearest existing sense. This simple online rule removes the need to fix the number of clusters and could serve other streaming clustering problems.
  • I3. Separate global vectors for contexts: contexts are represented by single-sense global vectors while target words get sense-specific ones. This keeps context representations stable and cheap, a design that suits other multi-prototype models.
  • I4. Evaluating with and without context: the paper reports several similarity measures (globalSim, avgSim, avgSimC, localSim) side by side. This shows where sense modelling helps (in-context similarity) and where it hurts (context-free analogies). The idea is useful when building benchmarks for ambiguous words in any language, for example a Ukrainian SCWS-style dataset.
  • I5. Triangle-inequality argument: this geometric point explains concretely why one vector per word distorts the space. It is a good diagnostic lens when checking odd nearest neighbours in existing embeddings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

151 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 146 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1113

@inproceedings{neelakantan2014efficient,
  title = {{Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space}},
  author = {Neelakantan, Arvind and Shankar, Jeevan and Passos, Alexandre and McCallum, Andrew},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1113},
  url = {https://doi.org/10.3115/v1/d14-1113}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring Continuous Word Representations for Dependency Parsing. Association for Computational Linguistics (ACL). find our card
  2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research (JMLR). find
  3. Peter F. Brown, Peter V. Desouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based N-gram models of natural language Computational Linguistics. find
  4. Ronan Collobert and Jason Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. International Conference on Machine learning (ICML). find
  5. Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. 2011. Multi-View Learning of Word Embeddings via CCA. Advances in Neural Information Processing Systems (NIPS). find
  6. John Duchi, Elad Hazan, and Yoram Singer 2011. Adaptive sub- gradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (JMLR). find
  7. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: the concept revisited. International Conference on World Wide Web (WWW). find
  8. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. International Joint Conference on Artificial Intelligence (IJCAI). find
  9. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving Word Representations via Global Context and Multiple Word Prototypes. Association of Computational Linguistics (ACL). find our card
  10. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple Semi-supervised Dependency Parsing. Association for Computational Linguistics (ACL). find our card
  11. Quoc V. Le and Tomas Mikolov. 2014 Distributed Representations of Sentences and Documents. International Conference on Machine Learning (ICML) find our card
  12. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011 Learning Word Vectors for Sentiment Analysis Association for Computational Linguistics (ACL) find our card
  13. Adam Meyerson. 2001 IEEE Symposium on Foundations of Computer Science. International Conference on Machine Learning (ICML) find
  14. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. Workshop at International Conference on Learning Representations (ICLR). find
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed Representations of Words and Phrases and their Compositionality. Advances in Neural Information Processing Systems (NIPS). find
  16. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013c. Exploiting Similarities among Languages for Machine Translation. arXiv. find
  17. Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find our card
  18. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. International Conference on Machine learning (ICML). find
  19. Arvind Neelakantan and Michael Collins. 2014. Learning Dictionaries for Named Entity Recognition using Minimal Supervision. European Chapter of the Association for Computational Linguistics (EACL). find
  20. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon Infused Phrase Embeddings for Named Entity Resolution. Conference on Natural Language Learning (CoNLL). find our card
  21. Lev Ratinov and Dan Roth. 2009. Design Challenges and Misconceptions in Named Entity Recognition. Conference on Natural Language Learning (CoNLL). find our card
  22. Siva Reddy, Ioannis P. Klapaftis, and Diana McCarthy. 2011. Dynamic and Static Prototype Vectors for Semantic Composition. International Joint Conference on Artificial Intelligence (IJCNLP). find
  23. Joseph Reisinger and Raymond J. Mooney. 2010a. Multi-prototype vector-space models of word meaning. North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) find our card
  24. Joseph Reisinger and Raymond Mooney. 2010b. A mixture model with sharing for lexical semantics. Empirical Methods in Natural Language Processing (EMNLP). find
  25. Cyrus Shaoul and Chris Westbury. 2010. The Westbury lab wikipedia corpus. find
  26. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011 Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. Advances in Neural Information Processing Systems (NIPS). find
  27. Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012. Cross-lingual Word Clusters for Direct Transfer of Linguistic Structure. North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  28. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word Representations: A Simple and General Method for Semi-Supervised Learning. Association for Computational Linguistics (ACL). find our card
  29. Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual Word Embeddings for Phrase-Based Machine Translation. Empirical Methods in Natural Language Processing. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neelakantan-2014-efficient-non-parametric-estimation-multiple/