Elamur Research⌕
☰
home / papers / 2015 / Do Multi-Sense Embeddings Improve Natural Language Understanding?

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Paper · 2015 · Jiwei Li and Dan Jurafsky

Summary

The paper asks whether word embeddings that keep a separate vector for each sense of an ambiguous word actually help real NLP tasks, rather than only artificial word-similarity benchmarks. The authors propose an unsupervised multi-sense model that adds Chinese Restaurant Processes (CRP) to a standard neural language model such as skip-gram. Each token has a global vector plus a growing set of sense vectors. A new sense is created only when context evidence and sense popularity favor it. The model reaches state-of-the-art results on matching human word similarity judgments.

They also build a three-step pipeline: learn sense embeddings, infer token senses, then compose phrase or sentence representations for classifiers. The evaluation covers five tasks (tagging words by grammatical category, detecting named entities, classifying sentiment, identifying semantic relations and measuring semantic relatedness), always comparing against single-vector embeddings of matched dimensionality. Multi-sense vectors help on some tasks (POS tagging, semantic relations, relatedness) but not on others (NER, sentiment). Simply doubling the dimensionality of ordinary embeddings often gives the same gain. The work matters because it shows that intrinsic benchmarks can overstate the value of an embedding method.

Key points

  • Sense learning uses a CRP. An occurrence joins an existing sense in proportion to two things: how often that sense has been used, and how well it fits the context (local neighbors plus document-level information). A hyperparameter γ governs how readily new senses open. Sense clustering and embedding training happen jointly.
  • Two sense-inference strategies are compared:
    • GREEDY picks the locally best sense for each token.
    • EXPECTATION uses a probability-weighted average of the sense vectors.
    • EXPECTATION performed better, likely because GREEDY makes costly hard errors with limited context.
  • Sentence-level tasks use three kinds of composition model: standard recurrent networks, LSTMs and recursive (tree-structured) networks.
  • Some multi-sense gains disappear when LSTMs are used. The authors attribute this to gating, which lets the model filter useful information out of a single mixed vector.
  • Doubling the dimensionality of baseline skip-gram vectors matches the multi-sense gains. Going up to 300 dimensions helps further but makes training much more expensive.

Interesting ideas

  • I1. Dimension-matched baselines. Comparing a richer representation against a plain one with the same total number of parameters or dimensions exposes gains that come only from capacity. This is a useful sanity check whenever a fancier embedding is proposed.
  • I2. CRP-style "open a new cluster only when needed". Weighting cluster fit by cluster popularity is more robust than a fixed similarity threshold. The same idea could suit any setting where the number of categories is unknown, such as variants of place names or product types.
  • I3. Soft over hard assignment. Representing a token as an expectation over its senses avoids errors from committing to one wrong sense. This is a general lesson for pipelines whose early stage makes discrete decisions.
  • I4. Extrinsic over intrinsic evaluation. A method can win on human-similarity benchmarks and still bring nothing to downstream tasks. New resources (for example, embeddings for Ukrainian) should therefore be validated inside real applications.
  • I5. Expect strong downstream models to absorb gains. Before investing in complex input features, check whether a gated model like an LSTM already recovers the same information.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States310 works · 284,480 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Lexical semantics & word embeddings · 40 works

Cite

Cite as Jiwei Li and Dan Jurafsky. 2015. Do Multi-Sense Embeddings Improve Natural Language Understanding?. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1200

@inproceedings{li2015do,
  title = {{Do Multi-Sense Embeddings Improve Natural Language Understanding?}},
  author = {Jiwei Li and Dan Jurafsky},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1200},
  url = {https://doi.org/10.18653/v1/d15-1200}
}

References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning, pages 137–186. Springer. find
  2. David M. Blei, Thomas L. Griffiths, Michael I. Jordan, and Joshua B. Tenenbaum. 2004. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16. find
  3. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1025–1035. our card
  4. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. our card
  6. Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press. find
  7. Thomas S Ferguson. 1973. A bayesian analysis of some nonparametric problems. The annals of statistics, pages 209–230. find
  8. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406– 414. ACM. find
  9. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Seman… our card
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  11. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for… our card
  12. Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In Advances in Neural Information Processing Systems, pages 2096–2104. find
  13. Walter Kintsch. 2001. Predication. Cognitive Science, 25(2):173–202. find
  14. Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2015. Topical word embeddings. In Twenty-Ninth AAAI Conference on Artificial Intelligence. find
  15. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. SemEval-2014. our card
  16. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH, pages 1045–1048. find
  17. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. our card
  18. Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning, pages 641–648. ACM. find
  19. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of EMNLP. our card
  20. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: Sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics. our card
  21. Barak A Pearlmutter. 1989. Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2):263–269. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. our card
  23. Luis Nieto Pina and Richard Johansson. 2014. A simple and efficient method to generate word sense representations. arXiv preprint arXiv:1412.6045. find
  24. Jim Pitman. 1995. Exchangeable and partially exchangeable random partitions. Probability theory and related fields, 102(2):145–158. find
  25. Lin Qiu, Yong Cao, Zaiqing Nie, and Yong Rui. 2014. Learning word representation considering proximity and ambiguity. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
  26. Joseph Reisinger and Raymond J Mooney. 2010. Multi-prototype vector-space models of word meaning. In NAACL. our card
  27. Herbert Rubenstein and John B Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  28. Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. A… our card
  29. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP. find
  30. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075. find
  31. Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2006. Hierarchical dirichlet processes. Journal of the american statistical association, 101(476). find
  32. Zhaohui Wu and C. Lee Giles. 2015. Sense-aware semantic analysis: A multi-prototype word representation model using wikipedia. In Twenty-Ninth AAAI Conference on Artificial Intelligence. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2015-do-multi-sense-embeddings-improve/