Elamur Research⌕
☰
home / papers / 2014 / A Unified Model for Word Sense Representation and Disambiguation

A Unified Model for Word Sense Representation and Disambiguation

Summary

Standard word embeddings give each word one vector, which blends the meanings of ambiguous words. Earlier multi-prototype approaches (Reisinger and Mooney, 2010; Huang et al., 2012) induce senses by clustering contexts. This has three drawbacks: the number of clusters must be chosen in advance, training runs offline, and the resulting prototypes are not linked to any dictionary sense, so they cannot be used directly for disambiguation.

Chen, Liu and Sun propose a WordNet-anchored framework in which learning sense vectors (WSR) and resolving which sense a word takes in context (WSD) reinforce one another. The method has three steps:

  1. Train word vectors with Skip-gram.
  2. Initialise each sense vector from gloss words that are similar to the target word, then use a knowledge-based disambiguator to label raw text with senses.
  3. Retrain with a modified Skip-gram objective that learns word and sense vectors jointly from the disambiguated occurrences.

The model improves agreement with human judgments on contextual word similarity compared with earlier WSR methods. It outperforms state-of-the-art supervised systems on domain-specific WSD and is competitive on coarse-grained all-words WSD. Because it needs only unlabeled text and a sense inventory with definitions, it can be applied to other lexicons.

Key points

  • Sense vector initialisation. The gloss is filtered to content words (noun, verb, adjective, adverb), excluding the target word itself. A word is kept only if its cosine similarity to the target exceeds a threshold δ, and the kept vectors are averaged.
    • For bank (sloping-land sense) with δ = 0.05, the kept words are sloping, land, slope, water, canoe, sat and river.
    • Weakly related words such as body, pulled, watch and currents are dropped.
  • Two disambiguation algorithms. Both start from a context vector equal to the mean of the sentence's content-word vectors, skip monosemous and non-content words, and choose the sense with the highest cosine similarity to the context.
    • L2R processes words in natural left-to-right order.
    • S2C processes words with fewer WordNet senses first; for example, lake (3 senses) is handled before bank and sat (10 senses each).
  • Confidence-gated context update. If the best sense beats the runner-up by more than a margin ε, that sense vector replaces the word vector inside the context. This sharpens the context for words still waiting to be disambiguated.
  • Evaluation. The model is tested on three tasks: contextual word similarity, domain-specific WSD and coarse-grained all-words WSD. The learned sense representations are released for download.
  • Inventory independence. WordNet is used in the experiments, but the authors state that any resource with sense distinctions and definitions could serve as input.

Interesting ideas

  • I1. Similarity-filtered gloss averaging. Building a starting vector for a dictionary entry from its definition, while discarding definition words unrelated to the headword, is a cheap way to seed sense or concept embeddings. It suits any language that has a dictionary but no sense-annotated corpus.
  • I2. Easiest-first decoding. Resolving low-ambiguity items before high-ambiguity ones, so that confident decisions inform harder ones, avoids searching over all sense combinations. The same ordering heuristic could apply to entity linking or toponym resolution.
  • I3. Margin-gated self-training. Only decisions with a clear winner are allowed to update the shared context. This limits error propagation when a system bootstraps from its own predictions, and the idea transfers to other iterative labelling loops.
  • I4. Mutual bootstrapping of representation and labelling. Better vectors produce better automatic sense labels, and better labels produce better vectors. This loop turns unlabeled text plus a lexicon into training data without manual annotation.
  • I5. Anchoring vectors to inventory entries. Tying each vector to an explicit sense ID, rather than an anonymous cluster, fixes the number of senses per word from the resource. It also makes outputs interpretable and lets a new sense be added just by writing its definition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 86 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1110

@inproceedings{chen2014unified,
  title = {{A Unified Model for Word Sense Representation and Disambiguation}},
  author = {Chen, Xinxiong and Liu, Zhiyuan and Sun, Maosong},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1110},
  url = {https://doi.org/10.3115/v1/d14-1110}
}

References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Oier Lopez De Lacalle, Aitor Soroa, and Informatika Fakultatea. 2009. Knowledge-based wsd and specific domains: Performing better than generic supervised wsd. In Proceedings of IJCAI, pages 1501–1506. find
  2. L Douglas Baker and Andrew Kachites McCallum. 1998. Distributional clustering of words for text classification. In Proceedings of SIGIR, pages 96– 103. find
  3. Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senécal, Fréderic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In Innovations in Machine Learning, pages 137–186. find
  4. Antoine Bordes, Jason Weston, Ronan Collobert, Yoshua Bengio, et al. 2011. Learning structured embeddings of knowledge bases. In Proceedings of AAAI, pages 301–306. find
  5. Yee Seng Chan, Hwee Tou Ng, and Zhi Zhong. 2007. Nus-pt: exploiting parallel texts for word sense disambiguation in the english all-words tasks. In Proceedings of SemEval, pages 253–256. find
  6. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167. find
  7. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, 12:2493–2537. find our card
  8. Montse Cuadros and German Rigau. 2006. Quality assessment of large scale knowledge resources. In Proceedings of EMNLP, pages 534–541. find
  9. Katrin Erk and Sebastian Pado. 2010. Exemplar-based models for word meaning in context. In Proceedings of ACL, pages 92–97. find
  10. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of WWW, pages 406– 414. find
  11. Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning sense-specific word embeddings by exploiting bilingual resources. In Proceedings of COLING, pages 497–507. find
  12. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873–882. find our card
  13. Rob Koeling and Diana McCarthy. 2007. Sussx: Wsd using automatically acquired predominant senses. In Proceedings of SemEval, pages 314–317. find
  14. Rob Koeling, Diana McCarthy, and John Carroll. 2005. Domain-specific sense distributions and predominant sense acquisition. In Proceedings of HLT-EMNLP, pages 419–426. find
  15. Linlin Li, Benjamin Roth, and Caroline Sporleder. 2010. Topic models for word sense disambiguation and token-based idiom detection. In Proceedings of ACL, pages 1138–1147. find
  16. Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and Maosong Sun. 2010. Automatic keyphrase extraction via topic decomposition. In Proceedings of EMNLP, pages 366–376. find
  17. Zhiyuan Liu, Xinxiong Chen, and Maosong Sun. 2011a. A simple word trigger method for social tag suggestion. In Proceedings of EMNLP, pages 1577– 1588. find
  18. Zhiyuan Liu, Xinxiong Chen, Yabin Zheng, and Maosong Sun. 2011b. Automatic keyphrase extraction by bridging vocabulary gap. In Proceedings of CoNLL, pages 135–144. find
  19. Zhiyuan Liu, Xinxiong Chen, and Maosong Sun. 2012. Mining the interests of chinese microbloggers via keyword extraction. Frontiers of Computer Science, 6(1):76–87. find
  20. Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge University Press Cambridge. find
  21. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Proceedings of ICLR. find
  22. Tomas Mikolov. 2012. Statistical Language Models Based on Neural Networks. Ph.D. thesis, Ph. D. thesis, Brno University of Technology. find
  23. George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker. 1993. A semantic concordance. In Proceedings of the workshop on HLT, pages 303– 308. find
  24. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. find our card
  25. Andriy Mnih and Geoffrey E Hinton. 2008. A scalable hierarchical distributed language model. In Proceedings of NIPS, pages 1081–1088. find
  26. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of the international workshop on artificial intelligence and statistics, pages 246–252. find
  27. Roberto Navigli and Mirella Lapata. 2010. An experimental study of graph connectivity for unsupervised word sense disambiguation. IEEE PAMI, 32(4):678–692. find
  28. Roberto Navigli and Paola Velardi. 2005. Structural semantic interconnections: a knowledge-based approach to word sense disambiguation. IEEE PAMI, 27(7):1075–1086. find
  29. Roberto Navigli, Kenneth C Litkowski, and Orin Hargraves. 2007. Semeval-2007 task 07: Coarse-grained english all-words task. In Proceedings of SemEval, pages 30–35. find our card
  30. Roberto Navigli. 2009. Word sense disambiguation: A survey. CSUR, 41(2):10. find
  31. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of EMNLP. find our card
  32. Simone Paolo Ponzetto and Roberto Navigli. 2009. Large-scale taxonomy mapping for restructuring and integrating wikipedia. In Proceedings of IJCAI, volume 9, pages 2083–2088. find
  33. Simone Paolo Ponzetto and Roberto Navigli. 2010. Knowledge-rich word sense disambiguation rivaling supervised systems. In Proceedings of ACL, pages 1522–1531. find
  34. Joseph Reisinger and Raymond J Mooney. 2010. Multi-prototype vector-space models of word meaning. In Proceedings of HLT-NAACL, pages 109– 117. find our card
  35. David E Rumelhart, Geoffrey E Hintont, and Ronald J Williams. 1986. Learning representations by backpropagating errors. Nature, 323(6088):533–536. find
  36. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. CSUR, 34(1):1–47. find
  37. Richard Socher, Cliff C Lin, Andrew Ng, and Chris Manning. 2011. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of ICML, pages 129–136. find
  38. Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. 2013. Parsing with compositional vector grammars. In Proceedings of ACL. find
  39. Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2008. Yago: A large ontology from wikipedia and wordnet. Web Semantics: Science, Services and Agents on the World Wide Web, 6(3):203–217. find
  40. Fei Tian, Hanjun Dai, Jiang Bian, Bin Gao, Rui Zhang, Enhong Chen, and Tie-Yan Liu. 2014. A probabilistic model for learning multi-prototype word embeddings. In Proceedings of COLING, pages 151–160. find
  41. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of A-CL, pages 384–394. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2014-unified-model-word-sense-representation/