Elamur Research
☰
home / papers / 2010 / Multi-Prototype Vector-Space Models of Word Meaning

Multi-Prototype Vector-Space Models of Word Meaning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N10-1013.pdf

Summary

Standard distributional models give each word type one centroid vector. Such a vector cannot represent ambiguous words: club resembles both bat and association, but those two words are unrelated. Reisinger and Mooney propose a multi-prototype alternative. For each word, its corpus occurrences are clustered with a mixture of von Mises-Fisher distributions (movMF) over unigram context features. Each cluster centroid then becomes a separate prototype vector. Word similarity is computed from pairwise prototype comparisons, either averaged or maximized. Contextual variants weight or select prototypes according to how well a given sentence fits each cluster. The number of clusters K connects two classic families: K = 1 gives a prototype model, and K equal to the number of occurrences gives an exemplar model. Models trained on Wikipedia and Gigaword correlate better with human similarity ratings than single-prototype and exemplar baselines, both for isolated words and words in context. Human evaluators also judged the near-synonyms they produced as better. The method needs no lexical resources and is modular, so it can be combined with other clustering methods or feature spaces.

Key points

  • Training data: a September 2009 English Wikipedia snapshot (2.8M articles, 2.05B words) and English Gigaword, third edition (6.6M articles, 3.9B words). Articles under 100 words were removed from both.
  • Features and clustering: unigrams in a 10-word window, weighted by tf-idf or χ², clustered with movMF. The authors chose movMF because it performed best in preliminary comparisons. Similarity uses cosine.
  • Similarity metrics:
    • AvgSim and MaxSim compare isolated words.
    • AvgSimC applies soft cluster assignment for words in context.
    • MaxSimC applies hard cluster assignment for words in context.
  • Robustness and plausibility: pooling prototypes from many values of K performs close to the best single clustering chosen after the fact. Variance among prototype similarities tracks disagreement between human annotators on WordSim-353.
  • Clusters are not human senses: the clusters are finer than WordNet senses and include odd ones, such as a "hurricane" sense of position. Prototype similarity shows essentially no correlation with the USim usage-similarity data (ρ = 0.04).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

148 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 142 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{reisinger2010multi,
  title = {{Multi-Prototype Vector-Space Models of Word Meaning}},
  author = {Reisinger, Joseph and Mooney, Raymond J.},
  booktitle = {Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/N10-1013/}
}

References

32 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre and Phillip Edmond. 2006. Word Sense Disambiguation: Algorithms and Applications (Text, Speech and Language Technology). Springer-Verlag New York, Inc., Secaucus, NJ, USA. find
  2. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proc. of NAACL-HLT-09, pages 19–27. find
  3. F. Gregory Ashby and Leola A. Alfonso-Reese. 1995. Categorization as probability density estimation. J. Math. Psychol., 39(2):216–233. find
  4. L. Douglas Baker and Andrew K. McCallum. 1998. Distributional clustering of words for text classification. In Proceedings of 21st International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 96–103. find
  5. Arindam Banerjee, Inderjit Dhillon, Joydeep Ghosh, and Suvrit Sra. 2005. Clustering on the unit hypersphere using von Mises-Fisher distributions. Journal of Machine Learning Research, 6:1345–1382. find
  6. Alexander Budanitsky and Graeme Hirst. 2006. Evaluating wordnet-based measures of lexical semantic relatedness. Computational Linguistics, 32(1):13–47. find our card
  7. James R. Curran and Marc Moens. 2002. Improvements in automatic thesaurus extraction. In Proceedings of the ACL-02 workshop on Unsupervised lexical acquisition, pages 59–66. find
  8. James R. Curran. 2004. From Distributional to Semantic Similarity. Ph.D. thesis, University of Edinburgh. College of Science. find
  9. Scott C. Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard A. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41:391–407. find
  10. Inderjit S. Dhillon and Dharmendra S. Modha. 2001. Concept decompositions for large sparse text data using clustering. Machine Learning, 42:143–175. find
  11. Katrin Erk, Diana McCarthy, Nicholas Gaylord Investigations on Word Senses, and Word Usages. 2009. Investigations on word senses and word usages. In Proc. of ACL-09. find
  12. Katrin Erk. 2007. A simple, similarity-based model for selectional preferences. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics. Association for Computer Linguistics. find
  13. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: the concept revisited. In Proc. of WWW-01, pages 406–414, New York, NY, USA. ACM. find
  14. Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proc. of IJCAI-07, pages 1606–1611. find
  15. David Graff. 2003. English Gigaword. Linguistic Data Consortium, Philadephia. find
  16. Tom L. Griffiths, Kevin. R. Canini, Adam N. Sanborn, and Daniel. J. Navarro. 2007. Unifying rational models of categorization via the hierarchical Dirichlet process. In Proc. of CogSci-07. find
  17. Zellig Harris. 1954. Distributional structure. Word, 10(23):146–162. find
  18. Lillian Lee. 1999. Measures of distributional similarity. In 37th Annual Meeting of the Association for Computational Linguistics, pages 25–32. find
  19. Dekang Lin and Patrick Pantel. 2002. Concept discovery from text. In Proc. of COLING-02, pages 1–7. find
  20. Dekang Lin, Shaojun Zhao, Lijuan Qin, and Ming Zhou. 2003. Identifying synonyms among distributionally similar words. In Proceedings of the Interational Joint Conference on Artificial Intelligence, pages 1492– 1493. Morgan Kaufmann. find
  21. Bradley C. Love, Douglas L. Medin, and Todd M. Gureckis. 2004. SUSTAIN: A network model of category learning. Psych. Review, 111(2):309–332. find
  22. Will Lowe. 2001. Towards a theory of semantic space. In Proceedings of the 23rd Annual Meeting of the Cognitive Science Society, pages 576–581. find
  23. Patrick Pantel and Dekang Lin. 2002. Discovering word senses from text. In Proc. of SIGKDD-02, pages 613– 619, New York, NY, USA. ACM. find
  24. Fernando C. N. Pereira, Naftali Tishby, and Lillian Lee. 1993. Distributional clustering of English words. In Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics (ACL-93), pages 183–190, Columbus, Ohio. find
  25. Daniel Ramage, Anna N. Rafferty, and Christopher D. Manning. 2009. Random walks for text semantic similarity. In Proc. of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4), pages 23–31. find
  26. Carl E. Rasmussen. 2000. The infinite Gaussian mixture model. In Advances in Neural Information Processing Systems, pages 554–560. MIT Press. find
  27. Yves Rosseel. 2002. Mixture models of categorization. J. Math. Psychol., 46(2):178–210. find
  28. Mark Sanderson. 1994. Word sense disambiguation and information retrieval. In Proc. of SIGIR-94, pages 142–151. find
  29. Hinrich Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–123. find our card
  30. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proc. of EMNLP-08. find
  31. Amos Tversky and Itamar Gati. 1982. Similarity, separability, and the triangle inequality. Psychological Review, 89(2):123–154. find
  32. Bram Vandekerckhove, Dominiek Sandra, and Walter Daelemans. 2009. A robust and extensible exemplar-based model of thematic fit. In Proc. of EACL 2009, pages 826–834. Association for Computational Linguistics. find