Elamur Research⌕
☰
home / papers / 2013 / Measuring Word Meaning in Context

Measuring Word Meaning in Context

Paper · 2013 · Katrin Erk, Diana McCarthy and Nicholas Gaylord

Summary

Classic word sense disambiguation assumes that every occurrence of a word maps to exactly one dictionary sense. Lexicographers, linguists and psychologists have questioned that assumption, and newer models (paraphrase sets, vector spaces, latent-sense distributions) express meaning in degrees. Evaluating them requires graded human data. Erk, McCarthy and Gaylord propose two rating-based annotation schemes completed by untrained native speakers:

  • WSsim: annotators score how well each WordNet sense fits a given occurrence on an ordinal scale.
  • Usim: annotators judge how similar two occurrences of the same lemma are.

A second, larger annotation round places both schemes alongside single-best-sense labeling and lexical substitution on identical data. Several raters judge each item, and their scores are averaged.

The ratings show good inter-annotator correlation and span the entire scale rather than collapsing to yes/no. They also correlate strongly with both the traditional sense labels and the paraphrase annotations. The work shows that graded meaning judgments are reliable, and it provides evaluation data better suited to context-sensitive meaning models. It also offers a way to audit sense inventories.

Key points

  • The second-round dataset applies four annotation types (WSsim, Usim, WSbest, lexical substitutes) to the same sentences, with eight annotators judging each item in parallel. It is publicly released.
  • For comparison, conventional WordNet-based sense tagging reaches only about 67–78% inter-annotator agreement.
  • In 23% to 46% of sentences, the same annotator rated more than one sense highly. Coarser sense groupings do not resolve this, and no fixed grouping accounts for the WSsim patterns.
  • Usim ratings almost always obey the triangle inequality. This supports distributional models that treat usage similarity as a metric, and it can serve as a validity check for future annotations.
  • WSsim and Usim were used to examine two existing coarse-grained sense groupings. The analysis flagged groups that may not match the intuitions of untrained native speakers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations🎓The University of Texas at AustinAustin · United States72 works · 55,722 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Lexical semantics & word embeddings · 40 works

Cite

Cite as Katrin Erk, Diana McCarthy and Nicholas Gaylord. 2013. Measuring Word Meaning in Context. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00142

@article{erk2013measuring,
  title = {{Measuring Word Meaning in Context}},
  author = {Katrin Erk, Diana McCarthy and Nicholas Gaylord},
  journal = {Computational Linguistics},
  year = {2013},
  doi = {10.1162/coli\_a\_00142},
  url = {https://doi.org/10.1162/coli_a_00142}
}

References

87 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, Eneko and Philip Edmonds, editors. 2007. Word Sense Disambiguation: Algorithms and Applications. Springer, Dordrecht. find
  2. Agirre, Eneko, Lluı́s Màrquez, and Richard Wicentowski, editors. 2007. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Prague. find
  3. Baroni, Marco and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1,183–1,193, Cambridge, MA. find
  4. Brown, Susan. 2008. Choosing sense distinctions for WSD: Psycholinguistic evidence. In Proceedings of ACL-08: HLT, Short Papers (Companion Volume), pages 249–252, Columbus, OH. find
  5. Brown, Susan. 2010. Finding Meaning: Sense Inventories for Improved Word Sense Disambiguation. Ph.D. thesis, University of Colorado at Boulder. find
  6. Burchardt, Aljoscha, Katrin Erk, Annette Frank, Andrea Kowalski, Sebastian Pado, and Manfred Pinkal. 2006. The SALSA corpus: A German resource for lexical semantics. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006), pages 969–974, Genoa. find
  7. Carpuat, Marine and Dekai Wu. 2007a. How phrase sense disambiguation outperforms word sense disambiguation for statistical machine translation. In Proceedings of the 11th Conference on Theoretical and Methodological Issues in Machine Translation (TMI 2007), pages 43–52, Skovde. find
  8. Carpuat, Marine and Dekai Wu. 2007b. Improving statistical machine translation using word sense disambiguation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007), pages 61–72, Prague. our card
  9. Chen, Jinying and Martha Palmer. 2009. Improving English verb sense disambiguation performance with linguistically motivated features and clear sense distinction boundaries. Journal of Language Resources and Evaluation (Special Issue on SemEval-2007), 43:181–208. find
  10. Coecke, Bob, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical foundations for a compositional distributed model of meaning. Lambek Festschrift, Linguistic Analysis, 36:345–384. find
  11. Coleman, Linda and Paul Kay. 1981. Prototype semantics: The English word “lie.” Language, 57:26–44. find
  12. Copestake, Ann and Ted Briscoe. 1995. Semi-productive polysemy and sense extension. Journal of Semantics, 12:15–67. find
  13. Cruse, D. A. 1995. Polysemy and related phenomena from a cognitive linguistic viewpoint. In Philip Saint-Dizier and Evelyne Viegas, editors, Computational Lexical Semantics. Cambridge University Press, pages 33–49. find
  14. Davis, Jason, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit Dhillon. 2007. Information-theoretic metric learning. In Proceedings of the 24th International Conference on Machine Learning, pages 209–216, Corvallis, OR. find
  15. Deschacht, Koen and Marie-Francine Moens. 2009. Semi-supervised semantic role labeling using the Latent Words Language Model. In Proceedings of EMNLP, pages 21–29, Singapore. find
  16. Dinu, Georgiana and Mirella Lapata. 2010. Measuring distributional similarity in context. In Proceedings of the 2010 find
  17. Conference on Empirical Methods in Natural Language Processing, pages 1,162–1,172, Cambridge, MA. Edmonds, Philip and Scott Cotton, editors. 2001. Proceedings of the SensEval-2 Workshop. Toulouse. See http://www.sle.sharp.co.uk/senseval. ↗
  18. Erk, Katrin and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 440–449, Singapore. find
  19. Erk, Katrin, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10–18, Suntec. find
  20. Erk, Katrin and Sebastian Pado. 2008. A structured vector space model for word meaning in context. In Proceedings of EMNLP-08, pages 897–906, Waikiki, HI. our card
  21. Erk, Katrin and Sebastian Pado. 2010. Exemplar-based models for word meaning in context. In Proceedings of the ACL 2010 Conference Short Papers, pages 92–97, Uppsala. find
  22. Erk, Katrin and Carlo Strapparava, editors. 2010. Proceedings of the 5th International Workshop on Semantic Evaluation(SemEval). Uppsala. find
  23. Frazier, Lyn and Keith Rayner. 1990. Taking on semantic commitments: Processing multiple meanings vs. multiple senses. Journal of Memory and Language, 29:181–200. find
  24. Gentner, Dedre and Virginia Gunn. 2001. Structural alignment facilitates the noticing of differences. Memory and Cognition, 21:565–577. find
  25. Gentner, Dedre and Arthur Markman. 1997. Structural alignment in analogy and similarity. American Psychologist, 52:45–56. find
  26. Grefenstette, Edward and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1,394–1,404, Edinburgh. find
  27. Hampton, James A. 1979. Polymorphous concepts in semantic memory. Journal of Verbal Learning and Verbal Behavior, 18:441–461. find
  28. Hampton, James A. 2007. Typicality, graded membership, and vagueness. Cognitive Science, 31:355–384. find
  29. Hanks, Patrick. 2000. Do word meanings exist? Computers and the Humanities, 34(1–2):205–215. find
  30. Hovy, Eduard H., Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (NAACL-2006), pages 57–60, New York. find
  31. Ide, Nancy and Yorick Wilks. 2006. Making sense about sense. In Eneko Agirre and Philip Edmonds, editors, Word Sense Disambiguation, Algorithms and Applications. Springer, Dordrecht, pages 47–73. find
  32. Kilgarriff, Adam. 1992. Polysemy. Ph.D. thesis, University of Sussex. find
  33. Kilgarriff, Adam. 1997. I don’t believe in word senses. Computers and the Humanities, 31(2):91–113. our card
  34. Kilgarriff, Adam. 2006. Word senses. In Eneko Agirre and Philip Edmonds, editors, Word Sense Disambiguation: Algorithms and Applications. Springer, Dordrecht, pages 29–46. find
  35. Kilgarriff, Adam and Martha Palmer, editors. 2000. Senseval: Special Issue of the Journal Computers and the Humanities, volume 34(1–2). Kluwer, Dordrecht. find
  36. Kilgarriff, Adam and Joseph Rosenzweig. 2000. Framework and results for English Senseval. Computers and the Humanities, 34(1-2):15–48. find
  37. Kintsch, Walter. 2007. Meaning in context. In T. K. Landauer, D. McNamara, S. Dennis, and W. Kintsch, editors, Handbook of Latent Semantic Analysis. Erlbaum, Mahwah, NJ, pages 89–105. find
  38. Klein, Devorah and Gregory Murphy. 2001. The representation of polysemous words. Journal of Memory and Language, 45:259–282. find
  39. Klein, Devorah and Gregory Murphy. 2002. Paper has been my ruin: Conceptual relations of polysemous senses. Journal of Memory and Language, 47:548–570. find
  40. Klepousniotou, Ekaterini. 2002. The processing of lexical ambiguity: Homonymy and polysemy in the mental lexicon. Brain and Language, 81:205–223. find
  41. Klepousniotou, Ekaterini, Debra Titone, and Caroline Romero. 2008. Making sense of word senses: The comprehension of polysemy depends on sense overlap. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6):1,534–1,543. find
  42. Krishnamurthy, Ramesh and Diane Nicholls. 2000. Peeling an onion: The lexicographers’ experience of manual sense-tagging. Computers and the Humanities, 34(1-2):85–97. find
  43. Landauer, Thomas and Susan Dumais. 1997. A solution to Plato’s problem: The Latent Semantic Analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104:211–240. find
  44. Landes, Shari, Claudia Leacock, and Randee Tengi. 1998. Building semantic concordances. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA. find
  45. Lefever, Els and Véronique Hoste. 2010. Semeval-2010 task 3: Cross-lingual word sense disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 15–20, Uppsala. find
  46. McCarthy, Diana. 2002. Lexical substitution as a task for wsd evaluation. In Proceedings of the ACL Workshop on Word Sense Disambiguation: Recent Successes and Future Directions, pages 109–115, Philadelphia, PA. find
  47. McCarthy, Diana and Roberto Navigli. 2007. SemEval-2007 task 10: English lexical substitution task. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 48–53, Prague. our card
  48. McCarthy, Diana and Roberto Navigli. 2009. The English lexical substitution task. Language Resources and Evaluation Special Issue on Computational Semantic Analysis of Language: SemEval-2007 and Beyond, 43(2):139–159. find
  49. McNamara, Timothy P. 2005. Semantic Priming: Perspectives from Memory and Word Recognition. Psychology Press, New York. find
  50. Mihalcea, Rada and Timothy Chklovski. 2003. Open Mind Word Expert: Creating large annotated data collections with web users’ help. In Proceedings of the EACL 2003 Workshop on Linguistically Annotated Corpora (LINC 2003), pages 53–60, Budapest. find
  51. Mihalcea, Rada, Timothy Chklovski, and Adam Kilgarriff. 2004. The SENSEVAL-3 English lexical sample task. In Rada Mihalcea and Phil Edmonds, editors, Proceedings SENSEVAL-3 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 25–28, Barcelona. our card
  52. Mihalcea, Rada and Phil Edmonds, editors. 2004. Proceedings SENSEVAL-3 Second International Workshop on Evaluating Word Sense Disambiguation Systems, Barcelona. find
  53. Mihalcea, Rada, Ravi Sinha, and Diana McCarthy. 2010. Semeval-2010 task 2: Cross-lingual lexical substitution. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 9–14, Uppsala. find
  54. Miller, George A., Claudia Leacock, Randee Tengi, and Ross T Bunker. 1993. A semantic concordance. In Proceedings of the ARPA Workshop on Human Language Technology, pages 303–308, Plainsboro, NJ. our card
  55. Mitchell, Jeff and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL-08: HLT, pages 236–244, Columbus, OH. find
  56. Mitchell, Jeff and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  57. Moon, Taesun and Katrin Erk. In press. An inference-based model of word meaning in context as a paraphrase distribution. ACM Transactions on Intelligent Systems and Technology special issue on paraphrasing. find
  58. Murphy, Gregory L. 1991. Meaning and concepts. In Paula Schwanenflugel, editor, The Psychology of Word Meanings. Lawrence Erlbaum Associates, Mahwah, NJ, pages 11–35. find
  59. Murphy, Gregory L. 2002. The Big Book of Concepts. MIT Press, Cambridge, MA. find
  60. Navigli, Roberto. 2009. Word sense disambiguation: a survey. ACM Computing Surveys, 41(2):1–69. find
  61. Navigli, Roberto, Kenneth C. Litkowski, and Orin Hargraves. 2007. SemEval-2007 task 7: Coarse-grained English all-words task. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 30–35, Prague. find
  62. Palmer, Martha, Hoa Trang Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13:137–163. find
  63. Passonneau, Rebecca, Ansaf Salleb-Aouissi, Vikas Bhardwaj, and Nancy Ide. 2010. Word sense annotation of polysemous words by multiple annotators. In Proceedings of LREC-7, pages 3,244–3,249, Valleta. find
  64. Pickering, Martin and Steven Frisson. 2001. Processing ambiguous verbs: Evidence from eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27:556–573. find
  65. Pradhan, Sameer, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. Semeval-2007 task 17: English lexical sample, SRL and all words. In 4th International Workshop on Semantic Evaluations (SemEval-4) at ACL-2007, pages 87–92, Prague. our card
  66. Preiss, Judita and David Yarowsky, editors. 2001. Proceedings of Senseval-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, Toulouse. find
  67. Pustejovsky, James. 1991. The generative lexicon. Computational Linguistics, 17(4):409–441. find
  68. Reddy, Siva, Ioannis P. Klapaftis, Diana McCarthy, and Suresh Manandhar. 2011. Dynamic and static prototype vectors for semantic composition. In Proceedings of The 5th International Joint Conference on Natural Language Processing 2011 (IJCNLP 2011), pages 210–218, Chiang Mai. find
  69. Reisinger, Joseph and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In Proceedings of Human Language Technologies: The 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109–117, Los Angeles, CA. our card
  70. Resnik, Philip and David Yarowsky. 2000. Distinguishing systems and distinguishing senses: New evaluation methods for word sense disambiguation. Natural Language Engineering, 5(3):113–133. find
  71. Rosch, Eleanor. 1975. Cognitive representations of semantic categories. Journal of Experimental Psychology: General, 104:192–233. find
  72. Rosch, Eleanor and Carolyn B. Mervis. 1975. Family resemblance: Studies in the internal structure of categories. Cognitive Psychology, 7:573–605. find
  73. Schütze, Hinrich. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–123. our card
  74. Senseval-2. 2001. Web page: http://www.sle.sharp.co.uk/senseval2. ↗
  75. Snyder, Benjamin and Martha Palmer. 2004. The English all-words task. In 3rd International Workshop on Semantic Evaluations (SensEval-3) at ACL-2004, pages 41–43, Barcelona. find
  76. Socher, Richard, Eric H. Huang, Jeffrey Pennin, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems 24, pages 801–809, Grenada. find
  77. Stokoe, Christopher. 2005. Differentiating homonymy and polysemy in information retrieval. In Proceedings of HLT/EMNLP-05, pages 403–410, Vancouver. find
  78. Taylor, John R. 2003. Linguistic Categorization. Oxford University Press, New York. find
  79. Thater, Stefan, Hagen Fürstenau, and Manfred Pinkal. 2010. Contextualizing semantic representations using syntactically enriched vector models. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 948–957, Uppsala. find
  80. Tuggy, David H. 1993. Ambiguity, polysemy and vagueness. Cognitive Linguistics, 4(2):273–290. find
  81. Tversky, Amos. 1977. Features of similarity. Psychological Review, 84(4):327–352. find
  82. Tversky, Amos and Itamar Gati. 1982. Similarity, separability, and the triangle inequality. Psychological Review, 89(2):123–154. find
  83. Van de Cruys, Tim, Thierry Poibeau, and Anna Korhonen. 2011. Latent vector weighting for word meaning in context. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1,012–1,022, Edinburgh. find
  84. Washtell, Justin. 2010. Expectation vectors: A semiotics inspired approach to geometric lexical-semantic representation. In Proceedings of the 2010 Workshop on GEometrical Models of Natural Language Semantics, pages 45–50, Uppsala. find
  85. Williams, John. 1992. Processing polysemous words in context: Evidence for interrelated meanings. Journal of Psycholinguistic Research, 21:193–218. find
  86. Zhong, Zhi and Hwee Tou Ng. 2010. It makes sense: A wide-coverage word sense disambiguation system for free text. In Proceedings of the ACL 2010 System Demonstrations, pages 78–83, Uppsala. find
  87. Zhong, Zhi, Hwee Tou Ng, and Yee Seng Chan. 2008. Word sense disambiguation using OntoNotes: An empirical study. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1,002–1,010, Honolulu, HI. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/erk-2013-measuring-word-meaning-context/