Elamur Research⌕
☰
home / papers / 2021 / RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)

RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)

Paper · 2021 · Sean Trott and Benjamin Bergen

Summary

Contextual embeddings do well on Word Sense Disambiguation. However, WSD only checks whether a model separates discrete senses. It does not test whether a model follows the graded, context-driven shifts in meaning that people perceive. The authors built RAW-C to test both abilities.

They took 112 ambiguous English words, adapted from earlier psycholinguistic stimuli, and wrote four near-identical sentences for each: two per meaning, usually differing in a single word, with inflection and part of speech held fixed. Human relatedness ratings were collected for all 672 resulting pairs, along with estimates of which sense dominates. Each word is also tagged as homonymous or polysemous using Merriam-Webster and the OED.

Annotators agreed well. Cosine distance between BERT or ELMo embeddings explains only part of the human ratings (R² = .37). Knowing whether two uses share a sense, and whether the word is a homonym, explains much more (R² = .66). This points to categorical sense structure in human judgments that distributional models miss.

Key points

  • RAW-C covers 112 ambiguous words (from an initial pool of 115) and 672 sentence pairs. It includes graded relatedness norms, sense-dominance estimates, and a homonymy/polysemy label taken from Merriam-Webster and the OED. It is released on GitHub (seantrott/raw-c).
  • Items are minimal pairs, for example a fruit/furry bat versus a wooden/baseball bat in the same frame. Target inflection and part of speech stay constant across all four sentences.
  • Inter-annotator agreement, measured leave-one-annotator-out, averaged 0.79. The released data contain only ratings averaged per pair, not single-trial responses.
  • Cosine distance from BERT and ELMo correlates with the human ratings (R² = .37). It systematically rates same-sense uses as less similar than humans do and different-sense homonyms as more similar than humans do. Sense and ambiguity-type labels reach R² = .66.
  • The paper positions RAW-C against datasets that test only one criterion: SimLex-999, WordSim-353, SCWS, WiC, CoSimLex and the WSD inventories. Only Haber and Poesio's 10-polyseme set tests both criteria.

Interesting ideas

  • I1. Split evaluation of word meaning into two separate criteria, discrete disambiguation and graded contextual modulation, so that a system cannot score well by passing only one.
  • I2. Build test items as minimal-pair sentence frames in which a single disambiguating word changes and morphology stays fixed, so any difference in output comes from meaning alone.
  • I3. Compare model similarity against human judgments separately within same-sense and different-sense groups, which exposes systematic compression or exaggeration rather than just one overall correlation.
  • I4. Record whether two meanings are homonymous (unrelated) or polysemous (related), plus which sense is dominant, as explicit lexical metadata.
  • I5. The authors suggest further annotating polysemy by its relation type, such as metaphor, metonymy or Animal-for-Meat, to see which relations can be recovered from text alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could report sense-choice accuracy and graded-similarity agreement as separate scores. A frame selector that merely picks the right label would then not hide poor handling of nuance.
  • I2 (frames): Controlled quartets like these could serve as a regression suite for VerbNet role and frame assignment. They would show exactly which context word flips the chosen frame.
  • I3 (learn): When tuning sense or lexical-unit choices against an LLM teacher, Mova could check for this bias by group. It could then flag cases where homonyms are treated as near neighbors.
  • I4 (world): The compiled WordNet-based ontology could store these two facts per lemma. The world builder could then prefer the dominant sense by default and refuse to merge entities across homonymous senses.
  • I5 (world): Typed sense links like these could be added to the knowledge graph. This would let the world model explain why, for example, "roast chicken" denotes food rather than an animal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California San DiegoSan Diego · United States89 works · 54,032 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Sean Trott and Benjamin Bergen. 2021. RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.550

@inproceedings{trott2021raw,
  title = {{RAW-C: Relatedness of Ambiguous Words in Context (A New Lexical Resource for English)}},
  author = {Sean Trott and Benjamin Bergen},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.550},
  url = {https://doi.org/10.18653/v1/2021.acl-long.550}
}

References

48 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carlos Santos Armendariz, Matthew Purver, Senja Pollak, Nikola Ljubešić, Matej Ulčar, Ivan Vulić, and Mohammad Taher Pilehvar. 2020. SemEval-2020 Task 3: Graded word similarity in context. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 36–49. find
  2. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198. ↗
  3. Katherine S Binder and Keith Rayner. 1998. Contextual strength does not modulate the subordinate bias effect: Evidence from eye fixations and self-paced reading. Psychonomic Bulletin & Review, 5(2):271– 276. ↗
  4. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020. Experience grounds language. arXiv preprint arXiv:2004.10151. ↗
  5. Gemma Boleda, Kristina Gulordava, and Laura Aina. 2019. Putting words in context: LSTM language models and lexical ambiguity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; 2019 Jul 28-Aug 2; Florence, Italy. Stroudsburg (PA): ACL; 2019. p. 3342–8. ACL (Association for Computational Linguistics). find
  6. Susan Windisch Brown. 2008a. Choosing sense distinctions for WSD: Psycholinguistic evidence. In Proceedings of ACL-08: HLT, Short Papers, pages 249–252. find
  7. Susan Windisch Brown. 2008b. Polysemy in the mental lexicon. Colorado Research in Linguistics, 21. ↗
  8. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. ↗
  9. Isabelle Dautriche. 2015. Weaving an ambiguous lexicon. Ph.D. thesis, Sorbonne Paris Cité. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Jeffrey L Elman. 2009. On the meaning of words and dinosaur bones: Lexical knowledge without a lexicon. Cognitive science, 33(4):547–582. ↗
  12. Manaal Faruqui and Chris Dyer. 2015. distributional word vector representations. preprint arXiv:1506.05230. Non-arXiv Christine Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. Cambridge, MA: MIT Press. ↗
  13. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406– 414. find
  14. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform.
  15. Dirk Geeraerts. 1993. Vagueness’s puzzles, polysemy’s vagaries. Cognitive Linguistics (includes Cognitive Linguistic Bibliography), 4(3):223–272. ↗
  16. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. Simverb-3500: A large-scale evaluation set of verb similarity. arXiv preprint arXiv:1608.00869. ↗
  17. Janosch Haber and Massimo Poesio. 2020a. Assessing polyseme sense similarity through co-predication acceptability and contextualised embedding distance. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 114–124. find
  18. Janosch Haber and Massimo Poesio. 2020b. Word sense distance in human similarity judgements and contextualised word embeddings. In Proceedings of the Probability and Meaning Conference (PaM 2020), pages 128–145. find
  19. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1406– 1414. find
  20. Patrick Hanks. 2000. Do word meanings exist? Computers and the Humanities, 34(1/2):205–215. ↗
  21. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. ↗
  22. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882.
  23. Adam Kilgarriff. 2007. Word senses. In Word Sense Disambiguation, pages 29–46. Springer. find
  24. Ekaterini Klepousniotou. 2002. The processing of lexical ambiguity: Homonymy and polysemy in the mental lexicon. Brain and language, 81(1-3):205– 223. ↗
  25. Ekaterini Klepousniotou and Shari R Baum. 2007. Disambiguating the ambiguity advantage effect in word recognition: An advantage for polysemous but not homonymous words. Journal of Neurolinguistics, 20(1):1–24. ↗
  26. Ekaterini Klepousniotou, Debra Titone, and Carolina Romero. 2008. Making sense of word senses: The comprehension of polysemy depends on sense overlap. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6):1534. ↗
  27. Caterina Lacerra, Michele Bevilacqua, Tommaso Pasini, and Roberto Navigli. 2020. CSI: A coarse sense inventory for 85% word sense disambiguation. In AAAI, pages 8123–8130. find
  28. Mallorie Leinenger and Keith Rayner. 2013. Eye movements while reading biased homographs: Effects of prior encounter and biasing context on reducing the subordinate bias effect. Journal of Cognitive Psychology, 25(6):665–681. ↗
  29. Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driving some sense into BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4656–4667, Online. Association for Computational Linguistics.
  30. Jiangtian Li and Marc F Joanisse. 2021. Word senses as clusters of meaning modulations: A computational model of polysemy. Cognitive Science, 45(4):e12955. ↗
  31. Anastasiya Lopukhina, Anna Laurinavichyute, Konstantin Lopukhin, and Olga Dragoy. 2018. The mental representation of polysemy across word classes. Frontiers in psychology, 9:192. ↗
  32. Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2020. Language models and word sense disambiguation: An overview and analysis. arXiv preprint arXiv:2008.11608. ↗
  33. George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993.
  34. Sathvik Nair, Mahesh Srinivasan, and Stephan Meylan. 2020. Contextualized word embeddings encode aspects of human-like word sense knowledge. arXiv preprint arXiv:2010.13057. ↗
  35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  36. Anna Yurchenko, Anastasiya Lopukhina, and Olga Dragoy. 2020. Metaphor is between metonymy and homonymy: Evidence from event-related potentials. Frontiers in Psychology, 11:2113. ↗
  37. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  38. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2018. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv:1808.09121. ↗
  39. Keith Rayner, Jeremy M Pacht, and Susan A Duffy. 1994. Effects of prior encounter and global discourse bias on the processing of lexically ambiguous words: Evidence from eye fixations. Journal of memory and language, 33(4):527–544. ↗
  40. Jennifer M Rodd, M Gareth Gaskell, and William D Marslen-Wilson. 2004. Modelling the effects of semantic ambiguity in word recognition. Cognitive science, 28(1):89–104. ↗
  41. Mahesh Srinivasan and Hugh Rabagliati. 2015. How concepts and conventions structure the lexicon: Cross-linguistic evidence from polysemy. Lingua, 157:124–152. ↗
  42. Mohamed Ali Hadj Taieb, Torsten Zesch, and Mohamed Ben Aouicha. 2020. A survey of semantic relatedness evaluation datasets and procedures. Artificial Intelligence Review, 53(6):4407–4448. ↗
  43. Sean Trott, Tiago Timponi Torrent, Nancy Chang, and Nathan Schneider. 2020. (Re) construing Meaning in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020). find
  44. David Tuggy. 1993. Ambiguity, polysemy, and vagueness. Cognitive linguistics, 4(3):273–290. ↗
  45. Salvador Valera. 2020. Polysemy versus homonymy. In Oxford Research Encyclopedia of Linguistics. ↗
  46. Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? Interpretable Word Sense Disambiguation with contextualized embeddings. arXiv preprint arXiv:1909.10430. ↗
  47. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753–5763. ↗
  48. Eiling Yee and Sharon L Thompson-Schill. 2016. Putting concepts into context. Psychonomic bulletin & review, 23(4):1015–1027. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/trott-2021-raw-c-relatedness-ambiguous-words/