WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1128.pdf
Summary
Standard word embeddings give each word a single vector, so they cannot show that one word form can carry different meanings in different contexts. Sense embeddings and contextualized models such as ELMo and BERT were built to fix this, but there was almost no benchmark that tests context sensitivity directly. The only established option, Stanford Contextual Word Similarity (SCWS), has low inter-annotator agreement. Even a model that ignores context can beat its estimated human ceiling. The authors present WiC, a binary classification task. Each item gives one target noun or verb in two sentences, and the system must decide whether the word has the same sense in both. The sentences come from lexicographer examples in WordNet, VerbNet and Wiktionary, linked through BabelNet. Item pairs whose senses were too close to tell apart were removed automatically. Because both sentences contain the same word, a static embedding cannot beat chance. In tests, BERT large scored best, but only about 14 points above a random baseline and well below human accuracy. This makes WiC a demanding, freely available benchmark for context-sensitive meaning representations.
Key points
- Dataset size: WiC has 5,428 training, 638 development and 1,400 test instances, each split evenly between positive and negative labels. No example sentence appears in more than one split, and only 36% of test target words also occur in training.
- Sense pruning: To limit WordNet's very fine sense distinctions, the authors dropped pairs whose senses were directly linked in the WordNet graph (including sister senses) or shared one of the 44 lexicographer supersenses.
- Human performance: Annotators who were not shown any sense inventory reached about 80% accuracy on the final data. On a sample of the pruned items they managed only 57%, which supports the pruning step.
- Model results: The best model, BERT large, scored roughly 64%, about 16 points below the human estimate. A simple cosine-threshold classifier beat an MLP. ELMo and Context2Vec did not beat a sentence bag-of-words baseline, and DeConf was the strongest multi-prototype model.
- Problems with SCWS: Pairwise annotator agreement on SCWS is only 0.35 Spearman ρ, and the mean agreement (the human ceiling) is 0.52. Google News Word2vec, which ignores context, reaches 0.65.
Cited by
- Muppet: Massive Multi-task Representations with Pre-Finetuning
- MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)
- On the Cross-lingual Transferability of Monolingual Representations
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Adapting Language Models to Compress Contexts
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Analysing Lexical Semantic Change with Contextualised Word Representations
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
- Generationary or «How We Went beyond Word Sense Inventories and Learned to Gloss»
- Large Scale Substitution-based Word Sense Induction
- Do Large Language Models Understand Word Senses?
- In the LLM era, Word Sense Induction remains unsolved
- GPT Understands, Too
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- A Comprehensive Overview of Large Language Models
- True Few-Shot Learning with Language Models
- Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- How many data points is a prompt worth?
- It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
- Language Models are Multilingual Chain-of-Thought Reasoners
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Emergent Abilities of Large Language Models
- BloombergGPT: A Large Language Model for Finance
- TIES-Merging: Resolving Interference When Merging Models
- Can Large Language Models Transform Computational Social Science?
Cite
@inproceedings{pilehvar2019wic,
title = {{WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations}},
author = {Pilehvar, Mohammad Taher and Camacho-Collados, Jose},
booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
year = {2019},
doi = {10.18653/v1/n19-1128},
eprint = {1808.09121},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/n19-1128}
}References
- Jose Camacho-Collados and Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743–788. find
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, Minneapolis, United States. find
- Haim Dubossarsky, Eitan Grossman, and Daphna Weinshall. 2018. Coming to your senses: on controls and evaluation sets in polysemy research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1732–1740, Brussels, Belgium. find
- Christiane Fellbaum, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA. find
- Lucie Flekova and Iryna Gurevych. 2016. Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization. In Proceedings of ACL. find
- Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics. find our card
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
- Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873–882, Jeju Island, Korea. find
- Karin Kipper-Schuler. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. Ph.D. thesis. find
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
- Roberto Navigli. 2006. Meaningful clustering of senses helps boost Word Sense Disambiguation performance. In Proceedings of COLING-ACL, pages 105–112, Sydney, Australia. find
- Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217– 250. find
- Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of EMNLP, pages 1059–1069, Doha, Qatar. find
- Maria Pelevina, Nikolay Arefyev, Chris Biemann, and Alexander Panchenko. 2016. Making sense of word embeddings. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 174–183. find
- M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL, New Orleans, LA, USA. find
- Mohammad Taher Pilehvar, Jose Camacho-Collados, Roberto Navigli, and Nigel Collier. 2017. Towards a Seamless Integration of Word Senses into Downstream NLP Applications. In Proceedings of ACL, Vancouver, Canada. find
- Mohammad Taher Pilehvar and Nigel Collier. 2016. De-conflated semantic representations. In Proceedings of EMNLP, pages 1680–1690, Austin, TX. find
- Massimiliano Mancini, Jose Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2017. Embedding words and senses together via joint knowledge-enhanced training. In Proceedings of CoNLL, pages 100–111, Vancouver, Canada. find
- Mohammad Taher Pilehvar, David Jurgens, and Roberto Navigli. 2013. Align, Disambiguate and Walk: a Unified Approach for Measuring Semantic Similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1341–1351, Sofia, Bulgaria. find
- Diana McCarthy, Marianna Apidianaki, and Katrin Erk. 2016. Word sense clustering and clusterability. Computational Linguistics. find
- Joseph Reisinger and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In Proceedings of ACL, pages 109–117. find our card
- Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional lstm. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. find our card
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. find
- Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
- Stefan Rüd, Massimiliano Ciaramita, Jens Müller, and Hinrich Schütze. 2011. Piggyback: Using search engines for robust cross-domain named entity recognition. In Proceedings of ACL-HLT, pages 965–975, Portland, Oregon, USA. find
- Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013. Learning semantic textual similarity with structural representations. In Proceedings of ACL (2), pages 714–718, Sofia, Bulgaria. find
- Rion Snow, Sushant Prakash, Daniel Jurafsky, and Andrew Y. Ng. 2007. Learning to merge word senses. In Proceedings of EMNLP, pages 1005– 1014, Prague, Czech Republic. find
- Yogarshi Vyas and Marine Carpuat. 2017. Detecting asymmetric semantic relations in context: A case-study on hypernymy detection. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 33–43. Association for Computational Linguistics. find
- George K. Zipf. 1949. Human Behaviour and the Principle of Least-Effort. Addison-Wesley, Cambridge, MA. find