Elamur Research⌕
☰
home / papers / 2025 / Investigating Idiomaticity in Word Representations

Investigating Idiomaticity in Word Representations

Paper · 2025 · Wei He, Tiago Kramer Vieira, Marcos Garcia et al.

Summary

Idiomatic noun compounds such as eager beaver have meanings that cannot be built from their parts. Systems that assume compositional meaning may therefore misread them. The authors test whether common word representation models actually encode this idiomaticity in English and Portuguese. They build a dataset of minimal pairs around noun compounds ranging from compositional to fully idiomatic. It includes human idiomaticity ratings at type and token level, paraphrases, and uses of each compound in both naturalistic and sense-neutral sentences, 32,200 sentences in total. Each compound is compared with a synonym, with synonyms of its individual words, and with random items. Two model-agnostic measures are introduced. Affinity checks whether a target sits closer to a related item than to a distractor. Scaled Similarity rescales a model's space using similarity to random items as a baseline. The models tested range from static embeddings to Llama2. Their similarity scores are often high, but they do not reliably capture idiomatic meaning. Contextualized models mostly rely on surface lexical cues rather than meaning from context. This exposes a weakness in handling non-compositional language.

Key points

  • Dataset: English and Portuguese noun compounds with human idiomaticity judgments, paraphrases, and naturalistic plus neutral sentences (32,200 sentences). Code and materials are on GitHub (risehnhew/Finding-Idiomaticity-in-Word-Representations).
  • Models evaluated: Eight off-the-shelf models, none fine-tuned: Llama2, BERT and its variants BERT ML and DistilBERT ML, SBERT ML, ELMo, GloVe and Word2Vec.
  • New metrics: Affinity and Scaled Similarity. They are designed to offset the inflated similarities that anisotropic embedding spaces tend to produce.
  • Robustness check 1: Removing compounds whose chosen synonym overlaps lexically with them leaves the conclusions unchanged, though the Spearman correlations become weaker.
  • Robustness check 2: z-score standardization against rogue dimensions (after Timkey and van Schijndel 2021) barely changes the similarities, with ρ mostly above 0.85 (e.g., SBERT ML at about 0.95–0.99).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations🔬The Alan Turing InstituteLondon · United Kingdom29 works · 4,368 citations🎓Universidade Federal do Rio Grande do SulPorto Alegre · Brazil8 works · 522 citations🎓Universidade de Santiago de CompostelaSantiago de Compostela · Spain13 works · 364 citations🎓University of ExeterExeter · United Kingdom12 works · 203 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart and Aline Villavicencio. 2025. Investigating Idiomaticity in Word Representations. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00546

@article{he2025investigating,
  title = {{Investigating Idiomaticity in Word Representations}},
  author = {Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart and Aline Villavicencio},
  journal = {Computational Linguistics},
  year = {2025},
  doi = {10.1162/coli\_a\_00546},
  url = {https://doi.org/10.1162/coli_a_00546}
}

References

37 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Acosta, Otavio, Aline Villavicencio, and Viviane Moreira. 2011. Identification and treatment of multiword expressions applied to information retrieval. In Proceedings of the Workshop on Multiword Expressions: from Parsing and Generation to the Real World, pages 101–109. find
  2. Adewumi, Tosin, Foteini Liwicki, and Marcus Liwicki. 2022. Vector representations of idioms in conversational systems. Sci, 4(4):37. https://doi.org/10 .3390/sci4040037 Akbik, Alan, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framewo… find
  3. Apidianaki, Marianna. 2022. From word types to tokens and back: A survey of approaches to word meaning representation and interpretation. Computational Linguistics, 49(2):465–523.
  4. Baroni, Marco, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: A collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. https://doi.org/10.1007/s10579-009-9081-4 Baroni, Marco, Georgiana Dinu, and… ↗
  5. Baroni, Marco and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. https://doi.org/10.1162/coli_a_00016 ↗
  6. Bojanowski, Piotr, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. https://doi.org/10.1162/tacl_a_00051 ↗
  7. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Quinta de…
  8. Chakrabarty, Tuhin, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159. https://doi.org/10.18653/v1 /2022.emnlp-…
  9. Contreras Kallens, Pablo and Morten H. Christiansen. 2022. Models of language and multiword expressions. Frontiers in Artificial Intelligence, 5:781962. https://doi.org/10.3389/frai.2022.781962, PubMed: 35252848 Cook, Paul, Afsaneh Fazly, and Suzanne Stevenson. 2008. The VNC-tokens dataset. In Proc… — [doi:10.3389/frai.2022.781962](https://doi.org/10.3389/frai.2022.781962) ↗
  10. Cordeiro, Silvio, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019. Unsupervised compositionality prediction of nominal compounds. Computational Linguistics, 45(1):1–57. https://doi.org/10.1162/coli a 00341 Dankers, Verna, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too c… ↗
  11. Fakharian, Samin and Paul Cook. 2021. Contextualized embeddings encode monolingual and cross-lingual knowledge of idiomaticity. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 23–32. https://doi.org/10.18653/v1/2021.mwe-1.4 Fazly, Afsaneh, Paul Cook, and Suzanne Steve… ↗
  12. Garcia, Marcos, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021b. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564. https://…
  13. Hashempour, Reyhaneh and Aline Villavicencio. 2020. Leveraging contextual embeddings and idiom principle for detecting idiomaticity in potentially idiomatic expressions. In Proceedings of the Workshop on the Cognitive Aspects of the Lexicon, pages 72–80. find
  14. Henderson, James. 2020. The unstoppable rise of computational linguistics in deep learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6294–6306. https://doi.org/10.18653/v1/2020.acl-main.561 Hupkes, Dieuwke, Verna Dankers, Mathijs Mul, and Eli… ↗
  15. Landauer, Thomas K. and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211. https://doi.org/10.1037//0033-295X.104.2.211 Lenci, Alessandro, Magnus Sahlgren, Patrick Je… ↗
  16. Liu, Chunxi, Qiaochu Zhang, Xiaohui Zhang, Kritika Singh, Yatharth Saraf, and Geoffrey Zweig. 2020. Multilingual graphemic hybrid ASR with massive data augmentation. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Co… find
  17. Liu, Emmy and Graham Neubig. 2022. Are representations built from the ground up? An empirical examination of local composition in language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9053–9073. https://doi.org/10.18653/v1 /2022.emnlp-main.… find
  18. Miller, George A. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39–41. https://doi.org/10.1145/219717.219748 Misra, Kanishka, Julia Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance… ↗
  19. Nandakumar, Navnita, Timothy Baldwin, and Bahar Salehi. 2019. How well do embedding models capture non-compositionality? A view from multiword expressions. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 27–34. https://doi.org/10.18653/v1/W19-2004 ↗
  20. Nedumpozhimana, Vasudevan and John Kelleher. 2021. Finding BERT’s idiomatic key. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 57–62. https://doi.org/10 .18653/v1/2021.mwe-1.7 Neelakantan, Arvind, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qimi…
  21. Nunberg, Geoffrey, Ivan A. Sag, and Thomas Wasow. 1994. Idioms. Language, 70:491–538. https://doi.org/10.1353/lan .1994.0007 Pennington, Jeffrey, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods i… ↗
  22. Peters, Matthew E., Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  23. Phelps, Dylan, Thomas Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. 2024. Sign of the times: Evaluating the use of large language models for idiomaticity detection. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD 2024). find
  24. Pires, Telmo, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001. https://doi.org/10.18653/v1/P19-1493 Prasad, Grusha, Marten van Schijndel, and Tal Linzen. 2019. U… ↗
  25. Ramisch, Carlos. 2023. Multiword Expressions in Computational Linguistics. Down the Rabbit Hole and Through the Looking Glass. Dissertation, Aix Marseille University. Available https://tel.archives -ouvertes.fr/tel-0421622 Reddy, Siva, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study o… ↗
  26. Reimers, Nils and Iryna Gurevych. 2019a. Alternative weighting schemes for ELMo embeddings. CoRR, abs/1904.02954. — [arXiv:1904.02954](https://arxiv.org/abs/1904.02954) find
  27. Reimers, Nils and Iryna Gurevych. 2019b. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992…
  28. Multiword expressions: A pain in the neck for NLP. In Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2002), pages 1–15. https://doi.org/10.1007/3-540-45715-1_1 Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 20… ↗
  29. Schuster, Tal, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan…
  30. Tayyar Madabushi, Harish, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 Task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 10…
  31. Tayyar Madabushi, Harish, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. AStitchInLanguageModels: Dataset and methods for the exploration of idiomaticity in pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3464–3477. htt…
  32. Wang, Alex, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355…
  33. Warstadt, Alex, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392. https://doi.org/10.1162/tacl a 00321 Wiedemann, G… ↗
  34. Wolf, Thomas, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. find
  35. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6 Yu, Lang and Allyson Ettinger. 2020. Assessing phrasal representa… ↗
  36. Zeng, Ziheng and Suma Bhat. 2022. Getting BART to ride the idiomatic train: Learning to represent idiomatic expressions. Transactions of the Association for Computational Linguistics, 10:1120–1137. https://doi.org/10.1162/tacl_a_00510 ↗
  37. Zeng, Ziheng and Suma Bhat. 2023. Unified representation for non-compositional and compositional expressions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11696–11710. https://doi.org/10.18653/v1/2023.findings-emnlp.783 Zeng, Ziheng, Kellen Cheng, Srihari Nanniyur… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2025-investigating-idiomaticity-word-representations/