Elamur Research⌕
☰
home / papers / 2026 / Measuring Idiomaticity in Text Embedding Models with epsilon-compositionality

Measuring Idiomaticity in Text Embedding Models with epsilon-compositionality

Summary

Recent NLP work has tested how text embedding models handle idioms, which are traditionally treated as non-compositional. However, it is unclear how this work connects to the formal semantics notion of compositionality. The authors start from the function and substitution definitions of compositionality (following Pagin and Westerståhl) and derive a continuous relaxation called ε-compositionality. Intuitively, a word occurs in an ε-compositional context when contextualizing it brings its representation closer to its counterparts, and in a non-compositional context when contextualization pushes it away.

They apply the measure to the ncimp dataset of minimal sentence pairs containing compositional and idiomatic multi-word expressions (MWEs). Idiomatic ε values are compared against compositional baselines with a Wilcoxon signed-rank test. The evaluated models include pooled embeddings from pre-trained encoders, contrastively post-trained sentence encoders and instruction-tuned embedding models. Most models separate idiomatic from literal phrases, but to different degrees.

The work matters because it gives a theoretically grounded, rather than purely human-rating-based, way to test idiomaticity sensitivity in embeddings.

Key points

  • Dataset: The ncimp dataset, which draws on work by Garcia et al. (2021) and He et al. (2025), labels noun-compound MWEs on a three-level scale: non-compositional (NC), partially compositional (PC) and compositional (C).
  • Statistical method: Non-compositionality is identified as a distribution shift in ε relative to compositional samples, tested with the Wilcoxon signed-rank test. Rank-biserial correlation measures effect size, and it is generally ordered C < PC < NC, most clearly for newer models. The PC class is harder to pinpoint.
  • Pooling choices: For bert, Modernbert and Deberta-v3, the pooling method (mean, cls, cls + sep) changes embedding scale but does not substantially change the p-values. This suggests the test handles magnitude confounders.
  • Limitations: The authors note ncimp quality problems, such as ungrammatical substitutions and very loose synonymy (e.g., apple/pear). A frequency confounder also remains uncontrolled.
  • Publication details: University of Oslo authors; EACL 2026, Long Papers, pp. 2239–2252.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.99

@inproceedings{wold2026measuring,
  title = {{Measuring Idiomaticity in Text Embedding Models with epsilon-compositionality}},
  author = {Wold, Sondre and Simon, Étienne and Velldal, Erik and Øvrelid, Lilja},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.99},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.99}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas. 2019. Measuring compositionality in representation learning. In International Conference on Learning Representations. find
  2. Colin Bannard, Timothy Baldwin, and Alex Lascarides. 2003. A statistical approach to the semantics of verb-particles. In Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment, pages 65–72, Sapporo, Japan. Association for Computational Linguistics. find
  3. Marco Baroni, Raffaella Bernardi, Roberto Zamparelli, and 1 others. 2014. Frege in space: A program for compositional distributional semantics. Linguistic Issues in language technology, 9:241–346. find
  4. Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. 2025. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831. find
  5. Mathieu Constant, Gülşen Eryiǧit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017. Survey: Multiword expression processing: A Survey. Computational Linguistics, 43(4):837–892. find our card
  6. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626. find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,… find
  8. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, and 1 others. 2023. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36:70293–70332. find our card
  9. James Fodor, Simon De Deyne, and Shinsuke Suzuki. 2025. Compositionality and sentence meaning: Comparing semantic parsing and transformers on a challenging sentence similarity dataset. Computational Linguistics, 51(1):139–190. find
  10. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. As… find our card
  11. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
  12. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations. find our card
  13. Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2025. Investigating idiomaticity in word representations. Computational Linguistics, 51:505–555. find
  14. Wilfrid Hodges. 2001. Formal features of compositionality. Journal of Logic, Language and Information, 10(1):7–28. find
  15. Theo Janssen. 1986. Foundations and applications of Montague grammar. Ph.D. thesis, University of Amsterdam. find
  16. Sinan Kurtyigit, Diego Frassinelli, Carina Silberer, and Sabine Schulte Im Walde. 2025. A couch potato is not a potato on a couch: Prompting strategies, image generation, and compositionality prediction for noun compounds. In Findings of the Association for Computational Linguistics: ACL 2025, pages… find
  17. Xianming Li and Jing Li. 2024. AoE: Angle-optimized embeddings for semantic textual similarity. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1825–1839, Bangkok, Thailand. Association for Computational Linguistics. find
  18. Yafu Li, Yongjing Yin, Yulong Chen, and Yue Zhang. 2021. On compositional generalization of neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  19. Samuel Lippl and Kim Stachenfeld. 2025. When does compositional structure yield compositional generalization? a kernel theory. In The Thirteenth International Conference on Learning Representations. find
  20. Kate McCurdy, Paul Soulos, Paul Smolensky, Roland Fernandez, and Jianfeng Gao. 2024. Toward compositional behavior in neural models: A survey of current views. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9323–9339, Miami, Florida, USA. Association… find
  21. Richard Montague and 1 others. 1970. Universal grammar. 1974, pages 222–46. find
  22. Niklas Muennighoff. 2022. SGPT: GPT sentence embeddings for semantic search. Preprint, arXiv:2202.08904. find our card
  23. Ryan M. Nefdt and Christopher Potts. 2024. Compositionality, chapter 1. MIT Press. Https://oecs.mit.edu/pub/e222wyjy. find
  24. Peter Pagin and Dag Westerståhl. 2010. Compositionality i: Definitions and variants. Philosophy Compass, 5(3):250–264. find
  25. Barbara Partee and 1 others. 1995. Lexical semantics and compositionality. An invitation to cognitive science: Language, 1:311–360. find
  26. Francis Jeffry Pelletier. 1994. The principle of semantic compositionality. Topoi, 13(1):11–24. find
  27. Francis Jeffry Pelletier. 2017. Compositionality and concepts—a perspective from formal semantics and philosophy of language. In Compositionality and concepts in linguistics and psychology, pages 31–94. Springer International Publishing Cham. find
  28. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics. find
  29. Parikshit Ram, Tim Klinger, and Alexander G. Gray. 2024. What makes models compositional? a theoretical view: With supplement. Preprint, arXiv:2405.02350. — [arXiv:2405.02350](https://arxiv.org/abs/2405.02350) find
  30. Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 210–218, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find our card
  31. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. arXiv preprint arXiv:1908.10084. find
  32. Magnus Sahlgren. 2008. The distributional hypothesis. Italian Journal of linguistics, 20:33–53. find
  33. Bahar Salehi, Paul Cook, and Timothy Baldwin. 2015. A word embedding approach to predicting the compositionality of multiword expressions. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 977–983,… find
  34. Sania Sinha, Tanawan Premsri, and Parisa Kordjamshidi. 2024. A survey on compositional learning of AI models: Theoretical and experimental practices. Transactions on Machine Learning Research. Survey Certification. find
  35. JFAK van Bethem, JAG Groenendijk, DHJ de Jong, MJB Stockhof, and HJ Verkuyl. 1991. Logic, Language, and Meaning, Volume 2: Intensional Logic and Intensional Grammar. University of Chicago Press, Chicago, IL. find
  36. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. find our card
  37. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. find our card
  38. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, and 1 others. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning an… find our card
  39. Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. 2023. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36:6941–6960. find
  40. Wlodek Zadrozny. 1994. From compositional to systematic semantics. Linguistics and philosophy, 17(4):329–342. find
  41. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wold-2026-measuring-idiomaticity-text-embedding-models/