Elamur Research⌕
☰
home / papers / 2025 / Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context

Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context

Paper · 2025 · Maggie Mi, Aline Villavicencio and Nafise Sadat Moosavi

Summary

LLMs score well on idiomaticity detection benchmarks, but this may reflect shortcuts rather than real use of context. In existing datasets such as MAGPIE, literal usages often differ in form from idiomatic ones (through passivisation or inserted modifiers), so models can rely on surface structure. To remove this cue, the authors built DICE (Dataset for Idiomatic Contrastive Evaluation). In DICE, each potentially idiomatic expression keeps an identical form in a figurative sentence and a literal one.

Figurative examples come from existing corpora. Literal counterparts were generated with GPT-4 and then checked by expert annotators. Models are evaluated on overall accuracy and on a strict consistency measure, which requires every literal and figurative instance of an expression to be classified correctly. The authors also examine how idiom frequency and model-assigned sentence likelihood relate to performance.

LLMs frequently fail when only context separates the two readings. Higher frequency improves literal detection but hurts figurative detection. Likelihood tracks accuracy for Llama models but not for Flan-T5. The study argues for evaluation that is sensitive to context rather than to memorised forms.

Key points

  • Dataset size and sources: DICE contains 2,066 sentences, split evenly into 1,033 literal and 1,033 figurative. They cover 402 unique expressions: 299 phrasal idioms and 103 idiomatic noun compounds.
  • Candidate selection:
    • Phrasal idioms were taken from the overlap of MAGPIE and SLIDE.
    • Noun compounds were taken from the overlap of NCTTI and AStitchInLanguageModels.
    • Compositional compounds were excluded.
    • Of 783 initial candidates, expressions were discarded if their idiomatic sense could not be overridden (e.g., "set eyes on").
  • Sentence length: Literal sentences average 15.4 words, while figurative ones average 28.1 words.
  • Models evaluated: 13 models, including GPT-3.5 Turbo, Llama 2, Llama 3 and Flan-T5 variants.
  • Frequency effect: In 9 of the 13 models, frequency showed no consistent correlation with accuracy.
  • Likelihood effect: Llama accuracy rises with sentence likelihood, especially on literal items. Flan-T5 shows negative or negligible correlation, possibly due to overconfidence.
  • Scope: The study covers English only. The code and data are available at github.com/mi-m1/dice.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations🎓University of ExeterExeter · United Kingdom12 works · 203 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Maggie Mi, Aline Villavicencio and Nafise Sadat Moosavi. 2025. Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.362

@inproceedings{mi2025rolling,
  title = {{Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context}},
  author = {Maggie Mi, Aline Villavicencio and Nafise Sadat Moosavi},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.362},
  url = {https://doi.org/10.18653/v1/2025.acl-long.362}
}

References

54 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John Ayto. 2020. The Oxford Dictionary of Idioms. find
  2. Oxford University Press. Anabela Barreiro, Johanna Monti, Brigitte Orliac, and Fernando Batista. 2013. When multiwords go bad in machine translation. In Proceedings of the Workshop on Multi-word Units in Machine Translation and Translation Technologies, Nice, France. find
  3. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  5. Marc Brysbaert, Paweł Mandera, and Emmanuel Keuleers. 2018. The word frequency effect in word processing: An updated review. Current directions in psychological science, 27(1):45–50. find
  6. Kellen Cheng and Suma Bhat. 2024. No context needed: Contextual quandary in idiomatic reasoning with pre-trained language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),…
  7. John Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 575–59…
  8. Albert Coil and Vered Shwartz. 2023. From chocolate bunny to chocolate crocodile: Do language models understand noun compounds? In Findings of the Association for Computational Linguistics: ACL 2023, pages 2698–2710, Toronto, Canada. Association for Computational Linguistics. find
  9. Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008. The vnc-tokens dataset. In Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions (MWE 2008), pages 19–22. find
  10. Citeseer. Brian C. Cronk, Susan D. Lima, and Wendy A. Schweigert. 1993. Idioms in sentences: Effects of frequency, literalness, and familiarity. Journal of Psycholinguistic Research, 22(1):59–82. find
  11. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Ireland…
  12. Francesca De Luca Fornaciari, Begoña Altuna, Itziar Gonzalez-Dios, and Maite Melero. 2024. A hard nut to crack: Idiom detection with conversational large language models. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 35–44, Mexico City, Mexico (Hybrid). A…
  13. Robert B Estill and Susan Kemper. 1982. Interpreting idioms. Journal of psycholinguistic research, 11(6):559–568. find
  14. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  15. Afsaneh Fazly, Paul Cook, and Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. Computational Linguistics, 35(1):61–103.
  16. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021a. Assessing the representations of idiomaticity in vector models with a noun compound dataset labeled at type and token levels. In Proceedings of the 59th Annual Meeting of the Association for Computati… find
  17. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021b. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551–3564, Online. A…
  18. Waseem Gharbieh, Virendra Bhavsar, and Paul Cook. 2016. A word embedding approach to identifying verb-noun idiomatic combinations. In Proceedings of the 12th Workshop on Multiword Expressions, pages 112–118, Berlin, Germany. Association for Computational Linguistics. find
  19. Raymond W. Gibbs and Gayle P. Gonzales. 1985. Syntactic frozenness in processing and remembering idioms. Cognition, 20(3):243–259. find
  20. Raymond W Gibbs and Nandini P Nayak. 1989. Psycholinguistic studies on the syntactic behavior of idioms. Cognitive Psychology, 21(1):100–138. find
  21. Raymond W Gibbs Jr, Nandini P Nayak, and Cooper Cutting. 1989. How to kick the bucket and not decompose: Analyzability and idiom processing. Journal of memory and language, 28(5):576– 593. find
  22. Maurice Gross. 1982. Une classification des phrases « figées » du français. Revue québécoise de linguistique, 11(2):151–185. find
  23. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France.
  24. European Language Resources Association. Andrea Horbach, Andrea Hensler, Sabine Krome, Jakob Prange, Werner Scholze-Stubenrecht, Diana Steffen, Stefan Thater, Christian Wellner, and Manfred Pinkal. 2016. A corpus of literal and idiomatic uses of German infinitive-verb compounds. In Proceedings of th… find
  25. R. Jackendoff. 1997. The Architecture of the Language Faculty. Linguistic inquiry monographs. MIT Press. Miloš Jakubíček, Adam Kilgarriff, Vojtěch Kovář, Pavel Rychlỳ, and Vít Suchomel. 2013. The tenten corpus family. In 7th international corpus linguistics conference CL, pages 125–127. find
  26. Valladolid. Charles Jochim, Francesca Bonin, Roy Bar-Haim, and Noam Slonim. 2018. SLIDE - a sentiment lexicon of common idioms. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  27. Ioannis Korkontzelos, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. SemEval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEv… find
  28. Marianna Kyriacou, Kathy Conklin, and Dominic Thompson. 2020. Passivizability of idioms: Has the wrong tree been barked up? Language and Speech, 63(2):404–435. PMID: 31106699. find
  29. Andreas Langlotz. 2006. find
  30. Idiomatic Creativity. John Benjamins. Maria Chiara Levorato and Cristina Cacciari. 1992. Children’s comprehension and production of idioms: the role of context and familiarity. Journal of Child Language, 19(2):415–433. find
  31. Siyan Li, Riley Carlson, and Christopher Potts. 2022. Systematicity in GPT-3’s interpretation of novel English noun compounds. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 717–728, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  32. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.
  33. Changsheng Liu and Rebecca Hwa. 2018. Heuristically informed unsupervised idiom usage recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1723–1731, Brussels, Belgium. Association for Computational Linguistics. find
  34. Pengfei Liu, Kaiyu Qian, Xipeng Qiu, and Xuanjing Huang. 2017. Idiom-aware compositional distributed semantics. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1204–1213, Copenhagen, Denmark. Association for Computational Linguistics. find
  35. R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Griffiths. 2024. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41):e2322420121. find
  36. Marilyn A Nippold and Mishelle Rudzinski. 1993. Familiarity and transparency in idiom explanation: A developmental study of children and adolescents. Journal of Speech, Language, and Hearing Research, 36(4):728–737. find
  37. Masanari Ohi, Masahiro Kaneko, Ryuto Koike, Mengsay Loem, and Naoaki Okazaki. 2024. Likelihood-based mitigation of evaluation bias in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 3237–3245, Bangkok, Thailand and virtual meeting. Association for…
  38. Dylan Phelps, Thomas M. R. Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. 2024. Sign of the times: Evaluating the use of large language models for idiomaticity detection. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2… find
  39. Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 66–76, Hong Kong, China. Association for Comput… find
  40. Frankie Robertson. 2019. A contrastive evaluation of word sense disambiguation systems for Finnish. In Proceedings of the Fifth International Workshop on Computational Linguistics for Uralic Languages, pages 42–54, Tartu, Estonia. Association for Computational Linguistics. find
  41. Bahar Salehi, Paul Cook, and Timothy Baldwin. 2014. Detecting non-compositional MWE components using Wiktionary. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1792–1797, Doha, Qatar. Association for Computational Linguistics. find
  42. Giancarlo Salton, Robert Ross, and John Kelleher. 2014. An empirical study of the impact of idioms on phrase based statistical machine translation of English to Brazilian-Portuguese. In Proceedings of the 3rd Workshop on Hybrid Approaches to Machine Translation (HyTra), pages 36–41, Gothenburg, Swed… find
  43. Wendy A Schweigert. 1986. The comprehension of familiar and less familiar idioms. Journal of Psycholinguistic Research, 15:33–45. find
  44. Marco Silvio Giuseppe Senaldi, Gianluca E. Lebani, and Alessandro Lenci. 2016. Lexical variability and compositionality: Investigating idiomaticity with distributional semantic models. In Proceedings of the 12th Workshop on Multiword Expressions, pages 21–31, Berlin, Germany. Association for Computa… find
  45. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376–382, Valencia…
  46. Caroline Sporleder, Linlin Li, Philip Gorinski, and Xaver Koch. 2010. Idioms in context: The IDIX corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find
  47. Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021. Do long-range language models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807–822, Online and Punta Cana, Dominican Republic. Association…
  48. David A Swinney and Anne Cutler. 1979. The access and processing of idiomatic expressions. Journal of verbal learning and verbal behavior, 18(5):523–534. find
  49. Patrizia Tabossi, Rachele Fanari, and Kinou Wolf. 2009. Why are idioms recognized fast? Memory & cognition, 37:529–540. find
  50. Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. AStitchInLanguageModels: Dataset and methods for the exploration of idiomaticity in pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3464–3477, Punt…
  51. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  52. Lowri Williams, Christian Bannister, Michael Arribas-Ayllon, Alun Preece, and Irena Spasić. 2015. The role of idioms in sentiment analysis. Expert Systems with Applications, 42(21):7375–7385. find
  53. Uriel Wèinreich. 1969. Problems in the Analysis of Idioms, pages 23–82. University of California Press, Berkeley. find
  54. Ziheng Zeng and Suma Bhat. 2021. Idiomatic Expression Identification using Semantic Compatibility. Transactions of the Association for Computational Linguistics, 9:1546–1562.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mi-2025-rolling-dice-idiomaticity-how-llms/