Elamur Research⌕
☰
home / papers / 2026 / Tug-of-war between idioms' figurative and literal interpretations in LLMs

Tug-of-war between idioms' figurative and literal interpretations in LLMs

Summary

The paper asks how autoregressive transformers handle idioms, which can be read both figuratively (e.g., "to die") and literally (e.g., physically kicking a bucket). The authors build a next-token prediction benchmark. They take 245 highly literal-plausible idioms from Cronk et al. (1993) and place each in a template whose continuation exposes the chosen reading. Llama3.3-70B-instruct generates figurative and literal paraphrases, contrasting continuation token sets, and disambiguating preceding contexts. Using knockout of MHSA/MLP sublayers and individual heads, activation patching, and a Query-preserved KV patching variant, they trace information flow mainly in Llama3.2-1B, with checks on larger and Qwen models. They find three stages. Early layers and specific heads fetch the figurative sense and dampen the literal one. Preceding context is used almost immediately, and middle-layer attention corrects conflicts. Finally, two routes compete: an intermediate path that favours the figurative reading and a direct bypass that keeps the literal one available. This gives mechanistic evidence about multiword-expression processing in LLMs and may inform efforts to improve idiom comprehension.

Key points

  • Data: 245 idioms from Cronk et al. (1993), embedded in the template "X (would) IDIOM because X was so/too/a/the". Figurative and literal candidate sets contain 20 tokens each. They were chosen by the largest logit differences between paraphrase-conditioned runs of Llama3.3-70B-instruct.
  • Models and methods: The main model is Llama3.2-1B, with additional runs on Llama3.1-8B, Qwen2.5-0.5B, and Qwen2.5-7B. Methods are mean/zero knockout, activation patching, and Query-preserved Key-Value patching.
  • Retrieval: Attention and MLP in roughly layers 0–3 retrieve the figurative meaning and suppress the literal one. The authors interpret this as a "detokenization" of the idiom.
  • Context: Idiom tokens draw on disambiguating context mostly in layers 2–6. Layer 0 is already important under figurative context, and later layers mainly consolidate the interpretation.
  • Limitations and resource: Interpretation is treated as binary, which ignores multiple figurative senses. Code and dataset are released at github.com/sori424/idiom_processing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.135

@inproceedings{oh2026tug,
  title = {{Tug-of-war between idioms' figurative and literal interpretations in LLMs}},
  author = {Oh, Soyoung and Huang, Xinting and Pink, Mathis and Hahn, Michael and Demberg, Vera},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.135},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.135}
}

References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sara D Beck and Andrea Weber. 2020. Context and literality in idiom processing: Evidence from self-paced reading. Journal of Psycholinguistic Research, 49(5):837–863. find
  2. Samuel A Bobrow and Susan M Bell. 1973. On catching on to idiomatic expressions. Memory & cognition, 1:343–346. find
  3. Cristina Cacciari and Sam Glucksberg. 1991. Chapter 9 understanding idiomatic expressions: The contribution of word meanings. In Greg B. Simpson, editor, Understanding Word and Sentence, volume 77 of Advances in Psychology, pages 217–240. North-Holland. find
  4. Cristina Cacciari and Patrizia Tabossi. 1988. The comprehension of idioms. Journal of memory and language, 27(6):668–683. find
  5. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bang… find our card
  6. Hakaze Cho, Mariko Kato, Yoshihiro Sakai, and Naoya Inoue. 2025. Revisiting in-context learning inference circuit in large language models. In The Thirteenth International Conference on Learning Representations. find
  7. Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352. find
  8. Brian C Cronk, Susan D Lima, and Wendy A Schweigert. 1993. Idioms in sentences: Effects of frequency, literalness, and familiarity. Journal of psycholinguistic research, 22:59–82. find
  9. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626. find our card
  10. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. Analyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16124–16170. find
  11. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12. find
  12. Afsaneh Fazly, Paul Cook, and Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. Computational Linguistics, 35(1):61–103. find our card
  13. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235. find
  14. Raymond W Gibbs. 1980. Spilling the beans on understanding and memory for idioms in conversation. Memory & cognition, 8(2):149–156. find
  15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  16. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research. find
  17. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033–76060. find our card
  18. Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understanding transformer memorization recall through idioms. find our card
  19. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 248–264. Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching, 2024. URL https://arxiv.org/abs/2404.15255. — [arXiv:2404.15255](https://arxiv.org/abs/2404.15255) ↗
  20. Edward Holsinger. 2013. Representing idioms: Syntactic and contextual effects on idiom processing. Language and speech, 56(3):373–394. find
  21. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. Position: The platonic representation hypothesis. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20617–20642. PMLR. find
  22. Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Indra Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023. Multi-lingual and multi-cultural figurative language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8… find
  23. Agne Knietaite, Adam Allsebrook, Anton Minkov, Adam Tomaszewski, Norbert Slinko, Richard Johnson, Thomas Pickard, Dylan Phelps, and Aline Villavicencio. 2024. Is less more? quality, quantity and context in idiom processing with natural language models. arXiv preprint arXiv:2405.08497. find
  24. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  25. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372. find
  26. Maggie Mi, Aline Villavicencio, and Nafise Sadat Moosavi. 2025. Rolling the DICE on idiomaticity: How LLMs fail to grasp context. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7314–7332, Vienna, Austria. Association for Comp… find
  27. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations. nostalgebraist. 2020. Interpreting gpt: The logit lens. https://www. lesswrong.com/p… find
  28. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oh-2026-tug-war-between-idioms-figurative/