Elamur Research⌕
☰
home / papers / 2026 / ID10M-JAM: Stress-Testing Idiom Identification Under Challenging Context

ID10M-JAM: Stress-Testing Idiom Identification Under Challenging Context

Summary

LLMs score well on idiom identification benchmarks, but it is unclear whether they stay accurate when the surrounding text pushes toward the wrong reading. The authors built ID10M-JAM, an adversarial version of the English and German test portions of ID10M. For each sentence containing a potential idiomatic expression (PIE), they prepended a fluent, coherent context that favours the opposite interpretation: literal-sounding cues before figurative uses, and figurative-sounding cues before literal ones. Candidate variants were generated with LLM assistance, then filtered and refined by human annotators so that each stays natural and has one clear reading for people. The task is framed as BIO sequence labeling with no list of candidate expressions provided. They tested open-weight, closed-weight and reasoning models under zero-shot and more advanced prompting. Human annotators remained largely accurate, while models made many errors. Better prompting, larger scale and explicit reasoning improved overall identification but did not make models reliably more robust to the misleading context. The benchmark therefore exposes weaknesses that clean test sets hide.

Key points

  • The source is the ID10M benchmark (Tedeschi et al., 2022), using 200 test sentences per language for EN and DE. ID10M's training sets contain 37,919 (EN) and 24,126 (DE) sentences. Italian and Spanish are left for future work.
  • Labels follow the B-IDIOM / I-IDIOM / O tagging scheme. Models must find idiom spans and also decide whether each use is figurative or literal.
  • In the human evaluation, two annotators reached 90.4% and 87.1% accuracy on the adversarial variants. Both found literal uses harder and tended to mark unclear items as ambiguous.
  • The main finding is that identification accuracy and robustness to adversarial context are not coupled. Sophisticated prompting does not reduce susceptibility to misleading context.
  • The data is released under CC BY-NC-SA 4.0 and the code under Apache-2.0 (github.com/Intellexus-DSI/ID10M-JAM). Because of budget limits, the cheaper models were run with three seeds and the expensive ones with a single seed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1045

@inproceedings{hashiloni2026id10m,
  title = {{ID10M-JAM: Stress-Testing Idiom Identification Under Challenging Context}},
  author = {Hashiloni, Kai Golan and Livyatan, Lior and Hefetz, Ofri and Mannor, Alon and Cohen, Bar and Bar, Kfir},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1045},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1045}
}

References

72 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Doğukan Arslan, Hüseyin Anıl Çakmak, Gülşen Eryiğit, and Joakim Nivre. 2025. Using LLMs to advance idiom corpus construction. In Proceedings of the 21st Workshop on Multiword Expressions (MWE 2025), pages 21–31, Albuquerque, New Mexico, U.S.A. Association for Computational Linguistics. find
  2. Anabela Barreiro, Johanna Monti, Brigitte Orliac, and Fernando Batista. 2013. When multiwords go bad in machine translation. In Proceedings of the Workshop on Multi-word Units in Machine Translation and Translation Technologies, Nice, France. find
  3. Christos Baziotis, Prashant Mathur, and Eva Hasler. 2023. Automatic evaluation and analysis of idioms in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3682– 3700, Dubrovnik, Croatia. Association for C… find our card
  4. J Briskilal and C.N. Subalalitha. 2022. An ensemble model for classifying idioms and literal texts using BERT and RoBERTa. Information Processing Management, 59(1):102756. find
  5. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  6. Amir DN Cohen, Hila Gonen, Ori Shapira, Ran Levy, and Yoav Goldberg. 2022. McPhraSy: Multi-context phrase similarity and clustering. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3538–3550, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other… find our card
  8. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find our card
  9. Mathieu Constant, Gülşen Eryiǧit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017. Survey: Multiword expression processing: A Survey. Computational Linguistics, 43(4):837–892. find our card
  10. Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2007. Pulling their weight: Exploiting syntactic forms for the automatic identification of idiomatic expressions in context. In Proceedings of the Workshop on A Broader Perspective on Multiword Expressions, pages 41–48, Prague, Czech Republic. Associa… find our card
  11. Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008. The VNC-Tokens dataset. In Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions, pages 19–22. find
  12. Thomas C. Cooper. 1998. Teaching idioms. Foreign Language Annals, 31(2):255–266. find
  13. Thomas C. Cooper. 1999. Processing of idioms by l2 learners of English. TESOL Quarterly, 33(2):233– 262. find
  14. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Ireland… find our card
  15. Francesca De Luca Fornaciari, Begoña Altuna, Itziar Gonzalez-Dios, and Maite Melero. 2024. A hard nut to crack: Idiom detection with conversational large language models. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 35–44, Mexico City, Mexico (Hybrid). A… find our card
  16. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in LL… find our card
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  18. GülŞen Eryiğit, Ali Şentaş, and Johanna Monti. 2022. Gamified crowdsourcing for idiom corpora construction. Natural Language Engineering, 29(4):909–941. find
  19. Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2018. Examining the tip of the iceberg: A data set for idiom translation. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  20. Afsaneh Fazly, Paul Cook, and Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. Computational Linguistics, 35(1):61–103. find our card
  21. Waseem Gharbieh, Virendra Bhavsar, and Paul Cook. 2016. A word embedding approach to identifying verb-noun idiomatic combinations. In Proceedings of the 12th Workshop on Multiword Expressions, pages 112–118, Berlin, Germany. Association for Computational Linguistics. find
  22. Andrew Gomes. 2025. Anatomy of an idiom: Tracing non-compositionality in language models. Preprint, arXiv:2511.16467. — [arXiv:2511.16467](https://arxiv.org/abs/2511.16467) find
  23. Hila Gonen, Terra Blevins, Alisa Liu, Luke Zettlemoyer, and Noah A. Smith. 2025. Does liking yellow imply driving a school bus? semantic leakage in language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human La… find
  24. Shreya Goyal, Sumanth Doddapaneni, Mitesh M. Khapra, and Balaraman Ravindran. 2023. A survey of adversarial defences and robustness in NLP. Preprint, arXiv:2203.06414. — [arXiv:2203.06414](https://arxiv.org/abs/2203.06414) find
  25. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association. find our card
  26. Najet Hadj Mohamed, Agata Savary, Cherifa Ben Khelil, Jean-Yves Antoine, Iskandar Keskes, and Lamia Hadrich-Belguith. 2024. Lexicons gain the upper hand in Arabic MWE identification. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2024,… find
  27. Lifeng Han, Gareth Jones, and Alan Smeaton. 2020. AlphaMWE: Construction of multilingual parallel corpora with MWE annotations. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 44–57, online. Association for Computational Linguistics. find
  28. Kai Golan Hashiloni, Ofri Hefetz, and Kfir Bar. 2025. Easy as PIE? identifying multi-word expressions with LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23771–23790, Suzhou, China. Association for Computational Linguistics. find
  29. Chikara Hashimoto and Daisuke Kawahara. 2009. Compilation of an idiom example database for supervised idiom identification. Language Resources and Evaluation, 43(4):355–384. 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. find
  30. Andrea Horbach, Andrea Hensler, Sabine Krome, Jakob Prange, Werner Scholze-Stubenrecht, Diana Steffen, Stefan Thater, Christian Wellner, and Manfred Pinkal. 2016. A corpus of literal and idiomatic uses of German infinitive-verb compounds. In Proceedings of the Tenth International Conference on Langu… find
  31. Yusuke Ide, Joshua Tanner, Adam Nohejl, Jacob Hoffman, Justin Vasselli, Hidetaka Kamigaito, and Taro Watanabe. 2025. CoAM: Corpus of all-type multiword expressions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27004– 27021,… find
  32. Ray S. Jackendoff. 1997. The Architecture of the Language Faculty, volume 28 of Linguistic Inquiry Monographs. MIT Press, Cambridge, MA; London, England. find
  33. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? a strong baseline for natural language attack on text classification and entailment. Preprint, arXiv:1907.11932. find our card
  34. Jisu Kim, Youngwoo Shin, Uiji Hwang, Jihun Choi, Richeng Xuan, and Taeuk Kim. 2025. Memorization or reasoning? exploring the idiom understanding of LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21678–21699, Suzhou, China. Association for Compu… find our card
  35. Wei He, Marco Idiart, Carolina Scarton, and Aline Villavicencio. 2024. Enhancing idiomatic representation in multiple languages via an adaptive contrastive triplet loss. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12473–12485, Bangkok, Thailand. Association for Comp… find
  36. Ioannis Korkontzelos, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. SemEval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEv… find
  37. Ofri Hefetz, Kai Golan Hashiloni, Alon Mannor, and Kfir Bar. 2025. Not just a piece of cake: Cross-lingual fine-tuning for idiom identification. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Associat… find
  38. Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Ift… find
  39. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages Dian… find our card
  40. Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023. Synthetic data generation with large language models for text classification: Potential and limitations. Preprint, arXiv:2310.07849. find our card
  41. Pengfei Liu, Kaiyu Qian, Xipeng Qiu, and Xuanjing Huang. 2017. Idiom-aware compositional distributed semantics. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1204–1213, Copenhagen, Denmark. Association for Computational Linguistics. find
  42. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9). find our card
  43. Maggie Mi, Aline Villavicencio, and Nafise Sadat Moosavi. 2025. Rolling the DICE on idiomaticity: How LLMs fail to grasp context. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7314–7332, Vienna, Austria. Association for Comp… find
  44. Mihai Nadǎş, Laura Dioşan, and Andreea Tomescu. 2025. Synthetic data generation using large language models: Advances in text and code. IEEE Access, 13:134615–134633. find
  45. Vasudevan Nedumpozhimana and John Kelleher. 2021. Finding BERT’s idiomatic key. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 57–62, Online. Association for Computational Linguistics. find
  46. Seoyoon Park, Hyeji Choi, Minseon Kim, Subin An, Xiaonan Wang, Gyuri Choi, and Hansaem Kim. 2025. FLUID QA: A multilingual benchmark for figurative language usage in dialogue across English, Chinese, and Korean. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processin… find
  47. Dylan Phelps, Thomas Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. 2024. Sign of the times: Evaluating the use of large language models for idiomaticity detection. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2024, p… find
  48. Howard R. Pollio, John M. Barlow, Howard J. Fine, and Marilyn R. Pollio. 1977. Psychology and the Poetics of Growth: Figurative Language in Psychology, Psychotherapy, and Education. Lawrence Erlbaum, Hillsdale, NJ. find
  49. Carlos Ramisch, Agata Savary, Bruno Guillaume, Jakub Waszczuk, Marie Candito, Ashwini Vaidya, Verginica Barbu Mititelu, Archna Bhatia, Uxoa Iñurrieta, Voula Giouli, Tunga Güngör, Menghan Jiang, Timm Lichte, Chaya Liebeskind, Johanna Monti, Renata Ramisch, Sara Stymne, Abigail Walsh, and Hongzhi Xu.… find our card
  50. Giancarlo Salton, Robert Ross, and John Kelleher. 2014. An empirical study of the impact of idioms on phrase based statistical machine translation of English to Brazilian-Portuguese. In Proceedings of the 3rd Workshop on Hybrid Approaches to Machine Translation (HyTra), pages 36–41, Gothenburg, Swed… find
  51. Agata Savary, Cherifa Ben Khelil, Carlos Ramisch, Voula Giouli, Verginica Barbu Mititelu, Najet Hadj Mohamed, Cvetana Krstev, Chaya Liebeskind, Hongzhi Xu, Sara Stymne, Tunga Güngör, Thomas Pickard, Bruno Guillaume, Eduard Bejček, Archna Bhatia, Marie Candito, Polona Gantar, Uxoa Iñurrieta, Albert… find
  52. Prateek Saxena and Soma Paul. 2020. EPIE dataset: A corpus for possible idiomatic expressions. Preprint, arXiv:2006.09479. — [arXiv:2006.09479](https://arxiv.org/abs/2006.09479) find
  53. Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun clustering. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1002–1010, Beijing, China. find
  54. Coling 2010 Organizing Committee. find
  55. Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, and Mihaela-Claudia Cercel. 2024. Investigating large language models for complex word identification in multilingual and multidomain setups. In Proceedings of the 2024 Conference on Empirical Methods in Natural Langu… find
  56. Caroline Sporleder and Linlin Li. 2009. Unsupervised recognition of literal and non-literal use of idiomatic expressions. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 754–762, Athens, Greece. Association for Computational Linguistics. find our card
  57. Caroline Sporleder, Linlin Li, Philip Gorinski, and Xaver Koch. 2010. Idioms in context: The IDIX corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find
  58. Varshini Subhash, Anna Bialas, Weiwei Pan, and Finale Doshi-Velez. 2023. Why do universal adversarial attacks work on large language models?: Geometry might be the answer. Preprint, arXiv:2309.00254. — [arXiv:2309.00254](https://arxiv.org/abs/2309.00254) find
  59. Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005, Singapore. Association for Computational Linguistics. find
  60. Joshua Tanner and Jacob Hoffman. 2023. MWE as WSD: Solving multiword expression identification with word sense disambiguation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 181–193, Singapore. Association for Computational Linguistics. find our card
  61. Shiva Taslimipoor, Sara Bahaadini, and Ekaterina Kochmar. 2020. MTLB-STRUCT @parseme 2020: Capturing unseen multiword expressions using multi-task learning and pre-trained masked language models. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 142–148, on… find
  62. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 107… find our card
  63. Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. AStitchInLanguageModels: Dataset and methods for the exploration of idiomaticity in pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3464–3477, Punt… find our card
  64. Simone Tedeschi, Federico Martelli, and Roberto Navigli. 2022. ID10M: Idiom identification in 10 languages. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2715–2726, Seattle, United States. Association for Computational Linguistics. find our card
  65. Su Nam Kim Timothy Baldwin. 2010. Handbook of Natural Language Processing, chapter 2:267-292. find
  66. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2021. Universal adversarial triggers for attacking and analyzing NLP. Preprint, arXiv:1908.07125. find
  67. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171. find our card
  68. Uriel Weinreich. 1969. Problems in the analysis of idioms. In Jaan Puhvel, editor, Substance and Structure of Language, pages 23–81. University of California Press, Berkeley and Los Angeles. find
  69. Parameter Other Models Epochs Batch size Learning rate 20 32 2e-5 *Table 4: Encoders fine-tuning hyper-parameters.* Lowri Williams, Christian Bannister, Michael Arribas-Ayllon, Alun Preece, and Irena Spasić. 2015. The role of idioms in sentiment analysis. Expert Systems with Applications, 42(21):73… find
  70. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025. Qwen2.5 technical report. Preprint, arXiv:241… find
  71. Ziheng Zeng and Suma Bhat. 2021. Idiomatic expression identification using semantic compatibility. Transactions of the Association for Computational Linguistics, 9:1546–1562. find our card
  72. Xingyi Zhao, Lu Zhang, Depeng Xu, and Shuhan Yuan. 2022. Generating textual adversaries with minimal perturbation. Preprint, arXiv:2211.06571. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/hashiloni-2026-id10m-jam-stress-testing-idiom/