Elamur Research⌕
☰
home / papers / 2026 / A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding

A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding

Summary

Potentially idiomatic expressions (PIEs) carry meanings rooted in a community's culture, and current language and vision-language models still struggle with them. Existing idiom corpora are also rarely aligned across languages, so cross-lingual comparison and transfer studies are hard to run. To fill this gap, the authors present XMPIE, a parallel benchmark that expands the two-language AdMIRe 1.0 design (English and Portuguese) to 34 languages. Language experts started from English seed PIEs and, following shared multilingual guidelines, produced counterparts in each target language along with visual material. Each PIE comes with five images that range from its figurative reading to its literal one, plus a distractor. The language set intentionally covers sociolinguistically and digitally endangered languages. With this resource, researchers can check whether a model's understanding of an idiom holds across languages and across modalities (text versus image). It also supports comparing how different cultures express shared concepts. The authors additionally run a benchmark evaluation of models on the data.

Key points

  • XMPIE covers 34 languages and contains 3054 PIEs and 7040 images. The abstract describes this as more than ten thousand items.
  • Each PIE has five image types: idiomatic, idiomatic-related, literal-related, literal, and a random distractor. The paper's example is "green fingers", meaning gardening skill.
  • Endangered languages are included, as classified by UNESCO and the European Language Equality (ELE) report:
    • Aromanian and Luxembourgish (sociolinguistically endangered)
    • Igbo, Catalan, Greek, Latvian and Lithuanian (digitally endangered)
  • The authors argue that classification-style idiom benchmarks do not fully reflect real comprehension. They also note that prior corpora are mostly non-parallel, including PARSEME, MAGPIE, ID10M and SemEval-2022 Task 2.
  • Contributors include researchers from Ukrainian institutions: Ternopil Volodymyr Hnatiuk National Pedagogical University and SET University.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5cvnbcoktfo2

@inproceedings{torunoluselamet2026parallel,
  title = {{A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding}},
  author = {Torunoğlu-Selamet, Dilara and Arslan, Doğukan and Wilkens, Rodrigo and He, Wei and Eryiğit, Doruk and Pickard, Thomas and Pagano, Adriana S. and Villavicencio, Aline and Eryiğit, Gülşen and Abuczki, Ágnes and Cardoso, Aida and Lazarenka, Alesia and Almassova, Dina and Mendes, Amália and Kanellopoulou, Anna and Brosa-Rodriguez, Antoni and Valkovska, Baiba and Wojtowicz, Beata and Pedersen, Bolette and Hidalgo-Ternero, Carlos Manuel and Liebeskind, Chaya and Jokić, Danka and Alves, Diego and Triantafyllidi, Eleni and Velldal, Erik and Philippy, Fred and Oleskeviciene, Giedre Valunaite and Rizgeliene, Ieva and Skadina, Inguna and Lobzhanidze, Irina and Haugen, Isabell Stinessen and Krito, Jauza Akbar and Marković, Jelena M. and Monti, Johanna and Sauca, Josue Alejandro and Zor, Kaja Dobrovoljc and Ugwuanyi, Kingsley O. and Rituma, Laura and Øvrelid, Lilja and Agro, Maha Tufail and Abjalova, Manzura and Chatzigrigoriou, Maria and Ramos, María del Mar Sánchez and Pendevska, Marija and Seyyedrezaei, Masoumeh and Shamsfard, Mehrnoush and Ahsan, Momina and Khan, Muhammad Ahsan Riaz and Norman, Nathalie Carmen Hau and Ayyıldız, Nilay Erdem and Hosseini-Kivanani, Nina and Ligeti-Nagy, Noémi and Naeem, Numaan and Kanishcheva, Olha and Yatsyshyna, Olha and Orel, Daniil and Giommarelli, Petra and Osenova, Petya and Garabik, Radovan and Semou, Regina E. and Rebechi, Rozane and Pranida, Salsabila Zahirah and Touileb, Samia and Nimb, Sanni and Ahmad, Sarfraz and Sharipova, Sarvinoz and Golan, Shahar and Ji, Shaoxiong and Aboh, Sopuruchi Christian and Sucur, Srdjan and Markantonatou, Stella and Olsen, Sussi and Tajalli, Vahide and Lipp, Veronika and Giouli, Voula and Eraydın, Yelda Yeşildal and Saaberi, Zahra and Xie, Zhuohan},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5cvnbcoktfo2},
  url = {https://doi.org/10.63317/5cvnbcoktfo2}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sophia Ananiadou, John McNaught, and Paul Thompson. 2012. The English Language in the Digital Age. META-NET White Paper Series. Springer Nature, Heidelberg. EC FP7 PSP Project METANET4U. find
  2. Doğukan Arslan, Rodrigo Wilkens, Wei He, Dilara Torunoğlu-Selamet, Thomas Pickard, Aline Villavicencio, Adriana S. Pagano, and Gülşen Eryiğit. 2026. MWE-2026 Shared Task 2: Ad-MIRe 2 - Advancing Multimodal Idiomaticity Representation. In Proceedings of the 22nd Workshop on Multiword Expressions (MWE… find
  3. Joanne Boisson, Luis Espinosa-Anke, and Jose Camacho-Collados. 2023. Construction artifacts in metaphor identification datasets. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 6581–6590, Singapore. Association for Computational Linguisti… find
  4. Jonathan Charteris-Black. 2002. Second language figurative proficiency: A comparative study of Malay and English. Applied Linguistics, 23(1):104–133. find
  5. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Ireland… find our card
  6. Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2025. Investigating idiomaticity in word Computational Linguistics, representations. 51(2):505–555. find
  7. Suzanne Irujo. 1986. Don’t put your leg in your mouth: Transfer in the acquisition of idioms in a second language. TESOL Quarterly, 20(2):287– 304. find
  8. Zoltan Kovecses. 2010. Metaphor: A practical introduction. Oxford university press. find
  9. Christopher Moseley. 2010. Atlas of the world’s languages in danger, 3 edition. UNESCO Publishing, Paris. find
  10. Dylan Phelps, Thomas Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. 2024. Sign of the times: Evaluating the use of large language models for idiomaticity detection. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE– UD 2024) at LREC–COLING… find
  11. Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for nlp. In Computational Linguistics and Intelligent Text Processing, pages 1–15, Berlin, Heidelberg. Springer Berlin Heidelberg. find our card
  12. Agata Savary, Daniel Zeman, Verginica Barbu Mititelu, Anabela Barreiro, Olesea Caftanatov, Marie-Catherine De Marneffe, Kaja Dobrovoljc, Gülşen Eryiğit, Voula Giouli, Bruno Guillaume, et al. 2024. Unidive: A cost action on universality, diversity and idiosyncrasy in language technology. In Proceedin… find
  13. Jannik Strötgen and Michael Gertz. 2012. Temporal tagging on different domains: Challenges, strategies, and gold standards. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 3746–3753, Istanbul, Turkey. European Language Resources Association… find our card
  14. Arkaitz Zubiaga. 2024. Natural language processing in the era of large language models: A survey. Frontiers in Artificial Intelligence, 6:1350306. 8. find
  15. Language Resource References Doğukan Arslan, Hüseyin Anıl Çakmak, Gülşen Eryiğit, and Joakim Nivre. 2025. Using LLMs to advance idiom corpus construction. In Proceedings of the 21st Workshop on Multiword Expressions (MWE 2025), pages 21–31, Albuquerque, New Mexico, U.S.A. Association for Computation… find
  16. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2025. CulturalBench: A robust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through human-AI red-teami… find
  17. Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008. The VNC-tokens dataset. Towards a Shared Task for Multiword Expressions (MWE 2008), page 19. find
  18. Silvio Cordeiro, Aline Villavicencio, Marco Idiart, and Carlos Ramisch. 2019. Unsupervised compositionality prediction of nominal compounds. Comput. Linguist., 45(1):1–57. find
  19. Gülşen Eryiğit, Ali Şentaş, and Johanna Monti. 2022. Gamified crowdsourcing for idiom corpora construction. Natural Language Engineering, 29(4):909–941. find
  20. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association. find our card
  21. Chikara Hashimoto and Daisuke Kawahara. 2009. Compilation of an idiom example database for supervised idiom identification. Language Resources and Evaluation, 43(4):355–384. find
  22. Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2025. Investigating idiomaticity in word representations. Computational Linguistics, 51(2):505–555. find
  23. Simran Khanuja, Sathyanarayanan Ramamoorthy, Yueqi Song, and Graham Neubig. 2024. An image speaks a thousand words, but can everyone listen? on image transcreation for cultural relevance. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 10… find
  24. Ioannis Korkontzelos, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. SemEval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEv… find
  25. Maggie Mi, Aline Villavicencio, and Nafise Sadat Moosavi. 2025. Rolling the DICE on idiomaticity: How LLMs fail to grasp context. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7314–7332, Vienna, Austria. Association for Comp… find
  26. Thomas Pickard, Aline Villavicencio, Maggie Mi, Wei He, Dylan Phelps, and Marco Idiart. 2025. Semeval-2025 task 1: Admire – advancing multimodal idiomaticity representation. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 2597–2609, Vienna, Austria. Ass… find our card
  27. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. Pm… find our card
  28. Siva Reddy, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 210–218, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find our card
  29. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cu… find
  30. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2025. Understanding figurative meaning through explainable visual entailment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech… find
  31. Agata Savary, Manfred Sailer, Yannick Parmentier, Michael Rosner, Victoria Rosén, Adam Przepiórkowski, Cvetana Krstev, Veronika Vincze, Beata Wójtowicz, Gyri Smørdal Losnegaard, Carla Parra Escartín, Jakub Waszczuk, Mathieu Constant, Petya Osenova, and Federico Sangati. 2015. PARSEME – PARSing and M… find
  32. Prateek Saxena and Soma Paul. 2020. Epie dataset: A corpus for possible idiomatic expressions. In Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8–11, 2020, Proceedings, page 87–94, Berlin, Heidelberg. Springer-Verlag. find
  33. Uliana Sentsova, Debora Ciminari, Josef Van Genabith, and Cristina España-Bonet. 2025. MultiCoPIE: A multilingual corpus of potentially idiomatic expressions for cross-lingual PIE disambiguation. In Proceedings of the 21st Workshop on Multiword Expressions (MWE 2025), pages 67–81, Albuquerque, New M… find our card
  34. Caroline Sporleder and Linlin Li. 2009. Unsupervised recognition of literal and non-literal use of idiomatic expressions. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 754–762, Athens, Greece. Association for Computational Linguistics. find our card
  35. Caroline Sporleder, Linlin Li, Philip Gorinski, and Xaver Koch. 2010. Idioms in context: The IDIX corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find
  36. Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. 2024. Eva-clip-18b: Scaling clip to 18 billion parameters. arXiv preprint arXiv:2402.04252. — [arXiv:2402.04252](https://arxiv.org/abs/2402.04252) find
  37. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022a. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 10… find our card
  38. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022b. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 10… find our card
  39. Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. AStitchInLanguageModels: Dataset and methods for the exploration of idiomaticity in pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3464–3477, Punt… find our card
  40. Simone Tedeschi, Federico Martelli, and Roberto Navigli. 2022. Id10m: Idiom identification in 10 languages. In Findings of the Association for Computational Linguistics: NAACL 2022, page 2715–2726. Association for Computational Linguistics. find our card
  41. Xiaoyu Tong, Zhi Zhang, Martha Lewis, and Ekaterina Shutova. 2025. Hummus: A dataset of humorous multimodal metaphor use. find
  42. Tongguan Wang, Mingmin Wu, Guixin Su, Dongyu Su, Yuxue Hu, Zhongqiang Huang, and Ying Sha. 2025. MChIRC: A multimodal benchmark for Chinese idiom reading comprehension. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applicati… find
  43. Ron Yosef, Yonatan Bitton, and Dafna Shahaf. 2023. IRFL: Image recognition of figurative language. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1044– 1058, Singapore. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/selamet-2026-parallel-cross-lingual-benchmark-multimodal/