Elamur Research⌕
☰
home / papers / 2025 / SemEval-2025 Task 1: AdMIRe - Advancing Multimodal Idiomaticity Representation

SemEval-2025 Task 1: AdMIRe - Advancing Multimodal Idiomaticity Representation

Summary

Idioms remain difficult for NLP systems because their meaning usually cannot be built from the meanings of their parts. Even strong LLMs handle figurative language inconsistently. The AdMIRe shared task tests whether models can link potentially idiomatic noun compounds to images that reflect the sense intended in context, in English and Brazilian Portuguese.

The organisers selected compounds with both a plausible literal reading and a plausible figurative reading, each of which could be depicted. Native speakers wrote scene descriptions, which were then rendered as cartoon-style images with Midjourney. LLaVA produced captions for the images, and humans checked them, so text-only systems could also take part.

The task has two subtasks:

  • Subtask A: rank five images according to how well they fit a compound used in a context sentence.
  • Subtask B: choose the image that correctly completes a three-panel sequence and decide whether the sense shown is literal or idiomatic.

The best systems matched human performance. They did this by combining several LLMs and VLMs as a mixture of experts and sending repeated queries. This came at substantial cost, which suggests that models' grasp of idiomaticity is still incomplete.

Key points

  • Static-image data (Subtask A): 100 English and 55 Brazilian Portuguese expressions. Each has five images: two depicting the figurative sense (one strongly, one mildly), two depicting the literal sense (one mildly, one strongly) and an unrelated distractor.
  • Sources of compounds: NCTTI, MAGPIE and FLUTE, plus crowdsourced items. Hyphens that separate idiomatic from literal Portuguese forms (e.g. ovelha-negra) were removed so models could not use them as a shortcut.
  • Subtask A splits: English has 70 training, 15 development and 15 test items. Portuguese has 32, 10 and 13. An extended evaluation set reuses all compounds in their opposite sense and gave more robust results than the small test sets.
  • Sequence data (Subtask B): 30 English items. Each has four candidate final images: the correct ending, two near-miss alternatives and the ending for the other sense.
  • Availability: the data is released under CC-BY-4.0 (doi.org/10.15131/shef.data.28436600.v1). Systems performed better on Portuguese than the organisers expected.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

30 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 26 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pickard2025semeval,
  title = {{SemEval-2025 Task 1: AdMIRe - Advancing Multimodal Idiomaticity Representation}},
  author = {Pickard, Thomas and Villavicencio, Aline and Mi, Maggie and He, Wei and Phelps, Dylan and Idiart, Marco},
  booktitle = {Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.semeval-1.330/}
}

References

62 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The authors would like to acknowledge the contributions of our Brazilian collaborators who made this work possible. In particular, our thanks go to Rozane Rebechi, Juliana Carvalho, Eduardo Victor, César Rennó Costa, Marina Ribeiro and their colleagues and students. We would also like to thank the m… find
  2. David Alfter. 2025. daalft at SemEval-2025 Task 1: Multi-step zero-shot multimodal idiomaticity ranking. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  3. This work was supported by the UKRI AI Centre for Doctoral Training in Speech and Language Technologies (SLT) and their Applications funded by UK Research and Innovation [grant number EP/S023062/1]. For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence… find
  4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… find our card
  5. Mohamed Badran, Youssof Nawar, and Nagwa El-Makky. 2025. AlexUNLP-NB at SemEval-2025 Task 1: A pipeline for idiom disambiguation and visual representation. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  6. Joanne Boisson, Luis Espinosa-Anke, and Jose Camacho-Collados. 2023. Construction Artifacts in Metaphor Identification Datasets. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6581–6590, Singapore. Association for Computational Linguistics. find
  7. Cristina Butnariu, Su Nam Kim, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2009. SemEval-2010 Task 9: The Interpretation of Noun Compounds Using Paraphrasing Verbs and Prepositions. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Direct… find
  8. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative Language Understanding through Textual Explanations. arXiv preprint. ArXiv:2205.12404 [cs]. find
  9. Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. 2022. Altclip: Altering the language encoder in clip for extended language capabilities. Preprint, arXiv:2211.06679. find
  10. Judith Clymo, Adam Zernik, and Shubham Gaur. 2025. UCSC NLP T6 at SemEval-2025 Task 1: Leveraging LLMs and VLMs for idiomatic understanding. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  11. Verna Dankers, Christopher Lucas, and Ivan Titov. 2022. Can transformer be too compositional? analysing idiom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3608–3626, Dublin, Ireland… find our card
  12. DeepSeek-AI. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  13. Yuming Fan, Dongming Yang, Zefeng Cai, and Binghuai Lin. 2025. CTYUN-AI at SemEval-2025 Task 1: Learning to rank for idiomatic expressions. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  14. Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Assessing the representations of idiomaticity in vector models with a noun compound dataset labeled at type and token levels. In Proceedings of the 59th Annual Meeting of the Association for Computatio… find
  15. Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the 12th language resources and evaluation conference, pages 279–287, Marseille, France. European Language Resources Association. find our card
  16. Wei He, Marco Idiart, Carolina Scarton, and Aline Villavicencio. 2024a. Enhancing idiomatic representation in multiple languages via an adaptive contrastive triplet loss. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12473–12485, Bangkok, Thailand. Association for Com… find
  17. Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2024b. Investigating idiomaticity in word representations. Computational Linguistics, pages 1–48. find
  18. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 13… find our card
  19. Saurav K. Aryal and Lawal Abdulmujeeb. 2025. Howard University-AI4PC at SemEval-2025 Task 1: Using GPT-4o and CLIP-ViLT to decode figurative language across text and images. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Lingui… find
  20. Maira Khatoon, Arooj Kiyani, Tehmina Doltana Farid, and Sadaf Abdul Rauf. 2025. FJWU_Squad at SemEval-2025 Task 1: An idiom visual understanding dataset for idiom learning. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguis… find
  21. Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, and Yadollah Yaghoobzadeh. 2025. Comparative Study of Multilingual Idioms and Similes in Large Language Models. arXiv preprint. ArXiv:2410.16461 [cs]. find our card
  22. Liu Lei, You Zhang, Jin Wang, Dan Xu, and Xuejie Zhang. 2025. YNU-HPCC at SemEval-2025 Task 1: Enhancing multimodal idiomaticity representation via LoRA and hybrid loss optimization. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computation… find
  23. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages… find our card
  24. Iris Hendrickx, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2013. SemEval-2013 task 4: Free paraphrases of noun compounds. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop… find
  25. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36. find our card
  26. Miloš Jakubíček, Adam Kilgarriff, Vojtěch Kovář, Pavel Rychlỳ, and Vít Suchomel. 2013. The TenTen corpus family. In 7th international corpus linguistics conference CL, volume 2013, pages 125–127. Valladolid. find
  27. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th Internatio… find our card
  28. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446. find
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. find
  30. Ann Maria Piho, Lara Eulenpesch, Julio Julio, Victoria Lohner, and Wiebke Petersen. 2025. Transformer25 at SemEval-2025 Task 1: A similarity-based approach. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  31. Thanet Markchom, Tong Wu, Liting Huang, and Huizhi Liang. 2025. UoR-NCL at SemEval-2025 Task 1: Using generative LLMs and CLIP models for multilingual multimodal idiomaticity representation. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Com… find
  32. Blake Matheny, Phuong Minh Nguyen, and Minh Le Nguyen. 2025. JNLP at SemEval-2025 Task 1: Multimodal idiomaticity representation with large language models. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  33. Maggie Mi, Aline Villavicencio, and Nafise Sadat Moosavi. 2024. Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context. arXiv preprint. find
  34. Joydeb Mondal and Pramir Sarkar. 2025. Modgenix at SemEval-2025 Task 1: Context aware vision language ranking (CAViLR) for multimodal idiomaticity understanding. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  35. OpenAI. 2024. Gpt-4o system card. arXiv:2410.21276. find our card
  36. Preprint, Ronghao Pan, Tomás Bernal-Beltrán, José Antonio García-Díaz, and Rafael Valencia-García. 2025. UMUTeam at SemEval-2025 Task 1: Leveraging multimodal and large language model for identifying and ranking idiomatic expressions. In Proceedings of the 19th International Workshop on Semantic Eva… find
  37. Dylan Phelps, Thomas M. R. Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. 2024. Sign of the times: Evaluating the use of large language models for idiomaticity detection. In Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2… find
  38. Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  39. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. Preprint, arXiv:1908.10084. find
  40. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2025. Understanding Figurative Meaning through Explainable Visual Entailment. arXiv preprint. ArXiv:2405.01474 [cs]. find
  41. Irena Spasić, Lowri Williams, and Andreas Buerki. 2020. Idiom-Based Features in Sentiment Analysis: Cutting the Gordian Knot. IEEE Transactions on Affective Computing, 11(2):189–199. find
  42. Syahrir Syahrir and St Hartina. 2021. The Analysis of Short Story Translation Errors (A Case Study of Types and Causes). IDEAS: Journal on English Language Teaching and Learning, Linguistics and Literature, 9(1). Number: 1. find
  43. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding. arXiv preprint. ArXiv:2204.10050 [cs]. find our card
  44. Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. AStitchInLanguageModels: Dataset and methods for the exploration of idiomaticity in pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3464–3477, Punt… find our card
  45. Simone Tedeschi, Federico Martelli, and Roberto Navigli. 2022. ID10M: Idiom Identification in 10 Languages. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2715–2726, Seattle, United States. Association for Computational Linguistics. find our card
  46. Sumiko Teng and Chuen Shin Yong. 2025. ChuenSumi at SemEval-2025 Task 1: Sentence transformer models and processing idiomacity. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  47. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020. find our card
  48. Lorenzo Vaiani, Davide Napolitano, and Luca Cagliero. 2025. PoliTo at SemEval-2025 Task 1: Beyond literal meaning: A chain-of-though approach for multimodal idiomacity understanding. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computation… find
  49. Arash Rasouli, Erfan Sadraiye, Omid Ghahroodi, Hamid Reza Rabiee, and Ehsaneddin Asgari. 2025. AIMA at SemEval-2025 Task 1: Bridging text and image for idiomatic knowledge extraction via mixture of experts. In Proceedings of the 19th International Aline Villavicencio, Francis Bond, Anna Korhonen, an… find
  50. Hao Wang, Fang Liu, Licheng Jiao, Jiahao Wang, Zehua Hao, Shuo Li, Lingling Li, Puhua Chen, and Xu Liu. 2024. Vilt-clip: Video and language tuning clip with multimodal prompt learning and scenario-guided optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6):5390–5400. find
  51. Ziheng Zeng and Suma Bhat. 2022. Getting BART to ride the idiomatic train: Learning to represent idiomatic expressions. Transactions of the Association for Computational Linguistics, 10:1120–1137. find
  52. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171. find our card
  53. Ziheng Zeng, Kellen Cheng, Srihari Nanniyur, Jianing Zhou, and Suma Bhat. 2023. IEKG: A commonsense knowledge graph for idiomatic expressions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14243–14264, Singapore. Association for Computational Lingui… find our card
  54. Yanan Wang, Dailin Li, Yicen Tian, Bo Zhang, Wang Jian, and Liang Yang. 2025. dutir914 at SemEval-2025 Task 1: An integrated approach for multimodal idiomaticity representations. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational L… find
  55. Yingzhou Zhao, Bowen Guan, Liang Yang, and Hongfei Lin. 2025. Zhoumou at SemEval-2025 Task 1: Leveraging multimodal data augmentation and large language models for enhanced idiom understanding. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for… find
  56. Lowri Williams, Christian Bannister, Michael Arribas-Ayllon, Alun Preece, and Irena Spasić. 2015. The role of idioms in sentiment analysis. Expert Systems with Applications, 42(21):7375–7385. find
  57. Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard, Wei-Wei Tu, Quanming Yao, Huan Zhao, and Isabelle Guyon. 2022. Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform. Patterns, 3(7):100543. find
  58. Majid Yazdani, Meghdad Farahmand, and James Henderson. 2015. Learning semantic composition to detect non-compositionality of multiword expressions. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1733–1742, Lisbon, Portugal. Association for Computatio… find
  59. Anar Yeginbergen, Elisa Sanchez, Andrea Jaunarena, and Ander Salaberria. 2025. HiTZ-Ixa at SemEval-2025 Task 1: Multimodal idiomatic language understanding. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  60. Ron Yosef, Yonatan Bitton, and Dafna Shahaf. 2023. IRFL: Image Recognition of Figurative Language. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1044–1058, Singapore. Association for Computational Linguistics. find
  61. Runyang You, Xinyue Mei, Mengyuan Zhou, XiaoLong Hou, and Lianxin Jiang. 2025. Pali-nlp at semeval-2025 task 1: Multimodal idiom recognition and alignment. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find
  62. Yue Yu, Jiarong Tang, and Ruitong Liu. 2025. TueCL at SemEval-2025 Task 1: Admire: Advancing multimodal idiomaticity representation. In Proceedings of the 19th International Workshop on Semantic Evaluations (SemEval-2025). Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/pickard-2025-semeval-2025-task-1-admire/