Elamur Research⌕
☰
home / papers / 2026 / I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes

I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes

Summary

Memes often carry non-literal meaning through the interplay of picture and caption. However, it is poorly understood how generative multimodal large language models (MLLMs) combine the two channels to recognise that meaning. The authors define a combined task: a model must decide which kinds of figurative meaning a meme carries and then justify that decision in natural language. They test eight open-weight MLLMs from the Aya, Gemma and Qwen families zero-shot on three existing English meme collections. The evaluation covers six figurative categories, binary and multi-label settings, and two prompt variants. To isolate each modality, models receive the full meme, the text alone, or the image with its caption erased. Human judges then rate whether explanations support the predicted label and stay faithful to the meme. Three findings stand out:

  • Every model tends to see figurative meaning even in literal memes.
  • Reliance on each modality depends on the type: irony/sarcasm leans on visuals, while metaphor needs both channels.
  • Correct labels often come with hallucinated or over-interpreted explanations.

Label accuracy alone therefore overstates how well models understand memes.

Key points

  • Evaluation data:
    • FigMemes test set: 1,372 memes from 4chan /pol/, multi-label. It covers allusion, exaggeration, irony, anthropomorphism, metaphor and contrast; its labels have a Fleiss' κ of 0.42.
    • Memotion 2 validation set: 1,500 memes, sarcasm only.
    • MET-Meme: 2,108 English memes, a balanced split of 1,054 metaphoric and 1,054 literal.
  • Models: Aya (8B, 32B), Gemma (4B, 12B, 27B) and Qwen (7B, 32B, 72B). F1 scores are averaged over five runs and compared against the earlier FigMemes baselines.
  • Image-only inputs: embedded text is detected and masked with PaddleOCR, then filled in with LaMa inpainting. The inpainting quality was checked manually.
  • Main error pattern: models systematically over-assign figurative labels to literal memes.
  • Availability: code and evaluation materials are released at github.com/mainlp/Figurative-Meaning-in-Memes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/55fa4fifm4pf

@inproceedings{zhou2026i,
  title = {{I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes}},
  author = {Zhou, Shijia and Mohammad, Saif M. and Plank, Barbara and Frassinelli, Diego},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/55fa4fifm4pf},
  url = {https://doi.org/10.63317/55fa4fifm4pf}
}

References

45 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arjun R. Akula, Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi, Sugato Basu, Leonidas Guibas, William T. Freeman, Yuanzhen Li, and Varun Jampani. 2023. MetaCLUE: Towards Comprehensive Visual Metaphors Research. In Proceedings of the IEEE/CVF Confere… find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Ha… find our card
  3. Luke Bates, Peter Ebert Christensen, Preslav Nakov, and Iryna Gurevych. 2025. A template is all you meme. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10443–10… find
  4. Christian Burgers, Elly A. Konijn, and Gerard J. Steen. 2016. Figurative Framing: Shaping Public Discourse Through Metaphor, Hyperbole, and Irony. Communication Theory, 26(4):410–430. find
  5. Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. 2022. Prompting for multimodal hateful meme classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 321–332, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  6. Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022a. It‘s not rocket science: Interpreting figurative language in narratives. Transactions of the Association for Computational Linguistics, 10:589–606. find
  7. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022b. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Associa… find our card
  8. Wangqun Chen, Fuqiang Lin, Guowei Li, and Bo Liu. 2024. A survey of automatic sarcasm detection: Fundamental theories, formulation, datasets, detection methods, and opportunities. Neurocomputing, 578:127428. find
  9. Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. 2025. PaddleOCR 3.0 Technical Report. find
  10. Susan R Fussell and Mallie M Moss. 2014. Figurative language in emotional communication. In Social and cognitive approaches to interpersonal communication, pages 113–141. Psychology Press. find
  11. Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest. In Proceedings of the 61st Annual Meeting of the Association for Computatio… find
  12. Eftekhar Hossain, Omar Sharif, and Mohammed Moshiul Hoque. 2022. MemoSen: A multimodal dataset for sentiment analysis of memes. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1542–1554, Marseille, France. European Language Resources Association. find
  13. EunJeong Hwang and Vered Shwartz. 2023. MemeCap: A Dataset for Captioning and Interpreting Memes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1433–1445, Singapore. Association for Computational Linguistics. find
  14. Hyewon Jang and Diego Frassinelli. 2024. Generalizable sarcasm detection is just around the corner, of course! In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4238–4249, M… find our card
  15. Hyewon Jang, Qi Yu, and Diego Frassinelli. 2023. Figurative language processing: A linguistically informed feature analysis of the behavior of language models and humans. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9816–9832, Toronto, Canada. Association for Computa… find
  16. Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Taylor Berg-Kirkpatrick. 2021. Investigating robustness of dialog models to popular figurative language constructs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7476–7485, Online and Punta Cana, Domini… find
  17. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. In Advances in Neural Information Processing Systems, volume 33, pages 2611–2624. Curran Associates, Inc. find our card
  18. Shreyas Kulkarni, Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2024. A report on the FigLang 2024 shared task on multimodal figurative language. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 115–119, Mexico City, Mexico (Hybrid). Association… find
  19. Gitanjali Kumari, Dibyanayan Bandyopadhyay, Asif Ekbal, and Vinutha B. NarayanaMurthy. 2024. CM-Off-Meme: Code-Mixed Hindi-English Offensive Meme Detection with Multi-Task Learning by Leveraging Contextual Knowledge. In Proceedings of the 2024 Joint International Conference on Computational Linguist… find
  20. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611… find
  21. Huiyuan Lai, Antonio Toral, and Malvina Nissim. 2023. Multilingual multi-figurative language detection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9254– 9267, Toronto, Canada. Association for Computational Linguistics. find
  22. George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press. find
  23. Chen Liu, Gregor Geigle, Robin Krebs, and Iryna Gurevych. 2022a. FigMemes: A Dataset for Figurative Language Identification in Politically-Opinionated Memes. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7069–7086, Abu Dhabi, United Arab Emirates. A… find
  24. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022b. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437–4452… find
  25. Ziyan Liu, Chunxiao Fan, Haoran Lou, Yuexin Wu, and Kaiwei Deng. 2025. MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 923–947, Vienna, Austria. Association f… find
  26. Leena Mathur, Paul Pu Liang, and Louis-Philippe Morency. 2024. Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20541–20560, Miami, Florida, USA. Association for Compu… find
  27. Abhilash Nandy, Yash Agarwal, Ashish Patwa, Millon Madhur Das, Aman Bansal, Ankit Raj, Pawan Goyal, and Niloy Ganguly. 2024. ***YesBut***: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models. In Proceedings of the 2024 Conference on Em… find
  28. Khoi P. N. Nguyen, Terrence Li, Derek Lou Zhou, Gabriel Xiong, Pranav Balu, Nandhan Alahari, Alan Huang, Tanush Chauhan, Harshavardhan Bala, Emre Guzelordu, Affan Kashfi, Aaron Xu, Suyesh Shrestha, Megan Vu, Jerry Wang, and Vincent Ng. 2025. MemeQA: Holistic Evaluation for Meme Understanding. In Pro… find
  29. Asaf Nissenbaum and Limor Shifman. 2017. Internet memes as contested cultural capital: The case of 4chan’s /b/ board. New Media & Society, 19(4):483–501. find
  30. Shraman Pramanick, Aniket Roy, and Vishal M. Patel Johns. 2022. Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 546–556. find
  31. Sathyanarayanan Ramamoorthy, Nethra Gunti, Shreyash Mishra, S Suryavardan, Aishwarya Reganti, Parth Patwa, Amitava Das, Tanmoy Chakraborty, Amit Sheth, Asif Ekbal, and Chaitanya Ahuja. 2022. Memotion 2: Dataset on sentiment and emotion analysis of memes. In De-Factify @AAAI, volume 3199 of CEUR Work… find
  32. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2025. Understanding figurative meaning through explainable visual entailment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech… find
  33. Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Björn Gambäck. 2020. SemEval-2020 task 8: Memotion analysis- the visuo-lingual metaphor! In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 759–773, Barcelo… find our card
  34. Limor Shifman. 2013. Memes in a Digital World: Reconciling with a Conceptual Troublemaker. Journal of Computer-Mediated Communication, 18(3):362–377. find
  35. Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. IMPLI: Investigating NLI models’ performance on figurative language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375–5388, Dublin, Ireland. Association for Computatio… find our card
  36. Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. 2021. Resolution-robust Large Mask Inpainting with Fourier Convolutions. find
  37. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, et al. 2025. Gemma 3 technical report. find
  38. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetune… find our card
  39. Bradley E Wiggins. 2019. The discursive power of memes in digital culture: Ideology, semiotics, and intertextuality. Routledge. find
  40. Bo Xu, Tingting Li, Junzhe Zheng, Mehdi Naseriparsa, Zhehuan Zhao, Hongfei Lin, and Feng Xia. 2022. MET-Meme: A Multimodal Meme Dataset Rich in Metaphors. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, pages 2887–2899, N… find
  41. Yanzhi Xu, Yueying Hua, Shichen Li, and Zhongqing Wang. 2024. Exploring Chain-of-Thought for Multi-modal Metaphor Detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 91–101, Bangkok, Thailand. Association for Computatio… find
  42. Akhila Yerukola, Saujas Vaduguru, Daniel Fried, and Maarten Sap. 2024. Is the pope catholic? yes, the pope is catholic. generative evaluation of non-literal intent resolution in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),… find
  43. Linhao Zhang, Jintao Liu, Li Jin, Hao Wang, Kaiwen Wei, and Guangluan Xu. 2024. GOME: Grounding-based metaphor binding with conceptual elaboration for figurative language illustration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18500–18510, Miami… find
  44. Naitian Zhou, David Jurgens, and David Bamman. 2024. Social meme-ing: Measuring linguistic variation in memes. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3005–3024, M… find
  45. Shijia Zhou, Siyao Peng, Simon M. Luebke, Jörg Haßler, Mario Haim, Saif M. Mohammad, and Barbara Plank. 2025. What Media Frames Reveal About Stance: A Dataset and Study about Memes in Climate Change Discourse. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-i-came-i-saw-i/