Elamur Research⌕
☰
home / papers / 2026 / “I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns?

“I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns?

Paper · 2026 · Naen Xu, Jiayi Sheng, Changjiang Li et al.

Summary

The paper asks whether vision-language models (VLMs) actually understand multimodal puns or only react to playful-looking content. In these puns, an image shows a literal object acting out a second meaning, and the caption triggers that second meaning. Earlier benchmarks were text-only or contained only positive examples. They also used one-sided yes/no questions, which cannot separate real comprehension from a model's tendency to agree.

The authors build an automatic pipeline that mines homophonic word pairs and polysemous nouns from the CMU Pronouncing Dictionary and WordNet. GPT-4o then generates captions and image descriptions. Hard non-pun distractors are created by swapping the pun word. The resulting benchmark, MultiPun, is used to test 11 VLMs on detection, localization and explanation, with each question asked in both direct and reversed form.

Most models over-interpret: they label distractors as puns and explain the wordplay imprecisely. Two remedies are proposed:

  • Pun-CoT, a structured verification prompt.
  • Pun-Tuning, fine-tuning on balanced data.

Together they raise F1 by 16.5% on average. This is a useful diagnostic for cross-modal humor reasoning.

Key points

  • MultiPun contains 445 puns (homophonic and homographic) and 890 non-pun distractors, produced by explicative and random word substitution.
  • Candidate pun tuples are filtered by:
    • Zipf frequency (>3.0 for homophones, >3.8 for homographs);
    • visually depictable WordNet noun categories;
    • lexical-file divergence and low path similarity (<0.1).
  • Prompts are given in both biased-to-pun and biased-to-non-pun versions to expose sycophancy and affirmative bias. TPR, TNR and F1 are reported.
  • Pun-CoT doubles the homophonic TNR of Qwen3-VL-8B-Thinking, from 0.387 to 0.776.
  • Pun-Tuning results on homophonic puns:
    • Qwen3-VL-30B-Instruct TNR rises from 0.209 to 0.948.
    • LLaVA-v1.6 Vicuna-13B F1 rises from 0.057 to 0.640.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓Beihang UniversityBeijing · China65 works · 8,452 citations🎓Hangzhou Dianzi UniversityHangzhou · China7 works · 1,061 citations🎓Qilu University of TechnologyJinan · China4 works · 3 citations🏢Palo Alto NetworksSanta Clara · United States3 works · 2 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li and Shouling Ji. 2026. “I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns?. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.444

@inproceedings{xu2026i,
  title = {{“I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns?}},
  author = {Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li and Shouling Ji},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.444},
  url = {https://doi.org/10.18653/v1/2026.acl-long.444}
}

References

53 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hengyu An, Minxi Li, Jinghuai Zhang, Naen Xu, Chunyi Zhou, Changjiang Li, Xiaogang Xu, Tianyu Du, and Shouling Ji. 2026. Aciarena: Toward unified evaluation for agent cascading injection. Preprint, arXiv:2604.07775. find
  2. Hengyu An, Jinghuai Zhang, Tianyu Du, Chunyi Zhou, Qingming Li, Tao Lin, and Shouling Ji. 2025. IPI-Guard: A novel tool dependency graph-based defense against indirect prompt injection in LLM agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 102… find
  3. Anthropic. 2025. Claude sonnet 4. https://www.anthropic.com/claude/sonnet. ↗
  4. Salvatore Attardo. 2018. Universals in puns and humorous wordplay. Cultures and traditions of wordplay and wordplay research, pages 89–110. find
  5. Salvatore Attardo. 2024. Linguistic theories of humor, volume 1. Walter de Gruyter GmbH & Co KG. find
  6. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. arXiv prepri…
  7. Carnegie Mellon University. 2015. The CMU pronouncing dictionary. http://www.speech.cs.cmu.edu/ cgi-bin/cmudict. ↗
  8. Jiwan Chung, Seungwon Lim, Jaehyun Jeon, Seungbeen Lee, and Youngjae Yu. 2024. Can visual language models resolve textual ambiguity with visual cues? let visual puns tell you! In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2452–2469, Miami, Florida,… find
  9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  11. Marta Dynel. 2009. Beyond a joke: Types of conversational humour. Language and linguistics compass, 3(5):1284–1299. find
  12. Zhe Hu, Tuo Liang, Jing Li, Yiren Lu, Yunlai Zhou, Yiran Qiao, Jing Ma, and Yu Yin. 2024. Cracking the code of juxtaposition: Can ai models understand the humorous contradictions. Advances in Neural Information Processing Systems, 37:47166–47188. find
  13. International Organization for Standardization. 2024a. ISO/TS 5777:2024 Health informatics — The architecture of internet healthcare service network. Technical Specification ISO/TS 5777:2024, International Organization for Standardization. find
  14. George A. Miller. 1992. WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992.
  15. International Organization for Standardization. 2024b. ISO/TS 5788:2024 Health informatics — Internet healthcare service pattern. Technical Specification ISO/TS 5788:2024, International Organization for Standardization. find
  16. Tristan Miller, Christian Hempelmann, and Iryna Gurevych. 2017. SemEval-2017 task 7: Detection and interpretation of English puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 58–68, Vancouver, Canada. Association for Computational Linguistics.
  17. Justine T Kao, Roger Levy, and Noah D Goodman. 2016. A computational model of linguistic humor in puns. Cognitive science, 40(5):1270–1285. find
  18. OpenAI. 2024. New embedding models and api updates. Blog post. find
  19. Guangchen Lan, Sipeng Zhang, Tianle Wang, Yuwei Zhang, Daoan Zhang, Xinpeng Wei, Xiaoman Pan, Hongming Zhang, Dong-Jun Han, and Christopher G Brinton. 2025. Mappo: Maximum a posteriori preference optimization with prior knowledge. arXiv preprint arXiv:2507.21183. — [arXiv:2507.21183](https://arxiv.org/abs/2507.21183) find
  20. Zixu Li, Zhiwei Chen, Haokun Wen, Zhiheng Fu, Yupeng Hu, and Weili Guan. 2025. Encoder: Entity mining and modification relation binding for composed image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5101–5109. find
  21. Zixu Li, Yupeng Hu, Zhiwei Chen, Qinlei Huang, Guozhi Qiu, Zhiheng Fu, and Meng Liu. 2026a. Retrack: Evidence-driven dual-stream directional anchor calibration network for composed video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 23373–23381. find
  22. OpenAI. 2025. Gpt-5 is here. https://openai.com/ gpt-5/. ↗
  23. Andrew Stott. 2014. Comedy. Routledge. find
  24. Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Tagyoung Chung, Jing Huang, Yang Liu, and Nanyun Peng. 2022. ExPUNations: Augmenting puns with keywords and explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4590–4605, Abu… find
  25. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  26. Zixu Li, Yupeng Hu, Zhiwei Chen, Shiqi Zhang, Qinlei Huang, Zhiheng Fu, and Yinwei Wei. 2026b. Habit: Chrono-synergia robust progressive learning framework for composed image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 6762–6770. find
  27. Xinyu Wang, Yue Zhang, and Liqiang Jing. 2025. Can large vision-language models understand multimodal sarcasm? In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 5340–5345. find
  28. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  29. Yebo Wu, Jingguang Li, Zhijiang Guo, and Li Li. 2025a. Elastic mixture of rank-wise experts for knowledge reuse in federated fine-tuning. arXiv preprint arXiv:2512.00902. — [arXiv:2512.00902](https://arxiv.org/abs/2512.00902) find
  30. Xuan Liu, Haoyang Shang, and Haojian Jin. 2025. Cobra: Programming cognitive bias in social agents using classic social science experiments. arXiv preprint arXiv:2509.13588. — [arXiv:2509.13588](https://arxiv.org/abs/2509.13588) find
  31. Yebo Wu, Jingguang Li, Zhijiang Guo, and Li Li. 2026a. Developmental federated tuning: A cognitive-inspired paradigm for efficient LLM adaptation. In The Fourteenth International Conference on Learning Representations. find
  32. Xuan Liu, Haoyang Shang, Zizhang Liu, Xinyan Liu, Yunze Xiao, Yiwen Tu, and Haojian Jin. 2026. Humanstudy-bench: Towards ai agent design for participant simulation. arXiv preprint arXiv:2602.00685. — [arXiv:2602.00685](https://arxiv.org/abs/2602.00685) find
  33. Ziqiang Liu, Feiteng Fang, Xi Feng, Xeron Du, Chenhao Zhang, Noah Wang, Qixuan Zhao, Liyang Fan, CHENGGUANG GAN, Hongquan Lin, and 1 others. 2024. Ii-bench: An image implication understanding benchmark for multimodal large language models. Advances in Neural Information Processing Systems, 37:46378–… find
  34. Yebo Wu, Jingguang Li, Chunlin Tian, Zhijiang Guo, and Li Li. 2025b. Memory-efficient federated fine-tuning of large language models via layer pruning. arXiv preprint arXiv:2508.17209. — [arXiv:2508.17209](https://arxiv.org/abs/2508.17209) find
  35. Yebo Wu, Li Li, Chunlin Tian, Tao Chang, Chi Lin, Cong Wang, and Cheng-Zhong Xu. 2024. Heterogeneity-aware memory efficient federated learning via progressive layer freezing. In 2024 IEEE/ACM 32nd International Symposium on Quality of Service (IWQoS), pages 1–10. IEEE. find
  36. Yebo Wu, Li Li, and Cheng-zhong Xu. 2025c. Breaking the memory wall for heterogeneous federated learning via progressive training. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pages 1623–1632. find
  37. Yebo Wu, Feng Liu, Ziwei Xie, Zhiyuan Liu, Changwang Zhang, Jun Wang, and Li Li. 2026b. Tsembed: Unlocking task scaling in universal multimodal embeddings. arXiv preprint arXiv:2603.04772. — [arXiv:2603.04772](https://arxiv.org/abs/2603.04772) find
  38. Naen Xu, Hengyu An, Shuo Shi, Jinghuai Zhang, Chunyi Zhou, Changjiang Li, Tianyu Du, Zhihui Fu, Jun Wang, and Shouling Ji. 2026a. When agents “misremember” collectively: Exploring the mandela effect in LLM-based multi-agent systems. In The Fourteenth International Conference on Learning Representati… find
  39. Naen Xu, Changjiang Li, Tianyu Du, Minxi Li, Wenjie Luo, Jiacheng Liang, Yuyuan Li, Xuhong Zhang, Meng Han, Jianwei Yin, and 1 others. 2024a. Copy-rightmeter: Revisiting copyright protection in text-to-image models. arXiv preprint arXiv:2411.13144. — [arXiv:2411.13144](https://arxiv.org/abs/2411.13144) find
  40. Naen Xu, Jinghuai Zhang, Changjiang Li, Hengyu An, Chunyi Zhou, Jun Wang, Boyu Xu, Yuyuan Li, Tianyu Du, and Shouling Ji. 2026b. Bridging the copyright gap: Do large vision-language models recognize and respect copyrighted content? In Proceedings of the AAAI Conference on Artificial Intelligence, vo… find
  41. Naen Xu, Jinghuai Zhang, Changjiang Li, Zhi Chen, Chunyi Zhou, Qingming Li, Tianyu Du, and Shouling Ji. 2025a. Videoeraser: Concept erasure in text-to-video diffusion models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5965–5994. find
  42. Zhenhua Xu, Dongsheng Chen, Shuo Wang, Jian Li, Chengjie Wang, Meng Han, and Yabiao Wang. 2026c. Adamarp: An adaptive multi-agent interaction framework for general immersive role-playing. Preprint, arXiv:2601.11007.
  43. Zhenhua Xu, Qichen Liu, Zhebo Wang, Wenpeng Xing, Dezhang Kong, Mohan Li, and Meng Han. 2025b. Fingerprint vector: Enabling scalable and efficient model fingerprint transfer via vector addition. Preprint, arXiv:2409.08846. — [arXiv:2409.08846](https://arxiv.org/abs/2409.08846) find
  44. Zhenhua Xu, Xubin Yue, Zhebo Wang, Qichen Liu, Xixiang Zhao, Jingxuan Zhang, Wenjun Zeng, Weng-peng Xing, Dezhang Kong, Changting Lin, and Meng Han. 2025c. Copyright protection for large language models: A survey of methods, challenges, and trends. Preprint, arXiv:2508.11548. — [arXiv:2508.11548](https://arxiv.org/abs/2508.11548) find
  45. Zhenhua Xu, Xixiang Zhao, Xubin Yue, Shengwei Tian, Changting Lin, and Meng Han. 2025d. CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proces…
  46. Zhijun Xu, Siyu Yuan, Lingjie Chen, and Deqing Yang. 2024b. “a good pun is its own reword”: Can large language models understand puns? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11766–11782, Miami, Florida, USA. Association for Computational Ling… find
  47. Zhijun Xu, Siyu Yuan, Yiqiao Zhang, Jingyu Sun, Tong Zheng, and Deqing Yang. 2025e. PunMemeCN: A benchmark to explore vision-language models’ understanding of Chinese pun memes. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18705–18721, Suzhou, Chin… find
  48. Zhiwei Yu, Hongyu Zang, and Xiaojun Wan. 2020. Homophonic pun generation with lexically constrained rewriting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2870–2876, Online. Association for Computational Linguistics. find
  49. Alessandro Zangari, Matteo Marcuzzo, Andrea Al-barelli, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2025. Pun unintended: LLMs and the illusion of humor understanding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27924–27959, Suzhou, China.… find
  50. Tuo Zhang, Tiantian Feng, Yibin Ni, Mengqin Cao, Ruy-ing Liu, Kiana Avestimehr, Katharine Butler, Yanjun Weng, Mi Zhang, Shrikanth Narayanan, and 1 others. 2025. Creating a lens of chinese culture: A multimodal dataset for chinese pun rebus art understanding. In Findings of the Association for Compu… find
  51. Yichao Zhou, Jyun-Yu Jiang, Jieyu Zhao, Kai-Wei Chang, and Wei Wang. 2020. “the boating store had its best sail ever”: Pronunciation-attentive contextualized pun recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 813–822, Online. Associatio… find
  52. Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 conference of the North American chapter of the Association for Computational Lingu… find
  53. Yanyan Zou and Wei Lu. 2019. Joint detection and location of English puns. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2117–2123, Minneapolis, Minnesota. Ass…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-i-see-what-you-did/