Elamur Research⌕
☰
home / papers / 2026 / JanusMM: A Benchmark for Self-Deprecation Understanding in Real-World Multimodal Conversations

JanusMM: A Benchmark for Self-Deprecation Understanding in Real-World Multimodal Conversations

Paper · 2026 · Xinyi Xu, Bingguang Hao, Yongyi Xiong et al.

Summary

Memes that make fun of oneself are a common way people signal emotions and intentions online. Often the picture and the caption seem unrelated on the surface, and the link between them is only implied. The authors ask whether multimodal large language models (MLLMs) can grasp this behaviour, and they present JanusMM as the first benchmark for it. They gathered 2,016 Chinese and English memes together with their surrounding dialogue from three settings: public social media, community group chats and one-on-one private messages. Annotators labelled each item in two stages, recording emotion, cross-modal reasoning, and the intent of both the sender and the receiver.

Evaluation has two tasks and six new metrics. The first task tests recognition and explanation of self-deprecation. The second asks the model to act as the person posting the meme and as the person replying to it. Across ten frontier MLLMs, recognition and reasoning were weak, explanations were often fabricated, and models understood the same meme differently depending on the role. The work exposes gaps in social and emotional alignment that matter for human–AI conversation.

Key points

  • Data sources: posts from Reddit and Rednote, chats from Discord and WeChat, and private dialogues contributed by 126 volunteers. All samples were manually verified, anonymised and filtered for toxic content.
  • Annotation reliability: Krippendorff's Alpha reached 0.81 on Emotion and 0.79 on Thinking, while Sender Reasoning scored 0.71 and Receiver Reasoning 0.74. Emotion labels draw on the NRC Emotion Lexicon and NTUSD.
  • Metrics: automatic metrics include Accuracy, METEOR, Greedy Match, BERTScore, BARTScore and a new IRC score that measures consistency between the initiator and responder roles. These are complemented by human-judged metrics.
  • Error analysis: an eight-category error taxonomy covers Social Criticism, Knowledge Disconnection, Attribution Fallacy, Textual Misunderstanding, Safety Misjudgment and other types.
  • Model-specific findings: Claude-Sonnet-4 (28.9%) and Claude-Sonnet-3.7 (26.4%) often over-read jokes as social criticism. Gemini-2.5-Pro and Doubao-1.5-Thinking-Pro show high rates of Knowledge Disconnection. Qwen models struggle with puns and with safety misjudgments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksEvaluation & benchmarks · 1,341 worksVision & language · 811 works

Cite

Cite as Xinyi Xu, Bingguang Hao, Yongyi Xiong, Zimo Chen, Xinchen Liu, Hongxin Guo, Xuelong Wang, Silin Zhou and Shihan Dou. 2026. JanusMM: A Benchmark for Self-Deprecation Understanding in Real-World Multimodal Conversations. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1116

@inproceedings{xu2026janusmm,
  title = {{JanusMM: A Benchmark for Self-Deprecation Understanding in Real-World Multimodal Conversations}},
  author = {Xinyi Xu, Bingguang Hao, Yongyi Xiong, Zimo Chen, Xinchen Liu, Hongxin Guo, Xuelong Wang, Silin Zhou and Shihan Dou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1116},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1116}
}

References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025a. Claude 3.7 sonnet and claude code. Accessed: 2024-05-22. find
  2. Anthropic. 2025b. Claude sonnet 4: Advanced reasoning and multimodal support for enterprise use cases. Accessed: 2025-02-28. find
  3. Jinze Bai, S Bai, S Yang, S Wang, S Tan, P Wang, J Lin, C Zhou, and J Zhou. 2023. Qwen-vl: A versatile vision-language model for. find
  4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation.
  5. Santosh Kumar Bharti, S Varadhaganapathy, Rajeev Kumar Gupta, Prashant Kumar Shukla, Mohamed Bouye, Simon Karanja Hingaa, and Amena Mahmoud. 2022. Text-based emotion recognition using deep learning approach. Computational Intelligence and Neuroscience, 2022(1):2645381. find
  6. Carlos Mauricio Castaño Díaz. 2013. Defining and characterizing the concept of internet meme. Ces Psicología, 6(2):82–104. find
  7. Yuyan Chen, Hao Wang, Songzhou Yan, Sijia Liu, Yueze Li, Yi Zhao, and Yanghua Xiao. 2024a. Emotionqueen: A benchmark for evaluating empathy of large language models. arXiv preprint arXiv:2409.13359. find
  8. Yuyan Chen, Yichen Yuan, Panjun Liu, Dayiheng Liu, Qinghao Guan, Mengfei Guo, Haiming Peng, Bang Liu, Zhixu Li, and Yanghua Xiao. 2024b. Talk funny! a large-scale humor response dataset with chain-of-humor interpretation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p… find
  9. Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805–110853. find
  10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  11. Jeffrey M Conte. 2005. A review and critique of emotional intelligence measures. Journal of organizational behavior, 26(4):433–440. find
  12. Laurence Devillers, Lori Lamel, and Ioana Vasilescu. 2003. Emotion detection in task-oriented spoken dialogues. In 2003 International Conference on Multimedia and Expo. ICME’03. Proceedings (Cat. No. 03TH8698), volume 3, pages III–549. IEEE. find
  13. Hang Du, Guoshun Nan, Sicheng Zhang, Binzhu Xie, Junrui Xu, Hehe Fan, Qimei Cui, Xiaofeng Tao, and Xudong Jiang. 2024. Docmsu: A comprehensive benchmark for document-level multimodal sarcasm understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17933–17941… find
  14. Maurice J Elias, L Hunter, and J Crees. 2001. Emotional intelligence. J. Ciarrochi., JP Forgas y JD Mayer (Eds.). Emotional Intelligence in everyday life: a scientific inquiry, pages 133–149. find
  15. Google. 2024. Gemini 2.5-flash: Efficient multimodal ai for real-time applications. Accessed: 2024-11-05. find
  16. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, and 1 others. 2025. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062. — Seed1.5-VL Technical Report
  17. Jinpeng Hu, Tengteng Dong, Gang Luo, Hui Ma, Peng Zou, Xiao Sun, Dan Guo, Xun Yang, and Meng Wang. 2024. Psycollm: Enhancing llm for psychological understanding and evaluation. IEEE Transactions on Computational Social Systems, 12(2):539–551. find
  18. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  19. Dongjin Kang, Sunghwan Kim, Taeyoon Kwon, Seungjun Moon, Hyunsouk Cho, Youngjae Yu, Dongha Lee, and Jinyoung Yeo. 2024. Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation. arXiv preprint arXiv:2402.13211. find
  20. Sören Krach, Frieder M Paulus, Maren Bodden, and Tilo Kircher. 2010. The rewarding nature of social interactions. Frontiers in behavioral neuroscience, 4:1141. find
  21. Bernhard Kratzwald, Suzana Ilić, Mathias Kraus, Stefan Feuerriegel, and Helmut Prendinger. 2018. Deep learning for affective computing: Text-based emotion recognition in decision support. Decision support systems, 115:24–35. find
  22. Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411– 433. find
  23. Lun-Wei Ku and Hsin-Hsi Chen. 2007. Mining opinions from the web: Beyond relevance retrieval. Journal of the American Society for Information Science and Technology, 58(12):1838–1850. find
  24. Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. 2025a. Natural language understanding and inference with mllm in visual question answering: A survey. ACM Computing Surveys, 57(8):1–36. find
  25. Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. 2025b. Natural language understanding and inference with mllm in visual question answering: A survey. ACM Computing Surveys, 57(8):1–36. find
  26. Anna Ladilova and Ulrike Schröder. 2022. Humor in intercultural interaction: A source for misunderstanding or a common ground builder? a multimodal analysis. Intercultural Pragmatics, 19(1):71–101. find
  27. Ao Li, Longwei Xu, Chen Ling, Jinghui Zhang, and Pengwei Wang. 2024. Emoverse: Exploring multimodal large language models for sentiment and emotion understanding. arXiv preprint arXiv:2412.08049. — [arXiv:2412.08049](https://arxiv.org/abs/2412.08049) find
  28. Deng Li, Bohao Xing, Xin Liu, Baiqiang Xia, Bihan Wen, and Heikki Kälviäinen. 2025. Deemo: De-identity multimodal emotion recognition and reasoning. arXiv preprint arXiv:2504.19549. — [arXiv:2504.19549](https://arxiv.org/abs/2504.19549) find
  29. Yunzhe Liu and Xinyi Xu. 2024. I mean i am a mouse: meets for bilingual multimodal meme sarcasm classification from large language models. In ACML, pages 1096–1111. find
  30. I Scott MacKenzie. 2024. Human-computer interaction: An empirical research perspective. find
  31. Saif M Mohammad and Peter D Turney. 2013. Nrc emotion lexicon. National Research Council, Canada, 2:234. find
  32. OpenAI. 2025. Gpt-4.1: Improved reasoning and context retention for complex tasks. Accessed: 2025-01-20. find
  33. Timothy J Owens. 1993. Accentuate the positive-and the negative: Rethinking the use of self-esteem, self-deprecation, and self-confidence. Social Psychology Quarterly, pages 288–299. find
  34. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE access, 7:100943–100953. find
  35. Oxford University Press. 2024. Self-deprecation: Oxford english dictionary. Accessed: 2024-05-22. find
  36. Huachuan Qiu and Zhenzhong Lan. 2024. Interactive agents: Simulating counselor-client psychological counseling via role-playing llm-to-llm interactions. arXiv preprint arXiv:2408.15787.
  37. Edward M Reingold and Robert E Tarjan. 1981. On a greedy heuristic for complete matching. SIAM Journal on Computing, 10(4):676–681. find
  38. Andrey Savchenko and Lyudmila Savchenko. 2025. Leveraging lightweight facial models and textual modality in audio-visual emotional understanding in-the-wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5778–5788. find
  39. Rossano Schifanella, Paloma De Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 24th ACM international conference on Multimedia, pages 1136–1145.
  40. ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, and 1 others. 2025. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914. — Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
  41. Susan A Speer. 2019. Reconsidering self-deprecation as a communication practice. British Journal of Social Psychology, 58(4):806–828. find
  42. Stephen L Tanner. 1996. The art of self-deprecation in american literary humor. Studies in American Humor, (3):54–65. find
  43. Ashwin TS and Gautam Biswas. 2024. Relating students cognitive processes and learner-centered emotions: An advanced deep learning approach. In Proceedings of the 26th International Conference on Multimodal Interaction, pages 575–584. find
  44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  45. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, 11(12):nwae403.
  46. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in neural information processing systems, 34:27263–27277.
  47. Tenggan Zhang, Xinjie Zhang, Jinming Zhao, Li Zhou, and Qin Jin. 2024. Escot: Towards interpretable emotional support dialogue systems. arXiv preprint arXiv:2406.10960. find
  48. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-janusmm-benchmark-self-deprecation-understanding/