Elamur Research⌕
☰
home / papers / 2026 / Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

Paper · 2026 · Ahmed Sharshar, Hosam Elgendy, Saad El Dine Ahmed et al.

Summary

The paper argues that detecting harmful humor is a safety problem that existing toxicity benchmarks cover poorly. Dark jokes often hide their offence behind cultural knowledge and multi-step inference rather than slurs. Most current resources are also English-only, built from text or single images, and rarely separate overt from covert harm.

To fill this gap, the authors hand-collected a benchmark of jokes in three modalities: text, images/memes and short videos. Text and images are in English and Arabic, including several dialects. Videos add a third, language-independent "Universal" group. Every item is first labeled Safe or Harmful. Harmful items are then split into Explicit, where the offence is visible on the surface, and Implicit, where decoding the offence needs semantic or cultural context.

The authors tested open and closed LLMs, VLMs and video LLMs on one shared task definition. Closed models clearly beat open ones, and all models do worse on Arabic than on English. The work highlights the need for safety alignment that is culturally grounded and able to reason.

Key points

  • Size of the resource: 3,000 texts (1,000 Arabic, 2,000 English), about 6,000 images (6,005 total) and 1,202 videos (Arabic, English, Universal). All were gathered manually from websites, with no automated scraping.
  • Implicit harm dominates the video portion: 843 of 1,202 clips are Implicit, versus 165 Safe and 194 Explicit.
  • Annotation process:
    • Seven volunteer annotators (4 men, 3 women) each labeled the whole dataset across all modalities.
    • Gold labels come from a per-item majority vote.
    • The Arabic data mixes Egyptian, Lebanese, Iraqi and Saudi dialects with MSA.
  • Main findings:
    • Proprietary models (GPT and Gemini families) are strongest.
    • Open-source video models often ignore audio or lose coherence on longer clips, especially for Arabic.
    • Every model family shows a gap between English and Arabic.
  • Release terms: annotations and schema are released under CC BY-NC-SA 4.0 for non-commercial research. Fine-grained harm-type labels (racism, religion, etc.) were deliberately left out to keep labeling consistent.

Interesting ideas

  • I1. Splitting positive cases into "visible on the surface" and "recoverable only through inference" shows whether a system truly reasons or just matches cues.
  • I2. Having every annotator label every item, instead of splitting the data among them, gives a full agreement picture for each example before the majority vote.
  • I3. The authors kept the label scheme coarse on purpose, because finer categories would lower agreement with a small labeling pool, and treated finer types as a later extension.
  • I4. A language-independent subset sits beside the language-specific ones, separating failures of language competence from failures of world or cultural reasoning.
  • I5. The authors propose making models state why a joke is harmful, which grounds the decision and limits hallucinated judgments.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): irony and implicature outputs in prag could carry a flag for lexically cued versus inferred readings, with test sets split the same way so a cue-spotting shortcut cannot hide.
  • I2 (learn): silver data could require all witness parsers to parse every sentence and store per-arc vote counts, so low-agreement items get routed to the LLM teacher tiers.
  • I3 (learn): new silver or teacher annotation layers could start with a two- or three-way distinction and add sub-labels only once agreement on the coarse layer is stable.
  • I4 (answer): fable and school-test QA could include items whose answer depends only on the story world, not on wording, as a control showing whether errors come from parsing or from reasoning.
  • I5 (frames): an irony or offence verdict could be emitted together with the frame and implicature chain that produced it, matching Mova's rule that every step is explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates47 works · 4,456 citations🎓Sofia University "St. Kliment Ohridski"Sofia · Bulgaria10 works · 2,341 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Psychology · 798 works Social Psychology · 129 works

🧭 Topics 4 Evaluation & benchmarks · 685 works Multilingual & cross-lingual NLP · 467 works Vision & language · 269 works Bias, fairness & safety · 209 works

Cite

Cite as Ahmed Sharshar, Hosam Elgendy, Saad El Dine Ahmed, Yasser Rohaim and Yuxia Wang. 2026. Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor. arXiv:2603.17759.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.17759 · arXiv 2603.17759

@misc{sharshar2026harm,
  title = {{Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor}},
  author = {Ahmed Sharshar, Hosam Elgendy, Saad {El Dine Ahmed}, Yasser Rohaim and Yuxia Wang},
  year = {2026},
  doi = {10.48550/arxiv.2603.17759},
  eprint = {2603.17759},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.17759}
}

References

54 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sajal Aggarwal, Ananya Pandey, and Dinesh Kumar Vishwakarma. 2023. Multimodal sarcasm recognition by fusing textual, visual and acoustic content via multi-headed attention for video dataset. In Proceedings of the 2023 World Conference on Communication and Computing (WCONF), pages 1–5. find
  2. Hend Alkhalifa, Fetoun AlZahrani, Hala Qawara, Reema AlRowais, Sawsan Alowa, and Luluh AlDhubayi. 2022. A dataset for detecting humor in arabic text. In Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022), pages 219–225. find
  3. Shahin Amiriparian, Lukas Christ, Alexander Kathan, Maurice Gerczuk, Niklas Müller, Steffen Klug, Lukas Stappen, Andreas König, Erik Cambria, Björn Schuller, and Simone Eulitz. 2024. find
  4. The MuSe 2024 multimodal sentiment analysis challenge: Social perception and humor recognition. In Proceedings of the 5th Multimodal Sentiment Analysis Challenge (MuSe 2024) Workshop. find
  5. Salvatore Attardo. 2017. The Linguistics of Humor: An Introduction. Oxford University Press. find
  6. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. https://creativecommons.org/licenses/by-nc-s a/4.0/legalcode.en Ashwin Baluja. 2024. Text is not all you need: Multimodal prompting helps llms understand humor. arXiv preprint arXiv:2412.05315. ↗ our card
  7. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2024. Allam: Large language models for arabic and english. ↗ our card
  8. Valentin Barriere, Nahuel Gomez, Leo Hemamou, Sofia Callejas, and Brian Ravenet. 2025. Standup4ai: A new multilingual dataset for humor detection in stand-up comedy videos. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 16951–16959. find
  9. Luis Chiruzzo, Santiago Castro, and Aiala Rosá. 2020. HAHA 2019 dataset: A corpus for humor analysis in spanish. In Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC), pages 5106–5112. find
  10. Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, et al. 2025. Aya vision: Advancing the frontier of multilingual multimodality. arXiv preprint arXiv:2505.08751. ↗
  11. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. our card
  12. Elisabetta Fersini, Francesca Gasparini, Giulia Rizzi, Aurora Saibene, Berta Chulvi, Paolo Rosso, Alyssa Lees, and Jeffrey Sorensen. 2022. Semeval-2022 task 5: Multimedia automatic misogyny identification. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 533– 549. our card
  13. Thomas E. Ford. 2015. The social consequences of disparagement humor: Introduction and overview. HUMOR: International Journal of Humor Research, 28(2):163–169. ↗
  14. Thomas E. Ford and Mark A. Ferguson. 2004. Social consequences of disparagement humor: A prejudiced norm theory. Personality and Social Psychology Review, 8(1):79–94. ↗
  15. Thomas E. Ford, Julie A. Woodzicka, Sarah R. Triplett, Adam O. Kochersberger, and C. J. Holden. 2014. Not all groups are equal: Differential vulnerability of social groups to the prejudice-releasing effects of disparagement humor. Basic and Applied Social Psychology, 36(6):546–558. ↗
  16. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, and Katie et. al Millican. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗ our card
  17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The llama 3 herd of models. find
  18. Gil Greengross and Geoffrey Miller. 2011. Humor ability reveals intelligence, predicts mating success, and is higher in males. Intelligence, 39(4):188–192. ↗
  19. Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed (Ehsan) Hoque. 2019. UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of EMNLP-IJCNLP 2019, pages 2046–2056. find
  20. Maram Hasanain, Mohammadi Akram Hasan, Firoj Ahmed, Reem Suwaileh, Mrittika Biswas, Wajdi Zaghouani, and Firoj Alam. 2024. Araieval 2024 shared task: Propagandistic technique detection in multimodal arabic content. In Proceedings of the Second Arabic Natural Language Processing Conference (ARABIC NLP 2024). find
  21. Zachary Horvitz, Jingru Chen, Rahul Aditya, Harshvardhan Srivastava, Robert West, Zhou Yu, and Kathleen McKeown. 2024. Getting serious about humor: Crafting humor datasets with unfunny large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 855–869. ↗
  22. Nabil Hossain, John Krumm, and Michael Gamon. 2019. "president vows to cut taxes hair": Dataset and analysis of creative text editing for humorous headlines. arXiv preprint arXiv:1906.00274. ↗
  23. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. Acegpt, localizing large language models in arabic. ↗ our card
  24. Lee Hyun, Sung-Bin Kim, Seungju Han, Youngjae Yu, and Tae-Hyun Oh. 2024. SMILE: A multimodal dataset for understanding laughter in video with language explanations. In Findings of NAACL 2024, pages 1148–1161. find
  25. Vedaant V. Jain, Felipe dos Santos Alves Feitosa, and Gabriel Kreiman. 2025. HumorDB: Can AI understand graphical humor? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). find
  26. Sophie Jentzsch and Kristian Kersting. 2023. Chatgpt is fun, but it is not funny! humor is still challenging large language models. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (WASSA), pages 325–340. ↗
  27. Sai Kartheek Reddy Kasu, Shankar Biradar, and Sunil Saumya. 2025. Deceptive humor: A synthetic multilingual benchmark dataset for bridging fabricated claims with humorous content. arXiv preprint arXiv:2503.16031. ↗
  28. Sai Kartheek Reddy Kasu, Mohammad Zia Ur Rehman, Shahid Shafi Dar, Rishi Bharat Junghare, Dhan-vin S. Namboodiri, and Nagendra Kumar. 2025. D-HUMOR: Dark humor understanding via multimodal open-ended reasoning – a benchmark dataset and method. In Proceedings of the IEEE International Conference on Data Mining (ICDM). ↗
  29. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. arXiv:2005.04790. ↗ our card
  30. Roberto Labadie Tamayo, Berta Chulvi, and Paolo Rosso. 2023. Everybody hurts, sometimes: Overview of hurtful humour at iberlef 2023: Detection of humour spreading prejudice in twitter. Procesamiento del Lenguaje Natural, 71:383–395. ↗
  31. Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang. 2025. Videochat-flash: Hierarchical compression for long-context video modeling. ↗
  32. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge. https://llava-vl.github.io/blog/2024-01-3 0-llava-next/. ↗
  33. Tyler Loakman, William Thorne, and Chenghua Lin. 2025. Comparing apples to oranges: A dataset & analysis of llm humour understanding from traditional puns to topical jokes. arXiv preprint arXiv:2507.13335. ↗
  34. Rod A. Martin and Thomas E. Ford. 2018. The Psychology of Humor: An Integrative Approach. Academic Press. ↗
  35. Delfina Sol Martinez Pandiani, Erik Tjong Kim Sang, and Davide Ceolin. 2025. ’Toxic’ memes: A survey of computational perspectives on the detection and explanation of meme toxicities. Online Social Networks and Media, 47:100317. ↗
  36. J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. Semeval 2021 task 7: Hahackathon, detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 105– 119. our card
  37. Tristan Miller, Christian Hempelmann, and Iryna Gurevych. 2017. SemEval-2017 task 7: Detection and interpretation of English puns. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 58–68. our card
  38. Abhinav Moudgil. 2016. Short jokes. https://www.kaggle.com/datasets/abhinavmoudgil95/sho rt-jokes. Kaggle dataset, accessed 27 December 2025. ↗
  39. OpenAI. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗ our card
  40. OpenAI. 2025. Gpt-5 system card. System card, OpenAI. find
  41. Victor M. Palma Preciado, Grigori Sidorov, Liana Ermakova, Anne-Gwenn Bosser, Tristan Miller, and Adam Jatowt. 2024. Overview of the CLEF 2024 JOKER task 2: Humour classification according to genre and technique. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF 2024) – Working Notes. find
  42. Badri N. Patro, Mayank Lunayach, Deepankar Srivastava, Sarvesh, Hunar Singh, and Vinay P. Namboodiri. 2021. Multimodal humor dataset: Predicting laughter tracks for sitcoms. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 576–585. find
  43. Taivo Pungas. 2017. A dataset of english plaintext jokes. find
  44. Victor Raskin. 1985. Semantic Mechanisms of Humor. D. Reidel. ↗
  45. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ↗ our card
  46. Mohammadamin Shafiei and Hamidreza Saffari. 2025. Not all jokes land: Evaluating large language models’ understanding of workplace humor. arXiv preprint arXiv:2506.01819. ↗
  47. Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Björn Gambäck. 2020. Semeval-2020 task 8: Memotion analysis - the visuo-lingual metaphor. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), pages 759–773. shuttie. 2023. Dad jokes dataset. https://huggingf ace.co/datasets/shuttie/dadjokes. Accessed: 2025-12-20. shuttie. 2024. Reddit /r/dadjokes dataset. https://hu ggingface.co/datasets/shuttie/reddit-dad jokes. Accessed: 2025-12-20. ↗ our card
  48. Alexey Tikhonov and Pavel Shtykovskiy. 2024. Humor mechanics: Advancing humor generation with multi-step reasoning. arXiv preprint arXiv:2405.07280. ↗
  49. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Ke-Yang Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. ↗ our card
  50. Orion Weller and Kevin Seppi. 2020. The rjokes dataset: A large scale humor collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC), pages 6136–6141. find
  51. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. ↗ our card
  52. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800. ↗ our card
  53. Alessandro Zangari, Matteo Marcuzzo, Andrea Albarelli, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2025. Pun unintended: LLMs and the illusion of humor understanding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 27924–27959. find
  54. Jinguo Zhu, Weiyun Wang, Zhangwei Gao, Zhe Chen, Hongjie Zhang, Jinhui Yin, Wenhao Li, et al. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. ↗ our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharshar-2026-harm-humor-multimodal-multilingual-benchmark/