Elamur Research⌕
☰
home / papers / 2024 / A Survey of Multimodal Sarcasm Detection

A Survey of Multimodal Sarcasm Detection

Paper · 2024 · Shafkat Farabi, Tharindu Ranasinghe, Diptesh Kanojia et al.

Summary

The paper addresses a gap in the literature: earlier surveys of sarcasm detection covered text alone, yet speakers often signal sarcasm through a mismatch with an attached image, their tone of voice, or their facial expression. The authors reviewed more than 60 publications from 2018–2023, drawn from Google Scholar, ACL Anthology, IEEE, ACM, Springer and CVF. They sort the field into two branches. The first is visuo-textual work, based on image–text posts from Twitter, Instagram, Tumblr and Facebook. The second is audio-visual-textual work, based on sitcom dialogue datasets such as MUStARD. For each branch the survey catalogues datasets and models and compares their reported scores. It traces how methods developed:

  • separate encoders (ResNet, BERT) combined by fusion modules,
  • then increasingly elaborate fusion schemes,
  • then fine-tuning of pretrained vision–language models like CLIP, VisualBERT and ViLBERT.

Two lessons recur across the reviewed work. Labels derived from hashtags introduce leaked cues, and auxiliary signals such as sentiment and emotion improve detection. The survey closes with four research directions: multilingual data, perspectivist annotation, joint modelling with related tasks, and multimodal LLMs.

Key points

  • MMSD (Cai et al., 2019) is built from Twitter posts and labelled by hashtags. It has a 19,818 / 2,410 / 2,409 train/validation/test split, with 10,560 sarcastic and 14,075 non-sarcastic items.
  • MMSD2.0 (Qin et al., 2023) fixes two problems in MMSD. It strips giveaway hashtags and emojis from the positive examples and manually relabels mis-annotated negatives. Its training set has 9,572 sarcastic and 10,240 non-sarcastic samples.
  • Sangwan et al. (2020) released two Instagram datasets: a hashtag-labelled Silver-Standard set of 20,000 posts and a human-annotated Gold-Standard set of 1,600 posts. MORE (Desai et al., 2022) adds natural-language explanations of the sarcasm along with a non-sarcastic paraphrase.
  • Audio-visual-textual datasets include MUStARD, MUStARD++, SEEmoji-MUStARD and a Spanish multimodal set. On these benchmarks, reported F1 scores mostly fall between about 70 and 78.
  • Notable models:
    • IWAN scores incongruity word by word, using BERT, ResNet and OpenSmile features.
    • MuLOT learns cross-modal incongruity through optimal transport.
    • Tiwari et al. add reader gaze features.
    • Bhosale et al. combine ViFi-CLIP with Wav2vec 2.0.

Interesting ideas

  • I1. Hashtag-labelled data contains shortcut cues, and MMSD2.0 showed that deleting them yields a more honest benchmark, which applies to any dataset built from automatic labels.
  • I2. Several systems treat sarcasm as a conflict between the polarity of the words and the polarity of the context, for example praise paired with a negative tone or image (IWAN scores this word by word); this makes the detection decision traceable.
  • I3. MORE pairs each sarcastic post with an explanation and a literal rewrite, turning detection into a task of recovering the intended meaning.
  • I4. Multi-task setups that add sentiment and emotion as auxiliary objectives improved sarcasm detection, suggesting that neighbouring labels sharpen a subtle one.
  • I5. The perspectivism direction argues for keeping each annotator's judgement instead of collapsing disagreement into a single gold label, because sarcasm is subjective.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Before trusting silver data, scan it for tokens that leak the label and remove them, then add a negative control that checks a model still scores well once those cues are gone.
  • I2 (frames): The irony module in prag could flag a clash between an utterance's evaluative frame and the state of the world it describes, and report which word caused the conflict as its explanation.
  • I3 (back): The generator could render an ironic utterance's intended meaning as plain text from the meaning graph, and such pairs could serve as test cases for the compressor.
  • I4 (frames): Evaluating speaker attitude alongside speech-act classification would give the irony decision an explicit sentiment input rather than a hidden guess.
  • I5 (learn): Preserve disagreement among witness parsers or LLM teacher tiers as a distribution, and use it to mark items as uncertain rather than forcing a majority vote.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Lancaster UniversityLancaster · United Kingdom35 works · 1,615 citations🎓Michigan State UniversityEast Lansing · United States23 works · 1,077 citations🎓George Mason UniversityFairfax · United States24 works · 1,288 citations🎓University of SurreyGuildford · United Kingdom21 works · 677 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 works

Cite

Cite as Shafkat Farabi, Tharindu Ranasinghe, Diptesh Kanojia, Yu Kong and Marcos Zampieri. 2024. A Survey of Multimodal Sarcasm Detection. arXiv:2410.18882.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.24963/ijcai.2024/887 · arXiv 2410.18882

@misc{farabi2024survey,
  title = {{A Survey of Multimodal Sarcasm Detection}},
  author = {Shafkat Farabi, Tharindu Ranasinghe, Diptesh Kanojia, Yu Kong and Marcos Zampieri},
  year = {2024},
  doi = {10.24963/ijcai.2024/887},
  eprint = {2410.18882},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.24963/ijcai.2024/887}
}

References

55 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sajal Aggarwal, Ananya Pandey, and Dinesh Kumar Vishwakarma. Multimodal sarcasm recognition by fusing textual, visual and acoustic content via multi-headed attention for video dataset. In WCONF, 2023. find
  2. Khalid Alnajjar and Mika Hämäläinen. ¡Qué maravilla! multimodal sarcasm detection in Spanish: a dataset and a baseline. In NAACL, 2021. find
  3. Amal Alqahtani, Lubna Alhenaki, and Abeer Alsheddi. Text-based sarcasm detection on social networks: A systematic review. IJACSA, 14(3), 2023. ↗
  4. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. find
  5. Rohan Badlani, Nishit Asnani, and Manan Rai. An ensemble of humour, sarcasm, and hate speech for sentiment classification in online reviews. In W-NUT 2019, November 2019. find
  6. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. NeurIPS, 2020.
  7. Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, Divyank Tiwari, Anjan Dutta, Xiatian Zhu, Pushpak Bhattacharyya, and Diptesh Kanojia. Sarcasm in sight and sound: Benchmarking and expansion to improve multimodal sarcasm detection. arXiv preprint arXiv:2310.01430, 2023. ↗
  8. Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sarcasm detection in Twitter with hierarchical fusion model. In ACL, 2019. ↗
  9. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an obviously perfect paper). In ACL, 2019. — Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
  10. Pranali Chaudhari and Chaitali Chandankhede. Literature survey of sarcasm detection. In WiSPNET, 2017. find
  11. Dushyant Singh Chauhan, Dhanush S R, Asif Ekbal, and Pushpak Bhattacharyya. Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In ACL, 2020.
  12. Dushyant Singh Chauhan, Gopendra Vikram Singh, Aseem Arora, Asif Ekbal, and Pushpak Bhattacharyya. An emoji-aware multitask framework for multimodal sarcasm detection. Knowledge-Based Systems, 257:109924, 2022. ↗
  13. Dipto Das and Anthony J Clark. Sarcasm detection on facebook: A supervised learning approach. In ICMI, 2018. find
  14. Dipto Das and Anthony J. Clark. Sarcasm detection on flickr using a cnn. In ICCBD, 2018. find
  15. Poorav Desai, Tanmoy Chakraborty, and Md Shad Akhtar. Nice perfume. how long did you marinate in it? multimodal sarcasm explanation. In AAAI, 2022. ↗
  16. Daijun Ding, Hu Huang, Bowen Zhang, Cheng Peng, Yangyang Li, Xianghua Fu, and Liwen Jing. Multi-modal sarcasm detection with prompt-tuning. In ACAIT, 2022. find
  17. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. ↗
  18. Florian Eyben, Martin Wöllmer, and Björn Schuller. Opensmile: The munich versatile and fast open-source audio feature extractor. In ACM MM, 2010. find
  19. Simona Frenda. The role of sarcasm in hate speech. a multilingual perspective. In SEPLN, 2018. find
  20. Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. Hybrid attention based multimodal network for spoken language classification. In ACL, 2018. find
  21. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. CASCADE: Contextual sarcasm detection in online discussion forums. In COLING, 2018.
  22. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. How do cultural differences impact the quality of sarcasm annotation?: A case study of Indian annotators and American text. In SIGHUM, 2016.
  23. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):1–22, 2017. ↗
  24. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. A large self-annotated corpus for sarcasm. In LREC, 2018.
  25. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. ↗
  26. Bin Liang, Chenwei Lou, Xiang Li, Lin Gui, Min Yang, and Ruifeng Xu. Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In ACM MM, 2021. find
  27. Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. Multi-modal sarcasm detection via cross-modal graph convolutional network. In ACL, 2022.
  28. Hongzhan Lin, Ziyang Luo, bo Wang, Ruichao Yang, and Jing Ma. Goat-bench: Safety insights to large multimodal models through meme-based social abuse. arXiv preprint arXiv:2401.01523, 2024. ↗
  29. Hui Liu, Wenya Wang, and Haoliang Li. Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement. In EMNLP, 2022.
  30. Yiyi Liu, Yequan Wang, Aixin Sun, Xuying Meng, Jing Li, and Jiafeng Guo. A dual-channel framework for sarcasm recognition by detecting sentiment conflict. In NAACL, 2022.
  31. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for v-l tasks. In NeurIPS, 2019. find
  32. Diana Maynard and Mark Greenwood. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC, 2014.
  33. JA Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. Semeval 2021 task 7: Hahackathon, detecting and rating humor and offense. In SemEval, 2021.
  34. Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhattacharyya. Predicting readers’ sarcasm understandability by modeling gaze behavior. In AAAI, 2016. find
  35. Silviu Oprea and Walid Magdy. iSarcasm: A dataset of intended sarcasm. In ACL, 2020.
  36. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In EMNLP, 2020.
  37. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In ACL, 2019.
  38. Shraman Pramanick, Aniket Roy, and Vishal M Patel. Multimodal learning using optimal transport for sarcasm and humor detection. In WACV, 2022. find
  39. Libo Qin, Shijue Huang, Qiguang Chen, Chenran Cai, Yudi Zhang, Bin Liang, Wanxiang Che, and Ruifeng Xu. MMSD2.0: Towards a reliable multi-modal sarcasm detection system. In ACL, 2023.
  40. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. ↗
  41. Hanoona Rasheed, Muhammad Uzair khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Finetuned clip models are efficient video learners. In CVPR, 2023. find
  42. Anupama Ray, Shubham Mishra, Apoorva Nunna, and Pushpak Bhattacharyya. A multimodal corpus for emotion recognition in sarcasm. In LREC, 2022.
  43. Kathrin Rothermich, Ayotola Ogunlana, and Natalia Jaworska. Change in humor and sarcasm use based on anxiety and depression symptom severity during the covid-19 pandemic. Journal of psychiatric research, 140:95–100, 2021. ↗
  44. Yalamanchili Salini and J. HariKiran. Sarcasm detection: A systematic review of methods and approaches. In ICSMDI, 2023. ↗
  45. Suyash Sangwan, Md Shad Akhtar, Pranati Behera, and Asif Ekbal. I didn’t mean what i wrote! exploring multimodality for sarcasm detection. In IJCNN, 2020. find
  46. Rossano Schifanella, Paloma De Juan, Joel Tetreault, and Liangliang Cao. Detecting sarcasm in multimodal social platforms. In ACM MM, 2016.
  47. Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP, 2019. ↗
  48. Yuan Tian, Nan Xu, Ruike Zhang, and Wenji Mao. Dynamic routing transformer network for multimodal sarcasm detection. In ACL, 2023.
  49. Divyank Tiwari, Diptesh Kanojia, Anupama Ray, Apoorva Nunna, and Pushpak Bhattacharyya. Predict and use: Harnessing predicted gaze to improve multimodal sarcasm detection. In EMNLP, 2023.
  50. Palak Verma, Neha Shukla, and AP Shukla. Techniques of sarcasm detection: A review. In ICACITE, 2021. find
  51. Xinyu Wang, Xiaowen Sun, Tan Yang, and Hongbo Wang. Building a bridge: A method for image-text sarcasm detection without pretraining on image-text data. In NLPBT, 2020.
  52. Tharindu Weerasooriya, Sujan Dutta, Tharindu Ranasinghe, Marcos Zampieri, Christopher Homan, and Ashiqur KhudaBukhsh. Vicarious offense and noise audit of offensive speech classifiers: Unifying human and machine disagreement on what is offensive. In EMNLP, 2023. find
  53. Yang Wu, Yanyan Zhao, Xin Lu, Bing Qin, Yin Wu, Jian Sheng, and Jinlong Li. Modeling incongruity between modalities for multimodal sarcasm detection. IEEE MultiMedia, 28(2):86–95, 2021. ↗
  54. Nan Xu, Zhixiong Zeng, and Wenji Mao. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In ACL, 2020.
  55. Xiaocui Yang, Wenfang Wu, Shi Feng, Ming Wang, Daling Wang, Yang Li, Qi Sun, Yifei Zhang, Xiaoming Fu, and Soujanya Poria. Mm-bigbench: Evaluating multimodal models on multimodal content comprehension tasks. arXiv preprint arXiv:2310.09036, 2023. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/farabi-2024-survey-multimodal-sarcasm-detection/