Elamur Research⌕
☰
home / papers / 2023 / Multi-modal Sarcasm Generation: Dataset and Solution

Multi-modal Sarcasm Generation: Dataset and Solution

Paper · 2023 · Wenye Zhao, Qingbao Huang, Dongsheng Xu et al.

Summary

Earlier work on sarcasm generation dealt only with text. On social media, however, sarcasm often comes from pairing a picture with a short post. Zhao et al. define Multi-modal Sarcasm Generation (MSG): given an image and the hashtags that name the target of the mockery, the system must write a human-like sarcastic description.

To support the task, they built MuSG, a set of 5000 image–text pairs. The data comes from the Twitter API and two earlier sarcasm-detection collections (Schifanella et al.; Cai et al.). It was cleaned of links, mentions, odd symbols, overly long texts and unsuitable images.

Their baseline, MTMSG, works in three steps:

  • It projects hashtags, detected visual objects and OCR tokens into one shared 768-dimensional space.
  • It lets multi-modal Transformer layers mix these inputs.
  • It decodes word by word with a dynamic pointer network, which can copy OCR words as well as pick from the vocabulary.

MTMSG outperforms every compared model on all automatic metrics. Human judges also rated its outputs for creativity, sarcasticness, coherence and image–text relation. The work matters as the first benchmark for producing, rather than merely detecting, sarcasm across modalities.

Key points

  • MuSG contains 5000 samples split 3536/723/741 (train/valid/test, about 5:1:1). Texts longer than 40 words were excluded, as were images that are text-only, have number-only OCR, contain too many objects or have low resolution.
  • The source of the sarcasm varies: 40.3% lies only in the image, 32.4% only in the hashtags and 27.3% in both. About a quarter of targets (25.8%) are small objects.
  • Life (28.4%) and politics (26.4%) are the largest content categories. Most reference sentences are declarative (56.3%) or imperative (40.9%), and only 2.8% are interrogative or other.
  • Ablations show clear drops without OCR tokens, without the pointer network and without iterative decoding. When each modality is tested alone, textual input contributes most.
  • A manual check of 100 failures found two causes: in 63%, the model missed intent carried by the OCR text; in 37%, the key sarcastic cue was in the image itself.

Interesting ideas

  • I1. Labelling each sample by where its sarcasm originates (picture, hashtag or both) makes the dataset diagnosable rather than a single opaque score.
  • I2. The input separates the target of sarcasm (the hashtag) from the cue that reveals the evaluative intent (OCR text), splitting "what is mocked" from "how the meaning flips".
  • I3. A copy pointer that complements the fixed vocabulary lets the generator reuse literal strings seen in the input.
  • I4. Single-modality and component-removal ablations show how much each information source actually contributes.
  • I5. Sorting a random sample of 100 errors by where the missing information lived gives a cheap, actionable failure map.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony module in prag could record whether the incongruity comes from the utterance or its context, so every ironic reading carries an explainable trigger.
  • I2 (frames): an irony frame could hold distinct slots for target and polarity cue, filled from different parts of the UD tree and discourse.
  • I3 (back): when generating from the meaning graph, names, numbers and rare words could be copied directly from world entities instead of being regenerated from a lexicon.
  • I4 (learn): run readers with one input stream or module disabled as negative controls, confirming that answers depend on the intended evidence.
  • I5 (learn): periodically sample failures and bin them by pipeline stage (tokens, tree, frames, world) to decide where repair rules or teacher effort should go.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Guangxi UniversityNanning · China2 works · 6 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 works

Cite

Cite as Wenye Zhao, Qingbao Huang, Dongsheng Xu and Peizhi Zhao. 2023. Multi-modal Sarcasm Generation: Dataset and Solution. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.346

@inproceedings{zhao2023multi,
  title = {{Multi-modal Sarcasm Generation: Dataset and Solution}},
  author = {Wenye Zhao, Qingbao Huang, Dongsheng Xu and Peizhi Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.346},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.346}
}

References

37 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jon Almazán, Albert Gordo, Alicia Fornés, and Ernest Valveny. 2014. Word spotting and recognition with embedded attributes. IEEE Trans. Pattern Anal. Mach. Intell., 36(12):2552–2566. ↗
  2. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. In Computer Vision - ECCV 2016 - 14th European Conference, volume 9909 of Lecture Notes in Computer Science, pages 382–398. Springer. ↗
  3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomás Mikolov. 2017. Enriching word vectors with subword information. Trans. Assoc. Comput. Linguistics, 5:135–146. ↗
  4. Christian Burgers, Margot Van Mulken, and Peter Jan Schellens. 2012. Verbal irony: Differences in usage across written genres. Journal of Language and Social Psychology, 31(3):290–310. find
  5. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 2506–2515.
  6. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 4619– 4629.
  7. Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020. Rˆ3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 7976–7986.
  8. Itsugun Cho, Dongyang Wang, Ryota Takahashi, and Hiroaki Saito. 2022. A personalized dialogue generator with implicit user persona detection. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 367–377. International Committee on Computational Linguistics. ↗
  9. Michael J. Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, pages 376–380. The Association for Computer Linguistics. Poorav Desai, Tanmoy Chakraborty, and Md.
  10. Shad Akhtar. 2022. Nice perfume. how long did you marinate in it? multimodal sarcasm explanation. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, pages 10563–10571. AAAI Press. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171–4186.
  12. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. 9th International Conference on Learning Representations, ICLR 2021. ↗
  13. Aditya Joshi, Anoop Kunchukuttan, Pushpak Bhattacharyya, and Mark James Carman. 2015. Sarcasm-bot: An open-source sarcasm-generation module for chatbots. In WISDOM Workshop at KDD. Shivani Kumar, Atharva Kulkarni, Md. find
  14. Shad Akhtar, and Tanmoy Chakraborty. 2022. When did you become so smart, oh wise one?! sarcasm explanation in multi-modal multi-party dialogues. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, pages 5956–5968. Association for Computational Linguistics. ↗
  15. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, volume 162 of Proceedings of Machine Learning Research, pages 12888– 12900. PMLR. ↗
  16. Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2021, pages 339–349. find
  17. Silviu Vlad Oprea, Steven R. Wilson, and Walid Magdy. 2022. Should a chatbot be sarcastic? understanding user preferences towards sarcasm generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, pages 7686–7700. Association for Computational Linguistics.
  18. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1383–1392.
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, ACL 2002, pages 311–318.
  20. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, pages 2249–2255. ↗
  21. Bin Liang, Chenwei Lou, Xiang Li, Lin Gui, Min Yang, and Ruifeng Xu. 2021. Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In ACM MM ’21: ACM Multimedia Conference, pages 4707–4715. find
  22. Lotem Peled and Roi Reichart. 2017. Sarcasm SIGN: interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, pages 1690–1700.
  23. Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. 2022. Multimodal sarcasm detection via cross-modal graph convolutional network. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pages 1767–1777.
  24. Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, NeurIps 2015, pages 91–99. ↗
  25. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization of ACL 2004, pages 74–81.
  26. Rossano Schifanella, Paloma De Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 2016 ACM Conference on Multimedia Conference, ACM MM 2016, pages 1136–1145.
  27. Nayu Liu, Xian Sun, Hongfeng Yu, Wenkai Zhang, and Guangluan Xu. 2020. Multistage fusion with forget gate for multimodal summarization in open-domain videos. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 1834–1845. find
  28. Abhijit Mishra, Tarun Tater, and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pages 6143–6153.
  29. Silviu Oprea, Steven R. Wilson, and Walid Magdy. 2021. Chandler: An explainable sarcastic response generator. In Proceedings of the 2021 Conference on Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Textcaps: A dataset for image captioning with reading comprehension. In Computer Vision - ECCV 2020 - 16th European Conference, volume 12347, pages 742–758.
  30. Jiajia Tang, Kang Li, Ming Hou, Xuanyu Jin, Wanzeng Kong, Yu Ding, and Qibin Zhao. 2022. MMT: multi-way multi-modal transformer for multimodal learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, pages 3458–3465. ijcai.org. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NeurIps 2017, pages 5998–6008.
  32. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, pages 4566–4575. IEEE Computer Society. ↗
  33. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Advances in neural information processing systems, 28. find
  34. Jiquan Wang, Lin Sun, Yi Liu, Meizhi Shao, and Zeng-wei Zheng. 2022. Multimodal sarcasm target identification in tweets. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, pages 8164–8175. Association for Computational Linguistics.
  35. Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 3777–3786.
  36. Ze Yang, Can Xu, Wei Wu, and Zhoujun Li. 2019. Read, attend and comment: A deep architecture for automatic news comment generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5076–5088. Association for Computational Linguistics. ↗
  37. ACL 2023 Responsible NLP Checklist find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2023-multi-modal-sarcasm-generation-dataset/