Elamur Research⌕
☰
home / papers / 2022 / When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues

When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues

Summary

The paper argues that detecting sarcasm is insufficient for dialogue systems, which also need to understand what a sarcastic remark actually means. It introduces Sarcasm Explanation in Dialogue (SED), a task in which a model receives a multi-party conversation ending in a sarcastic utterance and must generate a natural-language explanation of the irony. To support SED, the authors built WITS by extending the Hindi-English code-mixed MASAC corpus from the sitcom "Sarabhai v/s Sarabhai". They added episodes, defined dialogue contexts manually, and wrote explanations for each sarcastic dialogue. For modelling, they propose MAF (Modality Aware Fusion), a set of adapters for BART and mBART. MAF uses Multimodal Context-Aware Attention (MCA2) to condition attention on audio and video features, and a gated Global Information Fusion (GIF) to filter out multimodal noise. The trimodal variant MAF-TAV B outperforms text-only and conventional fusion baselines on most generation metrics and in human ratings, and it identifies the speaker of the sarcasm much more reliably. The work moves sarcasm research from classification toward interpretation in multimodal, code-mixed conversation.

Key points

  • WITS ("Why Is This Sarcastic") contains 2240 sarcastic dialogues and 9080 utterances, drawn from 55 episodes: the original 45 in MASAC plus 10 new ones. Dialogue contexts range from 2 to 27 utterances, and each dialogue includes up to 6 speakers.
  • Most utterances are code-mixed: 7526 of them, compared with 1453 in Hindi and 101 in English. Maya is the most frequent source of sarcasm, and Monisha is the most frequent target.
  • MAF-TAV B identifies the source of the sarcasm with 91.07% accuracy, compared with 77.23% for BART. However, accuracy on identifying the target drops slightly, from 52.67% to 46.42%.
  • In the human evaluation, 25 expert evaluators rated 30 test samples on a 0–5 scale. MAF-TAV B scored higher than BART on all three criteria:
    • coherence: 3.03 vs 2.73
    • relatedness to the dialogue: 3.11 vs 2.56
    • relatedness to the sarcasm: 2.77 vs 2.18
  • Fusion worked best when inserted before the sixth layer. The code and data are available at github.com/LCS2-IIITD/MAF.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

21 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 20 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.411 · arXiv 2203.06419

@inproceedings{kumar2022when,
  title = {{When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues}},
  author = {Kumar, Shivani and Kulkarni, Atharva and Akhtar, Md Shad and Chakraborty, Tanmoy},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.411},
  eprint = {2203.06419},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.411}
}

References

49 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The Ar-Sarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European La… find
  2. Salvatore Attardo, Jodi Eisterhold, Jennifery Hay, and Isabella Poggi. 2003. Multimodal markers of irony and sarcasm. Humor: International Journal of Humor Research, 16(2). find
  3. Manjot Bedi, Shivani Kumar, Md Shad Akhtar, and Tanmoy Chakraborty. 2021. Multi-modal sarcasm detection and humor classification in code-mixed conversations. IEEE Transactions on Affective Computing, pages 1–1. find
  4. Santosh Kumar Bharti, Korra Sathya Babu, and Sanjay Kumar Jena. 2017. Harnessing online news for sarcasm detection in hindi tweets. In Pattern Recognition and Machine Intelligence, pages 679– 686, Cham. Springer International Publishing. find
  5. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy. Association for Computational Linguistics. find our card
  6. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _Obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619– 4629, Flo… find our card
  7. Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020. Rˆ3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7976–7986, Online. Association for Computa… find our card
  8. The authors would like to acknowledge the support of the Ramanujan Fellowship (SERB, India), In-fosys Centre for AI (CAI) at IIIT-Delhi, and ihub-Anubhuti-iiitd Foundation set up under the NM-ICPS scheme of the Department of Science and Technology, India. find
  9. Alessandra Teresa Cignarella, Simona Frenda, Valerio Basile, Cristina Bosco, Viviana Patti, Paolo Rosso, et al. 2018. Overview of the evalita 2018 task on irony detection in italian tweets (ironita). In Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 20… find our card
  10. Herbert L Colston and Shauna B Keller. 1998. You’ll never believe this: Irony and hyperbole in expressing surprise. Journal of psycholinguistic research, 27(4):499–513. find
  11. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Pa… find our card
  12. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland, USA. Association for Computational Linguistics. find our card
  13. Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya-narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The kinetics human action video dataset. find
  14. Abhijeet Dubey, Aditya Joshi, and Pushpak Bhattacharyya. 2019. Deep models for converting sarcastic utterances into their non sarcastic interpretation. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, CoDS-COMAD ’19, page 289–292, New York, NY, U… find
  15. Roger Kreuz and Gina Caucci. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on Computational Approaches to Figurative Language, pages 1–4, Rochester, New York. Association for Computational Linguistics. find our card
  16. Herbert L. Colston. 1997. Salting a wound or sugaring a pill: The pragmatic functions of ironic criticism. Discourse Processes, 23(1):25–45. find
  17. Florian Eyben, Klaus R. Scherer, Björn W. Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y. Devillers, Julien Epps, Petri Laukka, Shrikanth S. Narayanan, and Khiet P. Truong. 2016. The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing.… find
  18. Debanjan Ghosh, Alexander R. Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755– 792. find our card
  19. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The role of conversation context for sarcasm detection in online interactions. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 186–196, Saarbrücken, Germany. Association for Computational Linguis… find our card
  20. Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. 2018. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  21. Md Kamrul Hasan, Sangwu Lee, Wasifur Rahman, Amir Zadeh, Rada Mihalcea, Louis-Philippe Morency, and Ehsan Hoque. 2021. Humor knowledge enriched transformer for understanding multimodal humor. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12972–12980. find
  22. Stacey L. Ivanko and Penny M. Pexman. 2003. Context incongruity and irony processing. Discourse Processes, 35(3):241–279. find
  23. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5). find our card
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  26. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  27. Abhijit Mishra, Tarun Tater, and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN… find our card
  28. Henri Olkoniemi, Henri Ranta, and Johanna K Kaaki-nen. 2016. Individual differences in the processing of written sarcasm and metaphor: Evidence from eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(3):433. find
  29. Shereen Oraby, Vrindavan Harrison, Amita Misra, Ellen Riloff, and Marilyn Walker. 2017. Are you serious?: Rhetorical questions and sarcasm in social media dialog. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 310–319, Saarbrücken, Germany. Association for Computa… find our card
  30. Reynier Ortega-Bueno, Francisco Rangel, D Hernández Farıas, Paolo Rosso, Manuel Montes-y Gómez, and José E Medina Pagola. 2019. Overview of the task on irony detection in spanish variants. In Proceedings of the Iberian languages evaluation forum (IberLEF 2019), co-located with 34th conference of the… find
  31. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1383–1392, Online. Association for Computational Linguistics. find our card
  32. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  33. Lotem Peled and Roi Reichart. 2017. Sarcasm SIGN: Interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1690–1700, Vancouver, Canada. Association for Computa… find our card
  34. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  35. Richard M. Roberts and Roger J. Kreuz. 1994. Why do people use figurative language? Psychological Science, 5(3):159–163. find
  36. Patricia Rockwell. 2007. Vocal features of conversational sarcasm: A comparison of methods. Journal of psycholinguistic research, 36(5):361–369. find
  37. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… find
  38. Himani Srivastava, Vaibhav Varshney, Surabhi Kumari, and Saurabh Srivastava. 2020. A novel hierarchical BERT architecture for sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 93–97, Online. Association for Computational Linguistics. find our card
  39. Sabina Tabacaru and Maarten Lemmens. 2014. Raised eyebrows as gestural triggers in humour: The case of sarcasm and hyper-understanding. The European Journal of Humour Research, 2(2):11–31. find
  40. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1010–1020, Melbourne, Australia. Association for Computational Linguistics. find our card
  41. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm — a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. Proceedings of the International AAAI Conference on Web and Social Media, 4(1):162– 169. find
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  43. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  44. Henry M Wellman. 2014. Making minds: How theory of mind develops. Oxford University Press. find
  45. Tao Xiong, Peiran Zhang, Hongbo Zhu, and Yihui Yang. 2019. Sarcasm detection with self-matching networks and low-rank bilinear pooling. In The World Wide Web Conference, WWW ’19, page 2115–2124, New York, NY, USA. Association for Computing Machinery. find
  46. Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3777–3786, Online. Association for Computational Lin… find our card
  47. Baosong Yang, Jian Li, Derek F. Wong, Lidia S. Chao, Xing Wang, and Zhaopeng Tu. 2019. Context-aware self-attention networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):387– 394. find
  48. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  49. Sahil Swami, Ankush Khandelwal, Vinay Singh, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. A corpus of english-hindi code-mixed tweets for sarcasm detection. arXiv preprint arXiv:1805.11869. — [arXiv:1805.11869](https://arxiv.org/abs/1805.11869) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/kumar-2022-when-did-you-become-so/