Elamur Research
☰
home / papers / 2019 / Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1906.01815

Summary

Most automatic sarcasm detection relies on written text alone. In speech, however, irony is often signalled by tone, stress, elongated syllables or a deadpan face. The authors address this gap by building MUStARD, a publicly released corpus of short TV-show clips labelled as sarcastic or not. They gathered candidate videos from YouTube searches (Friends, The Golden Girls, Sarcasmaholics Anonymous), episodes of The Big Bang Theory segmented using laughter detection, and non-sarcastic clips from MELD. Two annotators labelled the clips, and a balanced subset was kept. Each target utterance comes with its transcript, the preceding dialogue turns and the speaker's identity. Baseline classifiers combine text, audio and visual features through early fusion, with optional context and speaker inputs. They are tested in speaker-dependent and speaker-independent settings. Combining modalities consistently beat single-modality models. This shows that non-verbal signals carry useful sarcasm cues and opens a new task: detecting sarcasm within conversational context.

Key points

  • MUStARD contains 690 utterance videos, split evenly between sarcastic and non-sarcastic. They were drawn from an initial pool of 6,421 clips, of which 6,365 were annotated. About 61.3% of utterances are a single sentence.
  • Annotator agreement started very low (Kappa 0.1463). The annotators then reconciled their disagreements and re-labelled the videos.
  • The best multimodal variants cut the relative error rate in F-score by up to 12.9% compared with unimodal models.
  • With this small dataset, SVM classifiers outperformed neural models such as CNNs, which tended to overfit. Pooled conversational context gave little gain.
  • Speaker identity helped the text-only model in the speaker-dependent split. The text + video model appeared to pick up speaker tendencies implicitly. The authors suggest future work on better fusion methods (e.g., Tensor-Fusion), multi-party modelling and localising the active speaker.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

86 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 81 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1906.01815 · arXiv 1906.01815

@misc{castro2019multimodal,
  title = {{Towards Multimodal Sarcasm Detection (An \_Obviously\_ Perfect Paper)}},
  author = {Castro, Santiago and Hazarika, Devamanyu and Pérez-Rosas, Verónica and Zimmermann, Roger and Mihalcea, Rada and Poria, Soujanya},
  year = {2019},
  doi = {10.48550/arxiv.1906.01815},
  eprint = {1906.01815},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.01815}
}

References

43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113. find
  2. Salvatore Attardo, Jodi Eisterhold, Jennifer Hay, and Isabella Poggi. 2003. Multimodal markers of irony and sarcasm. Humor, 16(2):243–260. find
  3. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. ICWSM, 2:15. find
  4. Gregory A Bryant. 2010. Prosodic contrasts in ironic speech. Discourse Processes, 47(7):545–566. find
  5. Evgeny Byvatov, Uli Fechner, Jens Sadowski, and Gisbert Schneider. 2003. Comparison of support vector machine and artificial neural network systems for drug/nondrug classification. Journal of chemical information and computer sciences, 43(6):1882–1889. find
  6. Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:1812.08008. ↗
  7. Paula Carvalho, Luís Sarmento, Mário J Silva, and Eugénio De Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s so easy;-. In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. ACM. find
  8. Henry S Cheang and Marc D Pell. 2008. The sound of sarcasm. Speech communication, 50(5):366–381. find
  9. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107–116. Association for Computational Linguistics. find our card
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  12. Ruth Filik, Hartmut Leuthold, Katie Wallington, and Jemma Page. 2014. Testing theories of irony processing using eye-tracking and erps. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(3):811. find
  13. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. Proceedings of the 27th International Conference on Computational Linguistics, page 1837–1848. find our card
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. find
  15. Harold Hotelling. 1936. Relations between two sets of variates. Biometrika, 28(3/4):321–377. find
  16. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016a. How do cultural differences impact the quality of sarcasm annotation?: A case study of indian annotators and american text. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 95– 99. find
  17. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 757–762. find our card
  18. Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark J Carman. 2016b. Harnessing sequence labeling for sarcasm detection in dialogue from tv seriesfriends’. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 146–155. find
  19. Y Alex Kolchinski and Christopher Potts. 2018. Representing social media users for sarcasm detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1115–1121. find
  20. CC Liebrecht, FA Kunneman, and APJ van Den Bosch. 2013. The perfect solution for detecting sarcasm in tweets# not. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 29–37. New Brunswick, NJ: ACL. find our card
  21. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2018. Dialoguernn: An attentive rnn for emotion detection in conversations. arXiv preprint arXiv:1811.00405. Brian McFee, Matt McVicar, Stefan Balke, Carl Thomé, Vincent Lostanlen, Colin Raffel, Dana Lee, Oriol Nieto, Eric Battenberg, Dan Ellis, Ryuichi Yamamoto, Josh Moore, WZY, Rachel Bittner, Keunwoo Choi, Pius Friesch, Fabian-Robert Stöter, Matt Vollrath, Siddhartha Kumar, nehz, Simon Waloschek, Seth, Rimvydas Naktinis, Douglas Repetto, Curtis "Fjord" Hawthorne, CJ Carr, João Felipe Santos, JackieWu, Erik, and Adrian Holovaty. 2018. librosa/librosa: 0.6.2. ↗
  22. Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 377–387. find
  23. Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhattacharyya. 2016a. Predicting readers’ sarcasm understandability by modeling gaze behavior. In AAAI, pages 3747–3753. find
  24. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2016b. Harnessing cognitive features for sarcasm detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1095–1104. find
  25. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  26. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  27. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1601–1612. find our card
  28. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508. ↗ our card
  29. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. arXiv preprint arXiv:1905.02947. ↗
  30. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. find
  31. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. find
  32. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. find our card
  33. Patricia Rockwell. 2000. Lower, slower, louder: Vocal cues of sarcasm. Journal of Psycholinguistic Research, 29(5):483–495. find
  34. Kimiko Ryokai, Elena Durán López, Noura Howell, Jon Gillick, and David Bamman. 2018. Capturing, representing, and interacting with laughter. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, page 358. ACM. find
  35. R Schifanella, P de Juan, J Tetreault, L Cao, et al. 2016. Detecting sarcasm in multimodal social platforms. In ACM Multimedia, pages 1136–1145. ACM. find our card
  36. Joseph Tepperman, David Traum, and Shrikanth Narayanan. 2006. " yeah right": Sarcasm recognition for spoken dialogue systems. In Ninth International Conference on Spoken Language Processing. find
  37. Dominic Thompson, Ian G Mackenzie, Hartmut Leuthold, and Ruth Filik. 2016. Emotional responses to irony and emoticons in written language: evidence from eda and facial emg. Psychophysiology, 53(7):1054–1062. find
  38. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765–770. find
  39. Byron C Wallace, Eugene Charniak, et al. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1035–1044. find
  40. Byron C Wallace, Laura Kertz, Eugene Charniak, et al. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 512–516. find
  41. Silvio Amir Byron C Wallace, Hao Lyu, and Paula Carvalho Mário J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. CoNLL 2016, page 167. find our card
  42. Jennifer Woodland and Daniel Voyer. 2011. Context and intonation in the perception of sarcasm. Metaphor and Symbol, 26(3):227–239. find
  43. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114. find