Elamur Research⌕
☰
home / papers / 2019 / Towards Multimodal Sarcasm Detection (An \textitObviously Perfect Paper)

Towards Multimodal Sarcasm Detection (An \textitObviously Perfect Paper)

Paper · 2019 · Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas et al.

Summary

Sarcasm detection research has mostly worked on written text. Yet speakers often signal sarcasm through tone, stress, timing or a deadpan face, and these cues leave no trace in a transcript. The authors argue that sarcasm detection should be multimodal. To support this, they build MUStARD, a dataset of short audiovisual clips from TV series. Each clip is labelled as sarcastic or not and comes with the preceding dialogue turns and speaker identity. Candidate clips came from YouTube searches, a subset of MELD, and episodes of The Big Bang Theory segmented at audience laughter. Annotators labelled them, and a third person reconciled their disagreements. The final balanced set has 690 utterances. Baseline classifiers over text, audio and video features, combined by early fusion, show that mixing modalities beats any single one. The relative error reduction in F-score reaches 12.9%. The paper matters for two reasons. It released the first video-level sarcasm resource, and it frames sarcasm in conversational context as its own sub-task.

Key points

  • MUStARD draws on Friends (partly via 400 MELD clips), The Golden Girls, Sarcasmaholics Anonymous and The Big Bang Theory (seasons 1–8). The initial pool had 6,421 videos, of which 6,365 were annotated.
  • Agreement on the first Big Bang Theory batch (5,884 utterances) was very low, with Kappa 0.1463, and 98% of that batch was judged non-sarcastic. After the annotators discussed 20 disputed videos and re-labelled, Kappa rose to 0.2326, and a third annotator resolved the remaining conflicts.
  • The released set has 690 utterances, split evenly between sarcastic and non-sarcastic, each with its context turns and speaker label.
  • Models are evaluated in two setups:
    • speaker-dependent, where speakers overlap across splits;
    • speaker-independent, where no speaker appears in both training and test data.
  • Multimodal early fusion beat unimodal models by up to 12.9% relative error reduction. Averaged context features helped very little. A one-hot speaker vector helped the text model only in the speaker-dependent setup. SVMs outperformed CNNs on this small dataset.

Interesting ideas

  • I1. Testing both with and without speaker overlap shows how much a model leans on who is talking rather than on what is said, which protects against inflated scores.
  • I2. When speaker identity added nothing to the best multimodal model, the authors checked whether per-speaker prediction patterns matched the gold distribution, and concluded the model was already inferring speaker identity from its input.
  • I3. Plain averaging of context turns barely helped, and the authors blame the lost order of the conversation, which argues for structured context instead of pooling.
  • I4. Audience laughter served as a cheap automatic signal for segmenting episodes and finding likely humorous or sarcastic moments, with subtitle timestamps used to tighten the boundaries.
  • I5. The authors locate sarcasm in incongruity between channels, for instance positive wording delivered in a flat or negative manner.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluate the irony and speech-act components on held-out authors or characters as a negative control, so that memorised speaker habits are not counted as understanding.
  • I2 (learn): Run a similar check to detect hidden shortcuts, for example whether a frame classifier secretly keys on a character name.
  • I3 (frames): Model irony as a mismatch between an utterance and the ordered prior state of the dialogue, not as a bag of previous turns.
  • I4 (learn): Look for analogous weak signals in text, such as narrator reactions or "she said dryly", to mine silver candidates for irony and implicature.
  • I5 (frames): Detect contradictions between the literal evaluation in a frame and the situation in the text world, such as praise for an event that the world model marks as bad, and explain that clash.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations🎓Singapore University of Technology and DesignSingapore · Singapore33 works · 10,760 citations🎓University of MichiganAnn Arbor · United States36 works · 12,181 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 608 worksVision & language · 269 works

Cite

Cite as Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea and Soujanya Poria. 2019. Towards Multimodal Sarcasm Detection (An \textitObviously Perfect Paper). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1455

@inproceedings{castro2019multimodal,
  title = {{Towards Multimodal Sarcasm Detection (An textitObviously Perfect Paper)}},
  author = {Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea and Soujanya Poria},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1455},
  url = {https://doi.org/10.18653/v1/p19-1455}
}

References

43 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of twitter conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113.
  2. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. Proceedings of the 27th International Conference on Computational Linguistics, page 1837–1848.
  3. Salvatore Attardo, Jodi Eisterhold, Jennifer Hay, and Isabella Poggi. 2003. Multimodal markers of irony and sarcasm. Humor, 16(2):243–260. ↗
  4. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. ↗
  5. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. ICWSM, 2:15. find
  6. Harold Hotelling. 1936. Relations between two sets of variates. Biometrika, 28(3/4):321–377. ↗
  7. Gregory A Bryant. 2010. Prosodic contrasts in ironic speech. Discourse Processes, 47(7):545–566. ↗
  8. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016a. How do cultural differences impact the quality of sarcasm annotation?: A case study of indian annotators and american text. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 95– 99.
  9. Evgeny Byvatov, Uli Fechner, Jens Sadowski, and Gisbert Schneider. 2003. Comparison of support vector machine and artificial neural network systems for drug/nondrug classification. Journal of chemical information and computer sciences, 43(6):1882–1889. ↗
  10. Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:1812.08008. ↗
  11. Paula Carvalho, Luís Sarmento, Mário J Silva, and Eu-génio De Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s so easy;-. In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. ACM. find
  12. Henry S Cheang and Marc D Pell. 2008. The sound of sarcasm. Speech communication, 50(5):366–381. ↗
  13. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107–116. Association for Computational Linguistics.
  14. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE. find
  15. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 757–762.
  16. Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark J Carman. 2016b. Harnessing sequence labeling for sarcasm detection in dialogue from tv seriesfriends’. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 146–155. find
  17. Y Alex Kolchinski and Christopher Potts. 2018. Representing social media users for sarcasm detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1115–1121. ↗
  18. CC Liebrecht, FA Kunneman, and APJ van Den Bosch. 2013. The perfect solution for detecting sarcasm in tweets# not. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 29–37. New Brunswick, NJ: ACL.
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  20. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2018. Dialoguernn: An attentive rnn for emotion detection in conversations. arXiv preprint arXiv:1811.00405. ↗
  21. Ruth Filik, Hartmut Leuthold, Katie Wallington, and Jemma Page. 2014. Testing theories of irony processing using eye-tracking and erps. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(3):811. Brian McFee, Matt McVicar, Stefan Balke, Carl Thomé, Vincent Lostanlen, Colin Raffel, Dana Lee, Oriol Nieto, Eric Battenberg, Dan Ellis, Ryuichi Yamamoto, Josh Moore, WZY, Rachel Bittner, Keunwoo Choi, Pius Friesch, Fabian-Robert Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. ↗
  22. Stöter, Matt Vollrath, Siddhartha Kumar, nehz, Simon Waloschek, Seth, Rimvydas Naktinis, Douglas Repetto, Curtis "Fjord" Hawthorne, CJ Carr, João Felipe Santos, JackieWu, Erik, and Adrian Holovaty. 2018. librosa/librosa: 0.6.2. find
  23. Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 377–387.
  24. Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhattacharyya. 2016a. Predicting readers’ sarcasm understandability by modeling gaze behavior. In AAAI, pages 3747–3753. find
  25. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2016b. Harnessing cognitive features for sarcasm detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1095–1104. ↗
  26. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  27. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  28. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1601–1612. ↗
  29. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508. ↗
  30. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. arXiv preprint arXiv:1905.02947. ↗
  31. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. find
  32. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. find
  33. Patricia Rockwell. 2000. Lower, slower, louder: Vocal cues of sarcasm. Journal of Psycholinguistic Research, 29(5):483–495. ↗
  34. Kimiko Ryokai, Elena Durán López, Noura Howell, Jon Gillick, and David Bamman. 2018. Capturing, representing, and interacting with laughter. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, page 358. ACM. find
  35. R Schifanella, P de Juan, J Tetreault, L Cao, et al. 2016. Detecting sarcasm in multimodal social platforms. In ACM Multimedia, pages 1136–1145. ACM.
  36. Joseph Tepperman, David Traum, and Shrikanth Narayanan. 2006. " yeah right": Sarcasm recognition for spoken dialogue systems. In Ninth International Conference on Spoken Language Processing. find
  37. Dominic Thompson, Ian G Mackenzie, Hartmut Leuthold, and Ruth Filik. 2016. Emotional responses to irony and emoticons in written language: evidence from eda and facial emg. Psychophysiology, 53(7):1054–1062. ↗
  38. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765–770. ↗
  39. Byron C Wallace, Eugene Charniak, et al. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1035–1044.
  40. Byron C Wallace, Laura Kertz, Eugene Charniak, et al. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 512–516.
  41. Silvio Amir Byron C Wallace, Hao Lyu, and Paula Carvalho Mário J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. CoNLL 2016, page 167.
  42. Jennifer Woodland and Daniel Voyer. 2011. Context and intonation in the perception of sarcasm. Metaphor and Symbol, 26(3):227–239. ↗
  43. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/castro-2019-multimodal-sarcasm-detection-textitobviously-perfect/