Elamur Research⌕
☰
home / papers / 2022 / A Multimodal Corpus for Emotion Recognition in Sarcasm

A Multimodal Corpus for Emotion Recognition in Sarcasm

Summary

Sarcasm research has mostly focused on detecting whether an utterance is sarcastic. The emotion behind it, such as frustration, disgust, or ridicule, has received little attention, even though conversational systems need it to respond well. The authors address emotion recognition in sarcastic speech, which they describe as a new task. They start from the emotion-annotated version of MUStARD, a video corpus of sitcom clips. Models trained on it kept disagreeing with the gold labels, so the team re-annotated the data and corrected roughly half of the emotion labels. They then doubled the corpus with new sitcom clips, keeping a 50/50 sarcastic/non-sarcastic balance. Every utterance is labeled with implicit and explicit emotion, valence, arousal, and one of four sarcasm types. The resulting resource, MUStARD++, is benchmarked with text, audio, and video fusion models. These set baselines for emotion classification and valence/arousal regression, and the authors also report gains over the previous state of the art in sarcasm detection. Both data and code are publicly released.

Key points

  • Label corrections: The original MUStARD has 690 clips, 345 of them sarcastic. Model-based error analysis flagged 399 disagreements with the labels from Chauhan et al. (2020). Blind re-annotation then confirmed 343 wrong emotion labels: 255 in sarcastic clips and 88 in non-sarcastic ones. Many of the errors were sarcastic utterances marked Neutral.
  • New annotation layers: MUStARD++ adds valence and arousal scores and sarcasm-type tags following Camp (2012): Propositional, Embedded, Like-prefixed, and Illocutionary. Implicit and explicit emotions are labeled separately.
  • Annotation process: Seven annotators of varied age, gender, and education labeled perceived emotion, with final labels chosen by majority vote.
  • Benchmark results: Multiclass emotion F1 remains modest. Implicit emotion reaches about 33, while explicit emotion reaches about 41 (video+text with context). This shows the task is hard.
  • Availability: Data and code are released at github.com/apoorva-nunna/MUStARD_Plus_Plus. The paper text is CC-BY-NC-4.0, and the summary does not cover the dataset's own license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

18 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 17 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2206.02119

@inproceedings{ray2022multimodal,
  title = {{A Multimodal Corpus for Emotion Recognition in Sarcasm}},
  author = {Ray, Anupama and Mishra, Shubham and Nunna, Apoorva and Bhattacharyya, Pushpak},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/3d3nbg2do7ny},
  eprint = {2206.02119},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/3d3nbg2do7ny}
}

References

35 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bedi, M., Kumar, S., Akhtar, M. S., and Chakraborty, T. (2021). Multi-modal sarcasm detection and humor classification in code-mixed conversations. CoRR, abs/2105.09984. ↗
  2. Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. S. (2008). Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335. ↗
  3. Cai, Y., Cai, H., and Wan, X. (2019). Multi-modal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy, July. Association for Computational Linguistics. ↗ our card
  4. Camp, E. (2012). Sarcasm, pretense, and the semantics/pragmatics distinction. Noûs, 46(4):587–634. ↗
  5. Chen, S., Hsu, C., Kuo, C., Huang, T. K., and Ku, L. (2018). Emotionlines: An emotion corpus of multi-party conversations. CoRR, abs/1802.08379. ↗
  6. Cowie, R. and Cornelius, R. R. (2003). Describing the emotional states that are expressed in speech. Speech communication, 40(1-2):5–32. ↗
  7. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  8. Ekman, P. (1999). Basic emotions. Handbook of cognition and emotion, 98(45-60):16. find
  9. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). ↗
  10. Jadon, S. and Jasim, M. (2019). Video summarization using keyframe extraction and video skimming. arXiv preprint arXiv:1910.04792. ↗
  11. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2016). Automatic sarcasm detection: A survey. CoRR, abs/1602.03426. ↗ our card
  12. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2017). Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5), September. ↗ our card
  13. Joshi, A., Bhattacharyya, P., and Carman, M. J. (2018). Investigations in Computational Sarcasm. ↗
  14. Khodak, M., Saunshi, N., and Vodrahalli, K. (2017). A large self-annotated corpus for sarcasm. CoRR, abs/1704.05579. ↗ our card
  15. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461. ↗ our card
  16. Liu, Y., Albanie, S., Nagrani, A., and Zisserman, A. (2020). Use what you have: Video retrieval using representations from collaborative experts. ↗
  17. Maynard, D. G. and Greenwood, M. A. (2014). Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC 2014 Proceedings. find our card
  18. McFee, B., Lostanlen, V., McVicar, M., Metsai, A., Balke, S., Thomé, C., Raffel, C., Malek, A., Lee, D., Zalkow, F., Lee, K., Nieto, O., Mason, J., Ellis, D., Yamamoto, R., Seyfarth, S., Battenberg, E., , , Bittner, R., Choi, K., Moore, J., Wei, Z., Hidaka, S., nullmightybofo, Friesch, P., Stöter, F.-R., Hereñú, D., Kim, T., Vollrath, M., and Weiss, A. (2020). librosa/librosa: 0.7.2, January. find
  19. Mohammad, S. M. (2021). Ethics sheet for automatic emotion recognition and sentiment analysis. CoRR, abs/2109.08256. ↗
  20. Oprea, S. and Magdy, W. (2020). iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289. find our card
  21. Oraby, S., Harrison, V., Reed, L., Hernandez, E., Riloff, E., and Walker, M. (2016). Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31– 41, Los Angeles, September. Association for Computational Linguistics. find our card
  22. Pascual, S., Ravanelli, M., Serrà, J., Bonafonte, A., and Bengio, Y. ). Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks. In INTERSPEECH, 2019, pages 161–165. find
  23. Plutchick, R. (1980). Emotion: a psychoevolutionary synthesis. New York, Happer & Row. find
  24. Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2018). MELD: A multimodal multi-party dataset for emotion recognition in conversations. CoRR, abs/1810.02508. ↗ our card
  25. Preoţiuc-Pietro, D., Schwartz, H. A., Park, G., Eichstaedt, J., Kern, M., Ungar, L., and Shulman, E. (2016). Modelling valence and arousal in facebook posts. In Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 9–15. find
  26. Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683. ). ↗
  27. Russell, J. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39:1161– 1178, 12. ↗
  28. Sangwan, S., Akhtar, M. S., Behera, P., and Ekbal, A. (2020). I didn’t mean what i wrote! exploring multimodality for sarcasm detection. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  29. Schifanella, R., de Juan, P., Tetreault, J., and Cao, L. (2016). Detecting sarcasm in multimodal social platforms. Proceedings of the 24th ACM international conference on Multimedia, Oct. find our card
  30. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin. ↗
  31. Zafeiriou, S., Kollias, D., Nicolaou, M., Papaioannou, A., Zhao, G., and Kotsia, I. (2017). Aff-wild: Valence and arousal ‘in-the-wild’ challenge. pages 1980–1987, 07. find
  32. Zahiri, S. M. and Choi, J. D. (2017). Emotion detection on TV show transcripts with sequence-based convolutional neural networks. CoRR, abs/1708.04299. ↗
  33. Zvolenszky, Z. (2012). A gricean rearrangement of epithets. find
  34. Language Resource References Castro, Santiago and Hazarika, Devamanyu and Pérez-Rosas, Verónica and Zimmermann, Roger and Mihalcea, Rada and Poria, Soujanya. (2019). Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper). find our card
  35. Chauhan, Dushyant Singh and Dhanush, SR and Ekbal, Asif and Bhattacharyya, Pushpak. (2020). Sentiment and Emotion help Sarcasm? A Multi-task Learning Framework for Multi-Modal Sarcasm, Sentiment and Emotion Analysis. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ray-2022-multimodal-corpus-emotion-recognition-sarcasm/