Elamur Research⌕
☰
home / papers / 2023 / Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection

Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection

Paper · 2023 · Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams et al.

Summary

Sarcasm often cannot be recognized from words alone. Tone of voice and facial expression may be what turns a sentence ironic. This paper revisits MUStARD++, a sarcasm dataset of TV-show clips with text, audio and video. It makes two contributions. First, it re-benchmarks the dataset with modern pretrained encoders in place of the hand-crafted features used before (MFCC, spectrograms, OpenSMILE prosody, ResNet-152 frames). Video is encoded with ViFi-CLIP, speech with Wav2vec 2.0, and the modality features are joined through an existing fusion setup. This raises macro-F1 by more than 2 points over the earlier benchmark. Second, the original data has very uneven coverage of sarcasm types, so the authors build MUStARD++ Balanced. It adds clips from House MD, a show not used in earlier versions, labelled by three annotators with substantial agreement. Training and testing with the new clips adds about 2.4 more macro-F1 points. A controlled experiment shows the gain comes from diversity rather than from simply having more data. The work matters because it sets a stronger baseline and gives a more varied resource for multimodal irony research.

Key points

  • Dataset sizes:

    • MUStARD has 6,365 clips, of which 345 are sarcastic.
    • MUStARD++ has 1,202 clips (601 sarcastic, 601 not).
    • MUStARD++ Balanced has 1,365 clips (691 sarcastic, 674 not), with the new material from House MD.

  • Sarcasm types annotated:

    • Propositional (PRO) needs outside context.
    • Illocutionary (ILL) is carried by voice or gesture.
    • Embedded (EMB) has the incongruity inside the text.
    • Like-prefixed (LIKE) is rare; there were too few samples to test it separately.

  • Encoders: ViFi-CLIP for video and Wav2vec 2.0 for audio.

    • Better audio and video encoders add 3.39 points with dialogue context and 3.89 points without it.
    • The earlier MUStARD++ trimodal benchmark reached an F1 of about 70.2.

  • Distribution-shift test: some of the 952 original training clips were swapped for extension clips, keeping training size fixed (5 runs, majority vote). Class-wise accuracy rose:

    ClassBeforeAfter
    PRO0.7050.824
    ILL0.4840.545
    EMB0.6960.788
    Non-sarcastic0.6200.720

  • Error analysis:

    • Illocutionary sarcasm is the hardest type.
    • Most misclassified items are very short clips (under 2 seconds, e.g. "Nooo"), which carry too little context.

Interesting ideas

  • I1. To show that new data adds diversity and not just volume, the authors swap new samples into the training set without changing its size, then compare on a fixed, type-stratified test set
  • I2. The four-way sarcasm taxonomy separates irony visible in the text (embedded), irony that needs situational context (propositional) and irony carried only by delivery (illocutionary)
  • I3. Balancing the data by a finer category (sarcasm type) rather than only the binary label improved the minority types and also the non-sarcastic class
  • I4. Errors cluster on very short utterances that lack context, and the authors plan to flag or prune them

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when adding silver data from a new genre, swap at equal training size and evaluate on a frozen stratified test, so the effect of novelty is separated from the effect of size.
  • I2 (frames): the irony module in prag could label which kind of evidence supports an ironic reading and explicitly mark illocutionary-type cases as undecidable from text, instead of guessing.
  • I3 (learn): balance training data by construction or phenomenon type (e.g. rare UD relations, speech-act kinds), not just overall counts, and track per-type accuracy.
  • I4 (answer): treat very short or context-poor inputs as a known low-confidence zone and have readers abstain or ask for context, rather than forcing an interpretation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology BombayMumbai · India29 works · 2,104 citations🎓University of SurreyGuildford · United Kingdom21 works · 677 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 works

Cite

Cite as Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, Divyank Tiwari, Anjan Dutta, Xiatian Zhu, Pushpak Bhattacharyya and Diptesh Kanojia. 2023. Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection. arXiv:2310.01430.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.01430 · arXiv 2310.01430

@misc{bhosale2023sarcasm,
  title = {{Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection}},
  author = {Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, Divyank Tiwari, Anjan Dutta, Xiatian Zhu, Pushpak Bhattacharyya and Diptesh Kanojia},
  year = {2023},
  doi = {10.48550/arxiv.2310.01430},
  eprint = {2310.01430},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.01430}
}

References

50 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria, “Towards multimodal sarcasm detection (an _Obviously_ perfect paper),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  2. A. Ray, S. Mishra, A. Nunna, and P. Bhattacharyya, “A multimodal corpus for emotion recognition in sarcasm,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, 2022.
  3. M. Soleymani, D. Garcia, B. Jou, B. Schuller, S.-F. Chang, and M. Pantic, “A survey of multimodal sentiment analysis,” Image and Vision Computing, vol. 65, pp. 3–14, 2017. ↗
  4. D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine, “The hateful memes challenge: Detecting hate speech in multimodal memes,” Advances in Neural Information Processing Systems, vol. 33, pp. 2611–2624, 2020. ↗
  5. S. Suryawanshi, B. R. Chakravarthi, M. Arcan, and P. Buitelaar, “Multimodal meme dataset (multioff) for identifying offensive content in image and text,” in Proceedings of the second workshop on trolling, aggression and cyberbullying, 2020, pp. 32–41.
  6. P. Guo, Z. Chen, Y. Li, and H. Liu, “Audio-visual fusion network based on conformer for multimodal emotion recognition,” in Artificial Intelligence: Second CAAI International Conference, CICAI 2022, Beijing, China, August 27–28, 2022, Revised Selected Papers, Part II. Springer, 2023, pp. 315–326. find
  7. Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman, “Use what you have: Video retrieval using representations from collaborative experts,” arXiv preprint arXiv:1907.13487, 2019. ↗
  8. B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python.” in Proceedings of the 14th python in science conference, 2015. find
  9. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, June 2016. ↗
  10. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in ICML, 2021. ↗
  11. H. Rasheed, M. U. Khattak, M. Maaz, S. Khan, and F. S. Khan, “Fine-tuned clip models are efficient video learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6545–6554. ↗
  12. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf ↗
  13. A. Joshi, P. Bhattacharyya, and M. J. Carman, “Automatic sarcasm detection: A survey,” ACM Comput. Surv., vol. 50, no. 5, sep 2017. [Online]. Available: https://doi.org/10.1145/3124420 ↗
  14. M. Bouazizi and T. Otsuki Ohtsuki, “A pattern-based approach for sarcasm detection on twitter,” IEEE Access, vol. 4, pp. 5477–5488, 2016. ↗
  15. A. Reyes and P. Rosso, “Mining subjective knowledge from customer reviews: A specific case of irony detection,” in Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011). Portland, Oregon: Association for Computational Linguistics, Jun. 2011, pp. 118–124. [Online]. Available: https://aclanthology.org/W11-1715 ↗
  16. F. Barbieri, H. Saggion, and F. Ronzano, “Modelling sarcasm in Twitter, a novel approach,” in Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. Baltimore, Maryland: Association for Computational Linguistics, Jun. 2014, pp. 50–58. [Online]. Available: https://aclanthology.org/W14-2609 ↗
  17. C. Van Hee, E. Lefever, and V. Hoste, “SemEval-2018 task 3: Irony detection in English tweets,” in Proceedings of the 12th International Workshop on Semantic Evaluation. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 39–50. [Online]. Available: https://aclanthology.org/S18-1005 ↗
  18. R. A. Potamias, G. Siolas, and A. G. Stafylopatis, “A transformer-based approach to irony and sarcasm detection,” Neural Computing and Applications, vol. 32, no. 23, pp. 17 309–17 320, jun 2020. [Online]. Available: https://doi.org/10.1007%2Fs00521-020-05102-3 ↗
  19. E. Riloff, A. Qadir, P. Surve, L. De Silva, N. Gilbert, and R. Huang, “Sarcasm as contrast between a positive sentiment and negative situation,” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics, Oct. 2013, pp. 704–714. [Online]. Available: https://aclanthology.org/D13-1066 ↗
  20. T. Shangipour ataei, S. Javdan, and B. Minaei-Bidgoli, “Applying transformers and aspect-based sentiment analysis approaches on sarcasm detection,” in Proceedings of the Second Workshop on Figurative Language Processing. Online: Association for Computational Linguistics, Jul. 2020, pp. 67–71. [Online]. Available: https://aclanthology.org/2020.figlang-1.9 ↗
  21. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  22. D. Ma, S. Li, X. Zhang, and H. Wang, “Interactive attention networks for aspect-level sentiment classification,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, 2017, pp. 4068–4074. [Online]. Available: https://doi.org/10.24963/ijcai.2017/568 ↗
  23. I. Abu Farha and W. Magdy, “Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection,” in Proceedings of the Sixth Arabic Natural Language Processing Workshop. Kyiv, Ukraine (Virtual): Association for Computational Linguistics, Apr. 2021, pp. 21–31. [Online]. Available: https://aclanthology.org/2021.wanlp-1.3 ↗
  24. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440–8451. [Online]. Available: https://aclanthology.org/2020.acl-main.747 ↗
  25. M. Abdul-Mageed, A. Elmadany, and E. M. B. Nagoudi, “Arbert & marbert: Deep bidirectional transformers for arabic,” 2021. ↗
  26. J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1532–1543. [Online]. Available: https://aclanthology.org/D14-1162 ↗
  27. P. Goel, R. Jain, A. Nayyar, S. Singhal, and M. Srivastava, “Sarcasm detection using deep learning and ensemble learning,” Multimedia Tools and Applications, vol. 81, no. 30, pp. 43 229–43 252, 2022. ↗
  28. M. Khodak, N. Saunshi, and K. Vodrahalli, “A large self-annotated corpus for sarcasm,” 2018.
  29. V. Pérez-Rosas, R. Mihalcea, and L.-P. Morency, “Utterance-level multimodal sentiment analysis,” in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2013, pp. 973–982. find
  30. A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,” arXiv preprint arXiv:1606.06259, 2016. ↗
  31. C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Language resources and evaluation, vol. 42, no. 4, pp. 335–359, 2008. ↗
  32. S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L.-P. Morency, “Context-dependent sentiment analysis in user-generated videos,” in Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), 2017, pp. 873–883.
  33. N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, and S. Poria, “Multimodal sentiment analysis using hierarchical fusion with context modeling,” Knowledge-based systems, vol. 161, pp. 124–133, 2018. ↗
  34. D. Hazarika, S. Poria, A. Zadeh, E. Cambria, L.-P. Morency, and R. Zimmermann, “Conversational memory network for emotion recognition in dyadic dialogue videos,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 2122–2132. [Online]. Available: https://aclanthology.org/N18-1193 ↗
  35. Y. Wang, Y. Shen, Z. Liu, P. P. Liang, A. Zadeh, and L.-P. Morency, “Words can shift: Dynamically adjusting word representations using nonverbal behaviors,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 7216–7223. find
  36. H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. Póczos, “Found in translation: Learning robust joint representations by cyclic translations between modalities,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 6892–6899. find
  37. P. P. Liang, Z. Liu, A. B. Zadeh, and L.-P. Morency, “Multimodal language analysis with recurrent multistage fusion,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 150–161. ↗
  38. Y.-H. H. Tsai, P. P. Liang, A. Zadeh, L.-P. Morency, and R. Salakhutdinov, “Learning factorized multimodal representations,” arXiv preprint arXiv:1806.06176, 2018. ↗
  39. S. Sahay, E. Okur, S. H Kumar, and L. Nachman, “Low rank fusion based transformers for multimodal sequences,” in Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML). Seattle, USA: Association for Computational Linguistics, Jul. 2020, pp. 29–34. [Online]. Available: https://aclanthology.org/2020.challengehml-1.4 ↗
  40. Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 6558–6569. [Online]. Available: https://aclanthology.org/P19-1656 ↗
  41. Z. Liu, Y. Shen, V. B. Lakshminarasimhan, P. P. Liang, A. Bagher Zadeh, and L.-P. Morency, “Efficient low-rank multimodal fusion with modality-specific factors,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 2247–2256. [Online]. Available: https://aclanthology.org/P18-1209 ↗
  42. A. Bhat and A. Chauhan, “Multimodal sarcasm detection: A survey,” in 2022 IEEE Delhi Section Conference (DELCON), 2022, pp. 1–7. find
  43. S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria, “Towards multimodal sarcasm detection (an _obviously_ perfect paper),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Florence, Italy: Association for Computational Linguistics, 7 2019.
  44. A. Ray, S. Mishra, A. Nunna, and P. Bhattacharyya, “A multimodal corpus for emotion recognition in sarcasm,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, Jun. 2022, pp. 6992–7003. [Online]. Available: https://aclanthology.org/2022.lrec-1.756 ↗
  45. P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Y. Chen, P. Wu, M. A. Lee, Y. Zhu et al., “Multibench: Multiscale benchmarks for multimodal representation learning,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. ↗
  46. L.-W. Chen and A. Rudnicky, “Exploring wav2vec 2.0 fine tuning for improved speech emotion recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. find
  47. D. S. Chauhan, D. S R, A. Ekbal, and P. Bhattacharyya, “Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
  48. R. W. Picard, “Affective computing mit press,” Cambridge, Massachsusetts, p. 2, 1997. find
  49. S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” arXiv preprint arXiv:2004.10964, 2020. ↗
  50. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bhosale-2023-sarcasm-sight-sound-benchmarking-expansion/