Elamur Research⌕
☰
home / papers / 2023 / MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System

MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System

Summary

The paper argues that the standard multi-modal sarcasm benchmark, MMSD (Twitter text–image pairs, Cai et al., 2019), does not reliably measure multi-modal understanding. A text-only RoBERTa model beats the leading multi-modal system HKE, which suggests that models exploit textual shortcuts. The authors trace this to two flaws. First, hashtags and emoji tokens are distributed very unevenly between sarcastic and non-sarcastic posts. Second, any post lacking a tag like #sarcasm was automatically labeled negative, even though many such posts are sarcastic.

They release MMSD2.0, which strips hashtag and emoji words from the texts. It also has crowd workers relabel every originally negative example, with quality gated by a qualification test and an agreement check. Alongside the data, they propose multi-view CLIP, which builds on pre-trained CLIP. It combines evidence from a text view, an image view and a text–image interaction view, and it needs neither object detection nor graph construction. The authors report that it clearly outperforms earlier state-of-the-art baselines. The work matters because it shows that progress on the old benchmark may be inflated, and it offers a cleaner testbed with code and data on GitHub (JoeYing1019/MMSD2.0).

Key points

  • Evidence of shortcut learning: the text-only RoBERTa outperforms the multi-modal HKE on the original MMSD by 6.6%.

  • Hashtag and emoji imbalance: sarcastic samples average more than one hashtag word, while non-sarcastic ones average fewer than one. About 80.7% of emoji words occur in only one of the two classes.

  • Relabeling procedure:

    • All MMSD negatives, more than half the data, were re-annotated as Sarcasm, Not Sarcasm or Undecided.
    • Three experts resolved the Undecided cases.
    • Annotators had to reach 85% on a 100-sample onboarding test.
    • A second-annotator check on 1,000 samples gave Cohen's Kappa of 0.811.

  • Split sizes (positive/negative, MMSD → MMSD2.0):

    SplitSizeMMSDMMSD2.0
    Train19,8168,642/11,1749,572/10,240
    Validation2,410959/1,4511,042/1,368
    Test2,409959/1,4501,037/1,372

    MMSD2.0 has a more balanced class distribution.

  • Stated limitation: MMSD2.0 inherits the scale and diversity of MMSD. The annotators were paid postgraduate students at Chinese universities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

9 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.689 · arXiv 2307.07135

@inproceedings{qin2023mmsd2,
  title = {{MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System}},
  author = {Qin, Libo and Huang, Shijue and Chen, Qiguang and Cai, Chenran and Zhang, Yudi and Liang, Bin and Che, Wanxiang and Xu, Ruifeng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.689},
  eprint = {2307.07135},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.689}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  2. Nastaran Babanejad, Heidar Davoudi, Aijun An, and Manos Papagelis. 2020. Affective and contextual embedding for sarcasm detection. In Proceedings of the 28th International Conference on Computational Linguistics, pages 225–243, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  3. Bin Liang, Chenwei Lou, Xiang Li, Lin Gui, Min Yang, and Ruifeng Xu. 2021. Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In Proceedings of the 29th ACM International Conference on Multimedia, MM ’21, page 4707–4715, New York, NY, USA. Association for Computing Machinery. find
  4. Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. 2022. Multimodal sarcasm detection via cross-modal graph convolutional network. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1767– 1777, Dublin, Ireland. Association for Computational Linguistics. find our card
  5. Hui Liu, Wenya Wang, and Haoliang Li. 2022. Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement. find our card
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  7. Xiang Long, Chuang Gan, Gerard Melo, Xiao Liu, Yan-dong Li, Fu Li, and Shilei Wen. 2018. Multimodal keyless attention fusion for video classification. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). find
  8. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  9. Edwin Lunando and Ayu Purwarianti. 2013. Indonesian social media sentiment analysis with sarcasm detection. In 2013 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pages 195–198. find
  10. Mary L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia Medica, 22:276 – 282. ↗
  11. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1383–1392, Online. Association for Computational Linguistics. find our card
  12. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2(1–2):1–135. ↗
  13. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1601–1612, Osaka, Japan. ↗ our card
  14. The COL-ING 2016 Organizing Committee. find
  15. Rossano Schifanella, Paloma de Juan, Joel Tetreault, and LiangLiang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 24th ACM International Conference on Multimedia, MM ’16, page 1136–1145, New York, NY, USA. Association for Computing Machinery. find our card
  16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc. find
  17. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. ↗ our card
  18. Tao Xiong, Peiran Zhang, Hongbo Zhu, and Yihui Yang. 2019. Sarcasm detection with self-matching networks and low-rank bilinear pooling. In The World Wide Web Conference, WWW ’19, page 2115–2124, New York, NY, USA. Association for Computing Machinery. find
  19. Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3777–3786, Online. Association for Computational Linguistics. find our card
  20. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2449–2460, Osaka, Japan. find our card
  21. The COLING 2016 Organizing Committee. find
  22. Yangming Zhou, Qichao Ying, Zhenxing Qian, Sheng Li, and Xinpeng Zhang. 2022. Multimodal fake news detection via clip-guided learning. find
  23. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR. ↗ our card
  24. ACL 2023 Responsible NLP Checklist find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qin-2023-mmsd2-0-reliable-multi-modal/