Elamur Research⌕
☰
home / papers / 2022 / Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

Summary

The paper addresses multi-modal sarcasm detection: deciding whether a tweet and its attached image together convey sarcasm. Earlier systems mostly compared individual tokens with individual image patches and relied on limited image-derived knowledge such as attributes, ANPs or OCR text. The authors propose a hierarchical model that measures text–image agreement ("congruity") at two levels. At the atomic level, a multi-head cross attention module (MCA) aligns tokens with image objects. At the composition level, graph attention networks (GAT) run over a textual graph built from word dependencies and a visual graph built from spatial relations between object regions, so groups of words can be compared with groups of patches. Pairs with low congruity are treated as sarcastic. As external knowledge, the model adds image captions generated by the pre-trained ClipCap model (a CLIP prefix fed to GPT-2) and handles them like an extra "virtual" modality. On the public Twitter benchmark, the authors report gains over state-of-the-art baselines, with further improvement from the captions. This suggests that richer, open-vocabulary visual descriptions help expose subtle sarcasm.

Key points

  • The evaluation uses the public Twitter multi-modal sarcasm dataset introduced by Cai et al. (2019). The model outputs a binary label: sarcastic or not.
  • Architecture: pre-trained text and image encoders → MCA for atomic-level congruity → GAT for composition-level congruity. Atomic-level and composition-level features are concatenated for the final prediction.
  • Knowledge source: ClipCap captions (CLIP + GPT-2), compared against image attributes and ANPs. The authors acknowledge that their knowledge-fusion strategy does not suit ANPs or image attributes well.
  • Ablations:
    • A two-layer GAT performs best; deeper graphs degrade, likely due to over-smoothing.
    • For sentence embeddings, CLIP edges out USE, BERT CLS and Word Averaging. The authors still keep Word Averaging because it is simpler and adds fewer parameters.
  • Qualitative analysis:
    • Case studies show captions fixing errors made by the image-only model, such as a "fun" vs. "suffer" contrast.
    • Congruity maps show the composition-level module down-weighting image regions that the atomic level had wrongly treated as relevant.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

19 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.333 · arXiv 2210.03501

@inproceedings{liu2022multi,
  title = {{Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement}},
  author = {Liu, Hui and Wang, Wenya and Li, Haoliang},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.333},
  eprint = {2210.03501},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.333}
}

References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We present two possible limitations: 1) we only use the Twitter dataset for evaluation. However, to the best of our knowledge, this dataset is the only benchmark for the evaluation of multi-modal sar-Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2506–2515. Association for Computational Linguistics. find our card
  2. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for english. pages 169–174. Association for Computational Linguistics. find our card
  3. D Davidov, O Tsur, and A Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon (pp. 15–16). Retrieved from Association for Computational Linguistics website: https://www.aclweb. org/anthology/W10-2914. pdf. ↗ our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics. find
  5. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929. ↗
  6. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rah-wan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  7. Aniruddha Ghosh and Tony Veale. 2017. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 482–491, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  8. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5-6):602–610. ↗
  9. Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 2: Short Papers, pages 757–762. The Association for Computer Linguistics. find
  10. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. find our card
  11. Jiangnan Li, Hongliang Pan, Zheng Lin, Peng Fu, and Weiping Wang. 2021a. Sarcasm detection with commonsense knowledge. IEEE ACM Trans. Audio Speech Lang. Process., 29:3192–3201. ↗
  12. Ying Li, Hongwei Zhou, Yeyu Yin, and Jiaquan Gao. 2021b. Multi-label pattern image retrieval via attention mechanism driven graph convolutional network. In MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 300–308. ACM. find
  13. Bin Liang, Chenwei Lou, Xiang Li, Lin Gui, Min Yang, and Ruifeng Xu. 2021. Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 4707–4715. ACM. find
  14. Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. 2022. Multimodal sarcasm detection via cross-modal graph convolutional network. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1767–1777. Association for Computational Linguistics. find our card
  15. Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hong-tao Xie, Bin Wang, and Yongdong Zhang. 2020. Graph structured network for image-text matching. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10918–10927. Computer Vision Foundation / IEEE. find
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society. ↗
  17. Yaochen Liu, Yazhou Zhang, Qiuchi Li, Benyou Wang, and Dawei Song. 2021. What does your smile mean? jointly detecting multi-modal sarcasm and sentiment using quantum probability. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 871–880. Association for Computational Linguistics. find our card
  18. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Diana G Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Lrec 2014 proceedings. ELRA. find our card
  19. Ron Mokady, Amir Hertz, and Amit H. Bermano. 2021. Clipcap: CLIP prefix for image captioning. CoRR, abs/2111.09734. ↗
  20. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1383–1392. find our card
  21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR. ↗ our card
  22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  23. Rossano Schifanella, Paloma de Juan, Joel R. Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016, Amsterdam, The Netherlands, October 15-19, 2016, pages 1136–1145. ACM. find our card
  24. Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Trans. Assoc. Comput. Linguistics, 2:207–218. ↗ our card
  25. Hao Sun, Yuntao Li, and Yan Zhang. 2022. Conlearn: Contextual-knowledge-aware concept prerequisite relation learning with graph neural network. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pages 118–126. SIAM. find
  26. Hao Sun, Zijian Wu, Yue Cui, Liwei Deng, Yan Zhao, and Kai Zheng. 2021. Personalized dynamic knowledge-aware recommendation with hybrid explanations. In International Conference on Database Systems for Advanced Applications, pages 148–164. Springer. find
  27. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1010–1020, Melbourne, Australia. Association for Computational Linguistics. find our card
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008. find
  29. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI 2010 - 19th European Conference on Artificial Intelligence, Lisbon, Portugal, August 16-20, 2010, Proceedings, volume 215 of Frontiers in Artificial Intelligence and Applications, pages 765–770. IOS Press. ↗
  30. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations. Accepted as poster. ↗
  31. Tao Xiong, Peiran Zhang, Hongbo Zhu, and Yihui Yang. 2019. Sarcasm detection with self-matching networks and low-rank bilinear pooling. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 2115–2124. ACM. find
  32. Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020a. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3777–3786. Association for Computational Linguistics. find our card
  33. Xing Xu, Tan Wang, Yang Yang, Lin Zuo, Fumin Shen, and Heng Tao Shen. 2020b. Cross-modal attention with semantic consistence for image-text matching. IEEE Trans. Neural Networks Learn. Syst., 31(12):5412–5425. ↗

↑ back to top