Elamur Research⌕
☰
home / papers / 2020 / Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection

Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection

Summary

The paper tackles sarcasm detection in social media posts that combine text and an image. Earlier multi-modal approaches either concatenated features or fused modalities with handcrafted schemes, and they did not explicitly model incongruity. Incongruity, a clash between what is said and what is actually the case, is a defining trait of sarcasm. The authors build a BERT-based model that captures this clash in two places. Between modalities, a text-image matching component adapted from self-attention lets BERT token features serve as queries over ResNet-152 image region features, which act as keys and values. This lets the model focus on image areas that contradict the words. Within the text, a co-attention affinity matrix with max-pooling contrasts the tweet with its hashtags. The two incongruity signals are merged for classification. On a public multi-modal sarcasm dataset, the model sets a new state of the art. Error analysis suggests that text embedded in images is an important cue. This matters for sentiment analysis and opinion mining, where sarcasm can invert polarity.

Key points

  • Text is encoded with pre-trained Base BERT. Images are resized to 224×224 and passed through ResNet-152 with the final FC layer removed, which yields 49 region vectors of 2048 dimensions each. These are linearly projected to BERT's hidden size.
  • The intra-modality branch models text versus hashtags; a special token stands in when a post has no hashtags. Using the inter-modality attention design for this branch added parameters without a clear benefit, so the authors chose the affinity-matrix approach instead.
  • The reported F1 is 0.8292, which is 2.74% above the previous best method. Accuracy is 0.8605, precision 0.8087, and recall 0.8508. The number of stacked text-image matching layers was tuned over values from 1 to 7.
  • The authors inspected about 50 misclassified examples. Many errors involved meaningful text inside the image. Adding OCR text (via a General Character Recognition API) with an extra co-attention matrix raised F1 to 0.8618 and accuracy to 0.8875.
  • The remaining failures involve cases that need external knowledge, such as a speaker's facial expression or surrounding context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

34 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 24 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.124

@inproceedings{pan2020modeling,
  title = {{Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection}},
  author = {Pan, Hongliang and Lin, Zheng and Fu, Peng and Qi, Yatao and Wang, Weiping},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.124},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.124}
}

References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450. ↗ our card
  2. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2506–2515. find our card
  3. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, pages 4619– 4629. find our card
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. find
  5. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In EMNLP 2015, pages 1003–1012. find our card
  6. Raymond W Gibbs Jr, Raymond W Gibbs, and Jr Gibbs. 1994. The poetics of mind: Figurative thought, language, and understanding. Cambridge University Press. find
  7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. ↗
  8. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 328–339. find
  9. Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2017. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5):73:1–73:22. ↗ our card
  10. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics ACL 2015, pages 757–762. find our card
  11. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746–1751. find our card
  12. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  13. Bing Liu. 2010. Sentiment analysis and subjectivity. In Handbook of Natural Language Processing, Second Edition. find
  14. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 289–297. ↗
  15. Diana Maynard and Mark A. Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014, pages 4238–4243. find our card
  16. Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In ACL 2017, pages 377–387. find our card
  17. Bo Pang and Lillian Lee. 2007. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1–135. ↗
  18. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 8024–8035. ↗ our card
  19. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227–2237. find
  20. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In COLING 2016, pages 1601–1612. ↗ our card
  21. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, pages 704– 714. find our card
  22. Rossano Schifanella, Paloma de Juan, Joel R. Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016, Amsterdam, The Netherlands, October 15-19, 2016, pages 1136–1145. find our card
  23. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, pages 1010–1020. find our card
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998–6008. find
  25. Tao Xiong, Peiran Zhang, Hongbo Zhu, and Yihui Yang. 2019. Sarcasm detection with self-matching networks and low-rank bilinear pooling. In The World Wide Web Conference, WWW 2019, pages 2115–2124. find

↑ back to top