Elamur Research
☰
home / papers / 2020 / Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association

Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.349.pdf

Summary

Sarcasm in social media posts often appears only when a tweet's text is read against its attached picture. For example, a caption praising "perfect" weather may sit next to a photo of pouring rain. Text-only detectors miss such cases. Earlier multimodal systems mainly fused image and text features and did not reason about incongruity between them.

The authors propose D&R Net (Decomposition and Relation Network):

  • D-Net projects image and text features into a common subspace and into separate modality-specific subspaces. This implicitly captures where the two modalities agree and where they clash.
  • R-Net uses an attention mechanism between tweet words and adjective-noun pairs (ANPs) detected in the image. It weights ANPs by their detection confidence and explicitly models semantic links across modalities.

On a publicly available multimodal tweet dataset, the model reports state-of-the-art results (F1 80.60, accuracy 84.02). Its attention weights also provide word-level evidence for each decision. The study shows that modelling contrast between modalities, rather than simple fusion, is central to multimodal sarcasm detection.

Key points

  • Pipeline:
    • Top-K ANP extraction from each image.
    • Encoding: BiLSTM for text, pre-trained ResNet for images, and max-pooled adjective/noun embeddings for ANPs.
    • D-Net with shared, image and text layers.
    • R-Net with ANP-aware cross-modality attention.
    • A final classifier.
  • Full model: F1 80.60 and accuracy 84.02.
  • Ablations:
    • Removing D-Net causes the largest drop, to F1 77.63 (−2.97) and accuracy 82.27.
    • Removing R-Net gives F1 79.90 and accuracy 83.10.
  • ANPs vs. noun-only attributes: Replacing ANPs with the noun-only attributes of Cai et al. (2019) lowers F1 to 79.52.
  • ANP fusion: Probability-weighted ANP fusion outperforms MaxPool (79.80) and AvgPool (79.86).
  • Case studies: Attention visualisations show the model linking text such as "eclipse" or "lovely books" with contradicting visual ANPs, e.g. "rainy clouds" or "bad sign".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

34 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.349

@inproceedings{xu2020reasoning,
  title = {{Reasoning with Multimodal Sarcastic Tweets via Modeling Cross-Modality Contrast and Semantic Association}},
  author = {Xu, Nan and Zeng, Zhixiong and Mao, Wenji},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.349},
  url = {https://doi.org/10.18653/v1/2020.acl-main.349}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silvio Amir, Byron C Wallace, Hao Lyu, Paula Carvalho, and Mario J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of CoNLL, pages 167–177. find our card
  2. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of ICWSM, pages 574–577. find
  3. Christos Baziotis, Athanasiou Nikolaos, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. Ntua-slp at semeval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive rnns. In Proceedings of SemEval, pages 613–621. find
  4. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of ACL, pages 2506– 2515. find our card
  5. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of ACL-IJCNLP, pages 757–762. find our card
  6. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In Proceedings of EMNLP, pages 1006–1011. find our card
  7. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746–1751. find our card
  8. Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In Proceedings of ACL, pages 377–387. find
  9. Lotem Peled and Roi Reichart. 2017. Sarcasm sign: Interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of ACL, pages 1690–1700. find
  10. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of ACL, pages 4619–4629. find our card
  11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532– 1543. find
  12. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of CoNLL, pages 107–116. find our card
  13. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In Proceedings of COLING, pages 1601–1612. find our card
  14. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of EMNLP, pages 1615–1625. find our card
  15. Aniruddha Ghosh and Tony Veale. 2017. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of EMNLP, pages 482–491. find
  16. Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of ACL, pages 581–586. find our card
  17. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural networks, 18(5-6):602–610. find
  18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770–778. find
  19. Irazú Hernández-Farı́as, José-Miguel Benedı́, and Paolo Rosso. 2015. Applying basic features from sentiment analysis for automatic irony detection. In Proceedings of IbPRIA, pages 337–344. find
  20. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of WSDM, pages 97–106. find
  21. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of EMNLP, pages 704–714. find our card
  22. Rossano Schifanella, Paloma de Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of ACM MM, pages 1136–1145. find our card
  23. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of ACL, pages 1010–1020. find our card
  24. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm—a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Proceedings of ICWSM, pages 162–169. find
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 5998– 6008. find
  26. Penghui Wei, Wenji Mao, and Chen Guandan. 2019. A topic-aware reinforced model for weakly supervised stance detection. In Proceedings of AAAI, pages 7249–7256. find
  27. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING, pages 2449–2460. find our card