Elamur Research
☰
home / papers / 2019 / Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model

Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1239.pdf

Summary

Sarcasm detection on Twitter has mostly relied on text alone. However, many tweets include pictures that can reveal or confirm an ironic intent, for example cheerful words about the weather next to a photo of storm clouds. Cai, Cai and Wan address sarcasm detection for tweets that pair text with an image. They build and publicly release a new Twitter dataset for this task.

Their hierarchical fusion model treats three inputs as separate modalities: the text, the image, and a set of predicted image attributes (short descriptive words). The model combines them in three stages:

  • Attribute information sets the starting state of a Bi-LSTM that encodes the text.
  • Each modality's features are rebuilt with guidance from the others.
  • The refined vectors are merged through a learned weighted average before classification.

The full model beats text-only and simple concatenation baselines, and sign tests show the gains are significant. Ablations confirm that every fusion stage and every modality helps. The study shows that structured multimodal fusion, with attributes linking pixels to words, gives better sarcasm detection on social media.

Key points

  • Image features come from ResNet-50 V2 applied to 14×14 regions (196 regions) of a 448×448 image. Their mean serves as the image guidance vector.
  • An attribute predictor based on ResNet-101 is pre-trained on ImageNet and fine-tuned on 1000 attributes taken from COCO captions. It outputs five attributes per image, which are encoded with GloVe and weighted by attention.
  • The full model scores 0.8018 F-score and 0.8344 accuracy. Removing early fusion lowers F-score to 0.7880, and removing representation fusion lowers it to 0.7902.
  • Early fusion works better with attribute vectors than with image vectors (F-score 0.7787 with image vectors). This suggests attributes narrow the gap between text and visual representations.
  • Sign tests against the Text(Bi-LSTM), Concat(2) and Concat(3) baselines give p < 0.05. The error analysis shows the model misses sarcasm that depends on common-sense knowledge, such as recognizing an insulting gesture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 71 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1239

@inproceedings{cai2019multi,
  title = {{Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model}},
  author = {Cai, Yitao and Cai, Huiyu and Wan, Xiaojun},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1239},
  url = {https://doi.org/10.18653/v1/p19-1239}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. CoRR, abs/1607.00976. ↗ our card
  2. David Bamman and Noah A. Smith. 2015. Contextualized sarcasm detection on twitter. In ICWSM. find
  3. Christos Baziotis, Nikos Athanasiou, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. Ntua-slp at semeval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive rnns. arXiv preprint arXiv:1804.06659. ↗
  4. M. Bouazizi and T. Ohtsuki. 2015. Sarcasm detection in twitter: ”all your products are incredibly amazing!!!” - are they really? In 2015 IEEE Global Communications Conference (GLOBECOM), pages 1–6. find
  5. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. 2015. Abc-cnn: An attention based convolutional neural network for visual question answering. arXiv preprint arXiv:1511.05960. ↗
  6. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  7. Shelly Dews and Ellen Winner. 1995. Muting the meaning a social function of irony. Metaphor and Symbol, 10(1):3–19. Aniruddha Ghosh and Dr. find
  8. Tony Veale. 2016. Frack-ing sarcasm using neural network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169. Association for Computational Linguistics. find
  9. Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018a. Hybrid attention based multimodal network for spoken language classification. In Proceedings of the 27th International Conference on Computational Linguistics, Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018b. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2225–2235. Association for Computational Linguistics. find
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. CoRR, abs/1603.05027. ↗
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. CoRR, abs/1408.5882. ↗ our card
  13. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. Adam: CoRR, Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham. Springer International Publishing. ↗
  14. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing, pages 1532– 1543. find
  15. Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2015. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2539–2544. find
  16. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. CoRR, abs/1610.08815. ↗ our card
  17. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223. Dublin City University and Association for Computational Linguistics. find our card
  18. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP 2013 - 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pages 704–714. Association for Computational Linguistics (ACL). find our card
  19. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252. find our card
  20. Rossano Schifanella, Paloma de Juan, Joel R. Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. CoRR, abs/1608.02289. ↗ our card
  21. Yi Tay, Luu Anh Tuan, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. CoRR, abs/1805.02856. ↗ our card
  22. Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P. Xing. 2016. Select-additive learning: Improving cross-individual generalization in multimodal sentiment analysis. CoRR, abs/1609.05244. ↗
  23. Peng Wang, Qi Wu, Chunhua Shen, and Anton van den Hengel. 2017. The vqa-machine: Learning how to use existing vision algorithms to answer new questions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn, volume 4. find
  24. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. Thu ngn at semeval-2018 task 3: Tweet irony detection with densely connected lstm and multi-task learning. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 51–56. find
  25. Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton van den Hengel. 2016. What value do explicit high level concepts have in vision to language problems? In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 203–212. find
  26. Yang Yang, Jia Jia, Shumei Zhang, Boya Wu, Qicong Chen, Juanzi Li, Chunxiao Xing, and Jie Tang. 2014. How do your friends on social media disclose your emotions? In Twenty-Eighth AAAI Conference on Artificial Intelligence, pages 306–312. find
  27. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. CoRR, abs/1707.07250. ↗
  28. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In COLING. find our card