Elamur Research⌕
☰
home / papers / 2021 / What Does Your Smile Mean? Jointly Detecting Multi-Modal Sarcasm and Sentiment Using Quantum Probability

What Does Your Smile Mean? Jointly Detecting Multi-Modal Sarcasm and Sentiment Using Quantum Probability

Summary

The paper addresses the joint recognition of sarcasm and sentiment in multi-modal (text plus image) utterances. Both phenomena reflect uncertain, intuitive human judgement, and they are tightly linked because sarcasm can invert surface polarity. The authors use quantum probability (QP) as a modelling framework and present QPM, a QP-driven multi-task learner with three parts:

  • Encoder: each modality is encoded as a complex-valued superposition over basis units. Amplitude is meant to carry semantics and phase to carry sentiment or sarcastic orientation.
  • Fusion network: a quantum-like network uses tensor-product composition (density matrices) to capture context from neighbouring utterances. It uses an interference term to fuse textual and visual evidence non-linearly.
  • Measurement: incompatible (non-commuting) measurement operators produce the probabilities for both tasks.

Experiments on MUStARD and Memotion are reported to outperform state-of-the-art baselines. Analyses of the learned operators support the claim that the two tasks are incompatible and correlated. The work matters as an early attempt to bring a cognition-inspired probabilistic formalism into multi-modal, multi-task affect analysis.

Key points

  • Claimed novelty: QPM is presented as the first QP-based multi-task framework for joint multi-modal sarcasm and sentiment detection. It was evaluated on the MUStARD and Memotion benchmarks.
  • Ablation: the quantum incompatible measurement component contributes most to performance. It is followed by interference-based fusion, contextuality modelling, and the complex-valued representation.
  • Incompatibility test: the authors trained 1000 (MUStARD) and 800 (Memotion) pairs of sarcasm and sentiment operators. The commutation law was violated for all pairs.
  • Correlation scores: average inter-task correlation, based on quantum relative entropy, was 0.484 for MUStARD and 0.461 for Memotion.
  • Data analysis: 84% (MUStARD) and 74% (Memotion) of sarcastic samples carry explicit sentiment. About 38% and 36% respectively of ironic utterances are positive.
  • Error analysis: the multi-task setup corrects single-task errors where the literal meaning conflicts with the real sentiment. It still fails on cases that need external knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.74

@inproceedings{liu2021what,
  title = {{What Does Your Smile Mean? Jointly Detecting Multi-Modal Sarcasm and Sentiment Using Quantum Probability}},
  author = {Liu, Yaochen and Zhang, Yazhou and Li, Qiuchi and Wang, Benyou and Song, Dawei},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.74},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.74}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Bruza, Kirsty Kitto, Douglas Nelson, and Cathy McEvoy. 2009. Is there something quantum-like about the human mental lexicon? Journal of mathematical psychology, 53(5):362–377. ↗
  2. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515. find our card
  3. Erik Cambria, Soujanya Poria, and Amir Hussain. 2019. Speaker-independent multimodal sentiment analysis for big data. In Multimodal Analytics for Next-Generation Big Data Technologies and Applications, pages 13–43. Springer. ↗
  4. Noël Carroll and Noël E Carroll. 1999. Philosophy of art: A contemporary introduction. Psychology Press. find
  5. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _Obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629, Florence, Italy. Association for Computational Linguistics. ↗ our card
  6. Dushyant Singh Chauhan, Dhanush S R, Asif Ekbal, and Pushpak Bhattacharyya. 2020. Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4351–4360, Online. Association for Computational Linguistics. find our card
  7. Sébastien Designolle, Máté Farkas, and J˛edrzej Kaniewski. 2019. Incompatibility robustness of quantum measurements: a unified framework. New Journal of Physics, 21(11):113053. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT 2019: Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186. find
  9. Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. 2020. MEISD: A multimodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4441–4453, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  10. Dimitris Gkoumas, Qiuchi Li, Shahram Dehdashti, Massimo Melucci, Yijun Yu, and Dawei Song. 2021. Quantum cognitively motivated decision fusion for video sentiment analysis. arXiv preprint arXiv:2101.04406. ↗
  11. Feiran Huang, Xiaoming Zhang, Zhonghua Zhao, Jie Xu, and Zhoujun Li. 2019. Image–text sentiment analysis via deep multimodal attentive fusion. Knowledge-Based Systems, 167:26–37. ↗
  12. Yongyu Jiang, Peng Zhang, Hui Gao, and Dawei Song. 2020. A quantum interference inspired neural matching model for ad-hoc retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 19–28. find
  13. Akshi Kumar and Geetanjali Garg. 2019. Sentiment analysis of multimodal twitter data. Multimedia Tools and Applications, pages 1–17. ↗
  14. Lily Li, Or Levi, Pedram Hosseini, and David A Broni-atowski. 2020. A multi-modal method for satire detection using textual and visual cues. arXiv preprint arXiv:2010.06671. ↗
  15. Qiuchi Li, Dimitris Gkoumas, Christina Lioma, and Massimo Melucci. 2021. Quantum-inspired multimodal fusion for video sentiment analysis. Information Fusion, 65:58–71. ↗
  16. Qiuchi Li, Benyou Wang, and Massimo Melucci. 2019. Cnm: An interpretable complex-valued network for matching. In NAACL-HLT (1), pages 4139–4148. ↗
  17. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1383–1392. find our card
  18. Xinyu Wang, Xiaowen Sun, Tan Yang, and Hongbo Wang. 2020. Building a bridge: A method for image-text sarcasm detection without pretraining on image-text data. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 19–29. find our card
  19. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 527–536. find our card
  20. Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3718–3727, Online. Association for Computational Linguistics. find
  21. Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis. 2020. A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications, pages 1–12. ↗ our card
  22. Rossano Schifanella, Paloma de Juan, Joel Tetreault, and LiangLiang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 24th ACM International Conference on Multimedia, MM ’16, page 1136–1145, New York, NY, USA. Association for Computing Machinery. find our card
  23. Chhavi Sharma, Deepesh Bhageria, William Paka, Scott, Srinivas P Y K L, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Björn Gambäck. 2020. SemEval-2020 Task 8: Memotion Analysis-The Visuo-Lingual Metaphor! In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain. Association for Computational Linguistics. ↗
  24. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, Copenhagen, Denmark. Association for Computational Linguistics. find
  25. Peng Zhang, Jiabin Niu, Zhan Su, Benyou Wang, Liqun Ma, and Dawei Song. 2018a. End-to-end quantum-like language models with application to question answering. pages 5666–5673. find
  26. Peng Zhang, Zhan Su, Lipeng Zhang, Benyou Wang, and Dawei Song. 2018b. A quantum many-body wave function inspired language modeling approach. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 1303–1312. find
  27. Alessandro Sordoni, Jian-Yun Nie, and Yoshua Bengio. 2013. Modeling term dependencies with quantum language models for ir. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’13, page 653–662, New York, NY, USA. Association for Computing Machinery. find
  28. Yazhou Zhang, Dawei Song, Xiang Li, Peng Zhang, Panpan Wang, Lu Rong, Guangliang Yu, and Bo Wang. 2020. A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis. Information Fusion, 62:14–31. ↗
  29. Mingxing Tan and Quoc V. Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105–6114. ↗ our card
  30. Yazhou Zhang, Dawei Song, Peng Zhang, Panpan Wang, Jingfei Li, Xiang Li, and Benyou Wang. 2018c. A quantum-inspired multimodal sentiment analysis framework. Theoretical Computer Science, 752:21– 40. ↗
  31. Sagar Uprety, Dimitris Gkoumas, and Dawei Song. 2020. A survey of quantum theory inspired approaches to information retrieval. ACM Computing Surveys (CSUR), 53(5):1–39. ↗
  32. George-Alexandru Vlad, George-Eduard Zaharia, Dumitru-Clementin Cercel, Costin-Gabriel Chiru, and Stefan Trausan-Matu. 2020. Upb at semeval-2020 task 8: Joint textual and visual modeling in a multi-task learning architecture for memotion analysis. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1208–1214. find
  33. Benyou Wang, Qiuchi Li, Massimo Melucci, and Dawei Song. 2019. Semantic hilbert space for text representation learning. In The World Wide Web Conference on, pages 3293–3299. find

↑ back to top

This page in Markdown →