Elamur Research⌕
☰
home / papers / 2023 / Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection

Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection

Paper · 2023 · Divyank Tiwari, Diptesh Kanojia, Anupama Ray et al.

Summary

The paper asks whether eye-tracking signals can make it easier to recognize sarcasm in dialogue when combined with text, audio and video. The authors start from MUStARD++, a balanced collection of 1,202 TV-show clips, each with a target utterance and its preceding context. Five paid volunteers read 231 of these dialogues while an eye tracker recorded their gaze. Because recording gaze is slow and expensive, the authors then trained several models to predict gaze features for the remaining clips. They kept the predictor whose outputs agreed most closely with the real recordings, and checked its quality both directly and through the downstream task. Adding gaze as a fourth modality helped in every feature combination they tried. Real gaze gave up to 6.6 points of improvement on the annotated subset. Real plus predicted gaze gave 2.3 points on the full dataset, which is state of the art on MUStARD++. Predicted gaze alone still added 1.9 points. This suggests that cheap synthetic cognitive signals can partly stand in for costly human recordings. The code, gaze data and models are publicly released.

Key points

  • Gaze was recorded on 231 MUStARD++ instances (129 sarcastic, 102 non-sarcastic) by 5 participants, giving 1155 gaze samples. The sarcastic items cover all four sarcasm types.
  • For every participant, two-sample t-tests show significant differences between sarcastic and non-sarcastic readings (p < 0.05). This holds for average fixation duration and for IA regression path duration.
  • Ray et al. reported a baseline of 70.2% F1 using audio, text and video. The authors reproduce this baseline and add gaze through feature fusion with a feed-forward classifier.
  • Measured gains from gaze:
    • up to 6.6 points with collected gaze on the annotated subset;
    • 2.3 points with collected plus predicted gaze on the full data (SoTA);
    • 1.9 points with predicted gaze only.
  • Stated limitations:
    • agreement between annotators on sarcasm is only moderate;
    • multimodal sarcasm datasets are scarce;
    • qualitative analysis shows automatic key-frame selection can miss facial cues that appear only near the end of an utterance.

Interesting ideas

  • I1. Pay for a costly signal on a small slice of the data, train a predictor for the rest, and choose that predictor by its correlation with the real signal rather than by downstream score alone. This stretches scarce human annotation across a whole corpus.
  • I2. Before using a signal as a feature, test separately for each annotator whether it actually distinguishes the classes. This catches cues that only work for some people or that are just noise.
  • I3. Report gains three ways: real signal only, real plus synthetic, and synthetic only. This shows how much of the improvement survives when the human data is removed, which works as a natural control.
  • I4. In the motivating example, readers' regressions jumped back to a specific phrase in the earlier turn. That suggests sarcasm can be explained by linking the target utterance to the context span that contradicts it.
  • I5. The error analysis notes that utterances which open positively and end with an implied negative carry incongruity that models still miss.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Teacher-tier labels could be gathered on a small subset, a cheap model trained to imitate them, and the imitator kept only if it correlates with the gold subset before its silver output feeds training.
  • I2 (learn): Proposed irony or implicature cues could be required to pass per-source significance checks on silver data before being promoted into rules.
  • I3 (learn): Evaluations of silver-trained components could include a "silver-only" arm, so the value of witness-agreed data is measured against gold-only training.
  • I4 (frames): The irony detector could output an explicit link from the sarcastic utterance to the prior-context frame it clashes with, keeping the decision explainable.
  • I5 (frames): A pragmatic rule could compare the polarity of the asserted frame with the situation built from context and flag the mismatch as a candidate irony.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology BombayMumbai · India29 works · 2,104 citations🎓University of SurreyGuildford · United Kingdom21 works · 677 citations🔬IBM Research - IndiaNew Delhi · India9 works · 749 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 works

Cite

Cite as Divyank Tiwari, Diptesh Kanojia, Anupama Ray, Apoorva Nunna and Pushpak Bhattacharyya. 2023. Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.988

@inproceedings{tiwari2023predict,
  title = {{Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection}},
  author = {Divyank Tiwari, Diptesh Kanojia, Anupama Ray, Apoorva Nunna and Pushpak Bhattacharyya},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.988},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.988}
}

References

55 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abulaish and Ashraf Kamal. 2018. Self-deprecating sarcasm detection: An amalgamation of rule-based and machine learning approach. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pages 574–579. find
  2. David Bamman and Noah Smith. 2021. Contextualized sarcasm detection on twitter. Proceedings of the International AAAI Conference on Web and Social Media, 9(1):574–577. find
  3. George Bebis and Michael Georgiopoulos. 1994. Feed-forward neural networks. Ieee Potentials, 13(4):27– 31. ↗
  4. Yevgeni Berzak, Chie Nakamura, Suzanne Flynn, and Boris Katz. 2017. Predicting native language from gaze. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 541–551. find
  5. Aruna Bhat and Aditya Chauhan. 2022. Multimodal sarcasm detection: A survey. In 2022 IEEE Delhi Section Conference (DELCON), pages 1–7. find
  6. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. ↗
  7. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy. Association for Computational Linguistics.
  8. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019a. Towards multimodal sarcasm detection (an _Obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619– 4629, Florence, Italy. Association for Computational Linguistics.
  9. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019b. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy. Association for Computational Linguistics.
  10. Dushyant Singh Chauhan, Gopendra Vikram Singh, Aseem Arora, Asif Ekbal, and Pushpak Bhattacharyya. 2022. An emoji-aware multitask framework for multimodal sarcasm detection. Knowledge-Based Systems, 257:109924. ↗
  11. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning, 20(3):273–297. ↗
  12. Xiao Ding, Bowen Chen, Li Du, Bing Qin, and Ting Liu. 2022. CogBERT: Cognition-guided pre-trained language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3210–3225, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  13. David Freedman, Robert Pisani, and Roger Purves. 2007. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York. find
  14. Raymond W Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of experimental psychology: General, 115(1):3. ↗
  15. Sundesh Gupta, Aditya Shah, Miten Shah, Laribok Syiemlieh, and Chandresh Maurya. 2021. Filming multimodal sarcasm detection with attention. In International Conference on Neural Information Processing, pages 178–186. Springer. find
  16. Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2122–2132, New Orleans, Louisiana. Association for Computational Linguistics.
  17. K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). ↗
  18. Roger A Horn. 1990. The hadamard product. In Proc. Symp. Appl. Math, volume 40, pages 87–169. ↗
  19. Shruti Jadon and Mahmood Jasim. 2019. Video summarization using keyframe extraction and video skimming. arXiv preprint arXiv:1910.04792. ↗
  20. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762, Beijing, China. Association for Computational Linguistics. ↗
  21. Diptesh Kanojia, Prashant Sharma, Sayali Ghodekar, Pushpak Bhattacharyya, Gholamreza Haffari, and Malhar Kulkarni. 2021. Cognition-aware cognate detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3281–3292, Online. Association for Computational Linguistics. ↗
  22. Sigrid Klerke, Héctor Martínez Alonso, and Anders Søgaard. 2015a. Looking hard: Eye tracking for detecting grammaticality of automatically compressed sentences. In Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 2015), pages 97–105. find
  23. Sigrid Klerke, Sheila Castilho, Maria Barrett, and Anders Søgaard. 2015b. Reading metrics for estimating task efficiency with mt output. In Proceedings of the sixth workshop on cognitive aspects of computational language learning, pages 6–13. find
  24. Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. 2016. Improving sentence compression by learning to predict gaze. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1528–1533. find
  25. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. ↗
  26. Paul Pu Liang, Ziyin Liu, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 150– 161. ↗
  27. Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. 2019a. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv:1907.13487. ↗
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach.
  29. Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshmi-narasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2247–2256, Melbourne, Australia. Association for Computational Linguistics. ↗
  30. Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Poria. 2018. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge-based systems, 161:124–133. Computational Linguistics (Volume 1: Long Papers), pages 1095–1104, Berlin, Germany. Association for Computational Linguistics. ↗
  31. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2017b. Scanpath complexity: Modeling reading effort using gaze information. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1). find
  32. Keiron O’Shea and Ryan Nash. 2015. An introduction to convolutional neural networks. ↗
  33. Sandeep Mathias, Diptesh Kanojia, Abhijit Mishra, and Pushpak Bhattacharya. 2020a. A survey on using gaze behaviour for natural language processing. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4907–4913. International Joint Conferences on Artificial Intelligence Organization. Survey track. find
  34. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  35. Sandeep Mathias, Diptesh Kanojia, Kevin Patel, Samarth Agrawal, Abhijit Mishra, and Pushpak Bhattacharyya. 2018. Eyes are the windows to the soul: Predicting the rating of text quality using gaze behaviour. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2352–2362. ↗
  36. Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 973–982. find
  37. Sandeep Mathias, Rudra Murthy, Diptesh Kanojia, and Pushpak Bhattacharyya. 2020b. Cognitively aided zero-shot automatic essay grading. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 175–180, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI). ↗
  38. Brian McFee, Alexandros Metsai, Matt McVicar, Stefan Balke, Carl Thomé, Colin Raffel, Frank Zalkow, Ay-oub Malek, Dana, Kyungyun Lee, Oriol Nieto, Dan Ellis, Jack Mason, Eric Battenberg, Scott Seyfarth, Ryuichi Yamamoto, viktorandreevichmorozov, Ke-unwoo Choi, Josh Moore, Rachel Bittner, Shunsuke Hidaka, Ziyao Wei, nullmightybofo, Adam Weiss, Darío Hereñú, Fabian-Robert Stöter, Pius Friesch, Matt Vollrath, Taewoon Kim, and Thassilo. 2022. librosa/librosa: 0.9.1. find
  39. Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017a. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 377–387.
  40. Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhattacharyya. 2016a. Predicting readers’ sarcasm understandability by modeling gaze behavior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30. find
  41. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2016b. Harnessing cognitive features for sarcasm detection. In Proceedings of the 54th Annual Meeting of the Association for Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. ↗
  42. Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6892–6899. find
  43. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 873–883.
  44. Anupama Ray, Shubham Mishra, Apoorva Nunna, and Pushpak Bhattacharyya. 2022. A multimodal corpus for emotion recognition in sarcasm. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6992–7003, Marseille, France. European Language Resources Association.
  45. Rosa M Sáchez-Casas, José E García-Albea, and Christopher W Davis. 1992. Bilingual lexical processing: Exploring the cognate/non-cognate distinction. European Journal of Cognitive Psychology, 4(4):293–310. ↗
  46. Saurav Sahay, Eda Okur, Shachi H Kumar, and Lama Nachman. 2020. Low rank fusion based transformers for multimodal sequences. In Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML), pages 29–34, Seattle, USA. Association for Computational Linguistics. ↗
  47. Hollis S Scarborough, Susan Neuman, and David Dickinson. 2009. Connecting early language and literacy to later reading (dis) abilities: Evidence, theory, and practice. Approaching difficulties in literacy development: Assessment, pedagogy and programmes, 10:23–38. find
  48. William A Scott. 1955. Reliability of content analysis: The case of nominal scale coding. Public opinion quarterly, pages 321–325. ↗
  49. C Spearman. 1904. nthe proof and measurement of association between two things, oamerican j. find
  50. Harshvardhan Srivastava. 2022. Poirot at CMCL 2022 shared task: Zero shot crosslingual eye-tracking data prediction using multilingual transformer models. In CMCL Shared Task on Multilingual and crosslingual prediction of human reading behavior. ↗
  51. Ece Takmaz. 2022. Team DMG at CMCL 2022 shared task: Transformer adapters for the multi- and cross-lingual prediction of human reading behavior. In CMCL Shared Task on Multilingual and crosslingual prediction of human reading behavior. ↗
  52. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558– 6569, Florence, Italy. Association for Computational Linguistics.
  53. Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. arXiv preprint arXiv:1806.06176. ↗
  54. Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7216–7223. find
  55. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tiwari-2023-predict-use-harnessing-predicted-gaze/