Elamur Research⌕
☰
home / papers / 2017 / Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network

Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network

Summary

The paper addresses sentiment polarity and sarcasm classification of short texts. These are hard cases because the meaning often depends on semantic or pragmatic cues that the words themselves do not make explicit. Earlier cognitive NLP systems added eye-tracking information from human readers, but they relied on gaze features designed by hand. Such features are hard to engineer and may miss subtle textual effects.

The authors instead let a Convolutional Neural Network learn features directly from both text and gaze data. The network has two components:

  • Text component: follows Kim (2014), with static, non-static or multi-channel word embeddings.
  • Gaze component: applies 2-D convolution over participant-by-sequence matrices that encode fixation durations and fixation positions (saccades).

The pooled outputs of both components are merged and passed to a softmax classifier. The model was tested on two public datasets enriched with gaze data. Combining learned text and gaze features often beat both text-only CNNs and earlier systems built on handcrafted gaze and text features. This suggests that readers' eye movements carry useful signal for subtle classification tasks.

Key points

  • Gaze channels: the gaze component uses two input channels, FIXATION (min-max normalized durations) and SACCADE (word-id sequence of fixations). These are used alone or together as MULTI-CHANNEL GAZE, with each input shaped as a P × G matrix (participants × sequence length).
  • Text variants: STATIC TEXT, NON-STATIC TEXT and MULTICHANNEL TEXT were tested. Max pooling outperformed mean pooling.
  • Embedding size: dimensions between 50 and 300 were tried. Smaller embeddings gave slightly higher F-scores, and text+gaze consistently beat text-only across all sizes.
  • Channel effects: for sentiment, the saccade channel helped more than the fixation channel. For sarcasm, the two channels worked about equally well when used separately, but combining them lowered accuracy.
  • Learned features and resources: visualizations of the 150-dimensional merge-layer features show that adding gaze separates sarcastic from non-sarcastic texts more clearly. Datasets and resources are available at cfilt.iitb.ac.in/cognitive-nlp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

26 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 21 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1035

@inproceedings{mishra2017learning,
  title = {{Learning Cognitive Features from Gaze Data for Sentiment and Sarcasm Classification using Convolutional Neural Network}},
  author = {Mishra, Abhijit and Dey, Kuntal and Bhattacharyya, Pushpak},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1035},
  url = {https://doi.org/10.18653/v1/p17-1035}
}

References

45 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sentiment and sarcasm classification are two important problems in NLP and have been the focus of research for many communities for quite some time. Popular sentiment and sarcasm detection systems are feature based and are based on unigrams, bigrams etc. (Dave et al., 2003; Ng et al., 2006), syntactic properties (Martineau and Finin, 2009; Nakagawa et al., 2010), semantic properties (Balamurali et al., 2011). For sarcasm detection, supervised approaches rely on (a) Unigrams and Pragmatic features (González-Ibánez et al., 2011; Barbieri et al., 2014; Joshi et al., 2015) (b) Stylistic patterns (Davidov et al., 2010) and patterns related to situational disparity (Riloff et al., 2013) and (c) Hastag interpretations (Liebrecht et al., 2013; Maynard and Greenwood, 2014). Recent systems are based on variants of deep neural network built on the top of embeddings. A few representative works in this direction for sentiment analysis are based on CNNs (dos Santos and Gatti, 2014; Kim, 2014; Tang et al., 2014), RNNs (Dong et al., 2014; Liu et al., 2015) and combined archi-## References tecture (Wang et al., 2016). Few works exist on using deep neural networks for sarcasm detection, one of which is by (Ghosh and Veale, 2016) that uses a combination of RNNs and CNNs. Eye-tracking technology is a relatively new NLP, with very few systems directly making use of gaze data in prediction frameworks. find
  2. Klerke et al. (2016) present a novel multi-task learning approach for sentence compression using labeled data, while, Barrett and Søgaard (2015) discriminate between grammatical functions using gaze features. The closest works to ours are by Mishra et al. (2016b) and Mishra et al. (2016c) that introduce feature engineering based on both gaze and text data for sentiment and sarcasm detection tasks. These recent advancements motivate us to explore the cognitive NLP paradigm. find
  3. Cem Akkaya, Janyce Wiebe, and Rada Mihalcea. 2009. Subjectivity word sense disambiguation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1. ACL, pages 190–199. find
  4. AR Balamurali, Aditya Joshi, and Pushpak Bhattacharyya. 2011. Harnessing wordnet senses for supervised sentiment classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pages 1081–1091. find
  5. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Modelling sarcasm in twitter, a novel approach. ACL 2014 page 50. find our card
  6. Maria Barrett and Anders Søgaard. 2015. Using reading behavior to predict grammatical functions. In Proceedings of the Sixth Workshop on Cognitive Aspects of Computational Language Learning. Association for Computational Linguistics, Lisbon, Portugal, pages 1–5. Conclusion and Future Directions In this work, we proposed a multimodal ensemble of features, automatically learned using variants of CNNs from text and readers’ eye-movement data, for the tasks of sentiment and sarcasm classification. On multiple published datasets for which gaze information is available, our systems could often achieve significant performance improvements over (a) systems that rely on handcrafted gaze and textual features and (b) CNN based systems that rely on text input alone. An analysis of the learned features confirms that the combination of automatically learned features is indeed capable of representing deep linguistic subtleties in text that pose challenges to sentiment and sarcasm classifiers. Our future agenda includes: (a) optimizing the CNN framework hyper-parameters (e.g., filter width, dropout, embedding dimensions, etc.) to obtain better results, (b) exploring the applicability of our technique for document-level sentiment analysis and (c) applying our framework to related problems, such as emotion analysis, text summarization, and question-answering, where considering textual clues alone may not prove to be sufficient. find
  7. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12(Aug):2493–2537. find our card
  8. Kushal Dave, Steve Lawrence, and David M Pennock. 2003. Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of the 12th international conference on World Wide Web. ACM, pages 519–528. find
  9. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning. Association for Computational Linguistics, pages 107–116. find our card
  10. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In ACL (2). pages 49–54. find
  11. Cı́cero Nogueira dos Santos and Maira Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of COLING. find
  12. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of NAACL-HLT. pages 161–169. ↗ our card
  13. Justin Martineau and Tim Finin. 2009. Delta tfidf: An improved feature space for sentiment analysis. ICWSM 9:106. find
  14. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing coadaptation of feature detectors. arXiv preprint arXiv:1207.0580 . ↗
  15. Diana Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of LREC. find our card
  16. Aditya Joshi, Abhijit Mishra, Nivvedan Senthamilsel-van, and Pushpak Bhattacharyya. 2014. Measuring sentiment annotation complexity of text. In ACL (Daniel Marcu 22 June 2014 to 27 June 2014). ACL. find
  17. Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. 2016. The role of context types and dimensionality in learning word embeddings. In NAACL HLT 2016. pages 1030–1040. find our card
  18. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. Proceedings of 53rd Annual Meeting of the Association for Computational Linguistics, Beijing, China page 757. find our card
  19. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In HLT-NAACL. volume 13, pages 746–751. find
  20. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1746– 1751. find our card
  21. Abhijit Mishra, Diptesh Kanojia, and Pushpak Bhattacharyya. 2016a. Predicting readers’ sarcasm understandability by modeling gaze behavior. In Proceedings of AAAI. find
  22. Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. 2016. Improving sentence compression by learning to predict gaze. arXiv preprint arXiv:1604.03357 . ↗
  23. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2016b. Harnessing cognitive features for sarcasm detection. ACL 2016 page 156. ↗ our card
  24. Reinhold Kliegl, Ellen Grabner, Martin Rolfs, and Ralf Engbert. 2004. Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cognitive Psychology 16(1-2):262–284. ↗
  25. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2016c. Leveraging cognitive features for sentiment analysis. CoNLL 2016 page 156. find
  26. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. pages 1097–1105. find
  27. Tetsuji Nakagawa, Kentaro Inui, and Sadao Kurohashi. 2010. Dependency tree-based sentiment classification using crfs with hidden variables. In NAACL-HLT. Association for Computational Linguistics, pages 786–794. find
  28. Marta Kutas and Steven A Hillyard. 1980. Reading senseless sentences: Brain potentials reflect semantic incongruity. Science 207(4427):203–205. ↗
  29. Vincent Ng, Sajib Dasgupta, and SM Arifin. 2006. Examining the role of linguistic knowledge sources in the automatic identification and classification of reviews. In Proceedings of the COLING/ACL on Main conference poster sessions. Association for Computational Linguistics, pages 611–618. find
  30. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. In Proceedings of NAACL-HLT. pages 681– 691. find our card
  31. Christine Liebrecht, Florian Kunneman, and Antal van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets# not. WASSA 2013 page 29. find our card
  32. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 115–124. ↗
  33. Bing Liu and Lei Zhang. 2012. A survey of opinion mining and sentiment analysis. In Mining text data, Springer, pages 415–463. ↗
  34. Soujanya Poria, Erik Cambria, Gregoire Winterstein, and Guang-Bin Huang. 2014. Sentic patterns: Dependency-based rules for concept-level sentiment analysis. Knowledge-Based Systems 69:45–63. ↗
  35. Pengfei Liu, Shafiq R Joty, and Helen M Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In EMNLP. pages 1433–1443. find
  36. Keith Rayner and Susan A Duffy. 1986. Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition 14(3):191–201. ↗
  37. Titus Malsburg, Reinhold Kliegl, and Shravan Vasishth. 2015. Determinants of scanpath regularity in reading. Cognitive science 39(7):1675–1703. ↗
  38. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of Empirical Methods in Natural Language Processing. pages 704–714. find our card
  39. Raksha Sharma and Pushpak Bhattacharyya. 2013. Detecting domain dedicated polar words. In Proceedings of the International Joint Conference on Natural Language Processing. find
  40. Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 1701–1708. find
  41. Duyu Tang, Furu Wei, Bing Qin, Ting Liu, and Ming Zhou. 2014. Coooolll: A deep learning system for twitter sentiment classification. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). pages 208–212. find
  42. Jin Wang, Liang-Chih Yu, K. Robert Lai, and Xuejie Zhang. 2016. Dimensional sentiment analysis using a regional cnn-lstm model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 225–230. find
  43. Erroll Wood and Andreas Bulling. 2014. Eyetab: Model-based gaze estimation on unmodified tablet computers. In Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, pages 207–210. find
  44. Michiya Yamamoto, Hironobu Nakagawa, Koichi Egawa, and Takashi Nagamatsu. 2013. Development of a mobile tablet pc with gaze-tracking function. In Human Interface and the Management of Information. Information and Interaction for Health, Safety, Mobility and Complex Environments, Springer, pages 421–429. find
  45. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 . ↗ our card

↑ back to top

This page in Markdown →