Elamur Research⌕
☰
home / papers / 2018 / Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos

Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos

Summary

The paper tackles utterance-level emotion recognition in videos of two-person (dyadic) conversations. Earlier systems either ignored context or used LSTM-based context models, which summarize long histories poorly and weight all prior turns equally. Drawing on psychology, the authors model two forces: emotional inertia (a speaker's own earlier feelings persisting) and influence from the conversation partner. Their conversational memory network (CMN) represents each utterance with multimodal features (text, audio, visual). For a target utterance, it gathers separate recent histories for each speaker and encodes them with GRUs into memory cells. Attention then picks out the most relevant past turns, and the weighted memories are added to the target representation. This read-and-merge step repeats over several hops before classification. On a dyadic video dataset, CMN beats the bc-LSTM state of the art by 3–4% weighted accuracy. Attention visualizations show it can capture both a speaker's own mood and emotional shifts triggered by the partner. This makes it a useful foundation for empathetic dialogue systems.

Key points

  • Text features: a single-layer CNN with filter sizes 3, 4 and 5 (50 feature maps each), max-pooling, and a 100-neuron dense layer.
  • Audio features: openSMILE with the IS13 ComParE configuration produces 6373 features per utterance. These are z-standardized and projected down to 100 dimensions.
  • Speaker-only variant: CMN self, which ignores inter-speaker relations, still clearly surpasses the previous best methods.
  • Attention ablation: CMN without attention (CMN NA, uniform weights) performs worse than full CMN, confirming that filtering irrelevant history matters.
  • Hyperparameters: accuracy rises with history window K until it saturates, and 3–10 hops beat a single hop. Too many hops cause overfitting. The default settings are K = 20 and Q = 3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

76 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 69 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1193

@inproceedings{hazarika2018conversational,
  title = {{Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos}},
  author = {Hazarika, Devamanyu and Poria, Soujanya and Zadeh, Amir and Cambria, Erik and Morency, Louis-Philippe and Zimmermann, Roger},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1193},
  url = {https://doi.org/10.18653/v1/n18-1193}
}

References

54 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zakaria Aldeneh, Soheil Khorram, Dimitrios Dimitriadis, and Emily Mower Provost. 2017. Pooling acoustic and lexical features for the prediction of valence. find
  2. Noam Amir. 1998. Towards an automatic classification of emotions in speech. ICSLP, pages 699–702. find
  3. Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. 2010. Multimodal fusion for multimedia analysis: a survey. Multimedia systems, 16(6):345–379. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  5. James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281–305. find
  6. Dragos Datcu and L Rothkrantz. 2008. Semantic audiovisual data fusion for automatic emotion recognition. Euromedia’2008. find
  7. Dragoş Datcu and Léon JM Rothkrantz. 2011. Emotion recognition using bimodal data fusion. In International Conference on Computer Systems and Technologies, pages 122–128. ACM. find
  8. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. ICWSM, 13:1–10. find
  9. Frank Dellaert, Thomas Polzin, and Alex Waibel. 1996. Recognizing emotion in speech. In ICSLP, volume 3, pages 1970–1973. find
  10. Sidney K D’mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys (CSUR), 47(3):43. find
  11. Paul Ekman. 1993. Facial expression and emotion. American psychologist, 48(4):384. find
  12. Dario Bertero, Farhad Bin Siddique, Chien-Sheng Wu, Yan Wan, Ricky Ho Yin Chan, and Pascale Fung. 2016. Real-time speech emotion and sentiment recognition for interactive dialogue systems. In EMNLP, pages 1042–1047. find
  13. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In International Conference on Multimedia, pages 1459–1462. ACM. find
  14. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335. find
  15. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  16. Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004. Analysis of emotion recognition using facial expressions, speech and multimodal information. In ICMI, pages 205–211. find
  17. Erik Cambria, Soujanya Poria, Alexander Gelbukh, and Mike Thelwall. 2017. Sentiment analysis is a big suitcase. IEEE Intelligent Systems, 32(6):74–80. find
  18. Erik Cambria, Soujanya Poria, Devamanyu Hazarika, and Kenneth Kwok. 2018. SenticNet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings. In AAAI. find
  19. Ginevra Castellano, Loic Kessous, and George Caridakis. 2008. Emotion recognition through multiple modalities: face, body gesture, speech. Affect and emotion in human-computer interaction, pages 92– 103. find
  20. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  21. Shlomo Hareli and Anat Rafaeli. 2008. Emotion cycles: On the social influence of emotion in organizations. Research in organizational behavior, 28:35– 59. find
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  23. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In ACL 2014, volume 1, pages 655–665. find our card
  24. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP 2014, pages 1746–1751. find our card
  25. Peter Koval and Peter Kuppens. 2012. Changing emotion dynamics: individual differences in the effect of anticipatory social stress on emotional inertia. Emotion, 12(2):256. find
  26. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In ICML, pages 1378–1387. find our card
  27. Feng Liu and Sally Maitlis. 2014. Emotional dynamics and strategizing processes: A study of strategic conversations in top team meetings. Journal of Management Studies, 51(2):202–234. find
  28. Michael W Morris and Dacher Keltner. 2000. How emotions work: The social functions of emotional expression in negotiations. Research in organizational behavior, 22:1–50. find
  29. Costanza Navarretta, K Choukri, T Declerck, S Goggi, M Grobelnik, and B Maegaard. 2016. Mirroring facial expressions and emotions in dyadic conversations. In LREC. find
  30. Rosalind W Picard. 2010. Affective computing: from laughter to ieee. IEEE Transactions on Affective Computing, 1(1):11–17. find
  31. Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017a. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37:98–125. find
  32. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017b. Context-dependent sentiment analysis in user-generated videos. In ACL 2017, volume 1, pages 873–883. find our card
  33. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis-Philippe Morency. 2017c. Multi-level multiple attentions for contextual multimodal sentiment analysis. In ICDM 2017, pages 1033–1038. IEEE. find
  34. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In COLING 2016, pages 1601–1612. find our card
  35. Hiranmayi Ranganathan, Shayok Chakraborty, and Sethuraman Panchanathan. 2016. Multimodal emotion recognition using deep learning architectures. In WACV, pages 1–9. IEEE. find
  36. Jane M Richards, Emily A Butler, and James J Gross. 2003. Emotion regulation in romantic relationships: The cognitive consequences of concealing feelings. Journal of Social and Personal Relationships, 20(5):599–620. find
  37. Andrew Rosenberg. 2012. Classifying skewed data: Importance weighting to optimize average recall. In INTERSPEECH 2012. find
  38. Viktor Rozgic, Sankaranarayanan Ananthakrishnan, Shirin Saleem, Rohit Kumar, and Rohit Prasad. 2012. Ensemble of svm trees for multimodal emotion recognition. In APSIPA ASC, pages 1–4. find
  39. Johanna Ruusuvuori. 2013. Emotion, affect and conversation. The handbook of conversation analysis, pages 330–349. find
  40. Jack Sidnell and Tanya Stivers. 2012. The handbook of conversation analysis, volume 121. John Wiley & Sons. find
  41. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In NIPS 2015, pages 2440–2448. find
  42. Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In ICCV 2015, pages 4489–4497. find
  43. Panagiotis Tzirakis, George Trigeorgis, Mihalis A Nicolaou, Björn W Schuller, and Stefanos Zafeiriou. 2017. End-to-end multimodal emotion recognition using deep neural networks. IEEE JSTSP, 11(8):1301–1309. find
  44. Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2017. Select-additive learning: Improving generalization in multimodal sentiment analysis. In ICME, pages 949–954. find
  45. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916. find
  46. Martin Wöllmer, Angeliki Metallinou, Florian Eyben, Björn Schuller, and Shrikanth S Narayanan. 2010. Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling. In INTERSPEECH 2010. find
  47. Peng Xiaolan, Xie Lun, Liu Xin, and Wang Zhiliang. 2013. Emotional state transition model based on stimulus and personality characteristics. China Communications, 10(6):146–155. find
  48. Frank Xing, Erik Cambria, and Roy Welsch. 2018. Natural language based financial forecasting: A survey. Artificial Intelligence Review. find
  49. Liang Yang, Hong-fei LIN, and Wei GUO. 2011. Text-based emotion transformation analysis. Computer Engineering & Science, 9:026. find
  50. Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Augmenting end-to-end dialog systems with commonsense knowledge. In AAAI. find
  51. Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2017. Recent trends in deep learning based natural language processing. arXiv preprint arXiv:1708.02709. — [arXiv:1708.02709](https://arxiv.org/abs/1708.02709) find
  52. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In EMNLP 2017, pages 1103–1114. find our card
  53. Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2017. Emotional chatting machine: Emotional conversation generation with internal and external memory. arXiv:1704.01074. — [arXiv:1704.01074](https://arxiv.org/abs/1704.01074) find
  54. Mingli Song, Jiajun Bu, Chun Chen, and Nan Li. 2004. Audio-visual based emotion recognition-a new approach. In CVPR, volume 2. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hazarika-2018-conversational-memory-network-emotion-recognition/