Elamur Research⌕
☰
home / papers / 2018 / ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection

ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection

Summary

The paper tackles emotion recognition in dyadic conversational videos. The authors argue that earlier systems mostly tracked how a speaker's own previous utterances shape their current emotion ("emotional inertia") and ignored how the other participant affects them. Their model, ICON, first builds a multimodal vector for every utterance from text, audio and video. For a target utterance, it takes each speaker's recent utterances within a context window and encodes them with speaker-specific local GRUs. A global GRU then merges these outputs into memory cells that reflect interaction between speakers. Several attention-based read/write passes ("hops") over these memories produce a context summary, which repeatedly refines the target utterance's representation before the final prediction. On two benchmark datasets, including IEMOCAP, ICON beat prior state-of-the-art systems on both categorical and continuous emotion tasks. The work matters because it models interpersonal emotional dynamics explicitly, its attention weights are interpretable, and it is a step toward empathetic dialogue agents.

Key points

  • Architecture has four stages: extracting features from multiple modalities, a component capturing each speaker's influence on themselves, a dynamic component capturing global inter-speaker influence (DGIM), and a multi-hop memory.
  • Features per modality:
    • Text: a Kim-style CNN over 300-dimensional FastText embeddings.
    • Audio: openSMILE with the IS13 ComParE extractor, giving 6373 features that are reduced by L2-based selection.
    • Visual: a 3D-CNN for spatiotemporal facial cues.
  • Ablation results: removing multi-hop attention (a single hop) hurts more than dropping DGIM, and the full model performs best.
  • Attention analysis on correctly classified test cases:
    • In about 63% of cases, at least one top-5 attention weight falls on the 10 most recent utterances.
    • About 22% of top-attention weights go to utterances more than 30 steps back, indicating long-range emotional dependencies.
  • ICON extends the authors' earlier model, which used separate memory networks per speaker. ICON instead uses a shared interactive scheme with fewer trainable parameters and can, in principle, be extended to multi-party dialogue.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

71 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 69 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1280

@inproceedings{hazarika2018icon,
  title = {{ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection}},
  author = {Hazarika, Devamanyu and Poria, Soujanya and Mihalcea, Rada and Cambria, Erik and Zimmermann, Roger},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1280},
  url = {https://doi.org/10.18653/v1/d18-1280}
}

References

48 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pages 579–586. Association for Computational Linguis… find our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  3. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. find
  5. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335. find
  6. Tanya L Chartrand and John A Bargh. 1999. The chameleon effect: the perception–behavior link and social interaction. Journal of personality and social psychology, 76(6):893. find
  7. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462. ACM. find
  8. Reinhard Fiehler. 2002. How to do emotions with words: Emotionality in conversations. The verbal communication of emotions, pages 79–106. find
  9. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  10. Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… find our card
  11. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In ACL 2014, volume 1, pages 655–665. find our card
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP 2014, pages 1746–1751. find our card
  13. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  14. Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Bal-trušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pages 163–171. ACM. find
  15. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International Conference on Machine Learning, pages 1378–1387. find our card
  16. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  17. Peter Kuppens, Nicholas B Allen, and Lisa B Sheeber. 2010. Emotional inertia and psychological malad-justment. Psychological Science, 21(7):984–991. find
  18. Dragos Datcu and L Rothkrantz. 2008. Semantic audiovisual data fusion for automatic emotion recognition. Euromedia’2008. find
  19. Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder. 2012. The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Transactions on Affective Computing, 3(1):5–17. find
  20. Liyanage C De Silva and Pei Chi Ng. 2000. Bimodal emotion recognition. In Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on, pages 332–335. IEEE. find
  21. Sidney K D’mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys (CSUR), 47(3):43. find
  22. Paul Ekman. 1993. Facial expression and emotion. American psychologist, 48(4):384. find
  23. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, 521(7553):436. find
  24. Michael W Morris and Dacher Keltner. 2000. How emotions work: The social functions of emotional expression in negotiations. Research in organizational behavior, 22:1–50. find
  25. Costanza Navarretta, K Choukri, T Declerck, S Goggi, M Grobelnik, and B Maegaard. 2016. Mirroring facial expressions and emotions in dyadic conversations. In LREC. find
  26. Jérémie Nicolle, Vincent Rapp, Kévin Bailly, Lionel Prevost, and Mohamed Chetouani. 2012. Robust continuous prediction of human emotions using multiscale dynamic cues. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 501–508. ACM. find
  27. Rosalind W Picard. 2010. Affective computing: from laughter to ieee. IEEE Transactions on Affective Computing, 1(1):11–17. find
  28. Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017a. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37:98–125. find
  29. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017b. Context-dependent sentiment analysis in user-generated videos. In ACL 2017, volume 1, pages 873–883. find our card
  30. Jane M Richards, Emily A Butler, and James J Gross. 2003. Emotion regulation in romantic relationships: The cognitive consequences of concealing feelings. Journal of Social and Personal Relationships, 20(5):599–620. find
  31. Johanna Ruusuvuori. 2013. Emotion, affect and conversation. The handbook of conversation analysis, pages 330–349. find
  32. Björn Schuller, Michel Valster, Florian Eyben, Roddy Cowie, and Maja Pantic. 2012. find
  33. Avec 2012: the continuous audio/visual emotion challenge. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 449–456. ACM. find
  34. Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an integrated path-based and distributional method. arXiv preprint arXiv:1603.06076. find our card
  35. Mohammad Soleymani, Maja Pantic, and Thierry Pun. 2012. Multimodal emotion recognition in response to videos. IEEE transactions on affective computing, 3(2):211–223. find
  36. Carlo Strapparava and Rada Mihalcea. 2010. Annotating and identifying emotions in text. In Intelligent Information Access, pages 21–38. Springer. find
  37. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In NIPS 2015, pages 2440–2448. find
  38. Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In ICCV 2015, pages 4489–4497. find
  39. Panagiotis Tzirakis, George Trigeorgis, Mihalis A Nicolaou, Björn W Schuller, and Stefanos Zafeiriou. 2017. End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of Selected Topics in Signal Processing, 11(8):1301– 1309. find
  40. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916. find
  41. Peng Xiaolan, Xie Lun, Liu Xin, and Wang Zhiliang. 2013. Emotional state transition model based on stimulus and personality characteristics. China Communications, 10(6):146–155. find
  42. Liang Yang, Hong-fei LIN, and Wei GUO. 2011. Text-based emotion transformation analysis. Computer Engineering & Science, 9:026. find
  43. Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Augmenting end-to-end dialogue systems with commonsense knowledge. In AAAI, pages 4970–4977. find
  44. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114. find our card
  45. Mingli Song, Jiajun Bu, Chun Chen, and Nan Li. 2004. Audio-visual based emotion recognition-a new approach. In CVPR 2004, volume 2, pages II–II. IEEE. find
  46. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. In AAAI, pages 5634– 5641. find
  47. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  48. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018b. Multi-attention recurrent network for human communication comprehension. In AAAI, pages 5642–5649. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hazarika-2018-icon-interactive-conversational-memory-network/