Elamur Research⌕
☰
home / papers / 2020 / Towards Emotion-aided Multi-modal Dialogue Act Classification

Towards Emotion-aided Multi-modal Dialogue Act Classification

Summary

Dialogue Act Classification (DAC) labels the communicative intent of an utterance. Earlier work mostly relied on text alone and ignored how the speaker feels. This ACL 2020 paper argues that tone of voice, facial expression and emotional state all signal intent. A sarcastic "yes", for instance, can actually express disagreement. To explore this, the authors built EMOTyDA, a video dialogue resource. They took conversations from MELD and IEMOCAP, which already carry emotion labels, and manually added dialogue act tags based on the SWBD-DAMSL scheme. They then trained a deep multi-modal, multi-task network that learns DAC jointly with Emotion Recognition (ER) as a helper task. The network applies self-attention, inter-modal attention and inter-task attention over text, audio and video features. Adding non-textual modalities beat text-only models, and text plus visual input gave the best scores. Joint training with ER also outperformed single-task DAC, including a variant that receives emotion as an input feature. The work supplies a resource and baseline for emotion-aware, multi-modal intent detection in open-domain conversation.

Key points

  • EMOTyDA corpus: 1341 dyadic and multi-party dialogues, totalling 19,365 annotated utterance videos (about 22 hours). It combines 1039 MELD dialogues (9989 utterances) with all 302 IEMOCAP dialogues (9376 utterances). Each utterance includes text, audio, video, speaker ID and preceding context.
  • Tag sets: 12 dialogue act tags were chosen as the most frequent of the 42 SWBD-DAMSL labels, such as Greeting, Question, Statement-Opinion, Agreement, Disagreement and Backchannel. Emotions were harmonised to 10 IEMOCAP-style labels, with MELD "joy" mapped to "happy".
  • Annotation: three annotators trained in English linguistics labelled each utterance from the video and dialogue history. They did not see the emotion tags, which avoided bias toward particular DA–emotion pairs. Agreement above 80% was treated as reliable.
  • Modality results: text was the strongest single modality, and T+V was the best combination overall. The tri-modal model improved on all uni-modal baselines but fell short of T+V, apparently because of weak audio features. Dyadic dialogues were classified more accurately than multi-party ones.
  • Ablation and error analysis: using all three attention types (SA, IMA, ITA) together gave the best results. Remaining errors were linked to skewed label distribution, long utterances carrying multiple intents, and misrecognised emotions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Speech act theory

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.402

@inproceedings{saha2020emotion,
  title = {{Towards Emotion-aided Multi-modal Dialogue Act Classification}},
  author = {Saha, Tulika and Patra, Aditya and Saha, Sriparna and Bhattacharyya, Pushpak},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.402},
  url = {https://doi.org/10.18653/v1/2020.acl-main.402}
}

References

52 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The hcrc map task corpus. Language and speech, 34(4):351–366. Lisa Feldman. ↗
  2. Barrett, Michael Lewis, and Jeannette M. Haviland-Jones. 1993. Handbook of emotions. The Guilford Press. find
  3. Wauter Bosma and Elisabeth André. 2004. Exploiting emotions to disambiguate dialogue acts. In Proceedings of the 9th international conference on Intelligent user interfaces, pages 85–92. ACM. find
  4. Kristy Elizabeth Boyer, Joseph F Grafsgaard, Eun Young Ha, Robert Phillips, and James C Lester. 2011. An affect-enriched dialogue act classification model for task-oriented dialogue. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1190– 1199. Association for Computational Linguistics. find our card
  5. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335. ↗
  6. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4619–4629. find our card
  7. Roddy Cowie, Ellen Douglas-Cowie, Nicolas Tsapat-soulis, George Votsis, Stefanos Kollias, Winfried Fellenz, and John G Taylor. 2001. Emotion recognition in human-computer interaction. IEEE Signal processing magazine, 18(1):32–80. ↗
  8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. find
  9. Thomas Drugman and Abeer Alwan. 2011. Joint robust voicing detection and pitch estimation based on residual harmonics. In Twelfth Annual Conference of the International Speech Communication Association. find
  10. Thomas Drugman, Mark Thomas, Jon Gudnason, Patrick Naylor, and Thierry Dutoit. 2011. Detection of glottal closure instants from speech signals: A quantitative review. IEEE Transactions on Audio, Speech, and Language Processing, 20(3):994–1006. find
  11. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462. ACM. find
  12. Pascale Fung, Dario Bertero, Peng Xu, Ji Ho Park, Chien-Sheng Wu, and Andrea Madotto. 2018. Empathetic dialog systems. In The International Conference on Language Resources and Evaluation. European Language Resources Association. find
  13. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, volume 1, pages 517–520. IEEE. find
  14. Sergio Grau, Emilio Sanchis, Maria Jose Castro, and David Vilar. 2004. Dialogue act classification using a bayesian approach. In 9th Conference Speech and Computer. find
  15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. ↗
  16. Peter A Heeman and James F Allen. 1995. The trains 93 dialogues. Technical report, ROCHESTER UNIV NY DEPT OF COMPUTER SCIENCE. ↗
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  18. Chenyang Huang, Osmar Zaiane, Amine Trabelsi, and Nouha Dziri. 2018. Automatic dialogue generation with expressed emotions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 49–54. find
  19. Peter Lajos Ihasz and Victor Kryssanov. 2018. Emotions and intentions mediated with dialogue acts. In 2018 5th International Conference on Business and Industrial Research (ICBIR), pages 125–130. IEEE. find
  20. Anurag Illendula and Amit Sheth. 2019. Multimodal emotion classification. In Companion Proceedings of The 2019 World Wide Web Conference, pages 439–449. find
  21. Neha Jain, Shishir Kumar, Amit Kumar, Pourya Sham-solmoali, and Masoumeh Zareapoor. 2018. Hybrid deep neural networks for face emotion recognition. Pattern Recognition Letters, 115:101–106. ↗
  22. Dan Jurafsky. 1997. Switchboard swbd-damsl shallow-discourse-function annotation coders manual. Institute of Cognitive Science Technical Report. find
  23. Daniel Jurafsky, Rebecca Bates, Noah Coccaro, Rachel Martin, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Andreas Stolcke, Paul Taylor, and Carol Van Ess-Dykema. 1997. Automatic detection of discourse structure for speech recognition and understanding. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 88–95. IEEE. ↗
  24. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. arXiv preprint arXiv:1306.3584. ↗ our card
  25. John Kane and Christer Gobl. 2013. Wavelet maxima dispersion for breathy to tense voice discrimination. IEEE Transactions on Audio, Speech, and Language Processing, 21(6):1170–1179. ↗
  26. Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2012– 2021. find our card
  27. Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with crf. ↗ our card
  28. Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in dnn framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170–2178. find our card
  29. Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: harvesting opinions from the web. In Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI 2011, Alicante, Spain, November 14-18, 2011, pages 169–176. find
  30. Nicole Novielli and Carlo Strapparava. 2013. The role of affect analysis in dialogue act identification. IEEE Transactions on Affective Computing, 4(4):439–451. ↗
  31. Daniel Ortega, Chia-Yu Li, Gisela Vallejo, Pavel Denisov, and Ngoc Thang Vu. 2019. Context-aware neural-based dialog act classification on automatically generated transcriptions. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7265–7269. IEEE. find
  32. Pinelopi Papalampidi, Elias Iosif, and Alexandros Potamianos. 2017. Dialogue act semantic representation and classification using recurrent neural networks. SEMDIAL 2017 SaarDial, page 104. find
  33. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  34. Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pages 973–982. find
  35. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017a. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–883. find
  36. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis-Philippe Morency. 2017b. Multi-level multiple attentions for contextual multimodal sentiment analysis. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1033–1038. IEEE. find
  37. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 527–536. find our card
  38. Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. arXiv preprint arXiv:1904.02594. ↗ our card
  39. Norbert Reithinger and Martin Klesen. 1997. Dialogue act classification using language models. In Fifth European Conference on Speech Communication and Technology. ↗
  40. Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2019. A multilingual and multidomain study on dialog act recognition using character-level tokenization. Information, 10(3):94. ↗
  41. Tulika Saha, Saurabh Srivastava, Mauajama Firdaus, Sriparna Saha, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Exploring machine learning and deep learning frameworks for task-oriented dialogue act classification. pages 1–8. find
  42. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The icsi meeting recorder dialog act (mrda) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004. ↗ our card
  43. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373. ↗ our card
  44. Andreas Stolcke, Elizabeth Shriberg, Rebecca Bates, Noah Coccaro, Daniel Jurafsky, Rachel Martin, Marie Meteer, Klaus Ries, Paul Taylor, Carol Van Ess-Dykema, et al. 1998. Dialog act modeling for conversational speech. In AAAI Spring Symposium on Applying Machine Learning to Discourse Processing, pages 98–105. ↗
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998–6008. find
  46. Daan Verbree, Rutger Rienks, and Dirk Heylen. 2006. Dialogue-act tagging using smart feature selection; results on multiple corpora. In Spoken Language Technology Workshop, 2006. IEEE, pages 70–73. IEEE. amounting to a total of 13708 utterances across the dataset. Each utterance is annotated for the presence of 7 emotions namely sadness, anger, fear, joy, surprise, disgust, and neutral. find
  47. Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn W. Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):46–53. ↗
  48. Yue Yu, Siyao Peng, and Grace Hui Yang. 2019. Modeling long-range context for concurrent dialogue acts recognition. arXiv preprint arXiv:1909.00521. ↗
  49. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2236–2246. find our card
  50. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82–88. ↗
  51. Shiqing Zhang, Shiliang Zhang, Tiejun Huang, Wen Gao, and Qi Tian. 2018. Learning affective features with a hybrid deep model for audio–visual emotion recognition. IEEE Transactions on Circuits and Systems for Video Technology, 28(10):3030–3043. ↗
  52. Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Thirty-Second AAAI Conference on Artificial Intelligence. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saha-2020-emotion-aided-multi-modal-dialogue/