Elamur Research
☰
home / papers / 2018 / MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations

MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1810.02508

Summary

Research on emotion recognition in conversations (ERC) has been held back by the lack of a large multimodal corpus with more than two speakers per dialogue. Existing resources are either small and dyadic (IEMOCAP, SEMAINE) or text-only (EmotionLines). The authors build MELD from EmotionLines, which draws on the sitcom Friends. They locate each utterance's timestamps in episode subtitles and discard dialogues that span several scenes or episodes. They then cut the matching video clips and extract the audio. Three annotators relabel every utterance while watching the clips, and the final label is chosen by majority vote. Each utterance has one of seven emotion labels and a sentiment label. The paper benchmarks text-CNN, bcLSTM and DialogueRNN using text and audio features. Speaker-aware modelling and multimodal fusion both improve results, although audio features add only a small gain. MELD therefore gives a multi-party, multimodal testbed for context modelling, emotion-shift detection and empathetic dialogue systems.

Key points

  • MELD contains 1,433 dialogues (1,039 train, 114 dev, 280 test) and about 13.7k utterances (9,989 / 1,109 / 2,610). Each utterance comes with video, audio (16-bit PCM WAV) and transcript.
  • The seven emotion classes cover joy, surprise, sadness, fear, anger and disgust, along with a neutral class, and each utterance also carries a positive/negative/neutral sentiment label. The vague "non-neutral" label from EmotionLines was dropped.
  • Watching the clips raised inter-annotator agreement to a Fleiss' kappa of 0.43. This compares with 0.34 for EmotionLines and 0.4 for IEMOCAP. Only 89 utterances, spread over 11 dialogues, had no majority label and were removed.
  • DialogueRNN beats bcLSTM by 1–2%, and simple concatenation fusion beats unimodal models by roughly 1–3%. The best weighted F-score for sentiment is 67.56, from DialogueRNN with text and audio.
  • In 63% of DialogueRNN's correct predictions, the most-attended previous utterance came from a different speaker. Emotion shifts are detected with 66% recall, which falls to 36.7% when the new emotion must also be classified correctly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

335 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 328 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1810.02508 · arXiv 1810.02508

@misc{poria2018meld,
  title = {{MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations}},
  author = {Poria, Soujanya and Hazarika, Devamanyu and Majumder, Navonil and Naik, Gautam and Cambria, Erik and Mihalcea, Rada},
  year = {2018},
  doi = {10.48550/arxiv.1810.02508},
  eprint = {1810.02508},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1810.02508}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry, 25(1):49–59. find
  2. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  3. Sheng-Yeh Chen, Chao-Chun Hsu, Chuan-Chun Kuo, Lun-Wei Ku, et al. 2018. Emotionlines: An emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379. find
  4. Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR, abs/1412.3555. find
  5. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462. ACM. find
  6. Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. In NAACL, volume 1, pages 2122–2132. find
  7. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746– 1751. find our card
  8. Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In ACL (1), pages 973– 982. find
  9. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In ACL, pages 873–883. find
  10. Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. arXiv preprint arXiv:1905.02947. — [arXiv:1905.02947](https://arxiv.org/abs/1905.02947) find
  11. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. I know the feeling: Learning to converse with empathy. arXiv preprint arXiv:1811.00207. find
  12. Björn Schuller, Michel Valster, Florian Eyben, Roddy Cowie, and Maja Pantic. 2012. find
  13. Avec 2012: the continuous audio/visual emotion challenge. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 449–456. ACM. find
  14. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI, pages 3295–3301. find our card
  15. Martin Wollmer, Felix Weninger, Timo Knaup, Bjorn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):46–53. find
  16. Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. In ACL, volume 1, pages 994–1003. find our card
  17. Amir Zadeh, Tadas Baltrušaitis, and Louis-Philippe Morency. 2016a. Deep constrained local models for facial landmark detection. arXiv preprint arXiv:1611.08657. — [arXiv:1611.08657](https://arxiv.org/abs/1611.08657) find
  18. Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. DialogueRNN: An attentive RNN for emotion detection in conversations. Thirty-Third AAAI Conference on Artificial Intelligence. find
  19. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL, volume 1, pages 2236–2246. find our card
  20. Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder. 2012. The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent. Affective Computing, IEEE Transactions on, 3(1):5–17. find
  21. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016b. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82–88. find
  22. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532–1543. find
  23. Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2017. Emotional chatting machine: Emotional conversation generation with internal and external memory. arXiv preprint arXiv:1704.01074. — [arXiv:1704.01074](https://arxiv.org/abs/1704.01074) find