Elamur Research
☰
home / papers / 2018 / Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1208.pdf

Summary

Studying how people combine words, facial expressions and voice requires large, varied multimodal data, but earlier corpora were small and covered few speakers, topics and label types. The authors address this with CMU-MOSEI, a collection of YouTube monologue videos annotated for both sentiment and emotion. Videos were selected automatically: face detection confirmed a single speaker, moving-camera recordings were rejected, and each channel could contribute only a limited number of videos. Transcripts are manually written and aligned with the audio down to phonemes. The second contribution is the Dynamic Fusion Graph (DFG). This fusion module links unimodal, bimodal and trimodal representations through learned "efficacies," which make the fusion process inspectable. Inside the Memory Fusion Network pipeline, DFG beats previous models on sentiment and is competitive on emotion recognition. Its efficacies show that language and audio are fused consistently, while vision is used only when it carries useful signal. The release gives NLP researchers a large, freely available benchmark for multimodal affect analysis.

Key points

  • CMU-MOSEI comprises 23,453 labeled segments of video drawn from 1,000 different speakers (proxied by YouTube channels, with at most 10 videos each) and 250 topics. Its total running time is about 65 hours and 54 minutes.
  • In the authors' comparison table, it is the largest sentiment/emotion dataset that includes all three modalities (language, vision, audio) and both label types. Its predecessor, CMU-MOSI, has 2,199 clips from 98 speakers.
  • The topic distribution is fairly flat: reviews make up 16.2%, debate 2.9% and consulting 1.8%, with the remaining topics spread nearly evenly. The data is distributed through the CMU Multimodal Data SDK on GitHub.
  • DFG's interpretable efficacies show three recurring patterns:
    • language→(language, audio) links are consistently strong;
    • single-modality links to the final output are weak;
    • visual paths switch on only when the visual signal is informative.
  • Baselines include MFN, MARN, TFN, MV-LSTM, EF-LSTM, BC-LSTM and several language-only and acoustic-only models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

164 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 162 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1208

@inproceedings{zadeh2018multimodal,
  title = {{Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph}},
  author = {Zadeh, AmirAli Bagher and Liang, Paul Pu and Poria, Soujanya and Cambria, Erik and Morency, Louis-Philippe},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1208},
  url = {https://doi.org/10.18653/v1/p18-1208}
}

References

60 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paavo Alku, Tom Bäckström, and Erkki Vilkman. 2002. Normalized amplitude quotient for parametrization of the glottal flow. the Journal of the Acoustical Society of America 112(2):701–710. find
  2. Paavo Alku, Helmer Strik, and Erkki Vilkman. 1997. Parabolic spectral parameter—a new method for quantification of the glottal flow. Speech Communication 22(1):67–79. find
  3. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2017. Multimodal machine learning: A survey and taxonomy. arXiv preprint arXiv:1705.09406 . — [arXiv:1705.09406](https://arxiv.org/abs/1705.09406) find
  4. Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on. IEEE, pages 1–10. find
  5. Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Bal-trušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. find
  6. ACM, New York, NY, USA, ICMI 2017, pages 163–171. https://doi.org/10.1145/3136755.3136801. — [doi:10.1145/3136755.3136801](https://doi.org/10.1145/3136755.3136801) ↗
  7. Glen Coppersmith and Erin Kelly. 2014. Dynamic wordclouds and vennclouds for exploratory data analysis. In Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces. Association for Computational Linguistics, Baltimore, Maryland, USA, pages 22–29. find
  8. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Mach. Learn. 20(3):273–297. https://doi.org/10.1023/A:1022627411411. ↗
  9. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep—a collaborative voice analysis repository for speech technologies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960–964. find
  10. A. Dhall, R. Goecke, S. Lucey, and T. Gedeon. 2012. Collecting large, richly annotated facial-expression databases from movies. IEEE MultiMedia 19(3):34– 41. https://doi.org/10.1109/MMUL.2012.26. — [doi:10.1109/mmul.2012.26](https://doi.org/10.1109/mmul.2012.26) ↗
  11. Abhinav Dhall, O.V. Ramana Murthy, Roland Goecke, Jyoti Joshi, and Tom Gedeon. 2015. Video and image based emotion recognition challenges in the wild: Emotiw 2015. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, New York, NY, USA, ICMI ’15, pages 423–426. h… — [doi:10.1145/2818346.2829994](https://doi.org/10.1145/2818346.2829994) find
  12. Thomas Drugman and Abeer Alwan. 2011. Joint robust voicing detection and pitch estimation based on residual harmonics. In Interspeech. pages 1973– 1976. find
  13. Thomas Drugman, Mark Thomas, Jon Gudnason, Patrick Naylor, and Thierry Dutoit. 2012. Detection of glottal closure instants from speech signals: A quantitative review. IEEE Transactions on Audio, Speech, and Language Processing 20(3):994–1006. find
  14. Sanjay Bilakhia, Stavros Petridis, Anton Nijholt, and Maja Pantic. 2015. The mahnob mimicry database: A database of naturalistic human interactions. Pattern Recognition Letters 66(Supplement C):52 – 61. Paul Ekman, Wallace V Freisen, and Sonia Ancoli. Pattern Recognition in Human Computer Interactio… — [doi:10.1016/j.patrec.2015.03.005](https://doi.org/10.1016/j.patrec.2015.03.005) find
  15. Leo Breiman. 2001. Random forests. Mach. Learn. 45(1):5–32. https://doi.org/10.1023/A:1010933404324. ↗
  16. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Journal of Language Resources and Evaluation 42(4):335–359. https://doi.org/10.1007/s1057… — [doi:10.1007/s10579-008-9076-6](https://doi.org/10.1007/s10579-008-9076-6) ↗
  17. A. Graves, A. r. Mohamed, and G. Hinton. 2013. Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. pages 6645–6649. https://doi.org/10.1109/ICASSP.2013.6638947. — [doi:10.1109/icassp.2013.6638947](https://doi.org/10.1109/icassp.2013.6638947) ↗
  18. Michael Grimm, Kristian Kroschel, and Shrikanth Narayanan. 2008. The vera am mittag german audiovisual emotional speech database. In ICME. IEEE, pages 865–868. find
  19. Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmerman. 2018. Memn: Multimodal emotional memory network for emotion recognition in dyadic conversational videos. In NAACL. find
  20. Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th International Conference on Multimodal Interactions. ACM, pages 169–176. find
  21. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  22. Friedrich Max Müller. 1866. Lectures on the science of language: Delivered at the Royal Institution of Great Britain in April, May, & June 1861, volume 1. Longmans, Green. iMotions. 2017. goo.gl/1rh1JN. find
  23. Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency. 2016. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. find
  24. ACM, New York, NY, USA, ICMI 2016, pages 284– 288. https://doi.org/10.1145/2993148.2993176. — [doi:10.1145/2993148.2993176](https://doi.org/10.1145/2993148.2993176) ↗
  25. Facial expression analysis. Mohit Iyyer, Varun Manjunatha, Jordan L Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In ACL (1). pages 1681–1691. find
  26. Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2013. 3d convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1):221–231. https://doi.org/10.1109/TPAMI.2012.59. — [doi:10.1109/tpami.2012.59](https://doi.org/10.1109/tpami.2012.59) ↗
  27. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188 . find our card
  28. John Kane and Christer Gobl. 2013. Wavelet maxima dispersion for breathy to tense voice discrimination. IEEE Transactions on Audio, Speech, and Language Processing 21(6):1170–1179. find
  29. Wootaek Lim, Daeyoung Jang, and Taejin Lee. 2016. Speech emotion recognition using convolutional and recurrent neural networks. In Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2016 Asia-Pacific. IEEE, pages 1–4. find
  30. Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. 2017. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. find
  31. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational L… find our card
  32. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. pages 55–60. http://www.aclweb.org/anthology/P/P14/P14-50… ↗ our card
  33. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of EMNLP. pages 79–86. find
  34. Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2014. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proceedings of the 16th International Conference on Multimodal Interaction. ACM, New… — [doi:10.1145/2663204.2663260](https://doi.org/10.1145/2663204.2663260) find
  35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. volume 14, pages 1532– 1543. find
  36. Veronica Perez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-Level Multimodal Sentiment Analysis. In Association for Computational Linguistics (ACL). Sofia, Bulgaria. find
  37. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis-Philippe Morency. 2017a. Context dependent sentiment analysis in user generated videos. In Association for Computational Linguistics. find
  38. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis-Philippe Morency. 2017b. Context-dependent sentiment analysis in user-generated videos. In Association for Computational Linguistics. find
  39. Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. 2016. Convolutional mkl based multimodal emotion recognition and sentiment analysis. In Data Mining (ICDM), 2016 IEEE 16th International Conference on. IEEE, pages 439–448. find
  40. Shyam Sundar Rajagopalan, Louis-Philippe Morency, Tadas Baltrušaitis, and Roland Goecke. 2016. Extending long short-term memory for multi-view structured learning. In European Conference on Computer Vision. find
  41. Fabien Ringeval, Andreas Sonderegger, Jürgen S. Sauer, and Denis Lalanne. 2013. Introducing the recola multimodal corpus of remote collaborative and affective interactions. In FG. IEEE Computer Society, pages 1–8. find
  42. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In CVPR. IEEE Computer Society, pages 815–823. find our card
  43. M. Schuster and K.K. Paliwal. 1997. Bidirectional recurrent neural networks. Trans. Sig. Proc. 45(11):2673–2681. https://doi.org/10.1109/78.650093. — [doi:10.1109/78.650093](https://doi.org/10.1109/78.650093) ↗
  44. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts, et al. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP). Ci… find
  45. Rupesh K Srivastava, Klaus Greff, and Juergen Schmidhuber. 2015. Training very deep networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, Curran Associates, Inc., pages 2377–2385. http://papers.nips.cc/paper/585… ↗
  46. Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Washington, DC, USA, CVPR ’14, pages 1701–1708… — [doi:10.1109/cvpr.2014.220](https://doi.org/10.1109/cvpr.2014.220) find
  47. Edmund Tong, Amir Zadeh, Cara Jones, and Louis-Philippe Morency. 2017. Combating human trafficking with multimodal deep models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 1547–1556. find
  48. George Trigeorgis, Fabien Ringeval, Raymond Brueckner, Erik Marchi, Mihalis A Nicolaou, Björn Schuller, and Stefanos Zafeiriou. 2016. Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE In… find
  49. Sentiment analysis in an audio-visual context. IEEE Intelligent Systems 28(3):46–53. find
  50. Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America 123(5):3878. find
  51. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Empirical Methods in Natural Language Processing, EMNLP. find
  52. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. arXiv preprint arXiv:1802.00927 . — [arXiv:1802.00927](https://arxiv.org/abs/1802.00927) find
  53. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018b. Multi-attention recurrent network for human communication comprehension. arXiv preprint arXiv:1802.00923 . — [arXiv:1802.00923](https://arxiv.org/abs/1802.00923) find
  54. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016a. Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 . — [arXiv:1606.06259](https://arxiv.org/abs/1606.06259) find
  55. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016b. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems 31(6):82–88. find
  56. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23(10):1499–1503. find
  57. Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis C. M. Lau. 2015. A c-lstm neural network for text classification. CoRR abs/1511.08630. — [arXiv:1511.08630](https://arxiv.org/abs/1511.08630) find
  58. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnı́k, and Jürgen Schmidhuber. 2016. Recurrent Highway Networks. arXiv preprint arXiv:1607.03474 . find
  59. Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2016. Select-additive learning: Improving cross-individual generalization in multimodal sentiment analysis. arXiv preprint arXiv:1609.05244 . — [arXiv:1609.05244](https://arxiv.org/abs/1609.05244) find
  60. Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: find