Elamur Research⌕
☰
home / papers / 2017 / Tensor Fusion Network for Multimodal Sentiment Analysis

Tensor Fusion Network for Multimodal Sentiment Analysis

Summary

The paper addresses sentiment prediction for opinionated online videos, where spoken words, facial behavior and voice all shape the perceived sentiment. The authors split the problem into two parts. The first is modeling dynamics within a single modality, such as the messy, disfluent structure of speech. The second is modeling how modalities interact, for example when a smile resolves an ambiguous phrase. They propose the Tensor Fusion Network (TFN), which has three components:

  • a separate embedding subnetwork for each of language, visual and acoustic input;
  • a Tensor Fusion layer that builds a three-way Cartesian product of these embeddings, so unimodal, bimodal and trimodal interactions are represented explicitly;
  • an inference subnetwork whose output is adapted to classification or regression.

Experiments use CMU-MOSI, a corpus of YouTube movie-review utterances with sentiment labels. TFN beats earlier multimodal methods, and each of its unimodal subnetworks also outperforms specialized single-modality baselines. The work matters because it shows a practical alternative to simple concatenation (early fusion) and to vote averaging (late fusion).

Key points

  • Dataset: CMU-MOSI has 2199 opinion utterances from 93 speakers, with 26,295 words in total.
    • Five Mechanical Turk workers labeled each utterance on a seven-point scale borrowed from the Stanford Sentiment Treebank.
    • Agreement between annotators is Krippendorff's α = 0.77.
  • Tasks: binary classification, five-class classification and regression on [−3, 3]. Regression is scored with mean absolute error and correlation.
  • Language subnetwork: 300-dimensional GloVe vectors feed an LSTM with a forget gate and 128 hidden units. All time steps then pass through fully connected ReLU layers of size 128, producing a 128-dimensional embedding.
  • Evaluation protocol: five-fold cross-validation with no speaker overlap between train and test. Hyperparameters were chosen by grid search on held-out videos. Training used Adam, dropout of 0.15 and an L2 coefficient of 0.01, and all baselines used identical folds.
  • Baselines:
    • visual: 3DCNN, CNN-LSTM, LSTM-FA;
    • acoustic: Adieu-Net, SER-LSTM;
    • single-modality versions of multimodal systems: SAL-CNN, SVM-MD, CMKL, RF.

Interesting ideas

  • I1. Product-based fusion: combining modality embeddings through an outer-product-style operation gives the model explicit features for every pairwise and three-way interaction. This could suit any task where signals jointly change meaning, such as text plus metadata or speech plus prosody.
  • I2. Using every LSTM state: the full sequence of hidden states is passed to a dense layer instead of only the final state. Useful content early in a rambling utterance therefore survives filler that comes later, which could help with noisy transcripts or chat logs.
  • I3. Speaker-independent splits: keeping speakers disjoint between training and testing gives a more honest estimate of generalization to new people. This applies to any corpus with repeated authors or speakers.
  • I4. Diagnosing fusion by comparison: placing the fused model's predictions next to the unimodal models' outputs reveals whether a fusion strategy merely copies the strongest modality. Here, early fusion turned out to mostly mirror language.
  • I5. One backbone, several outputs: a single shared architecture handles binary, fine-grained and regression targets by swapping only the output layer. This makes it cheap to evaluate the same representation across several label granularities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

139 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 128 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1115

@inproceedings{zadeh2017tensor,
  title = {{Tensor Fusion Network for Multimodal Sentiment Analysis}},
  author = {Zadeh, Amir and Chen, Minghai and Poria, Soujanya and Cambria, Erik and Morency, Louis-Philippe},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1115},
  url = {https://doi.org/10.18653/v1/d17-1115}
}

References

58 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Basant Agarwal, Soujanya Poria, Namita Mittal, Alexander Gelbukh, and Amir Hussain. 2015. Concept-level sentiment analysis with dependency-based semantic parsing: a novel approach. Cognitive Computation 7(4):487–499. find
  2. Paavo Alku. 1992. Glottal wave analysis with pitch synchronous iterative adaptive inverse filtering. Speech communication 11(2-3):109–118. find
  3. Paavo Alku, Tom Bäckström, and Erkki Vilkman. 2002. Normalized amplitude quotient for parametrization of the glottal flow. the Journal of the Acoustical Society of America 112(2):701–710. find
  4. Paavo Alku, Helmer Strik, and Erkki Vilkman. 1997. Parabolic spectral parameter—a new method for quantification of the glottal flow. Speech Communication 22(1):67–79. find
  5. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision. pages 2425–2433. find
  6. Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on. IEEE, pages 1–10. find
  7. Young-Hyen Byeon and Keun-Chang Kwak. 2014. Facial expression recognition using 3d convolutional neural network. International Journal of Advanced Computer Science and Applications 5(12). find
  8. Erik Cambria, Dipankar Das, Sivaji Bandyopadhyay, and Antonio Feraco. 2017. A Practical Guide to Sentiment Analysis. Springer, Cham, Switzerland. find
  9. Erik Cambria, Soujanya Poria, Rajiv Bajpai, and Björn Schuller. 2016. SenticNet 4: A semantic resource for sentiment analysis based on conceptual primitives. In COLING. pages 2666–2677. find
  10. Donald G Childers and CK Lee. 1991. Vocal quality factors: Analysis, synthesis, and perception. the Journal of the Acoustical Society of America 90(5):2394–2410. find
  11. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep—a collaborative voice analysis repository for speech technologies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960–964. find
  12. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition… find our card
  13. Thomas Drugman and Abeer Alwan. 2011. Joint robust voicing detection and pitch estimation based on residual harmonics. In Interspeech. pages 1973– 1976. find
  14. Thomas Drugman, Mark Thomas, Jon Gudnason, Patrick Naylor, and Thierry Dutoit. 2012. Detection of glottal closure instants from speech signals: A quantitative review. IEEE Transactions on Audio, Speech, and Language Processing 20(3):994–1006. find
  15. Samira Ebrahimi Kahou, Vincent Michalski, Kishore Konda, Roland Memisevic, and Christopher Pal. 2015. Recurrent neural networks for emotion recognition in video. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, pages 467–474. find
  16. Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion 6(3-4):169–200. find our card
  17. Paul Ekman, Wallace V Freisen, and Sonia Ancoli. 1980. Facial signs of emotional experience. Journal of personality and social psychology 39(6):1125– 1134. find
  18. Hiroya Fujisaki and Mats Ljungqvist. 1986. Proposal and evaluation of models for the glottal source waveform. In Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP’86.. IEEE, volume 11, pages 1605–1608. find
  19. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with lstm. Neural computation 12(10):2451–2471. find
  20. Sayan Ghosh, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2016a. Representation learning for speech emotion recognition. In Interspeech 2016. pages 3603–3607. https://doi.org/10.21437/Interspeech.2016-692. — [doi:10.21437/interspeech.2016-692](https://doi.org/10.21437/interspeech.2016-692) ↗
  21. Sayan Ghosh, Eugene Laksana, Louis-Philippe Morency, and Stefan Scherer. 2016b. Representation learning for speech emotion recognition. find
  22. Interspeech 2016 pages 3603–3607. find
  23. Michael Glodek, Stephan Tschechne, Georg Lay-her, Martin Schels, Tobias Brosch, Stefan Scherer, Markus Kächele, Miriam Schmidt, Heiko Neumann, Günther Palm, et al. 2011. Multiple classifier systems for the classification of audio-visual emotional states. In Affective Computing and Intelligent Inte… find
  24. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  25. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pages 168– 177. find
  26. Mohit Iyyer, Varun Manjunatha, Jordan L Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In ACL (1). pages 1681–1691. find our card
  27. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences pages 656–666. find our card
  28. John Kane and Christer Gobl. 2013. Wavelet maxima dispersion for breathy to tense voice discrimination. IEEE Transactions on Audio, Speech, and Language Processing 21(6):1170–1179. find
  29. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  30. Jinkyu Lee and Ivan Tashev. 2015. High-level feature representation using recurrent neural network for speech emotion recognition. In INTERSPEECH. pages 1537–1540. find
  31. Wootaek Lim, Daeyoung Jang, and Taejin Lee. 2016. Speech emotion recognition using convolutional and recurrent neural networks. In Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2016 Asia-Pacific. IEEE, pages 1–4. find
  32. Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces. ACM, pages 169–176. find
  33. Behnaz Nojavanasghari, Tadas Baltrušaitis, Charles E Hughes, and Louis-Philippe Morency. 2016. Emore-act: a multimodal approach and dataset for recognizing emotional responses in children. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, pages 137–144. find
  34. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval 2(1–2):1–135. find
  35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. volume 14, pages 1532– 1543. find
  36. Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In ACL (1). pages 973– 982. find
  37. Soujanya Poria, Basant Agarwal, Alexander Gelbukh, Amir Hussain, and Newton Howard. 2014a. Dependency-based semantic parsing for concept-level text analysis. In International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pages 113–127. find
  38. Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion 37:98–125. find
  39. Soujanya Poria, Erik Cambria, and Alexander F Gelbukh. 2015. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In EMNLP. pages 2539–2544. find
  40. Soujanya Poria, Erik Cambria, Gregoire Winterstein, and Guang-Bin Huang. 2014b. Sentic patterns: Dependency-based rules for concept-level sentiment analysis. Knowledge-Based Systems 69:45–63. find
  41. Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. 2016. Convolutional mkl based multimodal emotion recognition and sentiment analysis. In Data Mining (ICDM), 2016 IEEE 16th International Conference on. IEEE, pages 439–448. find
  42. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts, et al. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP). Ci… find
  43. Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find our card
  44. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics 37(2):267–307. find our card
  45. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks pages 1556–1566. find
  46. Ingo R Titze and Johan Sundberg. 1992. Vocal intensity in speakers and singers. the Journal of the Acoustical Society of America 91(5):2936–2946. find
  47. Edmund Tong, Amir Zadeh, and Louis-Philippe Morency. 2017. Combating human trafficking with deep multimodal models. In Association for Computational Linguistics. find
  48. George Trigeorgis, Fabien Ringeval, Raymond Brueckner, Erik Marchi, Mihalis A Nicolaou, Björn Schuller, and Stefanos Zafeiriou. 2016. Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE In… find
  49. Michel Valstar, Jonathan Gratch, Björn Schuller, Fabien Ringeval, Dennis Lalanne, Mercedes Torres Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and Maja Pantic. 2016. find
  50. Avec 2016: Depression, mood, and emotion recognition workshop and challenge. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. ACM, pages 3–10. find
  51. Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2016. Select-additive learning: Improving cross-individual generalization in multimodal sentiment analysis. arXiv preprint arXiv:1609.05244 . — [arXiv:1609.05244](https://arxiv.org/abs/1609.05244) find
  52. Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems 28(3):46–53. find
  53. Bishan Yang and Claire Cardie. 2012. Extracting opinion expressions with semi-markov conditional random fields. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, page… find
  54. Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 4651–4659. find
  55. Amir Zadeh. 2015. Micro-opinion sentiment intensity analysis and summarization in online videos. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, pages 587– 591. find
  56. Amir Zadeh, Tadas Baltrušaitis, and Louis-Philippe Morency. 2017. Convolutional experts constrained local model for facial landmark detection. In Computer Vision and Pattern Recognition Workshop (CVPRW). IEEE. find
  57. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016a. Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 . — [arXiv:1606.06259](https://arxiv.org/abs/1606.06259) find
  58. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016b. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems 31(6):82–88. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zadeh-2017-tensor-fusion-network-multimodal-sentiment/