Elamur Research⌕
☰
home / papers / 2021 / Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

Paper · 2021 · Wei Han, Hui Chen and Soujanya Poria

Summary

Multimodal sentiment analysis (MSA) combines text, visual and acoustic signals from video segments to predict sentiment strength. Existing fusion methods are trained either through task-driven gradients or by reshaping feature-space geometry. Neither approach explicitly controls how task-relevant information travels from the raw modalities into the fused embedding. The authors propose MultiModal InfoMax (MMIM), which maximizes mutual information (MI) at two levels, trained jointly with the sentiment regression objective:

  • Input level: between pairs of unimodal representations.
  • Fusion level: between the fused vector and each unimodal representation.

Because exact MI cannot be computed, they optimize lower bounds. Some terms in those bounds are hard to evaluate, so they combine:

  • Parametric parts: neural predictors and Contrastive Predictive Coding (CPC).
  • A non-parametric part: a Gaussian Mixture Model (GMM) whose parameters are estimated without learning.

On two public benchmarks, including CMU-MOSEI, MMIM matches or exceeds state-of-the-art models. Ablations show that every MI term contributes. The authors present this as the first work to connect MI maximization with MSA, giving fusion an information-theoretic training signal.

Key points

  • Encoders: text is encoded with BERT, using the final-layer head token. Visual and acoustic streams use modality-specific unidirectional LSTMs. A stack of linear-activation layers fuses the outputs, and an MLP regressor makes the prediction.
  • Inter-modality bound: MI between modalities uses the Barber–Agakov lower bound, applied to the text–visual and text–acoustic pairs. Its entropy term is estimated with a class-wise GMM fed by a history memory of past samples.
  • Fusion-level bound: MI between the fusion output and the inputs is boosted with a CPC loss. The resulting per-modality scores show how strongly the fusion relies on each modality.
  • Ablations on CMU-MOSEI:
    • Dropping MI losses lowers performance.
    • Estimating entropy from only the current batch made training fail with NaN gradients.
    • Replacing the GMM with a single Gaussian hurt all metrics.
  • Further findings and code: loss tracing suggests the input-level MI term matters more than the fusion-level term. Code is available at github.com/declare-lab/Multimodal-Infomax.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Singapore University of Technology and DesignSingapore · Singapore33 works · 10,760 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksComputer Vision and Pattern Recognition · 1,282 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 608 worksVision & language · 269 worksSpeech & spoken language · 233 works

Cite

Cite as Wei Han, Hui Chen and Soujanya Poria. 2021. Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.723

@inproceedings{han2021improving,
  title = {{Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis}},
  author = {Wei Han, Hui Chen and Soujanya Poria},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.723},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.723}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Md Shad Akhtar, Dushyant Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multi-task learning for multimodal emotion recognition and sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin… find
  2. Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. 2016. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410. — [arXiv:1612.00410](https://arxiv.org/abs/1612.00410) find
  3. Rana Ali Amjad and Bernhard C Geiger. 2019. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence, 42(9):2225–2239. find
  4. Relja Arandjelovic and Andrew Zisserman. 2017. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, pages 609–617. find
  5. Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910. — [arXiv:1906.00910](https://arxiv.org/abs/1906.00910) find
  6. David Barber and Felix Agakov. 2004. The IM algorithm: A variational approach to information maximization. Advances in neural information processing systems, 16:201. find
  7. Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018. Mutual information neural estimation. In International Conference on Machine Learning, pages 531–540. PMLR. find
  8. Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Bal-trušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pages 163–171. find
  9. Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. 2020. Club: A contrastive log-ratio upper bound of mutual information. In International Conference on Machine Learning, pages 1779–1788. PMLR. find
  10. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep—a collaborative voice analysis repository for speech technologies. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 960–964. IEEE. find
  11. Julien Deonna and Fabrice Teroni. 2012. The emotions: A philosophical introduction. Routledge. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  13. Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internation…
  14. Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297–304. JMLR Workshop and Conference Proceedings. find
  15. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. CASCADE: Contextual sarcasm detection in online discussion forums. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1837–1848.
  16. Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. MISA: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1122–1131. find
  17. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738.
  18. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations. find
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  20. Marco F Huber, Tim Bailey, Hugh Durrant-Whyte, and Uwe D Hanebeck. 2008. On entropy approximation for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181–188. IEEE. find
  21. Mounir Ait Kerroum, Ahmed Hammouch, and Driss Aboutajdine. 2010. Textural feature selection by joint mutual information based on gaussian mixture model for multispectral image classification. Pattern Recognition Letters, 31(10):1168–1174. find
  22. Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: L…
  23. Sijie Mai, Haifeng Hu, and Songlong Xing. 2020. Modality to modality translation: An adversarial representation learning and graph fusion network for multimodal fusion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 164–172. find
  24. Prem Melville, Wojciech Gryc, and Richard D Lawrence. 2009. Sentiment analysis of blogs by combining lexical knowledge with text classification. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1275–1284. find
  25. Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces, pages 169–176. find
  26. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In ICML. find
  27. Mattias Nilsson, Harald Gustaftson, Søren Vang Andersen, and W Bastiaan Kleijn. 2002. Gaussian mixture model based mutual information estimation between frequency bands in speech. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages I–525. IEEE. find
  28. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  29. Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020. Integrating multimodal information in large pretrained transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2359…
  30. Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. 2020. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8992–8999. find
  31. Naftali Tishby and Noga Zaslavsky. 2015. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5. IEEE. find
  32. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019a. Multimodal Transformer for unaligned multimodal language sequences. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
  33. Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019b. Learning factorized multimodal representations. In International Conference on Representation Learning. find
  34. Yao-Hung Hubert Tsai, Martin Ma, Muqiao Yang, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Multimodal routing: Improving local and global interpretability of multimodal language analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pa… find
  35. Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2018. Deep graph infomax. In International Conference on Learning Representations. find
  36. Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6892–6899. find
  37. Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2017. Select-additive learning: Improving generalization in multimodal sentiment analysis. In 2017 IEEE International Conference on Multimedia and Expo (ICME), pages 949–954. IEEE. find
  38. Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. 2019. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171– 5180. PMLR. find
  39. Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):46–53. find
  40. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2020. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. IEEE Transactions on Affective Computing, pages 1–1. find
  41. Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. arXiv preprint arXiv:2102.04830. — [arXiv:2102.04830](https://arxiv.org/abs/2102.04830) find
  42. Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. The Journal of the Acoustical Society of America, 123(5):3878– 3878. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/han-2021-improving-multimodal-fusion-hierarchical-mutual/