Elamur Research⌕
☰
home / papers / 2018 / Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Summary

The paper tackles a scaling problem in multimodal fusion, which merges per-modality representations (language, audio, visual) into one joint vector. Tensor-based methods such as Tensor Fusion Network (TFN) capture cross-modal interactions well, but their tensors are built from outer products. As a result, dimensionality, parameter count and compute grow exponentially with the number of modalities, which also invites overfitting. The authors propose Low-rank Multimodal Fusion (LMF). LMF breaks the fusion weight tensor into a small number of modality-specific low-rank factors. Because the input tensor itself factorizes into the unimodal vectors, the output can be computed directly, without ever building the high-dimensional tensor, so cost grows linearly in the number of modalities. LMF is tested on three public datasets: CMU-MOSI for sentiment, POM for speaker traits and IEMOCAP for emotions. It beats TFN on every metric and matches or surpasses other state-of-the-art systems, while being much smaller and faster. This makes expressive tensor-style fusion practical for settings with more modalities.

Key points

  • Fusion complexity drops from O(d_y·∏d_m) in TFN to O(d_y·r·∑d_m) in LMF, where r is the rank.
  • With rank 4, the full LMF model has about 1.1M parameters, versus about 12.5M for TFN (nearly 11× more).
  • Speed was measured on an NVIDIA Quadro K4200:
    • Training: 1134.82 vs 340.74 inferences/sec for TFN.
    • Testing: 2249.90 vs 1177.17 inferences/sec for TFN.
  • Results on the three datasets:
    • CMU-MOSI: best MAE (0.912) and Corr (0.668) among compared models.
    • POM: top average scores over 16 speaker traits (Corr 0.396, Acc 42.8).
    • IEMOCAP: best F1 for every emotion (e.g., Angry 89.0).
  • Rank experiments on CMU-MOSI show that very low ranks already work well, while higher ranks make training less stable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Vision & language · Efficient models & compression

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1209

@inproceedings{liu2018efficient,
  title = {{Efficient Low-rank Multimodal Fusion With Modality-Specific Factors}},
  author = {Liu, Zhun and Shen, Ying and Lakshminarasimhan, Varun Bharadhwaj and Liang, Paul Pu and Zadeh, AmirAli Bagher and Morency, Louis-Philippe},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1209},
  url = {https://doi.org/10.18653/v1/p18-1209}
}

References

33 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Figure 4: The Impact of different rank settings on Model Performance: As the rank increases, the results become unstable and low rank is enough in terms of the mean absolute error.* Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette Chang, Sungbok Lee, and… — [doi:10.1007/s10579-008-9076-6](https://doi.org/10.1007/s10579-008-9076-6) find
  2. Lawrence S Chen, Thomas S Huang, Tsutomu Miyasato, and Ryohei Nakatsu. 1998. Multimodal human emotion/expression recognition. In Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on. IEEE, pages 366–371. find
  3. Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Bal-trušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. find
  4. ACM, New York, NY, USA, ICMI 2017, pages 163–171. https://doi.org/10.1145/3136755.3136801. — [doi:10.1145/3136755.3136801](https://doi.org/10.1145/3136755.3136801) ↗
  5. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20(3):273–297. find
  6. Liyanage C De Silva, Tsutomu Miyasato, and Ryohei Nakatsu. 1997. Facial emotion recognition using multi-modal information. In Information, Communications and Signal Processing, 1997. ICICS., Proceedings of 1997 International Conference on. IEEE, volume 1, pages 397–401. find
  7. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarepa collaborative voice analysis repository for speech technologies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960–964. find
  8. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847 . find our card
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9(8):1735– 1780. https://doi.org/10.1162/neco.1997.9.8.1735. ↗
  10. Guosheng Hu, Yang Hua, Yang Yuan, Zhihong Zhang, Zheng Lu, Sankha S Mukherjee, Timothy M Hospedales, Neil M Robertson, and Yongxin Yang. 2017. Attribute-enhanced face recognition with neural tensor fusion networks. find
  11. Othmar Koch and Christian Lubich. 2010. Dynamical tensor approximation. SIAM Journal on Matrix Analysis and Applications 31(5):2360–2375. find
  12. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 1381–1391. find
  13. Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th International Conference on Multimodal Interactions. ACM, pages 169–176. find
  14. Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency. 2016. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM, pages 284–288. find
  15. Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2014a. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proceedings of the 16th International Conference on Multimodal Interaction. ACM, pag… find
  16. Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2014b. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proceedings of the 16th International Conference on Multimodal Interaction. ACM, New… — [doi:10.1145/2663204.2663260](https://doi.org/10.1145/2663204.2663260) find
  17. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. find our card
  18. Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 973–982. find
  19. Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. 2016. Convolutional mkl based multimodal emotion recognition and sentiment analysis. In Data Mining (ICDM), 2016 IEEE 16th International Conference on. IEEE, pages 439–448. find
  20. Shyam Sundar Rajagopalan, Louis-Philippe Morency, Tadas Baltrušaitis, and Roland Goecke. 2016. Extending long short-term memory for multi-view structured learning. In European Conference on Computer Vision. find
  21. Ilya Razenshteyn, Zhao Song, and David P Woodruff. 2016. Weighted low rank approximations with provable guarantees. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing. ACM, pages 250–263. find
  22. Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2016. Select-additive learning: Improving cross-individual generalization in multimodal sentiment analysis. arXiv preprint arXiv:1609.05244 . — [arXiv:1609.05244](https://arxiv.org/abs/1609.05244) find
  23. Hongcheng Wang and Narendra Ahuja. 2008. A tensor approximation approach to dimensionality reduction. International Journal of Computer Vision 76(3):217–229. find
  24. Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems 28(3):46–53. find
  25. Torsten Wortwein and Stefan Scherer. 2017. What really mattersan information gain analysis of questions and reactions in automated ptsd screenings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, pages 15–20. find
  26. Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America 123(5):3878. find
  27. Ben P Yuhas, Moise H Goldstein, and Terrence J Sejnowski. 1989. Integration of acoustic and visual speech signals using neural networks. IEEE Communications Magazine 27(11):65–71. find
  28. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Empirical Methods in Natural Language Processing, EMNLP. find our card
  29. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. arXiv preprint arXiv:1802.00927 . — [arXiv:1802.00927](https://arxiv.org/abs/1802.00927) find
  30. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018b. Multi-attention recurrent network for human communication comprehension. arXiv preprint arXiv:1802.00923 . — [arXiv:1802.00923](https://arxiv.org/abs/1802.00923) find
  31. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016a. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 . — [arXiv:1606.06259](https://arxiv.org/abs/1606.06259) find
  32. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016b. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems 31(6):82–88. find
  33. Qiang Zhu, Mei-Chen Yeh, Kwang-Ting Cheng, and Shai Avidan. 2006. Fast human detection using a cascade of histograms of oriented gradients. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. IEEE, volume 2, pages 1491–1498. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2018-efficient-low-rank-multimodal-fusion/