Elamur Research⌕
☰
home / papers / 2020 / Integrating Multimodal Information in Large Pretrained Transformers

Integrating Multimodal Information in Large Pretrained Transformers

Summary

Pretrained Transformers such as BERT and XLNet accept only text. Applying them to multimodal language, where spoken words come with voice and facial cues, has therefore meant one of two compromises: discarding the nonverbal signals, or using the models only to produce word features for a separate fusion network. The authors propose the Multimodal Adaptation Gate (MAG), a lightweight attachment that leaves the original architecture unchanged. For each word, MAG pairs the lexical vector with its aligned acoustic and visual features. Gating turns these pairs into a displacement vector that shifts the word's internal representation, and the whole model is then fine-tuned end to end. The authors evaluate the approach on multimodal sentiment analysis using CMU-MOSI and CMU-MOSEI. MAG-BERT and MAG-XLNet outperform text-only fine-tuning, fusion models fed with BERT, XLNet or GloVe features (including MulT), and naive input-level fusion. On CMU-MOSI, MAG-XLNet is reported to match human-level performance. The work shows that large text-pretrained models can absorb nonverbal information with minimal overhead, and the code is publicly released.

Key points

  • On CMU-MOSEI, binary accuracy / F1 results are:
    • BERT category: MAG-BERT 84.7 / 84.5, plain BERT 83.9 / 83.9, MulT with BERT embeddings 83.5 / 82.9.
    • XLNet category: MAG-XLNet 85.6 / 85.7, XLNet 85.4 / 85.2, MulT with XLNet embeddings 84.1 / 83.7.
  • The authors tested inserting MAG after the embedding layer, after XLNet layers 1, 4, 6, 8 and 12, and at all layers. Early placement works best, since abstract upper-layer vectors are harder to shift with word-level nonverbal cues.
  • Simply concatenating or adding acoustic and visual features to the text input performs poorly, which supports the case for a dedicated gate.
  • Pretraining is essential. With randomly initialized weights, binary accuracy drops to 70.1 (MAG-BERT) and 70.7 (MAG-XLNet) on CMU-MOSI, and to 76.8 and 78.4 on CMU-MOSEI.
  • Qualitative examples show nonverbal cues changing predictions. Cues such as a tense voice, a frown, or a sarcastic tone can adjust sentiment intensity or even reverse the polarity predicted from text alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Vision & language

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.214

@inproceedings{rahman2020integrating,
  title = {{Integrating Multimodal Information in Large Pretrained Transformers}},
  author = {Rahman, Wasifur and Hasan, Md Kamrul and Lee, Sangwu and Zadeh, AmirAli Bagher and Mao, Chengfeng and Morency, Louis-Philippe and Hoque, Ehsan},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.214},
  url = {https://doi.org/10.18653/v1/2020.acl-main.214}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Bal-trušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pages 163–171. ACM. find
  2. Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry of bert. arXiv preprint arXiv:1906.02715. find our card
  3. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. find
  4. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep—a collaborative voice analysis repository for speech technologies. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 960–964. IEEE. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep 2 https://github.com/WasifurRahman/ BERT_multimodal_transformer bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  6. Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… find our card
  7. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  8. Paul Pu Liang, Ziyin Liu, Amir Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. arXiv preprint arXiv:1808.03920. find
  9. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  10. Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2014. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 50–… find
  11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  12. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  13. Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabas Poczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. arXiv preprint arXiv:1812.07809. — [arXiv:1812.07809](https://arxiv.org/abs/1812.07809) find
  14. Soujanya Poria, Amir Hussain, and Erik Cambria. 2018. Multimodal Sentiment Analysis, volume 8. Springer. find
  15. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. ↗
  16. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766. find our card
  17. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. arXiv preprint arXiv:1906.00295. find our card
  18. Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. arXiv preprint arXiv:1806.06176. — [arXiv:1806.06176](https://arxiv.org/abs/1806.06176) find
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  20. Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. arXiv preprint arXiv:1811.09362. — [arXiv:1811.09362](https://arxiv.org/abs/1811.09362) find
  21. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card
  22. Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America, 123(5):3878. find
  23. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250. find our card
  24. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  25. Amir Zadeh, Paul Pu Liang, Louis-Philippe Morency, Soujanya Poria, Erik Cambria, and Stefan Scherer. 2018b. Proceedings of grand challenge and workshop on human multimodal language (challenge-hml). In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML). find
  26. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018c. Multi-attention recurrent network for human communication comprehension. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  27. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259. — [arXiv:1606.06259](https://arxiv.org/abs/1606.06259) find
  28. AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018d. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rahman-2020-integrating-multimodal-information-large-pretrained/