Elamur Research
☰
home / papers / 2019 / Multimodal Transformer for Unaligned Multimodal Language Sequences

Multimodal Transformer for Unaligned Multimodal Language Sequences

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1656.pdf

Summary

People communicate through words, facial expressions and tone of voice, but these streams are recorded at different rates. As a result, they rarely line up in time, and related cues can be far apart. Earlier fusion models usually forced visual and acoustic features onto word boundaries before training. That step needs hand-crafted preprocessing and word timing metadata, and it captures only local interactions. Tsai et al. propose the Multimodal Transformer (MulT), an end-to-end extension of the Transformer that works directly on unaligned sequences. Its key component is directional pairwise crossmodal attention. For each pair of modalities, a target stream is repeatedly enriched with low-level features from a source stream, with attention spanning the whole utterance. The fused outputs are then passed to self-attention transformers for prediction. On CMU-MOSI, CMU-MOSEI and IEMOCAP, MulT sets new state-of-the-art results in both word-aligned and unaligned settings, beating prior methods by roughly 5–15% on most metrics. This suggests explicit alignment may be unnecessary for multimodal fusion.

Key points

  • MulT uses six directional crossmodal transformers (e.g., V → L, A → L, L → V). Each one stacks crossmodal attention blocks with residual connections and feed-forward sublayers, and it attends to the source modality's low-level features rather than its intermediate ones.
  • Evaluation covers three benchmarks: CMU-MOSI, CMU-MOSEI (sentiment) and IEMOCAP (emotion). Each is tested in both aligned and unaligned variants.
  • On unaligned CMU-MOSEI, the full MulT reaches 50.7 Acc-7, 81.6 binary accuracy/F1, 0.591 MAE and 0.691 correlation. The late- and early-fusion transformer baselines reach about 78.6–78.9 binary accuracy.
  • Ablations show the following:
    • Among unimodal models, language is by far the strongest (77.4 vs. 65.6 binary accuracy for audio).
    • Among the crossmodal branches, the one with language as the target performs best.
    • Adapting from intermediate-level features works slightly worse than adapting from low-level features.
  • Attention maps from the V → L branch link emotionally loaded words (e.g., "disappointing") to video segments with strong facial movement. This happens even without any alignment, and it includes off-diagonal, long-range links.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

165 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 164 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1656

@inproceedings{tsai2019multimodal,
  title = {{Multimodal Transformer for Unaligned Multimodal Language Sequences}},
  author = {Tsai, Yao-Hung Hubert and Bai, Shaojie and Liang, Paul Pu and Kolter, J. Zico and Morency, Louis-Philippe and Salakhutdinov, Ruslan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1656},
  url = {https://doi.org/10.18653/v1/p19-1656}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. find our card
  2. Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In International Conference on Learning Representations (ICLR). find
  3. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335. find
  4. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in ne… find our card
  5. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2018. Transformer-xl: Language modeling with longer-term dependency. find
  6. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  8. Sri Harsha Dumpala, Rupayan Chakraborty, and Sunil Kumar Kopparapu. 2019. Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS. find
  9. Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200. find our card
  10. Paul Ekman, Wallace V Freisen, and Sonia Ancoli. 1980. Facial signs of emotional experience. Journal of personality and social psychology, 39(6):1125. find
  11. Kathleen R Gibson, Kathleen Rita Gibson, and Tim Ingold. 1994. Tools, language and cognition in human evolution. Cambridge University Press. find
  12. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML. find
  13. Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Lin… find
  14. Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. arXiv preprint arXiv:1501.02598. find
  15. Paul Pu Liang, Ziyin Liu, Amir Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. EMNLP. find
  16. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130. find our card
  17. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Prismatic Inc, Steven J. Bethard, and David Mcclosky. 2014. The stanford corenlp natural language processing toolkit. In In ACL, System Demonstrations. find our card
  18. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696. find
  19. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933. find our card
  20. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  21. Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabas Poczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. AAAI. find
  22. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume… find
  23. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. find our card
  24. Imran Sheikh, Sri Harsha Dumpala, Rupayan Chakraborty, and Sunil Kumar Kopparapu. 2018. Sentiment analysis using imperfect views from spoken language and acoustic modalities. In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), pages 35–39. find
  25. Nitish Srivastava and Ruslan R Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222–2230. find
  26. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038. Association for Computational Linguistics… find our card
  27. Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? a targeted evaluation of neural machine translation architectures. arXiv preprint arXiv:1808.08946. find
  28. Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Learning factorized multimodal representations. ICLR. find
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  30. Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803. find
  31. Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. AAAI. find
  32. Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online segment to segment neural transduction. arXiv preprint arXiv:1609.08194. find
  33. Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America, 123(5):3878. find
  34. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  35. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82–88. find
  36. AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018b. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL. find our card