Elamur Research⌕
☰
home / papers / 2026 / EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness

EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness

Paper · 2026 · Yueru Sun, Yimeng Zhang, Haoyu Gu et al.

Summary

The paper asks how multimodal large language models (MLLM) actually weigh text, audio and video when recognizing emotion, especially when those signals contradict each other or one is absent. The authors build EmoMM, a bilingual benchmark drawn from CH-SIMS v2.0 (Chinese) and CMU-MOSI (English). It carries both unimodal and multimodal labels and is divided into aligned, conflict and missing-modality subsets. Using Shapley-style contribution metrics and attention analysis, they identify a failure pattern called Video Contribution Collapse (VCC): under conflict, models largely ignore visual evidence. They attribute this to the large number of redundant video tokens and a stable preference for other modalities. As a remedy they propose CHASE, a lightweight router that flags conflicting inputs and then adjusts attention in selected heads at inference time. CHASE rebalances weight across modalities and sharpens focus on informative visual tokens, without retraining the backbone. It raises accuracy for Qwen2.5-Omni-7B and VITA-1.5-8B, which is useful for building more trustworthy affect-aware systems.

Key points

  • EmoMM composition: 2,000 samples each from CH-SIMS v2.0 and CMU-MOSI, using a five-class scheme (N, WN, Neu, WP, P).
    • Five annotators supplied unimodal Text/Audio/Video labels for CMU-MOSI.
    • Subsets are EmoMM-Align, EmoMM-Conflict and EmoMM-Missing, with Missing further split into Missing-Align and Missing-Conflict.
  • Diagnostic metrics: PSMV, a Shapley-based measure of each modality's marginal contribution, and GTAR, which tracks agreement with the ground-truth-aligned modality.
    • Label agreement is highest between GT and text and lowest between text and video.
  • CHASE design: it combines an inter-modality bias with an intra-modality visual bias and intervenes on the top-K=4 attention heads.
    • It needs white-box access to model internals, so it cannot be applied to API-only models.
  • Gains for Qwen2.5-Omni-7B:
    • Overall accuracy rises from 44.53% to 49.96% (+5.43).
    • Conflict improves by +4.85, Missing by +6.03 and Align by +1.14.
    • For VITA-1.5-8B, Conflict improves by +5.16.
    • For reference, Gemini 3 Pro reaches 56.41% overall.
  • Ablation on Conflict (Qwen):
    • Replacing the router with random steering lowers accuracy to 45.63%.
    • Removing the inter-modality bias costs 3.22 points.
    • Removing the intra-modality bias costs 1.17 points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 works

Cite

Cite as Yueru Sun, Yimeng Zhang, Haoyu Gu, Nuo Chen, Dong She, Xianrong Yao, Yang Gao and Zhanpeng Jin. 2026. EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1018

@inproceedings{sun2026emomm,
  title = {{EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness}},
  author = {Yueru Sun, Yimeng Zhang, Haoyu Gu, Nuo Chen, Dong She, Xianrong Yao, Yang Gao and Zhanpeng Jin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1018},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1018}
}

References

40 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2019. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423–443. find
  2. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335–359. find
  3. Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander G. Hauptmann. 2024a. Emotion-LLaMA: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805– 110853. find
  4. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024b. VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-LLMs. arXiv preprint arXiv:2406.07476. find
  5. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-Audio technical report. arXiv preprint arXiv:2407.10759.
  6. Qi Fan, Hongyu Yuan, Haolin Zuo, Rui Liu, and Guanglai Gao. 2024. Leveraging retrieval augment approach for multimodal emotion recognition under missing modalities. arXiv preprint arXiv:2410.02804. — [arXiv:2410.02804](https://arxiv.org/abs/2410.02804) find
  7. Yiyang Fang, Jian Liang, Wenke Huang, He Li, Kehua Su, and Mang Ye. 2025. Catch your emotion: Sharpening emotion perception in multimodal large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 16029–16039. PMLR. find
  8. Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, Haoyu Cao, Di Yin, Long Ma, Xiawu Zheng, Rongrong Ji, Yunsheng Wu, Ran He, Caifeng Shan, and Xing Sun. 2024. VITA: Towards open-source interactive omni multimodal LLM. arXiv pre… — [arXiv:2408.05211](https://arxiv.org/abs/2408.05211) find
  9. Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, and Ran He. 2025a. VITA-1.5: Towards GPT-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957. — [arXiv:2501.01957](https://arxiv.org/abs/2501.01957) find
  10. Yao Fu, Qiong Liu, Qing Song, Pengzhou Zhang, and Gongdong Liao. 2025b. Multi-HM: A Chinese multimodal dataset and fusion framework for emotion recognition in human–machine dialogue systems. Applied Sciences, 15(8):4509. find
  11. OpenAI. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276.
  12. Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  13. OpenBMB. 2025. MiniCPM-o 2.6: A GPT-4o-level MLLM for vision, speech and live streaming. Technical blog and model card. find
  14. Google DeepMind. 2025. Gemini 3 pro model card. Technical report, Google DeepMind. find
  15. Vittorio Pipoli, Alessia Saporita, Federico Bolelli, Marcella Cornia, Lorenzo Baraldi, Costantino Grana, Rita Cucchiara, and Elisa Ficarra. 2025. MissRAG: Addressing the missing modality challenge in multimodal large language models. In Proceedings of the IEEE/CVF International Conference on Compute… find
  16. Zirun Guo, Tao Jin, and Zhou Zhao. 2024. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1714–1730. Association for Computation… find
  17. Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, and Xun Yang. 2025. Benchmarking and bridging emotion conflicts for multimodal emotion reasoning. In Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25). ACM. ArXiv:2508.01181. — [arXiv:2508.01181](https://arxiv.org/abs/2508.01181) find
  18. Nikola Kovačević, Christian Holz, Markus Gross, and Rafael Wampfler. 2024. On multimodal emotion recognition for human-chatbot interaction in the wild. In Proceedings of the 26th International Conference on Multimodal Interaction (ICMI ’24), pages 12–21. ACM. find
  19. Songning Lai, Xifeng Hu, Haoxuan Xu, Zhaoxia Ren, and Zhi Liu. 2023. Multimodal sentiment analysis: A survey. Information Sciences. find
  20. Chuangqi Li, Xinying Weng, Yifan Li, and Tianxin Zhang. 2025. Multimodal learning engagement assessment system: An innovative approach to optimizing learning engagement. International Journal of Human–Computer Interaction, 41(5):3474–3490. find
  21. Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, Jiangyan Yi, and Jianhua Tao. 2025. AffectGPT: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. In Proceedings of the 42nd Internati… find
  22. Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang, Yuanzhe Qiu, Tie Cheng, Xiaoteng Li, Hua Xu, and Kai Gao. 2022. Make acoustic and visual cues matter: CH-SIMS v2.0 dataset and AV-Mixup consistent module. In Proceedings of the 2022 International Conference on Multimodal Interaction, p… find
  23. Jiquan Ngiam, Aditya Khosla, Minjae Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multimodal deep learning. In Proceedings of the 28th International Conference on Machine Learning (ICML), pages 689–696. find
  24. OpenAI. 2025. Introducing GPT-5. Product announcement. find
  25. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527–536. Associ…
  26. Misha Sadeghi, Robert Richer, Bernhard Egger, Lena Schindler-Gmelch, Lydia Helene Rupp, Farnaz Rahimi, Matthias Berking, and Bjoern M. Eskofier. 2024. Harnessing multimodal approaches for depression detection using large language models and facial expressions. npj Mental Health Research, 3:66. find
  27. Lloyd S. Shapley. 1953. A value for n-person games. In Harold W. Kuhn and Albert W. Tucker, editors, Contributions to the Theory of Games II (Annals of Mathematics Studies, Vol. 28), pages 307–317. Princeton University Press. find
  28. Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. 2024. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint arXiv:2406.15704. — [arXiv:2406.15704](https://arxiv.org/abs/2406.15704) find
  29. THUIAR. 2022. CH-SIMS v2.0 project page. Project page. find
  30. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558–6569. A…
  31. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2.5-Omni technical report. arXiv preprint arXiv:2503.20215.
  32. Zhenrong Xu, Yuan Gao, Fang Wang, Longqian Zhang, Li Zhang, Junke Wang, and Jie Shu. 2025b. Depression detection methods based on multimodal fusion of voice and text. Scientific Reports, 15:21907. find
  33. Lijuan Yan, Xiaotao Wu, and Yi Wang. 2025. Student engagement assessment using multimodal deep learning. PLOS ONE, 20(6):e0325377. find
  34. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, 11(12):nwae403.
  35. Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3…
  36. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114. Association for Computational Linguistics.
  37. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. IEEE Intelligent Systems, 31(6):82–88. find
  38. AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume…
  39. Fan Zhang, Zebang Cheng, Chong Deng, Haoxuan Li, Zheng Lian, Qian Chen, Huadai Liu, Wen Wang, Yi-Fan Zhang, Renrui Zhang, Ziyu Guo, Zhihong Zhu, Hao Wu, Haixin Wang, Yefeng Zheng, Xiaojiang Peng, Xian Wu, Kun Wang, Xiangang Li, and 2 others. 2025. MME-Emotion: A holistic evaluation benchmark for emo… — [arXiv:2508.09210](https://arxiv.org/abs/2508.09210) find
  40. Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543–553. Association for Computational Linguistics…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-emomm-benchmarking-steering-mllm-multimodal/