Elamur Research⌕
☰
home / papers / 2026 / Beyond Modality Collapse: Taming Guided Modality Entropy for Omni-modal Emotion Reasoning

Beyond Modality Collapse: Taming Guided Modality Entropy for Omni-modal Emotion Reasoning

Paper · 2026 · Xian Zhao, Rui Hu, Yuxiang Zhang et al.

Summary

Omni-modal LLMs that process video, audio and text often handle emotion inference poorly. The authors trace this to "modality collapse": the model leans heavily on one channel, usually vision, and ignores acoustic and verbal evidence. They address it in two ways.

The first is OmniCoT, a pipeline for building training data. It produces reasoning traces divided by guided tokens: [vision], [audio], [dialogue] and [analysis]. Separate models extract the evidence for each channel. An LLM then checks whether that evidence actually supports the gold emotion label and discards samples that fail.

The second is DyME-GRPO, a GRPO variant. It adds a reward based on the entropy of the guided-token predictions, which pushes the model toward balanced and non-redundant use of the modalities.

Training Qwen2.5-Omni-7B with SFT on OmniCoT and then DyME-GRPO yields EmoOmni. The authors report state-of-the-art results on several emotion recognition and reasoning benchmarks, and EmoOmni keeps the base model's general skills. The approach is a reusable recipe for making multimodal models actually consult every input channel.

Key points

  • OmniCoT pipeline components:
    • Qwen3-VL produces visual descriptions.
    • Qwen2-Audio captures paralinguistic cues such as intonation and intensity.
    • Whisper transcribes the dialogue.
    • Qwen3-7B verifies each sample and writes the [analysis] step.
  • Effect of DyME-GRPO over the SFT-only model:
    • On MELD, UAR improved from 34.30 to 36.64, while WAR climbed to 46.53, up from 42.13.
    • On IEMOCAP, WAR reached 58.15, while UAR shifted slightly to 42.89. The authors attribute this to reduced majority-class bias.
  • Entropy-reward weight α: performance peaks at α = 0.1. Setting α = 0 hurts results, and large values make outputs less stable and less format-compliant.
  • Guided-token ablation: removing the guided tokens while keeping the reasoning text lowers performance. Training sets in this comparison were matched in size, and the baselines included the existing CoT datasets MERR and MER 2025.
  • General capabilities: on OmniBench (eight categories), EmoOmni matches or slightly exceeds the base model, so there is no catastrophic forgetting. Code is available at github.com/ZxShane/EmoOmni.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Beijing Jiaotong UniversityBeijing · China28 works · 1,754 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksSentiment & emotion · 1,035 works

Cite

Cite as Xian Zhao, Rui Hu, Yuxiang Zhang, Delai Qiu, Yining Wang, Shengping Liu, Jian Yu and Jitao Sang. 2026. Beyond Modality Collapse: Taming Guided Modality Entropy for Omni-modal Emotion Reasoning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1044

@inproceedings{zhao2026beyond,
  title = {{Beyond Modality Collapse: Taming Guided Modality Entropy for Omni-modal Emotion Reasoning}},
  author = {Xian Zhao, Rui Hu, Yuxiang Zhang, Delai Qiu, Yining Wang, Shengping Liu, Jian Yu and Jitao Sang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1044},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1044}
}

References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. arXiv prepri…
  2. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  3. Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, and 1 others. 2025. Emova: Empowering language models to see, hear and speak with vivid emotions. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5455–5466. find
  4. Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805–110853. find
  5. Yunfei Chu, Jin Xu, Qian Yang, Wei Wei, Chenxie Wei, Ziyue Leng, Xingjun Lv, Jingyong He, Junyang Zhu, Chang Zhou, and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  6. Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, and 1 others. 2025. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957. — [arXiv:2501.01957](https://arxiv.org/abs/2501.01957) find
  7. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  8. Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. 2020. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 2881–2889. find
  9. Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, and 1 others. 2024. Omnibench: Towards the future of universal omni-language models. arXiv preprint arXiv:2409.15272.
  10. Zheng Lian, Rui Liu, Kele Xu, Bin Liu, Xuefei Liu, Yazhou Zhang, Xin Liu, Yong Li, Zebang Cheng, Haolin Zuo, and 1 others. 2025. find
  11. Mer 2025: When affective computing meets large language models. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 13837–13842. find
  12. Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, and 1 others. 2023a. find
  13. Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning. In Proceedings of the 31st ACM international conference on multimedia, pages 9610– 9614. find
  14. Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao. 2024. Affectgpt: Dataset and framework for explainable multimodal emotion recognition. arXiv preprint arXiv:2407.07653. — [arXiv:2407.07653](https://arxiv.org/abs/2407.07653) find
  15. Zheng Lian, Licai Sun, Mingyu Xu, Haiyang Sun, Ke Xu, Zhuofan Wen, Shun Chen, Bin Liu, and Jianhua Tao. 2023b. Explainable multimodal emotion reasoning. CoRR. find
  16. Zhiyu Lin, Yifei Gao, Xian Zhao, Yunfan Yang, and Jitao Sang. 2025. Mind with eyes: from language reasoning to multimodal reasoning. arXiv preprint arXiv:2503.18071. — [arXiv:2503.18071](https://arxiv.org/abs/2503.18071) find
  17. Steven R Livingstone and Frank A Russo. 2018. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391. find
  18. Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang-Yan Gui, Jan Kautz, Yu-Xiong Wang, and Zhiding Yu. 2025. Argus: Vision-centric reasoning with grounded chain-of-thought. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14268–14280. find
  19. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.
  20. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning. PMLR.
  21. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  22. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  23. Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. 2024a. Visual cot: Advancing multimodal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612–8642. find
  24. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024b. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  25. Yudi Shi, Shangzhe Di, Qirui Chen, and Weidi Xie. 2025. Enhancing video-llm reasoning via agent-of-thoughts distillation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8523–8533. find
  26. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
  27. Andrea Tocchetti, Lorenzo Corti, Agathe Balayn, Mireia Yurrita, Philip Lippmann, Marco Brambilla, and Jie Yang. 2025. Ai robustness: a human-centered perspective on technological challenges and opportunities. ACM Computing Surveys, 57(6):1–38. find
  28. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025a. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  29. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025b. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765.
  30. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  31. Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. 2025b. Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis. arXiv preprint arXiv:2501.09502. — [arXiv:2501.09502](https://arxiv.org/abs/2501.09502) find
  32. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
  33. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vi…
  34. Jiaxing Zhao, Xihan Wei, and Liefeng Bo. 2025a. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.05379. — [arXiv:2503.05379](https://arxiv.org/abs/2503.05379) find
  35. Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, and 1 others. 2025b. Humanomni: A large vision-speech language model for human-centric video understanding. arXiv preprint arXiv:2501.15111. — [arXiv:2501.15111](https://arxiv.org/abs/2501.15111) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-beyond-modality-collapse-taming-guided/