Elamur Research⌕
☰
home / papers / 2026 / CAIR: Causal Adaptive Information-based Reinforcement Learning for Multimodal Emotion Reasoning

CAIR: Causal Adaptive Information-based Reinforcement Learning for Multimodal Emotion Reasoning

Paper · 2026 · Fengyu Zhang, Bin Liu, Jianhua Tao et al.

Summary

The paper targets a weakness of multimodal emotion reasoning models that the authors call causal degeneracy. Models trained with SFT, or with RL rewards based only on accuracy or likelihood, often give fluent explanations that do not actually support the predicted emotion. CAIR is an RL framework built on GRPO. It starts from an SFT stage and treats the generated rationale as a causal mediator between the video and text input and the emotion label. A dual-path intrinsic verifier compares two predictions: one from the input alone and one from the input plus the rationale. The resulting Causal Mediation Reward (CMR) scores a rationale by how much it reduces predictive uncertainty. A Cognitive-load Adaptive Optimization (CLO) mechanism, motivated by the information bottleneck, gives more weight to ambiguous, high-entropy samples. CAIR reaches state-of-the-art accuracy on MTMEUR and competitive results on EmoBench-M, and its rationales are more faithful. The authors conclude that reward design matters more than model scale for genuine emotional reasoning.

Key points

  • CAIR reaches 73.80% accuracy on MTMEUR and 68.5% on the SCEA subset of EmoBench-M. It beats specialized SFT baselines by up to 14.4%.
  • The rationale's value is measured as the shift between the model's prediction without it and with it, where adding the rationale is treated as an intervention. Shuffled or irrelevant rationales sharply reduce this information gain (ΔH).
  • On CA-MER, CLO raises accuracy on samples with contradictory multimodal cues to 68.7%, compared with 58.4% for Uniform RL and 61.2% for Curriculum RL. Stability also improves: IQR falls to 3.8, from 8.2 and 6.5 respectively.
  • Compared with SFT, CAIR's prediction confidence clusters above 0.85, and the variance of its predicted probabilities is 42% narrower.
  • Limitations include:
    • reliance on ground-truth labels when computing rewards;
    • evaluation mostly on English benchmarks;
    • sensitivity to the base model's calibration.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🔬Institute of AutomationBeijing · China87 works · 14,883 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,744 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,685 worksSentiment & emotion · 1,035 worksVision & language · 811 works

Cite

Cite as Fengyu Zhang, Bin Liu, Jianhua Tao, Zhuofan Wen, Shun Chen, Hailiang Yao and Zhengqi Wen. 2026. CAIR: Causal Adaptive Information-based Reinforcement Learning for Multimodal Emotion Reasoning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1263

@inproceedings{zhang2026cair,
  title = {{CAIR: Causal Adaptive Information-based Reinforcement Learning for Multimodal Emotion Reasoning}},
  author = {Fengyu Zhang, Bin Liu, Jianhua Tao, Zhuofan Wen, Shun Chen, Hailiang Yao and Zhengqi Wen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1263},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1263}
}

References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 1(2):3.
  2. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  3. Linda Camras. 1980. Emotion: a psychoevolutionary synthesis. find
  4. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, and 1 others. 2024. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472–19495. find
  5. Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024a. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805–110853. find
  6. Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and 1 others. 2024b. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476.
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  9. Yumeng Fu, Junjie Wu, Zhongjie Wang, Meishan Zhang, Yulin Wu, and Bingquan Liu. 2025. Bemerc: Behavior-aware mllm-based framework for multimodal emotion recognition in conversation. arXiv preprint arXiv:2503.23990. — [arXiv:2503.23990](https://arxiv.org/abs/2503.23990) find
  10. Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, and Xun Yang. 2025. Benchmarking and bridging emotion conflicts for multimodal emotion reasoning. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 5528–5537. find
  11. He Hu, Yucheng Zhou, Lianzhong You, Hongbo Xu, Qianning Wang, Zheng Lian, Fei Richard Yu, Fei Ma, and Laizhong Cui. 2025a. Emobench-m: Benchmarking emotional intelligence for multimodal large language models. arXiv preprint arXiv:2502.04424. — [arXiv:2502.04424](https://arxiv.org/abs/2502.04424) find
  12. Jinpeng Hu, Hongchang Shi, Chongyuan Dai, Zhuo Li, Peipei Song, and Meng Wang. 2025b. Beyond emotion recognition: A multi-turn multimodal emotion understanding and reasoning benchmark. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 5814–5823. find
  13. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, and 1 others. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395.
  14. Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. 2024a. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14271–14280. find
  15. Zhaopei Huang, Jinming Zhao, and Qin Jin. 2024b. Ecr-chain: Advancing generative language models to better emotion-cause reasoners through reasoning chains. arXiv preprint arXiv:2405.10860. — [arXiv:2405.10860](https://arxiv.org/abs/2405.10860) find
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  17. Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. 2024. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700–13710. find
  18. Garapati Keerthana and Manik Gupta. 2025. Towards emotionally intelligent and responsible reinforcement learning. arXiv preprint arXiv:2511.10573. — [arXiv:2511.10573](https://arxiv.org/abs/2511.10573) find
  19. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  20. Yuanyuan Sun and Ting Zhou. 2025. Dialoguemllm: transforming multimodal emotion recognition in conversation through instruction-tuned mllm. IEEE Access. find
  21. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, page 6558.
  22. Cong Wang, Changfeng Gao, Yang Xiang, Zhihao Du, Keyu An, Han Zhao, Qian Chen, Xiangang Li, Yingming Gao, and Ya Li. 2025a. Rrpo: Robust reward policy optimization for llm-based emotional tts. arXiv preprint arXiv:2512.04552. — [arXiv:2512.04552](https://arxiv.org/abs/2512.04552) find
  23. Peisong Wang, Ruotian Ma, Bang Zhang, Xingyu Chen, Zhiwei He, Kang Luo, Qingsong Lv, Qingxuan Jiang, Zheng Xie, Shanyi Wang, and 1 others. 2025b. Rlver: Reinforcement learning with verifiable emotion rewards for empathetic agents. arXiv preprint arXiv:2507.03112. — [arXiv:2507.03112](https://arxiv.org/abs/2507.03112) find
  24. Pengcheng Li, Botao Zhao, Zuheng Kang, Junqing Peng, Xiaoyang Qu, Yayun He, and Jianzong Wang. 2025. Emo-rl: Emotion-rule-based reinforcement learning enhanced audio-language model for generalized speech emotion recognition. In Findings of the Association for Computational Linguistics: EMNLP 2025, p… find
  25. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  26. Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao. 2024. Affectgpt: Dataset and framework for explainable multimodal emotion recognition. arXiv preprint arXiv:2407.07653. — [arXiv:2407.07653](https://arxiv.org/abs/2407.07653) find
  27. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  28. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 5971–5984.
  29. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12585– 12602.
  30. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.
  31. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. find
  32. Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. 2025a. Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis. arXiv preprint arXiv:2501.09502. — [arXiv:2501.09502](https://arxiv.org/abs/2501.09502) find
  33. Qu Yang, Mang Ye, and Bo Du. 2024. Emollm: Multimodal emotional understanding meets large language models. arXiv preprint arXiv:2406.16442. — [arXiv:2406.16442](https://arxiv.org/abs/2406.16442) find
  34. Ting Yang, Li Chen, and Huimin Wang. 2025b. Towards open-ended emotional support conversations in llms via reinforcement learning with future-oriented rewards. arXiv preprint arXiv:2508.12935. — [arXiv:2508.12935](https://arxiv.org/abs/2508.12935) find
  35. Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807–8817. find
  36. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL, pages 2236–2246.
  37. Hang Zhang, Xin Li, and Lidong Bing. 2023. Videollama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858.
  38. Ren Zhang, Guoliang Xu, Tianyu Zhou, Junjun Zhao, Bo Yu, Zhicheng Zhang, and Jianqin Yin. 2025. Emo-synergy: Synergizing task-specific and general experts for multimodal emotion recognition. In Proceedings of the 3rd International Workshop on Multimodal and Responsible Affective Computing, pages 30–… find
  39. Jiaxing Zhao, Xihan Wei, and Liefeng Bo. 2025. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.05379. — [arXiv:2503.05379](https://arxiv.org/abs/2503.05379) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-cair-causal-adaptive-information-based/