Elamur Research⌕
☰
home / papers / 2026 / QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis

QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis

Paper · 2026 · Yitong Zhu, Yuxuan Jiang, Guanxuan Jiang et al.

Summary

Real-world multimodal sentiment analysis (MSA) systems receive text, audio and video that may be noisy, partially lost, or both at once. Prior robust methods usually treat these defects as separate categories or train for one fixed corruption ratio, so they break when test conditions shift. The authors propose a Continuous Reliability Spectrum. It places every input on a single scale, from clean data through gradual noise corruption to complete modality absence. It is formalized as a stochastic process with three evaluation protocols. On top of this, they build QA-MoE, a mixture-of-experts model. It represents each modality's features as a Gaussian distribution and uses the learned variance (aleatoric uncertainty, with no quality labels) to compute a reliability score. That score reweights the router, so unreliable inputs activate fewer experts. When everything is degraded, a learned dataset-level prior serves as a fallback. Experiments on MSA and intent recognition benchmarks report competitive or state-of-the-art results. A single trained model handles unseen noise and missingness levels without retraining, which the authors call "One-Checkpoint-for-All". This matters for deploying affective models with imperfect sensors.

Key points

  • Three protocols: The spectrum unifies Modality Missingness (Bernoulli drop rate η), Quality Degradation (Gaussian noise of intensity λ) and Stochastic Mixture (both λ and η sampled jointly).
  • Benchmarks: CMU-MOSI and CMU-MOSEI for MSA, plus IEMOCAP and MIntRec as cross-task datasets.
  • Ablation on CMU-MOSI (spectrum training):
    • The full model reaches MAE 0.525, ACC7 54.5, ACC2 88.8 and F1 89.1.
    • Removing quality gating raises MAE to 0.884.
    • Removing the variance term raises MAE to 0.795.
    • Removing the universal fallback (y_prior) raises MAE to 0.780 and drops F1 to 45.9.
  • Reliability score validity: Average MAE falls monotonically across reliability bins, from 1.42 for the least reliable inputs to 0.58 for the most reliable.
  • Expert count: With 8 experts in total, activating 3 per input gives the best trade-off. Using 1 underfits and using all 8 overfits.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China67 works · 1,765 citations🎓Aarhus UniversityAarhus · Denmark7 works · 12 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksVision & language · 811 worksQuestion answering · 487 works

Cite

Cite as Yitong Zhu, Yuxuan Jiang, Guanxuan Jiang, Bojing Hou, Peng Yuan Zhou, Ge Lin and Yuyang Wang. 2026. QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1461

@inproceedings{zhu2026qa,
  title = {{QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis}},
  author = {Yitong Zhu, Yuxuan Jiang, Guanxuan Jiang, Bojing Hou, Peng Yuan Zhou, Ge Lin and Yuyang Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1461},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1461}
}

References

42 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume…
  2. Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: An open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–10. find
  3. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. Iemocap: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335–359. find
  4. Lei Cai, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji. 2018. Deep adversarial learning for multi-modality missing data completion. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 1158–1166, New York, NY, USA. Associat… find
  5. Kezhou Chen, Shuo Wang, Huixia Ben, Shengeng Tang, and Yanbin Hao. 2025. Mixture of multimodal adapters for sentiment analysis. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long P… find
  6. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. 2024 IEEE/CVF Conference on… find
  7. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep — a collaborative voice analysis repository for speech technologies. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 960–964. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics.
  9. Yiyang Fang, Wenke Huang, Guancheng Wan, Kehua Su, and Mang Ye. 2025. Emoe: Modality-specific enhanced dynamic emotion experts. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14314–14324. find
  10. Zirun Guo, Tao Jin, and Zhou Zhao. 2024. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1726–1736, Bangkok, Thailand. Associat… find
  11. Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9180–9192, Online and Punta Cana, Dominican R…
  12. Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. Misa: Modality-invariant and -specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM International Conference on Multimedia, MM ’20, page 1122–1131, New York, NY, USA. Association for Computing Machin… find
  13. Kang He, Boyu Chen, Yuzhe Ding, Fei Li, Chong Teng, and Donghong Ji. 2025a. Pase: Prototype-aligned calibration and shapley-based equilibrium for multimodal sentiment analysis. Preprint, arXiv:2511.17585. — [arXiv:2511.17585](https://arxiv.org/abs/2511.17585) find
  14. Xilin He, Haijian Liang, Boyi Peng, Weicheng Xie, Muhammad Haris Khan, Siyang Song, and Zitong Yu. 2025b. Msamba: Exploring multimodal sentiment analysis with state space models. Proceedings of the AAAI Conference on Artificial Intelligence, 39(2):1309–1317. find
  15. Guanxuan Jiang, Shirao Yang, Yuyang Wang, and Pan Hui. 2026. When trust collides: Exploring human-llm cooperation intention through the prisoner’s dilemma. International Journal of Human-Computer Studies, page 103740. find
  16. Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. 2023. Gcnet: Graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8419–8432. find
  17. Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, and 1 others. 2021. Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 2021(DB1):1. find
  18. Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: L…
  19. Mengmeng Ma, Jian Ren, Long Zhao, S. Tulyakov, Cathy Wu, and Xi Peng. 2021. Smil: Multimodal learning with severely missing modality. ArXiv, abs/2103.05677. find
  20. Sijie Mai, Shiqin Han, and Haifeng Hu. 2025. Supervised attention mechanism for low-quality multimodal data. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21377–21397, Suzhou, China. Association for Computational Linguistics. find
  21. Sijie Mai, Ya Sun, Aolin Xiong, Ying Zeng, and Haifeng Hu. 2024. Multimodal boosting: Addressing noisy modalities and identifying modality contribution. IEEE Transactions on Multimedia, 26:3018–3033. find
  22. Sijie Mai, Ying Zeng, and Haifeng Hu. 2023. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. Trans. Multi., 25:4121–4134. find
  23. David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 2023. 4m: massively multimodal masked modeling. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc… find
  24. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  25. Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019. Found in translation: learning robust joint representations by cyclic translations between modalities. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative… find
  26. Chengkai Shi and Yunhua Zhang. 2025. Mmkt: Multimodal sentiment analysis model based on knowledge-enhanced and text-guided learning. Applied Sciences, 15(17). find
  27. Kaili Sun, Zhiwen Xie, Mang Ye, and Huyin Zhang. 2024. Contextual augmented global contrast for multimodal intent recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26963–26973. find
  28. Kaiwei Sun and Mi Tian. 2025. Sequential fusion of text-close and text-far representations for multimodal sentiment analysis. In Proceedings of the 31st International Conference on Computational Linguistics, pages 40–49, Abu Dhabi, UAE. Association for Computational Linguistics. find
  29. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558– 6569,…
  30. Yuanzhi Wang, Yong Li, and Zhen Cui. 2023. Incomplete multimodality-diffused emotion recognition. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. find
  31. Wenxin Xu, Hexin Jiang, and Xuefeng Liang. 2024. Leveraging knowledge of modality experts for incomplete multimodal learning. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 438–446, New York, NY, USA. Association for Computing Machinery. find
  32. Yang Yang, Xunde Dong, and Yupeng Qiang. 2024. CLGSI: A multimodal sentiment analysis framework based on contrastive learning guided by sentiment intensity. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2099–2110, Mexico City, Mexico. Association for Computational L… find
  33. Amir Zadeh, Minghai Chen, Soujanya Poria, E. Cambria, and Louis philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Conference on Empirical Methods in Natural Language Processing.
  34. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82–88. find
  35. Hanlei Zhang, Hua Xu, Xin Wang, Qianrui Zhou, Shaojie Zhao, and Jiayan Teng. 2022. Mintrec: A new dataset for multimodal intent recognition. In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 1688–1697, New York, NY, USA. Association for Computing Machinery. find
  36. Haoyu Zhang, Wenbin Wang, and Tianshu Yu. 2024. Towards robust multimodal sentiment analysis with incomplete data. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. find
  37. Haoyu Zhang, Yinan Zhang, Chaolong Ying, Xiaoying Tang, and Tianshu Yu. 2025. Improving task-specific multimodal sentiment analysis with general mllms via prompting. In Advances in Neural Information Processing Systems. find
  38. Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. 2023. Meta-transformer: A unified framework for multimodal learning. Preprint, arXiv:2307.10802. — [arXiv:2307.10802](https://arxiv.org/abs/2307.10802) find
  39. Jinming Zhao, Ruichen Li, and Qin Jin. 2021. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc… find
  40. Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, and Ning An. 2025a. Multimodal invariant sentiment representation learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14743–14755, Vienna, Austria. Association for Computational Linguistics. find
  41. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2024. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. Preprint, arXiv:2310.01852. — [arXiv:2310.01852](https://arxiv.org/abs/2310.01852) find
  42. Yitong Zhu, Lei Han, Guanxuan Jiang, PengYuan Zhou, and Yuyang Wang. 2025b. Hierarchical moe: Continuous multimodal emotion recognition with incomplete and asynchronous inputs. Preprint, arXiv:2508.02133. — [arXiv:2508.02133](https://arxiv.org/abs/2508.02133) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2026-qa-moe-continuous-reliability-spectrum/