Elamur Research⌕
☰
home / papers / 2026 / DEAR: Distributional Error-Aware Reliability for Robust Multimodal Sentiment Analysis with Missing Modalities

DEAR: Distributional Error-Aware Reliability for Robust Multimodal Sentiment Analysis with Missing Modalities

Paper · 2026 · Shihao Zou, Wei Wei and Yongshuo Zhang

Summary

Multimodal sentiment analysis (MSA) systems are usually trained on complete text, audio and video, but in deployment some modalities are often partly or fully absent, and accuracy drops sharply. The authors treat this as a domain shift, with full-modality inputs as the source and incomplete inputs as the target. Their framework, DEAR, has two parts. The first is a reconstruction module, HDCR. Instead of matching features point by point with MSE, it uses Maximum Mean Discrepancy (MMD) in an RKHS to pull completed features back onto the distribution of the original features. The second is a reliability estimator, SURE (built from a Transfer Difficulty Estimator and a Monotonic Weighting Network). It scores how far each modality has drifted for each sample and gives drifted modalities less weight. A learnable gate in the SR-DS dual-stream design then blends a cross-modal fusion stream with a conservative reliability-weighted stream. On MOSI, MOSEI and SIMS, DEAR is reported to beat recent baselines such as CorrKD and HRLF across most missing-modality settings. This matters for sentiment systems that must stay dependable when inputs are noisy or incomplete.

Key points

  • Simulating missing data: missingness follows the LNLN protocol, with random rates from 0% to 100%. Missing audio and visual features are set to zero, and missing text tokens are replaced with [UNK]. The authors also test settings where whole modalities are dropped.
  • Distribution alignment on MOSI: after 100 epochs, the Wasserstein distance between reconstructed and original features was about 0.31 with MSE-based reconstruction, versus about 0.007 with the HDCR loss.
  • Single-modality comparison (Table 4): F1 scores are reported for all six available-modality subsets. Language alone is the most informative single modality, and DEAR achieves the best results in most patterns, with an average F1 gain over HRLF on MOSEI.
  • Reliability analysis: results cover 10 missingness levels averaged over 3 seeds. As data loss grows, the gate shifts weight toward the robust stream, and audio-visual weights fall to about 0.15 while emphasis moves to text.
  • Limitations stated by the authors:
    • Some baseline scores were copied from earlier papers rather than rerun.
    • HDCR costs somewhat more compute than MSE-based completion.
    • More complex real-world joint missingness patterns remain untested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksVision & language · 657 works

Cite

Cite as Shihao Zou, Wei Wei and Yongshuo Zhang. 2026. DEAR: Distributional Error-Aware Reliability for Robust Multimodal Sentiment Analysis with Missing Modalities. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1517

@inproceedings{zou2026dear,
  title = {{DEAR: Distributional Error-Aware Reliability for Robust Multimodal Sentiment Analysis with Missing Modalities}},
  author = {Shihao Zou, Wei Wei and Yongshuo Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1517},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1517}
}

References

43 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine Learning, 79:151–175. find
  2. Bingfeng Chen, Qihan Ouyang, Yongqi Luo, Boyan Xu, Ruichu Cai, and Zhifeng Hao. 2024. S 2 gsl: Incorporating segment to syntactic enhanced graph structure learning for aspect-based sentiment analysis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 1… find
  3. Yi Ding, Neethu Robinson, Chengxuan Tong, Qiuhao Zeng, and Cuntai Guan. 2024. Lggnet: Learning from local-global-graph representations for brain-computer interface. IEEE Transactions on Neural Networks and Learning Systems, 35:9773–9786. find
  4. Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. 2006. A kernel method for the two-sample-problem. In Advances in Neural Information Processing Systems, pages 513–520. find
  5. Jiwei Guo, Jiajia Tang, Weichen Dai, Yu Ding, and Wanzeng Kong. 2022. Dynamically adjust word representations using unaligned multimodal information. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3394–3402. find
  6. Zirun Guo, Tao Jin, and Zhou Zhao. 2024. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 1726–1736. find
  7. Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9180–9192.
  8. Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. MISA: modality-invariant and -specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1122–1131. find
  9. Ye He, Shihao Zou, Yuzhe Chen, and Xianying Huang. 2024. C 3 lpgcn:integrating contrastive learning and cooperative learning with prompt into graph convolutional network for aspect-based sentiment analysis. In Findings of the Association for Computational Linguistics: NAACL, pages 3237–3247. find
  10. Geoffrey E Hinton and Sam Roweis. 2002. Stochastic neighbor embedding. In Advances in neural information processing systems. find
  11. Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. 2021. What makes multi-modal learning better than single (provably). In Advances in Neural Information Processing Systems, pages 10944–10956. find
  12. Mingcheng Li, Dingkang Yang, Yuxuan Lei, Shunli Wang, Shuaibing Wang, Liuzhen Su, Kun Yang, Yuzheng Wang, Mingyang Sun, and Lihua Zhang. 2024a. A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities. In Proceedings of the AAAI Conference on Artifici… find
  13. Mingcheng Li, Dingkang Yang, Yang Liu, Shunli Wang, Jiawei Chen, Shuaibing Wang, Jinjie Wei, Yue Jiang, Qingyao Xu, Xiaolu Hou, Mingyang Sun, Ziyun Qian, Dongliang Kou, and Lihua Zhang. 2024b. Toward robust incomplete multimodal sentiment analysis via hierarchical representation learning. In Advance… find
  14. Mingcheng Li, Dingkang Yang, Xiao Zhao, Shuaibing Wang, Yan Wang, Kun Yang, Mingyang Sun, Dongliang Kou, Ziyun Qian, and Lihua Zhang. 2024c. Correlation-decoupled knowledge distillation for multimodal sentiment analysis with incomplete modalities. In IEEE/CVF Conference on Computer Vision and Patter… find
  15. Wenjin Tian, Xianying Huang, and Shihao Zou. 2025. Multi-condition guided diffusion network for multimodal emotion recognition in conversation. In Findings of the Association for Computational Linguistics: NAACL, pages 3215–3227. find
  16. Yong Li, Yuanzhi Wang, and Zhen Cui. 2023. Decoupled multimodal distilling for emotion recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 6631–6640. find
  17. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558–6569.
  18. Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. 2023. Gcnet: Graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:8419–8432. find
  19. Di Wang, Xutong Guo, Yumin Tian, Jinhui Liu, Lihuo He, and Xuemei Luo. 2023a. TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis. Pattern Recognition, 136:109259. find
  20. Jingjun Liang, Ruichen Li, and Qin Jin. 2020. Semi-supervised multi-modal emotion recognition with cross-modal distribution matching. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2852–2861. find
  21. Zhizhong Liu, Bin Zhou, Dianhui Chu, Yuhang Sun, and Lingqiang Meng. 2024. Modality translation-based multimodal sentiment analysis under uncertain missing modalities. Information Fusion, 101:101973. find
  22. Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. 2021. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2554–2562. find
  23. Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. In The Thirty-Third AAAI Conference on Artificial Intelligence, pages 6892–6899. find
  24. Wasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Mohammed E. Hoque. 2020. Integrating multimodal information in large pre-trained transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pa… find
  25. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 8821–8831.
  26. Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin. 2022. Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation. In Proceedings of the 30st ACM International Conference on Multimedia, pages 3722– 3729. find
  27. Di Wang, Shuai Liu, Quan Wang, Yumin Tian, Lihuo He, and Xinbo Gao. 2023b. Cross-modal enhancement network for multimodal sentiment analysis. IEEE Transactions on Multimedia,, 25:4909–4921. find
  28. Yuanzhi Wang, Zhen Cui, and Yong Li. 2023c. Distribution-consistent modal recovering for incomplete multimodal learning. In IEEE International Conference on Computer Vision, pages 21968– 21977. find
  29. Yuanzhi Wang, Yong Li, and Zhen Cui. 2023d. Incomplete multimodality-diffused emotion recognition. In Advances in Neural Information Processing Systems, volume 36, pages 17117–17128. find
  30. Dingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. 2022. Disentangled representation learning for multimodal emotion recognition. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1642–1651. find
  31. Mingzheng Yang, Kai Zhang, Yuyang Ye, Yanghai Zhang, Runlong Yu, and Min Hou. 2025. Decoupling and reconstructing: A multimodal sentiment analysis framework towards robustness. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2025, Montreal, Canada… find
  32. Shuo Yin and Guoqiang Zhong. 2024. Textgt: A double-view graph transformer on text for aspect-based sentiment analysis. In Thirty-Eighth AAAI Conference on Artificial Intelligence, pages 19404–19412. find
  33. Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. CH-SIMS: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3…
  34. Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, pages 10790–10797. find
  35. Ziqi Yuan, Wei Li, Hua Xu, and Wenmeng Yu. 2021. Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4400–4407. find
  36. Ziqi Yuan, Yihe Liu, Hua Xu, and Kai Gao. 2024. Noise imitation based adversarial training for robust multimodal sentiment analysis. IEEE Transactions on Multimedia, 26:529–539. find
  37. Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2236–2246.
  38. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82–88. find
  39. Haoyu Zhang, Wenbin Wang, and Tianshu Yu. 2024. Towards robust multimodal sentiment analysis with incomplete data. In Advances in Neural Information Processing Systems. find
  40. Haoyu Zhang, Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, and Tianshu Yu. 2023. Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 756–767. find
  41. Xiangmin Zhang, Wei Wei, and Shihao Zou. 2025. Modal feature optimization network with prompt for multimodal sentiment analysis. In Proceedings of the 31st International Conference on Computational Linguistics, COLING, pages 4611–4621. find
  42. Sicheng Zhao, Zhenhua Yang, Henglin Shi, Xiaocheng Feng, Lingpengkun Meng, Bing Qin, Chenggang Yan, Jianhua Tao, and Guiguang Ding. 2025. Sdrs: Sentiment-aware disentangled representation shifting for multimodal sentiment analysis. IEEE Transactions on Affective Computing, pages 1–13. find
  43. Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, and Ning An. 2025. Proxy-driven robust multimodal sentiment analysis with incomplete data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 22123–22138. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2026-dear-distributional-error-aware-reliability/