Elamur Research⌕
☰
home / papers / 2026 / Fast Retrieval and Slow Reasoning for Explainable Multimodal Sentiment Analysis

Fast Retrieval and Slow Reasoning for Explainable Multimodal Sentiment Analysis

Paper · 2026 · Aoqiang Zhu, Min Hu and Yan Xing

Summary

Typical multimodal sentiment analysis (MSA) models merge language, audio and video holistically and weight every modality and time step alike. This lets irrelevant signals, such as recurring backgrounds or a speaker's habitual rhythm, distort predictions and makes it hard to see which cues drove a decision. Drawing on Kahneman's dual-process theory, the authors propose FRSR (Fast Retrieval and Slow Reasoning), which splits the work into two stages:

  • Fast Pathway: a cheap Conv1D-based salience scorer, stabilized by auxiliary unimodal supervision, keeps only the Top-K most sentiment-relevant tokens per modality.
  • Slow Pathway: learnable reasoning tokens perform hierarchical cross-modal reasoning over that small evidence set only.

Inputs use BERT, librosa and OpenFace features. On MOSI, MOSEI and SIMS, FRSR reaches competitive accuracy. Compared with the LNLN baseline, it trains over twice as fast per epoch, has slightly fewer parameters and copes better with noise. Because the retained cues are explicit, users can inspect the evidence behind each prediction. This offers a route to cheaper, more transparent multimodal affect models.

Key points

  • Top-K setting: on MOSI, K = 4 is best overall (Acc-2 85.07/87.20, F1 85.22/87.15, MAE 0.661, Corr 0.819). It beats keeping the full 50-token sequence (MAE 0.695). A fixed K = 4 is used across all datasets.
  • Efficiency vs. LNLN: 7.856 s vs. 17.143 s per epoch (about 2.18× faster), with 111.302M vs. 115.965M parameters.
  • Missing modalities (MOSI): text alone gives Acc-7 45.42, while audio+visual without text gives only 22.71. Adding modalities to text improves results further.
  • Benchmarks and features: MOSI, MOSEI and SIMS. Features come from BERT (text), librosa (audio) and OpenFace (visual), each refined by two Transformer layers.
  • Stated limitations: a static K cannot adapt to inputs of different information density, and cues missed by the Fast Pathway cannot be recovered later. The reasoning tokens also remain latent and do not map onto human-readable concepts such as sarcasm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hefei University of TechnologyHefei · China17 works · 1,702 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 4Research subjects and languages the work is about.Reasoning · 1,685 worksSentiment & emotion · 1,035 worksVision & language · 811 worksInformation retrieval & search · 552 works

Cite

Cite as Aoqiang Zhu, Min Hu and Yan Xing. 2026. Fast Retrieval and Slow Reasoning for Explainable Multimodal Sentiment Analysis. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhu2026fast,
  title = {{Fast Retrieval and Slow Reasoning for Explainable Multimodal Sentiment Analysis}},
  author = {Aoqiang Zhu, Min Hu and Yan Xing},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1519},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1519}
}

References

46 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 59–66. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  3. Andrius Dzedzickis, Gediminas Vaičiūnas, Karolina Lapkauskaitė, Darius Viržonis, and Vytautas Bučin-skas. 2025. Recent advances in human–robot interaction: robophobia or synergy. Journal of Intelligent Manufacturing, 36(4):2281–2307. find
  4. Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9180–9192.
  5. Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, pages 1122–1131. find
  6. Jian Huang, Yanli Ji, Zhen Qin, Yang Yang, and Heng Tao Shen. 2023. Dominant single-modal supplementary fusion (simsuf) for multimodal sentiment analysis. IEEE Transactions on Multimedia, 26:8383–8394. find
  7. Daniel Kahneman. 2003. Maps of bounded rationality: Psychology for behavioral economics. American economic review, 93(5):1449–1475. find
  8. Daniel Kahneman. 2011. macmillan. find
  9. Jiang Li, Xiaoping Wang, and Zhigang Zeng. 2025. Tracing intricate cues in dialogue: Joint graph structure and sentiment dynamics for multimodal emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  10. Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: L…
  11. Sijie Mai, Ying Zeng, Shuangjia Zheng, and Haifeng Hu. 2022. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis. IEEE Transactions on Affective Computing, 14(3):2276– 2289. find
  12. Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python. SciPy, 2015:18–24. find
  13. Arpan Phukan, Santanu Pal, and Asif Ekbal. 2024. Hybrid quantum-classical neural network for multimodal multitask sarcasm, emotion, and sentiment analysis. IEEE Transactions on Computational Social Systems, 11(5):5740–5750. find
  14. Fan Qian, Jiqing Han, Yongjun He, Tieran Zheng, and Guibin Zheng. 2023. Sentiment knowledge enhanced self-supervised learning for multimodal sentiment analysis. In Findings of the association for computational linguistics: ACL 2023, pages 12966–12978. find
  15. Tao Shi and Shao-Lun Huang. 2023. Multiemo: An attention-based correlation-aware multimodal fusion framework for emotion recognition in conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14752–14766. find
  16. Upendra Singh, Kumar Abhishek, and Hiteshwar Kumar Azad. 2024. A survey of cutting-edge multimodal sentiment analysis. ACM Computing Surveys, 56(9):1–38. find
  17. Keith E. Stanovich and Richard F. West. 2000. Individual differences in reasoning: Implications for the rationality debate. Behavioral and Brain Sciences, 23:645–726. find
  18. Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. 2023. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis. IEEE Transactions on Affective Computing, 15(1):309–325. find
  19. Chuanqi Tao, Jiaming Li, Tianzi Zang, and Peng Gao. 2025. A multi-focus-driven multi-branch network for robust multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1547–1555. find
  20. Mani Kumar Tellamekala, Shahin Amiriparian, Björn W Schuller, Elisabeth André, Timo Giesbrecht, and Michel Valstar. 2023. Cold fusion: Calibrated and ordinal latent distribution fusion for uncertainty-aware multimodal emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligenc… find
  21. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, page 6558.
  22. Di Wang, Xutong Guo, Yumin Tian, Jinhui Liu, LiHuo He, and Xuemei Luo. 2023. Tetfn: A text enhanced transformer fusion network for multimodal sentiment analysis. Pattern Recognition, 136:109259. find
  23. Di Wang, Shuai Liu, Quan Wang, Yumin Tian, Lihuo He, and Xinbo Gao. 2022. Cross-modal enhancement network for multimodal sentiment analysis. IEEE Transactions on Multimedia, 25:4909–4921. find
  24. Pan Wang, Qiang Zhou, Yawen Wu, Tianlong Chen, and Jingtong Hu. 2025. Dlf: Disentangled-language-focused multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 21180–21188. find
  25. Yuhua Wen, Qifei Li, Yingying Zhou, Yingming Gao, Zhengqi Wen, Jianhua Tao, and Ya Li. 2025. Dashfusion: Dual-stream alignment with hierarchical bottleneck fusion for multimodal sentiment analysis. IEEE Transactions on Neural Networks and Learning Systems. find
  26. Sheng Wu, Dongxiao He, Xiaobao Wang, Longbiao Wang, and Jianwu Dang. 2025. Enriching multimodal sentiment analysis through textual emotional descriptions of visual-audio content. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1601–1609. find
  27. Dingkang Yang, Haopeng Kuang, Shuai Huang, and Lihua Zhang. 2022. Learning modality-specific and -agnostic representations for asynchronous multimodal language sequences. In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 1708–1717. Association for Computing Machiner… find
  28. Jiuding Yang, Yakun Yu, Di Niu, Weidong Guo, and Yu Xu. 2023. Confede: Contrastive feature decomposition for multimodal sentiment analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7617–7630. find
  29. Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 3…
  30. Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10790–10797. find
  31. Yakun Yu, Mingjun Zhao, Shi-ang Qi, Feiran Sun, Baoxun Wang, Weidong Guo, Xiaoli Wang, Lei Yang, and Di Niu. 2023. Conki: Contrastive knowledge injection for multimodal sentiment analysis. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13610–13624. find
  32. Ziqi Yuan, Wei Li, Hua Xu, and Wenmeng Yu. 2021. Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In Proceedings of the 29th ACM international conference on multimedia, pages 4400–4407. find
  33. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114.
  34. Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259. find
  35. AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume…
  36. Jiandian Zeng, Jiantao Zhou, and Caishi Huang. 2023. Exploring semantic relations for social media sentiment analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2382–2394. find
  37. Haoyu Zhang, Wenbin Wang, and Tianshu Yu. 2024. Towards robust multimodal sentiment analysis with incomplete data. Advances in Neural Information Processing Systems, 37:55943–55974. find
  38. Haoyu Zhang, Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, and Tianshu Yu. 2023a. Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 756–767. find
  39. Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. 2023b. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pages 41753–41769. PMLR. find
  40. Wei Zhang, Feng Qiu, Suzhen Wang, Hao Zeng, Zhimeng Zhang, Rudong An, Bowen Ma, and Yu Ding. 2022. Transformer-based multimodal information fusion for facial expression analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2428–2437. find
  41. Yazhou Zhang, Dawei Song, Xiang Li, Peng Zhang, Panpan Wang, Lu Rong, Guangliang Yu, and Bo Wang. 2020. A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis. Information Fusion, 62:14–31. find
  42. Aoqiang Zhu, Min Hu, Xiaohua Wang, and Fuji Ren. 2026. Beneficial noise learning for robust multimodal fusion. IEEE Transactions on Multimedia. find
  43. Aoqiang Zhu, Min Hu, Xiaohua Wang, Yan Xing, Yiming Tang, Jiaoyun Yang, Ning An, and Fuji Ren. 2025a. EaNet: Enhanced multimodal awareness alignment network for multimodal aspect-based sentiment analysis. IEEE Transactions on Affective Computing. find
  44. Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, and Ning An. 2025b. Proxy-driven robust multimodal sentiment analysis with incomplete data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22123–22138. find
  45. Aoqiang Zhu, Min Hu, Xiaohua Wang, Jiaoyun Yang, Yiming Tang, and Fuji Ren. 2024. KEBR: Knowledge enhanced self-supervised balanced representation for multimodal sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5732–5741. find
  46. Yan Zhuang, Yanru Zhang, Zheng Hu, Xiaoyue Zhang, Jiawen Deng, and Fuji Ren. 2024. Glomo: Global-local modal fusion for multimodal sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1800–1809. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2026-fast-retrieval-slow-reasoning-explainable/