Elamur Research⌕
☰
home / papers / 2026 / Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

Paper · 2026 · Jiayu Zhang, Shuo Ye, Qilang Ye et al.

Summary

Audio-Visual Question Answering (AVQA) models usually assume that video, sound and the question text are all present. In real deployments a sensor may fail or a stream may drop, and accuracy then falls sharply. Most earlier fixes for missing modalities generate substitute features from the remaining inputs. The authors argue that such synthesized features drift toward generic, shared content and lose the fine detail needed for reasoning, such as one specific instrument's timbre.

Their framework, R²ScP, replaces generation with retrieval. The available modality queries a memory bank built in a unified embedding space (ImageBind), and the closest real features of the missing modality are fetched. A Context-aware Adaptive Purification (CAP) module then compares these candidates with the present modalities and suppresses inconsistent noise. Modality-specific experts are trained separately first and then combined through a soft router.

On Music-AVQA and other benchmarks, R²ScP outperforms earlier incomplete-modality methods. Its advantage grows as more data is missing, which suggests retrieval is a practical route to robust multimodal QA.

Key points

  • Retrieval step: a memory bank of key–feature pairs is indexed with ImageBind embeddings. Candidates are ranked by cosine similarity to the available modality, and the top-n are kept. The best setting is n = 3, since more candidates add noise.
  • Purification and training: CAP's purification budget peaks at k = 5, following an inverted-U trend. Training uses a two-stage mixture-of-experts scheme: independent expert training, then expert mixing. The router gives retrieved features weights comparable to those of the real visual or audio inputs.
  • Robustness to missing data: the method was tested at 30%, 50% and 70% missing rates. R²ScP leads at every level, while baselines such as Missing-AVQA and SimMLM degrade more steeply.
  • Choice of retrieval corpus: an in-domain Music-AVQA corpus gives 70.72% accuracy, compared with 66.92% for VGGSound and 66.51% for AVQA. This indicates that domain match matters most, and that corpus size helps within the same domain.
  • Limitations: results depend on the quality of retrieved samples. Missing modalities are handled only at inference time, not during training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations🎓Tsinghua Shenzhen International Graduate SchoolShenzhen · China28 works · 3,035 citations🎓Nankai UniversityTianjin · China24 works · 762 citations🎓Great Bay UniversityDongguan · China3 works · 3 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 3Research subjects and languages the work is about.Vision & language · 657 worksQuestion answering · 441 worksSpeech & spoken language · 480 works

Cite

Cite as Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang and Zitong YU. 2026. Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.961

@inproceedings{zhang2026retrieving,
  title = {{Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification}},
  author = {Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang and Zitong YU},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.961},
  url = {https://doi.org/10.18653/v1/2026.acl-long.961}
}

References

39 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lei Cai, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji. 2018. Deep adversarial learning for multi-modality missing data completion. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1158–1166. find
  2. Huilin Chen, Miaomiao Cai, Fan Liu, Zhiyong Cheng, Richang Hong, and Meng Wang. 2025. I3-mrec: Invariant learning with information bottleneck for incomplete modality recommendation. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 6133–6142. find
  3. Zailong Chen, Lei Wang, Peng Wang, and Peng Gao. 2023. Question-aware global-local video understanding network for audio-visual question answering. IEEE Transactions on Circuits and Systems for Video Technology, 34(5):4109–4119. find
  4. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190.
  5. Hongyeob Kim, Inyoung Jung, Dayoon Suh, Youjia Zhang, Sangmin Lee, and Sungeun Hong. 2025. Question-aware gaussian experts for audio-visual question answering. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13681–13690. find
  6. Guangyao Li, Henghui Du, and Di Hu. 2024a. Boosting audio visual question answering via key semantic-aware cues. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5997–6005. find
  7. Guangyao Li, Wenxuan Hou, and Di Hu. 2023. Progressive spatio-temporal perception for audio-visual question answering. In Proceedings of the 31st ACM international conference on multimedia, pages 7808– 7816. find
  8. Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19108–19118. find
  9. Sijie Li, Chen Chen, and Jungong Han. 2025a. Simmlm: A simple framework for multi-modal learning with missing modality. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 24068–24077. find
  10. Zhangbin Li, Dan Guo, Jinxing Zhou, Jing Zhang, and Meng Wang. 2024b. Object-aware adaptive-positivity learning for audio-visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3306–3314. find
  11. Ziyi Li, Wei-Long Zheng, and Bao-Liang Lu. 2025b. Multimodal emotion recognition with missing modality via a unified multi-task pre-training framework. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 5717–5725. find
  12. Xun Lin, Xiaobao Guo, Taorui Wang, Yingjie Ma, Jiajian Huang, Jiayu Zhang, Junzhe Cao, and Zitong Yu. 2025. find
  13. Svc 2025: the first multimodal deception detection challenge. In Proceedings of the 1st International Workshop & Challenge on Subtle Visual Computing, pages 59–64. find
  14. Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. 2022. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18177–18186. find
  15. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
  16. Kyu Ri Park, Hong Joo Lee, and Jung Uk Kim. 2024. Learning trimodal relation for audio-visual question answering with missing modality. In European Conference on Computer Vision, pages 42–59. Springer. find
  17. Baoqi Pei, Yifei Huang, Guo Chen, Jilan Xu, Yali Wang, Limin Wang, Tong Lu, Yu Qiao, and Fei Wu. 2025. Guiding audio-visual question answering with collective question reasoning. International Journal of Computer Vision, pages 1–18. find
  18. Pengjie Tang, Jiayu Zhang, Hanli Wang, Yunlan Tan, and Yun Yi. 2025. Srvc-la: Sparse regularization of visual context and latent attention based model for video description. Neurocomputing, 630:129639. find
  19. Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. 2017. Missing modalities imputation via cascaded residual autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1405–1414. find
  20. Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. 2023a. Multimodal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15878–15887. find
  21. Taorui Wang, Xun Lin, Yong Xu, Qilang Ye, Dan Guo, Sergio Escalera, Ghada Khoriba, and Zitong Yu. 2026. Micro-gesture recognition: A comprehensive survey of datasets, methods, and challenges. Machine Intelligence Research, 23(2):308–330. find
  22. Yuanzhi Wang, Yong Li, and Zhen Cui. 2023b. Incomplete multimodality-diffused emotion recognition. Advances in Neural Information Processing Systems, 36:17117–17128. find
  23. Renjie Wu, Hu Wang, Hsiang-Ting Chen, and Gustavo Carneiro. 2024. Deep multimodal learning with missing modality: A survey. arXiv preprint arXiv:2409.07825. — [arXiv:2409.07825](https://arxiv.org/abs/2409.07825) find
  24. Xinyu Xie, Yawen Cui, Tao Tan, Xubin Zheng, and Zitong Yu. 2024. Fusionmamba: Dynamic feature enhancement for multimodal image fusion with mamba. Visual Intelligence, 2(1):37. find
  25. Jiayi Xin, Sukwon Yun, Jie Peng, Inyoung Choi, Jenna L Ballard, Tianlong Chen, and Qi Long. 2025. I2moe: Interpretable multimodal interaction-aware mixture-of-experts. arXiv preprint arXiv:2505.19190. — [arXiv:2505.19190](https://arxiv.org/abs/2505.19190) find
  26. Wenxin Xu, Hexin Jiang, and Xuefeng Liang. 2024. Leveraging knowledge of modality experts for incomplete multimodal learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 438–446. find
  27. Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. 2022. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pages 3480–3491. find
  28. Qilang Ye, Wei Zeng, Meng Liu, Jie Zhang, Yupeng Hu, Zitong Yu, and Yu Zhou. 2026a. When eyes and ears disagree: Can mllms discern audio-visual confusion? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 11955–11963. find
  29. Qilang Ye, Yu Zhou, Lian He, Jie Zhang, Xuanming Guo, Jiayu Zhang, Mingkui Tan, Weicheng Xie, Yue Sun, Tao Tan, and 1 others. 2026b. Sugar: Learning skeleton representation with visual-motion knowledge for action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 4… find
  30. Shuo Ye, Lixin Chen, Qiaoqi Li, Jiayu Zhang, Chaomeng Chen, and Shutao Xia. 2026c. Ika2: Internal knowledge adaptive activation for robust recognition in complex scenarios. Machine Intelligence Research, 23(2):429–443. find
  31. Zhi Zeng, Jiaying Wu, Minnan Luo, Herun Wan, Xiangzheng Kong, Zihan Ma, Guang Dai, and Qinghua Zheng. 2025. Imol: Incomplete-modality-tolerant learning for multi-domain fake news video detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo…
  32. Jiayu Zhang, Xun Lin, Jiajian Huang, Shuo Ye, Xiaobao Guo, Dongliang Zhu, Ruimin Hu, Dan Guo, Yanyan Liang, Zitong Yu, and 1 others. 2026. Multimodal deception detection: A survey. Machine Intelligence Research, 23(2):284–307. find
  33. Jiayu Zhang, Pengjie Tang, Yunlan Tan, and Hanli Wang. 2025a. Mgtr-miss: More ground truth retrieving based multimodal interaction and semantic supervision for video description. Neural Networks, page 107817. find
  34. Jiayu Zhang, Qilang Ye, Shuo Ye, Xun Lin, Zihan Song, and Zitong Yu. 2025b. Av-master: Dual-path comprehensive perception makes better audio-visual question answering. arXiv preprint arXiv:2510.18346. — [arXiv:2510.18346](https://arxiv.org/abs/2510.18346) find
  35. Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, and Yefeng Zheng. 2022. mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In International conference on medical image computing and computer-assiste… find
  36. Zhihui Zhang, Luanyuan Dai, Qika Lin, Yunfeng Diao, Guangyin Jin, Yufei Guo, Jing Zhang, and Xiaoshuai Hao. 2025c. Synergistic prompting for robust visual recognition with missing modalities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1881–1890. find
  37. Jinming Zhao, Ruichen Li, and Qin Jin. 2021. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc… find
  38. Xujian Zhao, Yixin Wang, and Peiquan Jin. 2025. Audio-visual adaptive fusion network for question answering based on contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10483–10491. find
  39. Shenghao Zhu, Yifei Chen, Weihong Chen, Yuanhan Wang, Chang Liu, Shuo Jiang, Feiwei Qin, and Changmiao Wang. 2025. Bridging the gap in missing modalities: Leveraging knowledge distillation and style matching for brain tumor segmentation. In International Conference on Medical Image Computing and Com… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-retrieving-recover-incomplete-audio-visual/