Elamur Research⌕
☰
home / papers / 2026 / WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

Paper · 2026 · Yingjian Zhu, Xinming Wang, Kun Ding et al.

Summary

WikiSeeker targets Knowledge-Based Visual Question Answering (KB-VQA), where answers need encyclopedic facts that the image alone does not contain. The authors identify two weaknesses in current multi-modal RAG systems. First, retrieval usually uses only the image and ignores the question text. Second, vision-language models (VLMs) serve only as answer generators, even though text-only LLMs extract answers from retrieved passages more effectively. WikiSeeker addresses both problems. It builds a fine-grained knowledge base of aligned <image, section> pairs and retrieves with weighted, concatenated visual and text embeddings, then applies a multi-modal reranker. The VLM takes on two new roles. As the Refiner, trained with GRPO reinforcement learning, it rewrites the question using visual cues. As the Inspector, it judges whether the retrieved context is sufficient. If so, a text LLM produces the answer; if not, the VLM answers from its own knowledge. The system reports state-of-the-art results on EVQA, InfoSeek and M2KR. This suggests that assigning VLMs to query rewriting and context checking pays off more than using them as generators.

Key points

  • Retrieval: Query and KB vectors concatenate image and text embeddings, with a weight α balancing the two modalities, and are matched by cosine similarity. Overly long Wikipedia sections are summarized by an LLM.
  • Refiner gains: The RL-trained Refiner, built on a 3B VLM, raises EVQA retrieval R@1 from 14.59 to 25.45 and R@20 from 43.35 to 57.98. It beats image captioning and zero-shot query expansion done with Qwen2.5-VL-7B-Instruct.
  • Ablation: Without the Refiner or Inspector, accuracy is 42.35% on EVQA and 31.68% on InfoSeek. Adding the Refiner gives 53.92% and 43.47%, and the full system reaches 55.62% and 44.72%.
  • Inspector routing: The Inspector routes between Qwen3-VL-8B-Instruct (VLM) and Qwen2.5-7B-Instruct (LLM). An oracle ensemble of both models would reach 58.87% on EVQA and 47.28% on InfoSeek, so routing could still improve.
  • Limitations: The routing rule is hard rather than learned, and retrieval is single-pass only, so multi-hop questions are not supported. Code is announced at github.com/zhuyjan/WikiSeeker.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 worksQuestion answering · 441 works

Cite

Cite as Yingjian Zhu, Xinming Wang, Kun Ding, Ying Wang, Bin Fan and Shiming Xiang. 2026. WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.268

@inproceedings{zhu2026wikiseeker,
  title = {{WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering}},
  author = {Yingjian Zhu, Xinming Wang, Kun Ding, Ying Wang, Bin Fan and Shiming Xiang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.268},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.268}
}

References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425–2433. find
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. arXiv prepr…
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025b. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  4. Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPRW), pag… find
  5. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  6. Lin Chen, Yingjian Zhu, Qi Yang, Xin Niu, Kun Ding, and Shiming Xiang. 2025. Sam-mi: A mask-injected framework for enhancing open-vocabulary semantic segmentation with sam. arXiv preprint arXiv:2511.20027. — [arXiv:2511.20027](https://arxiv.org/abs/2511.20027) find
  7. Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 149…
  8. Changin Choi, Wonseok Lee, Jungmin Ko, and Wonjong Rhee. 2025. Multimodal iterative rag for knowledge-intensive visual question answering. arXiv preprint arXiv:2509.00798. — [arXiv:2509.00798](https://arxiv.org/abs/2509.00798) find
  9. Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2025. Augmenting multimodal llms with self-reflective tokens for knowledge-based visual question answering. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 9199–9209. find
  10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  11. Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. 2023. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pa… find
  12. Pu Jian, Donglei Yu, and Jiajun Zhang. 2024. Large language models know what is key visual entity: An LLM-assisted multimodal retrieval for VQA. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 10939–10956. find
  13. Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. 2025. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. arXiv preprint arXiv:2503.00223. — [arXiv:2503.00223](https://arxiv.org/abs/2503.00223) find
  14. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547.
  15. Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. PreFLMR: Scaling up fine-grained late-interaction multi-modal retrievers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 5294–5316. find
  16. Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, and Lifu Huang. 2024. Rora-vlm: Robust retrieval-augmented vision language models. arXiv preprint arXiv:2410.08876. — [arXiv:2410.08876](https://arxiv.org/abs/2410.08876) find
  17. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems (NeurIPS), 36:53728–53741.
  18. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision (ECCV), pages 146–162. Springer.
  19. Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pages 8876–8884. find
  20. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  21. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3195–3204. find
  22. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems (ECCS), pages 1279–1297.
  23. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems (NeurIPS), 37:124198– 124235. find
  24. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389.
  25. Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. 2023. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vi… find
  26. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1527–1536. find
  27. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 35:2773…
  28. Yang Tian, Fan Liu, Jingyuan Zhang, V. W., Yupeng Hu, and Liqiang Nie. 2025. CoRe-MMRAG: Cross-source knowledge reconciliation for multimodal RAG. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 32967–32982. find
  29. Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Henge. 2017a. Explicit knowledge-based reasoning for visual question answering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), pages 1290– 1296. find
  30. Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017b. Fvqa: Fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(10):2413–2427. find
  31. Xinming Wang, Jian Xu, Aslan H Feng, Yi Chen, Haiyang Guo, Fei Zhu, Yuanqi Shao, Minsi Ren, Hongzhu Yi, Sheng Lian, and 1 others. 2025a. The hitchhiker’s guide to autonomous research: A survey of scientific agents. TechRxiv.August 07, 2025. DOI:10.36227/techrxiv175459840.02185500/V1. — [doi:10.36227/techrxiv175459840.02185500/v1](https://doi.org/10.36227/techrxiv175459840.02185500/v1) find
  32. Wei Yang, Jingjing Fu, Rui Wang, Jinyu Wang, Lei Song, and Jiang Bian. 2025. OMGM: Orchestrate multiple granularities and modalities for efficient multimodal retrieval. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 24545–24563. find
  33. Xinming Wang, Jian Xu, Bin Yu, Sheng Lian, Hongzhu Yi, Yi Chen, Yingjian Zhu, Boran Wang, Hongming Yang, Han Hu, and 1 others. 2025b. Mr-align: Meta-reasoning informed factuality alignment for large reasoning models. arXiv preprint arXiv:2510.24794.
  34. Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Yuxuan Zhao, Zehua Xie, and 1 others. 2024. mr 2̂ ag: Multimodal retrieval-reflection-augmented generation for knowledge-based vqa. arXiv preprint arXiv:2411.15041. — [arXiv:2411.15041](https://arxiv.org/abs/2411.15041) find
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS), 35:24824–24837.
  36. Xiaojun Xue, Chunxia Zhang, Zhendong Niu, and Xindong Wu. 2022. Multi-level attention map network for multimodal sentiment analysis. IEEE Transactions on Knowledge and Data Engineering, 35(5):5105–5118. find
  37. Yibin Yan and Weidi Xie. 2024. EchoSight: Advancing visual-language models with Wiki knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2024 (EMNLP), pages 1538–1551. find
  38. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXi… find
  39. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  40. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, and 1 others. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  41. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL).
  42. Yingjian Zhu, Ying Wang, Yuyang Hong, Ruohao Guo, Kun Ding, Xin Gu, Bin Fan, and Shiming Xiang. 2026. Seavis: Sound-enhanced association for online audio-visual instance segmentation. arXiv preprint arXiv:2603.01431. — [arXiv:2603.01431](https://arxiv.org/abs/2603.01431) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2026-wikiseeker-rethinking-role-vision-language/