Elamur Research⌕
☰
home / papers / 2025 / VISA: Retrieval Augmented Generation with Visual Source Attribution

VISA: Retrieval Augmented Generation with Visual Source Attribution

Paper · 2025 · Xueguang Ma, Shengyao Zhuang, Bevan Koopman et al.

Summary

Most RAG systems that cite sources point only to whole documents. This makes users search long, dense pages to find the actual evidence. Highlighting text passages could help, but it needs extra engineering to map chunks back to the source and control over how documents render, which PDFs often do not allow. The authors propose VISA, which works on document screenshots instead of extracted text. A vision-language model reads the query and one or more retrieved page images. It then returns a short answer, the ID of the supporting document, and bounding-box coordinates marking the evidence region. The whole output is produced as ordinary next-token generation, with no changes to the model architecture. For training and evaluation, the authors built two datasets: Wiki-VISA, from Natural Questions on rendered Wikipedia pages, and Paper-VISA, from PubLayNet biomedical PDFs with synthetic queries. Strong models such as Qwen2-VL-72B localize evidence poorly without task-specific training. Fine-tuning on the new data gives large gains. The work points toward RAG pipelines that are visual end-to-end and easier for users to verify.

Key points

  • Datasets:
    • Wiki-VISA uses NQ short answers as generation targets and the HTML boxes of NQ long answers as attribution targets. Its multi-page screenshots span up to four pages.
    • Paper-VISA covers biomedical scientific PDFs and includes passages, tables and figures.
    • FineWeb-VISA serves as additional training data.
  • Effect of adding FineWeb-VISA to Wiki-VISA training:
    • Bounding-box accuracy on Wiki-VISA rises from 54.2% to 58.2%.
    • Accuracy on cases spanning more than one page rises from 50.1% to 61.7%.
  • Cross-domain transfer: Training on Paper+FineWeb barely changes in-domain results. However, zero-shot accuracy on Wiki-VISA jumps from 0.2% to 36.1%.
  • Combined training: Training on Wiki+Paper+FineWeb together reaches 58.1% box accuracy on Wiki-VISA and 67.6% on Paper-VISA.
  • Error analysis (50 Wiki-VISA cases):
    • 43 boxes pointed to the wrong region.
    • 4 boxes were misplaced or wrongly sized.
    • 3 boxes had a granularity mismatch, such as marking a table cell instead of the whole passage.
  • Limitations: The multi-candidate setting was capped at three documents because of memory constraints.
  • Availability: Code, data and checkpoints are announced at github.com/castorini/visa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada67 works · 13,592 citations🎓The University of QueenslandBrisbane · Australia14 works · 4,603 citations🏛️Commonwealth Scientific and Industrial Research OrganisationCanberra · Australia15 works · 1,898 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Xueguang Ma, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Wenhu Chen and Jimmy Lin. 2025. VISA: Retrieval Augmented Generation with Visual Source Attribution. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1456

@inproceedings{ma2025visa,
  title = {{VISA: Retrieval Augmented Generation with Visual Source Attribution}},
  author = {Xueguang Ma, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Wenhu Chen and Jimmy Lin},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1456},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1456}
}

References

38 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen tau Yih. 2024. Reliable, adaptable, and attributable language models with retrieval. arXiv:2403.03187.
  2. Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, D… — Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
  3. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, page 213–229, Berlin, Hei…
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879.
  5. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313– 9332,…
  6. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural…
  7. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, 521(7553):436–444. find
  8. Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2024. M3DocRAG: Multi-modal retrieval is what you need for multi-page multi-document understanding. find
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th Internation…
  10. Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. 2016. R-FCN: Object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. find
  11. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. ShowUI: One vision-language-action model for generalist GUI. In NeurIPS 2024 Workshop on Open-World Agents. find
  12. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Intern…
  13. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Celine Hudelot, and Pierre Colombo. 2025. ColPali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations. find
  14. Jeremy J. Foster. 1979. The Use of Visual Cues in Text, pages 189–203. Springer US, Boston, MA. find
  15. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Association for Computational Linguistics.
  16. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. arXiv:2312.10997.
  17. Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. 2024. Source-aware training enables knowledge attribution in language models. In First Conference on Language Modeling.
  18. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024. Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6492– 6505, Miami, Florida, USA. Association for Computational…
  19. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. ExpertQA: Expert-curated questions and attributed answers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  20. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  21. Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2024. Fine-tuning or retrieval? comparing knowledge injection in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 237–250, Miami, Florida, USA. Association for Computational Linguisti…
  22. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb datasets: Decanting the web for the finest text data at scale. arXiv:2406.17557.
  23. Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. 2024. GLaMM: Pixel grounding large multimodal model. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13009… find
  24. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring attribution in natural language generation models. Computational Linguistics, 49(4):777–840.
  25. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779–788. find
  26. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  27. Monica Riedler and Stefan Langer. 2024. Beyond text: Optimizing rag with multimodal inputs for industrial applications. Preprint, arXiv:2410.21943. — [arXiv:2410.21943](https://arxiv.org/abs/2410.21943) find
  28. Xi Ye, Ruoxi Sun, Sercan Arik, and Tomas Pfister. 2024. Effective large language model adaptation for improved grounding and citation generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume… find
  29. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024. VisRAG: Vision-based retrieval-augmented generation on multi-modality documents. Preprint, arXiv:2410.10594. — [arXiv:2410.10594](https://arxiv.org/abs/2410.10594) find
  30. Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. 2023. BuboGPT: Enabling visual grounding in multi-modal llms. arXiv:2307.08581. — [arXiv:2307.08581](https://arxiv.org/abs/2307.08581) find
  31. Zhong-Qiu Zhao, Peng Zheng, Shou tao Xu, and Xindong Wu. 2019. Object detection with deep learning: A review. arXiv:1807.05511. — [arXiv:1807.05511](https://arxiv.org/abs/1807.05511) find
  32. Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019. PubLayNet: Largest Dataset Ever for Document Layout Analysis . In 2019 International Conference on Document Analysis and Recognition (IC-DAR), pages 1015–1022, Los Alamitos, CA, USA. IEEE Computer Society. find
  33. Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. 2023. Object detection in 20 years: A survey. arXiv:1905.05055. — [arXiv:1905.05055](https://arxiv.org/abs/1905.05055) find
  34. Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua… — [arXiv:2412.03555](https://arxiv.org/abs/2412.03555) find
  35. John Sweller. 2011. Chapter two - cognitive load theory. volume 55 of Psychology of Learning and Motivation, pages 37–76. Academic Press. find
  36. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing vision-language model’s perception of the w…
  37. Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2024. Mmed-rag: Versatile multimodal rag system for medical vision language models. Preprint, arXiv:2410.13085. — [arXiv:2410.13085](https://arxiv.org/abs/2410.13085) find
  38. Crop, Absolute No Random Crop Normalized Value No Bounding Box Wiki-VISA bbx ans 54.2 58.8 56.4 0 65.2 65.6 64.4 67.6 Paper-VISA bbx ans 27.8 1.7 0.1 0 Model 36.2 36.9 37.2 35.2 *Table 5: Impact of bounding box target representation and cropping strategies during training on Wiki-VISA in the single… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2025-visa-retrieval-augmented-generation-visual/