Elamur Research⌕
☰
home / papers / 2026 / ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

Paper · 2026 · António Loison, Quentin Macé, Antoine Edy et al.

Summary

Real-world retrieval-augmented generation must handle documents whose key content lives in tables, charts and images. It must also combine evidence spread across many pages and point to exact source locations. Existing benchmarks usually test only one of these aspects, stay text-only, or score retrieval and generation separately. The authors build ViDoRe V3, an end-to-end multimodal RAG benchmark over large, visually rich professional corpora. Queries come from three sources: human annotators working from raw pages, humans working only from summaries (which discourages simple lookup questions), and an LLM-based synthetic pipeline. A VLM narrows down candidate pages. Human annotators then rate page relevance, draw bounding boxes around evidence and write answers, which are merged into one reference answer. Across state-of-the-art pipelines, image-based retrievers beat text-based ones, and late-interaction models and textual reranking give clear gains. Visual or hybrid context also improves generated answers. Current systems remain weak on non-textual content, open-ended questions and precise visual grounding. The benchmark gives practitioners a realistic, commercially usable testbed for these gaps.

Key points

  • The benchmark has 10 corpora (7 English, 3 French) with about 26,000 pages and 3,099 human-verified queries. Each query is offered in 6 languages, and two datasets are kept as a private test set.
  • Domains include Finance, Computer Science, Energy, Pharmaceuticals, Human Resources, Industrial Maintenance, Telecom and Physics.
  • Queries follow a two-axis taxonomy:
    • 7 query types: open-ended, extractive, numerical, multi-hop, compare-contrast, boolean and enumerative.
    • 3 query formats: question, keyword and instruction.
  • Construction used roughly 12,000 annotation hours and about 3,000 NVIDIA H100 GPU hours (about 200 kg CO2e). Tools included Docling, Qwen3-235B, NeMo Data Designer and Qwen2.5-VL 32B. The benchmark is integrated into MTEB.
  • Bounding-box tests compared models against human annotators on each page:
    • Qwen3-VL-30B-A3B and Gemini 3 Pro both marked evidence on the same pages as humans only 16–17% of the time.
    • 26–27% of pages that humans annotated received no model box at all, which points to recall as the main weakness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 552 works

Cite

Cite as António Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Balough, Gabriel de Souza P. Moreira, Bo Liu, Manuel Faysse, Celine Hudelot and Gautier Viaud. 2026. ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.755

@inproceedings{loison2026vidore,
  title = {{ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios}},
  author = {António Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Balough, Gabriel {de Souza P. Moreira}, Bo Liu, Manuel Faysse, Celine Hudelot and Gautier Viaud},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.755},
  url = {https://doi.org/10.18653/v1/2026.acl-long.755}
}

References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, and Ehsaneddin Asgari. 2025. Ask in any modality: A comprehensive survey on multimodal retrieval-augmented generation. Preprint, arXiv:2502.08826. find
  2. Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, and 1 others. 2024. Docling technical report. arXiv preprint arXiv:2408.09869. — [arXiv:2408.09869](https://arxiv.org/abs/2408.09869) find
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:25…
  4. Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pages 160–172. Springer. find
  5. Antoine Chaffin. 2025. Gte-moderncolbert. find
  6. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv preprint. Version Number: 3. find
  7. Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2024a. M3docrag: Multi-modal retrieval is what you need for multi-page multi-document understanding. Preprint, arXiv:2411.04952. find
  8. Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2024b. M3docrag: Multimodal retrieval is what you need for multi-page multi-document understanding. arXiv preprint arXiv:2411.04952. find
  9. Max Conti, Manuel Faysse, Gautier Viaud, Antoine Bosselut, Céline Hudelot, and Pierre Colombo. 2025. Context is gold to find the gold passage: Evaluating and training contextual document embeddings. Preprint, arXiv:2505.24782. find
  10. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. Preprint, arXiv:2405.06211.
  11. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. Colpali: Efficient document retrieval with vision language models. Preprint, arXiv:2407.01449.
  12. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. Preprint, arXiv:2305.14627.
  13. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. Preprint, arXiv:2312.10997.
  14. Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Sedigheh Eslami, Scott Martens, Bo Wang, Nan Wang, and Han Xiao. 2025. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval. Preprint, arXiv:2506.18902. find
  15. Jina AI. 2025. jina-reranker-m0: Multilingual multimodal document reranker. Accessed: 2025-12-22. seed data. https://github.com/NVIDIA-NeMo/ DataDesigner. GitHub Repository. ↗
  16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. Preprint, arXiv:2005.11401.
  17. Nomic Team. 2025. Nomic embed multimodal: Interleaved text, image, and screenshots for visual document retrieval. find
  18. Liquid AI. 2025. Lfm2 technical report. arXiv preprint arXiv:2511.23404. — [arXiv:2511.23404](https://arxiv.org/abs/2511.23404) find
  19. NVIDIA Ingest Development Team. 2024. NVIDIA Ingest: An accelerated pipeline for document ingestion. find
  20. Frank Liu, Kenneth Enevoldsen, Roman Solomatin, Isaac Chung, Tom Aarsen, and Zoltán Fődi. 2025. Introducing rteb: A new standard for retrieval evaluation. find
  21. Xiangyu Peng, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, and Chien-Sheng Wu. 2025. Unidoc-bench: A unified benchmark for document-centric multimodal rag. Preprint, arXiv:2510.03663. — [arXiv:2510.03663](https://arxiv.org/abs/2510.03663) find
  22. Xing Han Lù. 2024. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring. Preprint, arXiv:2407.03618. — [arXiv:2407.03618](https://arxiv.org/abs/2407.03618) find
  23. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  24. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024a. Unifying multimodal retrieval via document screenshot embedding. Preprint, arXiv:2406.11251.
  25. Xueguang Ma, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Wenhu Chen, and Jimmy Lin. 2024b. Visa: Retrieval augmented generation with visual source attribution. Preprint, arXiv:2412.14457.
  26. Quentin Macé, António Loison, and Manuel Faysse. 2025. Vidore benchmark v2: Raising the bar for visual retrieval. Preprint, arXiv:2505.17166. — [arXiv:2505.17166](https://arxiv.org/abs/2505.17166) find
  27. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2025. Smolvlm: Redefining small and ef… find
  28. Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. 2021a. Infographicvqa. Preprint, arXiv:2104.12756. — [arXiv:2104.12756](https://arxiv.org/abs/2104.12756) find
  29. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021b. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  30. Leland McInnes, John Healy, and James Melville. 2020. Umap: Uniform manifold approximation and projection for dimension reduction. Preprint, arXiv:1802.03426.
  31. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037.
  32. NeMo Data Designer Team. 2025. Nemo data designer: A framework for generating synthetic data from scratch or based on your own Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel, Michael Tang, and 1 others. 2024. Bright: A realist…
  33. Rikiya Takehi, Benjamin Clavié, Sean Lee, and Aamir Shakir. 2025. Fantastic (small) retrievers and how to train them: mxbai-edge-colbert-v0 tech report. Preprint, arXiv:2510.14880. — [arXiv:2510.14880](https://arxiv.org/abs/2510.14880) find
  34. Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. Preprint, arXiv:2401.15391.
  35. Paul Teiletche, Quentin Macé, Max Conti, Antonio Loison, Gautier Viaud, Pierre Colombo, and Manuel Faysse. 2025. Modernvbert: Towards smaller visual document retrievers. arXiv preprint arXiv:2510.01149. — [arXiv:2510.01149](https://arxiv.org/abs/2510.01149) find
  36. Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, and Andrew Drozdov. 2025. Freshstack: Building realistic benchmarks for evaluating retrieval on technical documents. Preprint, arXiv:2504.13128. — [arXiv:2504.13128](https://arxiv.org/abs/2504.13128) find
  37. Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anck-aert, Ernest Valveny, and 1 others. 2023. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF International Conferenc… find
  38. Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. 2025. Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents. arXiv preprint arXiv:2502.18017.
  39. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, and 1 others. 2024. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37:113569–113697.
  40. Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay, and Leonid Karlinsky. 2025. Real-mm-rag: A real-world multi-modal retrieval benchmark. arXiv preprint arXiv:2502.12342.
  41. Mengyao Xu, Gabriel Moreira, Ronay Ak, Radek Osmulski, Yauhen Babakhin, Zhiding Yu, Benedikt Schifferer, and Even Oldridge. 2025. Llama nemore-triever colembed: Top-performing text-image retrieval model. Preprint, arXiv:2507.05513. — [arXiv:2507.05513](https://arxiv.org/abs/2507.05513) find
  42. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025a. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. Preprint, arXiv:2410.10594. — [arXiv:2410.10594](https://arxiv.org/abs/2410.10594) find
  43. Wenhan Yu, Wang Chen, Guanqiang Qi, Weikang Li, Yang Li, Lei Sha, Deguo Xia, and Jizhou Huang. 2025b. Bbox docvqa: A large scale bounding box grounded dataset for enhancing reasoning in document visual question answer. Preprint, arXiv:2511.15090. — [arXiv:2511.15090](https://arxiv.org/abs/2511.15090) find
  44. Zero Entropy. 2025. Introducing zerank-2. Accessed: 2025-12-22. find
  45. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  46. Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. 2022. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4857–4866. *Figure 8: Examples from the ViDoRe V3 datasets. Featuring va… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/loison-2026-vidore-v3-comprehensive-evaluation-retrieval/