Elamur Research⌕
☰
home / papers / 2025 / REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark

REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark

Paper · 2025 · Navve Wasserman, Roi Pony, Oshri Naparstek et al.

Summary

Existing benchmarks for multi-modal document retrieval in RAG do not reflect how retrieval works in practice. Their queries often assume the user is looking at a specific page, they reuse the page's own wording, their corpora are easy to tell apart, and relevant pages are sometimes missed in labeling. REAL-MM-RAG addresses this with an automatically built benchmark of long, specialized documents and slides, such as IBM financial reports and FlashSystem technical material, that mix text, tables and figures.

VLMs generate the queries, LLMs filter them so they read like natural user requests, and a VLM-based check verifies relevance against pages to reduce false negatives. Queries are also reworded at three increasing levels to test semantic understanding rather than keyword overlap.

Evaluation shows that current retrievers struggle with table-heavy financial content and lose accuracy as rewording grows. To address this, the authors built a rephrased version of the ColPali training data and a new finance-table training set. Fine-tuning on these produced state-of-the-art results, giving the field both a stronger testbed and practical remedies.

Key points

  • The benchmark targets four properties: multi-modal pages, high difficulty from many similar pages within one domain, realistic RAG-style queries with no page references, and exhaustive relevance labeling.
  • Averaged NDCG@5 drops sharply from rephrasing level 0 to level 3, for example from 71.3 to 50.6 for ColPali and from 73.9 to 54.5 for ColQwen. BM25 and BGE-M3 on captions score lower still.
  • The RobCol models were created by fine-tuning ColPali-v1.2 and ColQwen2-v1.0 for one epoch on ColPali training data with half its queries rephrased by LLaMA-3-70B. They gain an average of 11.1 NDCG@5 at level 3 without hurting performance on unrephrased queries.
  • The RobTabCol models add the rephrased finance-table data and outperform the base models by 25–30 NDCG@5 when financial benchmark queries are reworded, and by 6–9 on benchmarks outside finance. RobTabColQwen reaches 83.9 at level 0 and 71.7 at level 3.
  • The fine-tuned models also beat baselines on ViDoRe V2. The authors note two limitations: the queries are VLM-generated, and the benchmark does not test multi-page reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksVision & language · 657 worksInformation retrieval & search · 473 works

Cite

Cite as Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay and Leonid Karlinsky. 2025. REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1528

@inproceedings{wasserman2025real,
  title = {{REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark}},
  author = {Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay and Leonid Karlinsky},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1528},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1528}
}

References

40 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. 2024. Pixtral 12b. arXiv preprint arXiv:2410.07073. find
  2. Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2016. Uqv100: A test collection with query variability. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 725–728. find
  3. Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2017. Retrieval consistency in the presence of query variations. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval, pages 395–404. find
  4. Rodger Benham, J Shane Culpepper, Luke Gallagher, Xiaolu Lu, and Joel M Mackenzie. 2018. Towards efficient and effective query variant generation. In DESIRES, pages 62–67. find
  5. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. 2024. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726. find
  6. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  7. Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  8. Yihao Ding, Kaixuan Ren, Jiabin Huang, Siwen Luo, and Soyeon Caren Han. 2024. Mvqa: A dataset for multimodal information retrieval in pdf-based visual question answering. arXiv preprint arXiv:2404.12720. — [arXiv:2404.12720](https://arxiv.org/abs/2404.12720) find
  9. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. Colpali: Efficient document retrieval with vision language models. arXiv preprint arXiv:2407.01449. find
  10. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288–2292.
  11. Tim Hagen, Harrisen Scells, and Martin Potthast. 2024. Revisiting query variation robustness of transformer models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4283–4296. find
  12. Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944. — [arXiv:2311.11944](https://arxiv.org/abs/2311.11944) find
  13. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. find
  14. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
  15. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39– 48.
  16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
  17. Xiaolu Lu, Oren Kurland, J Shane Culpepper, Nick Craswell, and Ofri Rom. 2019. Relevance modeling with multiple query variations. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, pages 27–34. find
  18. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. 2024a. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406.11251.
  19. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. 2024b. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523. find
  20. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.
  21. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  22. Gustavo Penha, Arthur Câmara, and Claudia Hauff. 2022. Evaluating the robustness of retrieval pipelines with query variation generators. In European conference on information retrieval, pages 397–412. Springer. find
  23. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PM…
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  25. Rita Ramos, Desmond Elliott, and Bruno Martins. 2023. Retrieval-augmented image captioning. arXiv preprint arXiv:2302.08268. find
  26. S Robertson, Steve Walker, Susan Jones, and MHB GATFORD. 1994. Okapi at 3. In Proceedings of the 3rd Text REtrieval Conference (-3), pages 109–126. find
  27. Georgios Sidiropoulos and Evangelos Kanoulas. 2022. Analysing the robustness of dual encoders for dense retrieval against misspellings. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2132–2136. find
  28. Alan F Smeaton and Fergus Kelledy. 1998. User-chosen phrases in interactive query formulation for information retrieval. In 20th Annual BCS-IRSG Colloquium on IR. BCS Learning & Development. find
  29. Ray Smith. 2007. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE. find
  30. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21. find
  31. Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. 2023. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13636–13645. find
  32. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the…
  33. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  34. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808.
  35. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. 2024. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594. — [arXiv:2410.10594](https://arxiv.org/abs/2410.10594) find
  36. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.
  37. Xinyi Zheng, Doug Burdick, Lucian Popa, Peter Zhong, and Nancy Xin Ru Wang. 2021. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. Winter Conference for Applications in Computer Vision (WACV). find
  38. Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. 2022. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4857–4866. find
  39. Hongyi Zhu, Jia-Hong Huang, Stevan Rudinac, and Evangelos Kanoulas. 2024. Enhancing interactive image retrieval with query rewriting using large language models and vision language models. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 978–987. find
  40. Guido Zuccon, Joao Palotti, and Allan Hanbury. 2016. Query variations and their effect on comparing information retrieval systems. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 691–700. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wasserman-2025-real-mm-rag-real-world/