Elamur Research⌕
☰
home / papers / 2026 / Beyond Single View: A Comprehensive Benchmark for Medical Multimodal Large Language Models on Multi-Image Understanding

Beyond Single View: A Comprehensive Benchmark for Medical Multimodal Large Language Models on Multi-Image Understanding

Paper · 2026 · Dexuan Xu, Yuan Jiayin, Jianing Wang et al.

Summary

Medical multimodal LLMs are usually tested on one image at a time, but clinicians routinely combine evidence from adjacent slices, follow-up scans, and different modalities. The authors present MedMultiBench, a benchmark built specifically to measure how well models reason over several medical images at once. It contains 11,392 expert-verified samples in 13 subtasks, organized into four dimensions: Joint Reasoning, Comparative Analysis, Comprehensive Perception (for example, spatial ordering and temporal change), and In-Context Learning. The samples were assembled from public resources and passed through filtering, label alignment, and LLM-assisted text rewriting, with medical professionals checking the results. The authors evaluated 13 state-of-the-art models. They found that strong single-image performance does not carry over reliably to multi-image settings. Open-source models deteriorate as more images are added, while closed-source systems such as Gemini3-Pro stay largely stable. The benchmark exposes a practical weakness in long visual-context processing that matters for real clinical workflows.

Key points

  • Scale and structure: MedMultiBench has 11,392 samples across 13 subtasks. Its taxonomy sorts the ways images relate to one another into four kinds: continuity across space and time, heterogeneity between different views, correspondence of differences between images, and reference based on semantic meaning.
  • Data sources: The samples come from IU-Xray, MedPix, GMAI, OmniMedVQA, MedFrameQA, Med-MIM, and MMMU. GPT-5 rewrote some text stylistically, and all rewritten text was manually verified by medical experts.
  • Effect of image count: Gemini3-Pro reaches its best accuracy, 69.7%, with 10 input images. Open-source models such as Qwen3-VL-8B and Lingshu-7B often fall below 30% when given many images.
  • Few-shot results: On IU-Xray, Qwen3-VL-8B and Lingshu-7B score above 80 on BERTScore. On MedPix, Gemini3-Pro leads at every shot count, while some models plateau or slightly decline at 5 shots.
  • Limitations: Volumes are approximated with sequences of 2D slices rather than native 3D data such as NIfTI. Open-ended answers are scored with Rouge-L and BERTScore, which may not reflect expert clinical judgment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksVision & language · 811 works

Cite

Cite as Dexuan Xu, Yuan Jiayin, Jianing Wang, Yanyuan Chen, Hanpin Wang and Yu Huang. 2026. Beyond Single View: A Comprehensive Benchmark for Medical Multimodal Large Language Models on Multi-Image Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1263

@inproceedings{xu2026beyond,
  title = {{Beyond Single View: A Comprehensive Benchmark for Medical Multimodal Large Language Models on Multi-Image Understanding}},
  author = {Dexuan Xu, Yuan Jiayin, Jianing Wang, Yanyuan Chen, Hanpin Wang and Yu Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1263},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1263}
}

References

40 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923.
  3. Bytedance. 2025. Doubao. find
  4. Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, and 1 others. 2024. HuatuoGPT-Vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280. find
  5. Junlong Cheng, J Ye, Z Deng, J Chen, T Li, H Wang, Y Su, Z Huang, J Chen, L Jiang, and 1 others. 2023. SAM-Med2D. arXiv preprint arXiv:2308.16184. — [arXiv:2308.16184](https://arxiv.org/abs/2308.16184) find
  6. Deepmind. 2025. Gemini 3 Pro Model Card. Technical report, Google. find
  7. Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2015. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–31… find
  8. Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. PathVQA: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. — [arXiv:2003.10286](https://arxiv.org/abs/2003.10286) find
  9. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, and 1 others. 2025. GLM-4.1 V-Thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006.
  10. Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. 2024. Omnimed-VQA: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183. find
  11. Yefan Huang, Xiaoli Wang, Feiyan Liu, and Guofeng Huang. 2022. OVQA: A clinically generated visual question answering dataset. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2924–2938. find
  12. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276.
  13. Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. 2024. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483.
  14. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10. find
  15. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564.
  16. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024. MVbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206.
  17. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81.
  18. Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE. find
  19. Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, and 1 others. 2024. MIBench: Evaluating multimodal large language models over multiple images. arXiv preprint arXiv:2407.15272. find
  20. Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, and 1 others. 2024. MMIU: Multimodal multi-image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.02718. — [arXiv:2408.02718](https://arxiv.org/abs/2408.02718) find
  21. MoonshotAI. 2025. Moonshot. find
  22. Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR.
  23. OpenAI. 2025. GPT-5 System Card. Technical report, OpenAI. find
  24. Yan Shu, Chi Liu, Robin Chen, Derek Li, and Bryan Dai. 2025. Fleming-VL: Towards universal medical visual reasoning with multimodal llms. arXiv preprint arXiv:2511.00916. — [arXiv:2511.00916](https://arxiv.org/abs/2511.00916) find
  25. Irene Siragusa, Salvatore Contino, Massimo La Ciura, Rosario Alicata, and Roberto Pirrone. 2025. Med-Pix 2.0: A comprehensive multimodal biomedical data set for advanced ai applications with retrieval augmented generation and knowledge graphs. Data Science and Engineering, pages 1–17. find
  26. Yuhao Tang, Haichen Yang, Liyan Zhang, and Ye Yuan. 2024. Work like a doctor: Unifying scan localizer and dynamic generator for automated computed tomography report generation. Expert Systems with Applications, 237:121442. find
  27. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  28. Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, and 1 others. 2024. Towards generalist biomedical ai. Nejm Ai, 1(3):AIoa2300138. find
  29. Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, and 1 others. 2025. MuirBench: A comprehensive benchmark for robust multi-image understanding. In The Thirteenth International Conference on Learning Representations. find
  30. Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hanpin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, and 1 others. 2024. MLeVLM: Improve multi-level progressive capabilities based on multimodal large language model for medical visual question answering. In Findings of the Association f… find
  31. Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, and 1 others. 2025. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044. — [arXiv:2506.07044](https://arxiv.org/abs/2506.07044) find
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  33. Xikai Yang, Juzheng Miao, Yuchen Yuan, Jiaze Wang, Qi Dou, Jinpeng Li, and Pheng-Ann Heng. 2025b. Medical large vision language models with multi-image visual ability. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 402–412. Springer. find
  34. Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, and 1 others. 2024. GMAI-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37:94327–94427. find
  35. Suhao Yu, Haojin Wang, Juncheng Wu, Cihang Xie, and Yuyin Zhou. 2025. MedFrameQA: A multi-image medical vqa benchmark for clinical reasoning. arXiv preprint arXiv:2505.16964. — [arXiv:2505.16964](https://arxiv.org/abs/2505.16964) find
  36. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vi…
  37. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  38. Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. PMC-VQA: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415. — [arXiv:2305.10415](https://arxiv.org/abs/2305.10415) find
  39. Bingchen Zhao, Yongshuo Zong, Letian Zhang, and Timothy Hospedales. 2024. Benchmarking multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv preprint arXiv:2406.12742. — [arXiv:2406.12742](https://arxiv.org/abs/2406.12742) find
  40. Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. MedXpertQA: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-beyond-single-view-comprehensive-benchmark/