Elamur Research⌕
☰
home / papers / 2025 / Can Multimodal Large Language Models Understand Spatial Relations?

Can Multimodal Large Language Models Understand Spatial Relations?

Paper · 2025 · Jingping Liu, Ziyan Liu, Zhedong Cen et al.

Summary

Spatial relation reasoning is a basic skill for multimodal large language models (MLLMs), but existing benchmarks have several weaknesses. Some depend on bounding boxes. Some label relations in ways that conflict with real-world physics, rarely test viewpoint changes, or contain questions that commonsense alone can answer without the image. To fix this, the authors built SpatialMQA, a manually annotated multiple-choice benchmark drawn from COCO2017 images, with no bboxes. Each question asks how two entities are positioned relative to each other, using a gravity-based coordinate system centered on the observer. Questions use one of three viewpoints: an outside observer, a first-person view (the object itself), or a third-person view (another living being in the scene). Annotators followed guidelines that exclude image-independent questions, and a three-round procedure controlled quality. The authors tested closed models (GPT-4o, Gemini-1.5-flash) and open models (LLaVA, SpaceLLaVA). Even the best system falls far short of humans. This shows that real visual-spatial understanding, especially under perspective shifts, remains an open problem.

Key points

  • SpatialMQA contains 5,392 samples (3,780 train, 1,076 test) covering 128 subject/object types. The samples were chosen from about 30,000 pre-selected COCO2017 images.
  • Every answer is one of six spatial relations: on/above, below, behind, in front of, to the left of, or to the right of. Each question offers 2–6 options.
  • The best result is 48.14% accuracy from instruction-tuned SpaceLLaVA. GPT-4o reaches 40.20%, while humans score 98.40%.
  • Unlike VSR, where only about 6% of items involve perspective, SpatialMQA systematically includes first- and third-person questions.
  • The benchmark is released on Hugging Face (liuziyan/SpatialMQA) under CC-BY 4.0. The authors note two limitations: the training set is small, and only basic relations are covered.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🎓East China University of Science and TechnologyShanghai · China14 works · 87 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Jingping Liu, Ziyan Liu, Zhedong Cen, Yan Zhou, Yinan Zou, Weiyan Zhang, Haiyun Jiang and Tong Ruan. 2025. Can Multimodal Large Language Models Understand Spatial Relations?. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.31

@inproceedings{liu2025can,
  title = {{Can Multimodal Large Language Models Understand Spatial Relations?}},
  author = {Jingping Liu, Ziyan Liu, Zhedong Cen, Yan Zhou, Yinan Zou, Weiyan Zhang, Haiyun Jiang and Tong Ruan},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.31},
  url = {https://doi.org/10.18653/v1/2025.acl-long.31}
}

References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Anja Belz, Adrian Muscat, Pierre Anguill, Mouhamadou Sow, Gaétan Vincent, and Yassine Zinessabah. 2018. Spatialvoc2k: A multilingual dataset of images with annotations and features for spatial relations between objects. In Proceedings of the 11th International Conference on Natural Language Generati… find
  3. Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. 2024. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642. — [arXiv:2406.13642](https://arxiv.org/abs/2406.13642) find
  4. Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. 2024. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465. find
  5. An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. 2024. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584. — [arXiv:2406.01584](https://arxiv.org/abs/2406.01584) find
  6. Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. 2023. Can vision-language models think from a first-person perspective? arXiv preprint arXiv:2311.15596. — [arXiv:2311.15596](https://arxiv.org/abs/2311.15596) find
  7. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36.
  8. Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. 2024. Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. arXiv preprint arXiv:2406.05756. find
  9. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306.13394.
  10. Haoxiang Gao, Yaqian Li, Kaiwen Long, Ming Yang, and Yiqing Shen. 2024. A survey for foundation models in autonomous driving. arXiv preprint arXiv:2402.01105. — [arXiv:2402.01105](https://arxiv.org/abs/2402.01105) find
  11. Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. 2020. Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. Advances in Neural Information Processing Systems, 33:10514–10525. find
  12. Zixian Guo, Bowen Dong, Zhilong Ji, Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo. 2023. Texts as images in prompt tuning for multi-label image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2808–2817. find
  13. Zachary Hawes and Daniel Ansari. 2020. What explains the relationship between spatial and mathematical skills? a review of evidence from brain and behavior. Psychonomic bulletin & review, 27:465–482. find
  14. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  15. Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.
  16. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE CVPR, pages 2901–2910.
  17. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pag…
  18. Fangyu Liu, Guy Emerson, and Nigel Collier. 2023a. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651. find
  19. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.
  20. Weihao Liu, Fangyu Lei, Tongxu Luo, Jiahe Lei, Shizhu He, Jun Zhao, and Kang Liu. 2023b. Mmhqa-icl: Multimodal in-context learning for hybrid question answering over text, tables and images. arXiv preprint arXiv:2309.04790. — [arXiv:2309.04790](https://arxiv.org/abs/2309.04790) find
  21. Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. Ptuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602.
  22. Michael J Proulx, Orlin S Todorov, Amanda Taylor Aiken, and Alexandra A de Sousa. 2016. Where am i? who am i? the relation between spatial cognition, social cognition and individual differences in the built environment. Frontiers in psychology, 7:158846. find
  23. Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. 2023. Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning. In The Twelfth International Conference on Learning Representations. find
  24. Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217–223. find
  25. Hugo Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, et al. 2023. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023. URL https://huggingface. co/blog/… find
  26. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  27. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.
  28. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  29. Junjie Wang, Wei Li, Yinjian Wang, Ran Tao, and Qian Du. 2023. Representation-enhanced status replay network for multisource remote-sensing image classification. IEEE Transactions on Neural Networks and Learning Systems. find
  30. Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. 2024. T-sciq: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19162–19170. find
  31. Chuan Wen, Dinesh Jayaraman, and Yang Gao. 2024. Can transformers capture spatial relations between objects? arXiv preprint arXiv:2403.00729. — [arXiv:2403.00729](https://arxiv.org/abs/2403.00729) find
  32. Kaiyu Yang, Olga Russakovsky, and Jia Deng. 2019. Spatialsense: An adversarially crowdsourced benchmark for spatial relation recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2051–2060. find
  33. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178.
  34. Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. 2024. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs. arXiv preprint arXiv:2401.02582. — [arXiv:2401.02582](https://arxiv.org/abs/2401.02582) find
  35. Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. 2023. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168–5191. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-can-multimodal-large-language-models/