Elamur Research⌕
☰
home / papers / 2026 / RotBench: Evaluating Multi-modal Large Language Models on Identifying Image Rotation

RotBench: Evaluating Multi-modal Large Language Models on Identifying Image Rotation

Paper · 2026 · Tianyi Niu, Jaemin Cho, Elias Stengel-Eskin et al.

Summary

The paper asks whether multimodal LLMs can tell how far an image has been turned counter-clockwise: 0°, 90°, 180°, or 270°. The authors built RotBench from Spatial-MM photos of lifestyle scenes, portraits, and landscapes. A two-stage human filter removed images whose orientation is ambiguous. The benchmark has 350 images, split into RotBench-Large (300) and RotBench-Small (50). RotBench-Small also comes with human baseline answers. Each image is posed as a four-option multiple-choice question, and accuracy is scored separately for each angle.

Open and proprietary models were tested, including GPT-5, o3, and Gemini-2.5-Pro. Models usually recognize upright images, and stronger ones also detect upside-down images. No model reliably tells 90° from 270°. Several fixes help little or inconsistently:

  • adding captions, depth maps, bounding boxes, or scene graphs;
  • chain-of-thought prompting;
  • fine-tuning.

A majority-voting scheme across re-rotated copies of the image works better, but it multiplies inference cost. The study exposes a basic spatial-reasoning weakness in current MLLMs. This matters for applications with moving cameras, such as robotics or first-person video.

Key points

  • RotBench construction: 350 Spatial-MM images, with a single annotator in Stage 1 and three evaluators checking flagged images in Stage 2. Code and data are available at github.com/tianyiniu/RotBench.
  • Models and prompting: Models evaluated include Qwen-2.5-VL-7B-Instruct, GPT-4o, GPT-4.1, o3, GPT-5, Gemini-2.0-Flash, Gemini-2.5-Flash, and Gemini-2.5-Pro. Chain-of-thought often helped on 270° while hurting 90°.
  • Normalized rotation voting on RotBench-Small: The model predicts the angle of four re-rotated copies, each prediction is shifted back by the added rotation, and a majority vote decides. This raised GPT-4o to 0.80–0.88 at every angle; zero-shot 270° accuracy had been 0.21. Qwen-2.5-VL-7B-Instruct became more balanced, at roughly 0.52–0.56 per angle.
  • Fine-tuning: Qwen-2.5-VL-7B-Instruct was trained on 1000 MS COCO images and evaluated on RotBench-Large. Accuracy at 180° rose to about 0.8, while 90° and 270° oscillated against each other, suggesting two competing local optima.
  • Limitations: Only the four cardinal angles were tested, so real-world performance on arbitrary angles may be even lower.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations🎓University of North Carolina at Chapel HillChapel Hill · United States67 works · 16,474 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Tianyi Niu, Jaemin Cho, Elias Stengel-Eskin and Mohit Bansal. 2026. RotBench: Evaluating Multi-modal Large Language Models on Identifying Image Rotation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.259

@inproceedings{niu2026rotbench,
  title = {{RotBench: Evaluating Multi-modal Large Language Models on Identifying Image Rotation}},
  author = {Tianyi Niu, Jaemin Cho, Elias Stengel-Eskin and Mohit Bansal},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.259},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.259}
}

References

57 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang. 2023. More context, less distraction: Visual classification by inferring and conditioning on contextual attributes. CoRR. find
  2. Ahmad Mustafa Anis, Hasnain Ali, and Saquib Sarfraz. 2025. On the limitations of vision-language models in understanding image transforms. Preprint, arXiv:2503.09837. — [arXiv:2503.09837](https://arxiv.org/abs/2503.09837) find
  3. Anthropic. 2025. Claude 4.1 system card. Technical report, Anthropic. Accessed October 2025. find
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:25…
  5. Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. 2023. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint. find
  6. Weijing Chen, Linli Yao, and Qin Jin. 2023a. Rethinking benchmarks for cross-modal image-text retrieval. In Proceedings of the 46th International ACM SI-GIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 1241–1251. ACM. find
  7. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, and 10 others.… — PaLI: A Jointly-Scaled Multilingual Language-Image Model
  8. Taco S. Cohen and Max Welling. 2016. Group equivariant convolutional networks. Preprint, arXiv:1602.07576. find
  9. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, and 31 others. 2024a.… — Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  10. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, and 31 others. 2024b.… — Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: efficient finetuning of quantized llms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
  12. Zeyu Feng, Chang Xu, and Dacheng Tao. 2019. Self-supervised representation learning by rotation feature decoupling. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10356–10366. find
  13. Philipp Fischer, Alexey Dosovitskiy, and Thomas Brox. 2015. Image orientation estimation with convolutional networks. In German Conference on Pattern Recognition. find
  14. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024a. Mme: A comprehensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394.
  15. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024b. Blink: Multimodal large language models can see but not perceive. Preprint, arXiv:2404.12390. find
  16. Google Gemini Team. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. https://storage.googleapis.com/deepmind-media/ gemini/gemini_v2_5_report.pdf. ↗
  17. Gemma, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, an… find
  18. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  19. Demis Hassabis and Koray Kavukcuoglu. 2024. Introducing gemini 2.0: our new ai model for the agentic era. https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/. ↗
  20. Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations. find
  21. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  22. Ujash Joshi and Michael Guerzhoy. 2017. Automatic photo orientation detection with convolutional neural networks. In 2017 14th Conference on Computer and Robot Vision (CRV), page 103–108. IEEE. find
  23. Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What‘s “up” with vision-language models? investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9161– 9175, Singapore. Association for Computational Ling… find
  24. Jinwoo Lee, Hyunsung Go, Hyunjoon Lee, Sunghyun Cho, Minhyuk Sung, and Junho Kim. 2021. Ctrl-c: Camera calibration transformer with line-classification. Preprint, arXiv:2109.02259. — [arXiv:2109.02259](https://arxiv.org/abs/2109.02259) find
  25. Jinwoo Lee, Minhyuk Sung, Hyunjoon Lee, and Junho Kim. 2020. Neural geometric parser for single image camera calibration. Preprint, arXiv:2007.11855. — [arXiv:2007.11855](https://arxiv.org/abs/2007.11855) find
  26. Jongmin Lee, Byungjin Kim, Seungwook Kim, and Minsu Cho. 2023. Learning rotation-equivariant features for visual correspondence. Preprint, arXiv:2303.15472. — [arXiv:2303.15472](https://arxiv.org/abs/2303.15472) find
  27. Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi. 2025. A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges. Preprint, arXiv:2501.02189. — [arXiv:2501.02189](https://arxiv.org/abs/2501.02189) find
  28. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft coco: Common objects in context. Preprint, arXiv:1405.0312.
  29. Zhiqiu Lin, Jin Sun, Abe Davis, and Noah Snavely. 2020. Visual chirality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12295–12303. find
  30. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. Preprint, arXiv:2304.08485.
  31. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and 1 others. 2023b. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
  32. Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, and 57 others. 2025. find
  33. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. Preprint, arXiv:2503.01743.
  34. Agnieszka Mikołajczyk and Michał Grochowski. 2018. Data augmentation for improving deep learning in image classification problem. In 2018 International Interdisciplinary PhD Workshop (IIPhDW), pages 117–122. find
  35. OpenAI. 2025a. Introducing GPT-4.1 in the api. https://openai.com/index/gpt-4-1/. ↗
  36. OpenAI. 2025b. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/. ↗
  37. OpenAI. 2025c. Introducing OpenAI o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. ↗
  38. OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, and 400 others. 2024. Gpt-4o sys…
  39. Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. Preprint, arXiv:1712.04621. — [arXiv:1712.04621](https://arxiv.org/abs/1712.04621) find
  40. Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. 2025. Capture: Evaluating spatial reasoning in vision language models via occluded object counting. Preprint, arXiv:2504.15485. find
  41. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.
  42. Cyrus Rashtchian, Charles Herrmann, Chun-Sung Ferng, Ayan Chakrabarti, Dilip Krishnan, Deqing Sun, Da-Cheng Juan, and Andrew Tomkins. 2023. Substance or style: What does your image embedding know? arXiv preprint arXiv:2307.05610. — [arXiv:2307.05610](https://arxiv.org/abs/2307.05610) find
  43. Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. 2024. Sam 2: Segm… find
  44. R N Shepard and J Metzler. 1971. Mental rotation of three-dimensional objects. Science, 171(3972):701– 703. find
  45. Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Reza Haf, and Yuan-Fang Li. 2024. An empirical analysis on spatial reasoning capabilities of large multimodal models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21440–21455, Miami, Florida, U… find
  46. Connor Shorten and Taghi M. Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1). find
  47. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. Preprint, arXiv:2401.06209.
  48. Vishaal Udandarao, Max F Burg, Samuel Albanie, and Matthias Bethge. 2024. Visual data-type understanding does not emerge from scaling vision-language models. In The Twelfth International Conference on Learning Representations. find
  49. Muhammad Usama, Syeda Aishah Asim, Syed Bilal Ali, Syed Talal Wasim, and Umair Bin Mansoor. 2025. Analysing the robustness of vision-language-models to common corruptions. Preprint, arXiv:2504.13690. — [arXiv:2504.13690](https://arxiv.org/abs/2504.13690) find
  50. S G Vandenberg and A R Kuse. 1978. Mental rotations, a group test of three-dimensional spatial visualization. Percept. Mot. Skills, 47(2):599–604. find
  51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  52. Wenqi Xian, Zhengqi Li, Matthew Fisher, Jonathan Eisenmann, Eli Shechtman, and Noah Snavely. 2019. Uprightnet: Geometry-aware camera orientation estimation from single images. Preprint, arXiv:1908.07070. — [arXiv:1908.07070](https://arxiv.org/abs/1908.07070) find
  53. Renjun Xu, Kaifan Yang, Ke Liu, and Fengxiang He. 2023. e(2)-equivariant vision transformer. Preprint, arXiv:2306.06722. — [arXiv:2306.06722](https://arxiv.org/abs/2306.06722) find
  54. Ruijie Xu, Yong Shi, and Zhiquan Qi. 2024. Image orientation estimation based on deep learning - a survey. Procedia Computer Science, 242:1193–1197. 11th International Conference on Information Technology and Quantitative Management (ITQM 2024). find
  55. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. Preprint, arXiv:2303.15343.
  56. Yikang Zhou, Tao Zhang, Shilin Xu, Shihao Chen, Qianyu Zhou, Yunhai Tong, Shunping Ji, Jiangning Zhang, Lu Qi, and Xiangtai Li. 2025. Are they the same? exploring visual correspondence shortcomings of multimodal llms. Preprint, arXiv:2501.04670. — [arXiv:2501.04670](https://arxiv.org/abs/2501.04670) find
  57. Guangming Zhu, Liang Zhang, Youliang Jiang, Yixuan Dang, Haoran Hou, Peiyi Shen, Mingtao Feng, Xia Zhao, Qiguang Miao, Syed Afaq Ali Shah, and Mohammed Bennamoun. 2022. Scene graph generation: A comprehensive survey. Preprint, arXiv:2201.00443. — [arXiv:2201.00443](https://arxiv.org/abs/2201.00443) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/niu-2026-rotbench-evaluating-multi-modal-large/