Elamur Research⌕
☰
home / papers / 2026 / Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

Paper · 2026 · Rohit Sinha, Aditya Sanjiv Kanade, Sai Srinivas Kancheti et al.

Summary

Multimodal LLMs perform well on standard vision-language benchmarks. It is less clear whether they can carry out the mental operations humans use on visual puzzles, such as rotating, folding or recombining shapes. Existing benchmarks also often allow models to lean on language priors instead of the image.

The authors present MIND'S EYE, a multiple-choice benchmark of eight programmatically generated tasks. The tasks draw on well-known psychometric assessments, including those that ask people to mentally turn objects or imagine folded sheets of paper. They are grouped under an Abstraction–Relation–Transformation (ART) taxonomy based on Carroll's model of fluid intelligence. Items avoid world knowledge, allow parametric control of difficulty, and include distractors built to expose specific reasoning errors.

The authors evaluated 18 open and closed MLLMs against a human baseline. Humans reach about 80% accuracy, while the strongest models stay under 50%. Model accuracy barely changes across difficulty levels, which suggests the underlying operations are missing rather than merely strained by harder items. Prompting helps only some task types.

The benchmark gives a controlled diagnostic for visuospatial reasoning and points to concrete weaknesses for future model design.

Key points

  • Taxonomy and tasks: Eight visuo-cognitive tasks are organised under the ART taxonomy, which covers pattern induction, analogical mapping and mental manipulation. The authors present it as the first benchmark meeting all six of their diagnostic criteria, including keyed distractors and scalable generation.
  • Overall results: Across 18 MLLMs, humans average about 80% accuracy and top models stay below 50%. The biggest gaps are on Transformation and Abstraction tasks.
  • Effect of difficulty:
    • Human accuracy falls from above 0.80 on Easy items to roughly 0.25 on Hard ones.
    • Model accuracy stays flat at about 0.20–0.45 regardless of difficulty.
  • Effect of prompting:
    • Meta-task and Step-by-step prompts raise Abstraction scores by about +1.3 pts.
    • Every alternative prompt lowers Transformation scores, with Hint prompting dropping by about −0.9 pts.
  • Further experiments and resources:
    • Fine-tuning and RL-based alignment were also tested on a strong open-source model.
    • Attention analysis shows models find the relevant image regions but fail to reason over them.
    • Generators, seeds and scoring code are released at github.com/microsoft/Mind-s-Eye.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft ResearchBengaluru · India13 works · 3,602 citations🎓Indian Institute of Technology HyderabadSangāreddi · India4 works · 108 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksVision & language · 657 works

Cite

Cite as Rohit Sinha, Aditya Sanjiv Kanade, Sai Srinivas Kancheti, Vineeth N. Balasubramanian and Tanuja Ganu. 2026. Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2124

@inproceedings{sinha2026mind,
  title = {{Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs}},
  author = {Rohit Sinha, Aditya Sanjiv Kanade, Sai Srinivas Kancheti, Vineeth N. Balasubramanian and Tanuja Ganu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2124},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2124}
}

References

76 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, and Akshay Nambi. 2024. Promptwizard: Task-aware prompt optimization framework. arXiv preprint arXiv:2405.18369. — [arXiv:2405.18369](https://arxiv.org/abs/2405.18369) find
  2. Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. 2025. Thought anchors: Which llm reasoning steps matter? Preprint, arXiv:2506.19143. find
  3. Lorelle Burton. 2003. Examining the relation between visual imagery and spatial ability tests. International Journal of Testing, 3(3):277–291. find
  4. John B. Carroll. 1993. Human cognitive abilities: A survey of factor-analytic studies. Cambridge University Press. find
  5. Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. 2025. Arc-agi-2: A new challenge for frontier ai reasoning systems. Preprint, arXiv:2505.11831. find
  6. François Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547. find
  7. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In The Twelfth International Conference on Learning Representations. find
  8. Paul De Boeck and Mark Wilson. 2003. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer. find
  9. Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. 2024. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedi… find
  10. Ruth B. Ekstrom, John W. French, and Harry H. Harman. 1976a. Manual for kit of factor-referenced cognitive tests. find
  11. Ruth B. Ekstrom, John W. French, Harry H. Harman, and Diran Dermen. 1976b. Kit of Factor-Referenced Cognitive Tests. Educational Testing Service, Princeton, NJ. find
  12. Susan E. Embretson. 1983. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93(1):179–197. find
  13. Susan E. Embretson and Steven P. Reise. 2013. Item Response Theory for Psychologists. Psychology Press. find
  14. Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Naraya-naraju, Xinze Guan, and Xin Eric Wang. 2025. Grit: Teaching mllms to think with images. arXiv preprint arXiv:2505.15879. — [arXiv:2505.15879](https://arxiv.org/abs/2505.15879) find
  15. François Fleuret, Tingting Li, Charles Dubout, Eric K. Wampler, Steven Yantis, and Donald Geman. 2011. Comparing machines and humans on a visual categorization test. PNAS, 108(43):17621–17625. find
  16. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024,… find
  17. Deepanway Ghosal, Vernon Toh, Yew Ken Chia, and Soujanya Poria. 2025. AlgoPuzzleVQA: Diagnosing multimodal reasoning challenges of language models with algorithmic multimodal puzzles. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Li… find
  18. J. P. Guilford and Wayne S. Zimmerman. 1948. The guilford-zimmerman aptitude survey. Journal of Applied Psychology, 32:24–34. find
  19. Graeme S. Halford, William H. Wilson, and Steven Phillips. 2010. Relational complexity and reasoning. Cognitive Science, 34(8):1451–1476. find
  20. Douglas R Hofstadter. 1979. Gödel, Escher, Bach: An Eternal Golden Braid. Basic Books. find
  21. Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang, Youliang Yuan, Xiaoyuan Liu, Wenxuan Wang, Wenxiang Jiao, Pinjia He, Zhaopeng Tu, and Haodong Duan. 2025a. Human cognitive benchmarks reveal foundational visual gaps in mllms. Preprint, arXiv:2502.16435. — [arXiv:2502.16435](https://arxiv.org/abs/2502.16435) find
  22. Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang, Youliang Yuan, Xiaoyuan Liu, Wenxuan Wang, Wenxiang Jiao, Pinjia He, Zhaopeng Tu, and Haodong Duan. 2025b. Human cognitive benchmarks reveal foundational visual gaps in mllms. Preprint, arXiv:2502.16435. — [arXiv:2502.16435](https://arxiv.org/abs/2502.16435) find
  23. Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR.
  24. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR.
  25. Aditya Kanade and Tanuja Ganu. 2025. Do you see me : A multidimensional benchmark for evaluating visual perception in multimodal llms. Preprint, arXiv:2506.02022. find
  26. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  27. Maithilee Kunda, Keith McGreggor, and Ashok Goel. 2012. Reasoning on the raven’s advanced progressive matrices test with iconic visual representations. In Proceedings of the Cognitive Science Society, Sapporo, Japan. find
  28. Michael Lan, Philip Torr, and Fazl Barez. 2024. Towards interpretable sequence continuation: Analyzing shared circuits in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12576– 12601, Miami, Florida, USA. Association for Computa… find
  29. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  30. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2023a. Seed-bench-2: Benchmarking multimodal large language models. find
  31. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13299–13308. find
  32. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023b. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125.
  33. Junnan Li, Dongxu Li, Caiming Xiong, and Steven CH Hoi. 2023c. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML.
  34. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024a. Mmbench: Is your multi-modal model an all-around player? In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–Octob…
  35. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024b. Mmbench: Is your multi-modal model an all-around player? In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–Octob…
  36. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024a. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR).
  37. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024b. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. Preprint, arXiv:2310.02255.
  38. Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Preprint, arXiv:2209.09513. find
  39. Suzanne A. McFall, Jeanette C. Deitz, and Terry K. Crowe. 1993. Test-retest reliability of the test of visual perceptual skills with children with learning disabilities. American Journal of Occupational Therapy, 47(9):819–824. find
  40. Kevin S McGrew. 2005. The Cattell-Horn-Carroll theory of cognitive abilities. Springer. find
  41. Kevin C. Moen, Michael R. Beck, Samuel M. Saltz-mann, and 1 others. 2020. Strengthening spatial reasoning: elucidating the attentional and neural mechanisms associated with mental rotation skill development. Cognitive Research: Principles and Implications, 5(1):20. find
  42. Allen Newell. 1994. Unified theories of cognition. Harvard University Press. find
  43. Weili Nie, Zhiding Yu, Lei Mao, Ankit B. Patel, Yuke Zhu, and Animashree Anandkumar. 2020a. Bongard-logo: a new benchmark for human-level concept learning and reasoning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran… find
  44. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020b. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901.
  45. Pranshu Pandya, Vatsal Gupta, Agney S Talwarr, Tushar Kataria, Dan Roth, and Vivek Gupta. 2025. NTSEBENCH: Cognitive reasoning benchmark for vision language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3680–3708, Albuquerque, New Mexico. Association for Com… find
  46. Tan-Hanh Pham, Phu-Vinh Nguyen, Dang The Hung, Bui Trong Duong, Vu Nguyen Thanh, Chris Ngo, Tri Quang Truong, and Truong-Son Hy. 2025. find
  47. Iqbench: How ”smart” are vision-language models? a study with human iq tests. Preprint, arXiv:2505.12000. — [arXiv:2505.12000](https://arxiv.org/abs/2505.12000) find
  48. Riverside Publishing. 2009. Cognitive Abilities Test (CogAT) Form 7, Paper Folding Items. Riverside. Psychometric test battery. find
  49. Achyuta Rajaram, Neil Chowdhury, Antonio Torralba, Jacob Andreas, and Sarah Schwettmann. 2024. Automatic discovery of visual circuits. find
  50. John Raven. 2000. Raven’s progressive matrices. Handbook of Nonverbal Assessment, pages 223– 237. find
  51. Yufan Ren, Konstantinos Tertikas, Shalini Maiti, Junlin Han, Tong Zhang, Sabine Süsstrunk, and Filippos Kokkinos. 2025. Vgrp-bench: Visual grid reasoning puzzle benchmark for large vision-language models. Preprint, arXiv:2503.23064. — [arXiv:2503.23064](https://arxiv.org/abs/2503.23064) find
  52. W. Joel Schneider and Kevin S. McGrew. 2018. Intelligence in education: Cattell-horn-carroll theory and assessment. Psychology in the Schools, 55(1):7–43. find
  53. Alessandro Serra, Francesco Ortu, Emanuele Panizon, Lucrezia Valeriani, Lorenzo Basile, Alessio Ansuini, Diego Doimo, and Alberto Cazzaniga. 2025. The narrow gate: Localized image-text communication in native multimodal models. Preprint, arXiv:2412.06646. — [arXiv:2412.06646](https://arxiv.org/abs/2412.06646) find
  54. Roger N Shepard and Jacqueline Metzler. 1971. Mental rotation of three-dimensional objects. Science. find
  55. Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. 2025. Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge. Preprint, arXiv:2504.10342. find
  56. Ilias Stogiannidis, Steven McDonagh, and Sotirios A Tsaftaris. 2024. Mind the Gap: Benchmarking spatial reasoning in vision-language models. arXiv preprint arXiv:2403.19707. — [arXiv:2403.19707](https://arxiv.org/abs/2403.19707) find
  57. Ilias Stogiannidis, Steven McDonagh, and Sotirios A. Tsaftaris. 2025a. Mind the gap: Benchmarking spatial reasoning in vision-language models. Preprint, arXiv:2503.19707. — [arXiv:2503.19707](https://arxiv.org/abs/2503.19707) find
  58. Ilias Stogiannidis, Steven McDonagh, and Sotirios A. Tsaftaris. 2025b. Mind the gap: Benchmarking spatial reasoning in vision-language models. arXiv preprint arXiv:2503.19707. — [arXiv:2503.19707](https://arxiv.org/abs/2503.19707) find
  59. Alane Suhr, Adam Trischler, Jackie Chi Kit Cheung, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In ACL.
  60. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penc… find
  61. Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR.
  62. Preprint, Louis Leon Thurstone. 1938. Primary Mental Abilities. University of Chicago Press, Chicago. find
  63. Sinan Urgun and Seçkin Arı. 2025. An analysis of architectural impact on llm-based abstract visual reasoning: A systematic benchmark on raven-fair. Preprint, arXiv:2511.11916. — [arXiv:2511.11916](https://arxiv.org/abs/2511.11916) find
  64. Steven G. Vandenberg and Allan R. Kuse. 1978a. Mental rotations, a group test of three-dimensional spatial visualization. Perceptual and Motor Skills, 47(2):599–604. find
  65. Steven G Vandenberg and Allan R Kuse. 1978b. Mental rotations, a group test of three-dimensional spatial visualization. Perceptual and Motor Skills, 47(2):599–604. find
  66. Steven G. Vandenberg and Allan R. Kuse. 1978c. Mental rotations, a group test of three-dimensional spatial visualization. Perceptual and Motor Skills, 47(2):599–604. find
  67. Siting Wang, Minnan Pei, Luoyang Sun, Cheng Deng, Kun Shao, Zheng Tian, Haifeng Zhang, and Jun Wang. 2025. Spatialviz-bench: An mllm benchmark for spatial visualization. Preprint, arXiv:2507.07610. — [arXiv:2507.07610](https://arxiv.org/abs/2507.07610) find
  68. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. find
  69. Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  70. Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. 2025. MLLMs know where to look: Training-free perception of small visual details with multimodal LLMs. In The Thirteenth International Conference on Learning Representations. find
  71. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? Preprint, arXiv:2403.14624.
  72. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R…
  73. Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, and Jinguo Zhu. 2025a. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. Preprint, arXiv:2504.15279. — [arXiv:2504.15279](https://arxiv.org/abs/2504.15279) find
  74. Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, and Jinguo Zhu. 2025b. Visulogic: A benchmark for evaluating visual reasoning in multimodal large language models. find
  75. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.
  76. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sinha-2026-mind-s-eye-benchmark-visual/