Elamur Research⌕
☰
home / papers / 2026 / MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?

MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?

Paper · 2026 · Yuandong Wang, Yao Cui, Yuxin Zhao et al.

Summary

MathSight examines whether vision-language models (VLMs) actually use images when solving university-level math problems, or whether they mostly depend on the textual problem statement. Earlier multimodal math benchmarks usually supply one clean figure per problem, so they cannot show how much the visual channel contributes. The authors built a controlled benchmark from proprietary, never-published PDF materials. GPT-4o extracted the problems, and human experts verified their difficulty. Each multimodal item comes in four forms: the original figure, a hand-drawn redraw, a printed-and-photographed version, and a text-only condition.

Several strong VLMs were tested, including GPT-5, Gemini 2.5 Pro, Claude, Qwen3-VL, InternVL and GLM. The three image variants produced no statistically significant performance differences. Removing the image often matched or improved accuracy, and the value of visual input dropped as difficulty rose. The authors conclude that current VLMs largely process the text rather than truly look at the figure. This matters for anyone building or evaluating visually grounded reasoning systems.

Key points

  • Dataset construction: more than 20,000 source files were screened, yielding about 2,600 candidate problems. The final benchmark keeps 661 multimodal items, 603 of them graduate-level and 30 proving problems, plus a separate set of 1,387 text-only problems for evaluating LLMs.
  • Visual variants: five graduate students with different handwriting redrew the figures by hand. Printed copies were photographed with a phone, then cropped and resized to match the originals.
  • Headline result: Qwen3-VL-235B-A22B reached 50.53% accuracy with no image. With images it scored 40.85% (original), 38.73% (hand-drawn) and 39.33% (photo). Its text-only score also beat GPT-5 given the original image (45.39%).
  • Consistency across variants: only about 18% of items received different outcomes across the image versions. For GPT-5, 34.19% of items were correct in all three versions and 48.11% were wrong in all three.
  • Coverage and limitations: problems span six domains, heavily skewed toward Calculus (82.15%) and Applied Math (11.8%). Text-only LLMs given image captions stayed far behind VLMs, roughly 24–29% accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Capital Normal UniversityBeijing · China2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksMathematics · 177 worksComputational Theory and Mathematics · 252 worksApplied Mathematics · 46 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksVision & language · 657 works

Cite

Cite as Yuandong Wang, Yao Cui, Yuxin Zhao, Zhen Yang, Yangfu Zhu and Zhenzhou Shao. 2026. MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2198

@inproceedings{wang2026mathsight,
  title = {{MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?}},
  author = {Yuandong Wang, Yao Cui, Yuxin Zhao, Zhen Yang, Yangfu Zhu and Zhenzhou Shao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2198},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2198}
}

References

33 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. Computing Research Repository, arXiv:1905.13319.
  2. Anthropic. 2025a. Claude 3.7 sonnet system card. https://assets.anthropic.com/m/785e231869ea8b3b/original/ claude-3-7-sonnet-system-card.pdf. ↗
  3. Anthropic. 2025b. Introducing claude 4. https://www.anthropic.com/news/claude-4. ↗
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:25…
  5. Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep think with confidence. Preprint, arXiv:2508.15260. find
  6. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang, Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, and 178 others. 2025. Seed1.5-vl technical report. arXiv preprint arXiv:250…
  7. Himanshu Gupta, Shreyas Verma, Ujjwala Anantheswaran, Kevin Scaria, Mihir Parmar, Swaroop Mishra, and Chitta Baral. 2024. Polymath: A Challenging Multi-Modal Mathematical Reasoning Benchmark. Computing Research Repository, arXiv:2410.14702. — [arXiv:2410.14702](https://arxiv.org/abs/2410.14702) find
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. Computing Research Repository, arXiv:2009.03300.
  9. Felix Chen, Hangjie Yuan, Yunqiu Xu, Tao Feng, Jun Cen, Pengwei Liu, Zeying Huang, and Yi Yang. 2025. Mathflow: Enhancing the perceptual flow of mllms for visual mathematical problems. arXiv preprint arXiv:2503.16549. find
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. Computing Research Repository, arXiv:2103.03874.
  11. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889–7901.
  12. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, and 1 others. 2025. Glm-4.1 v-thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning. Computing Research Repository, arXiv:2507.01006.
  13. Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, and Sergei Tilga. 2024. U-math: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs. Computing Research Repository, arXiv:2412.03205.
  14. Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang. 2024. Mamo: A Mathematical Modeling Benchmark with Solvers. Computing Research Repository, arXiv:2405.13144. find
  15. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Computing Research Repository, arXiv:2110.14168.
  16. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Com…
  17. Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, and Michael P. Brenner. 2024. Hardmath: A Benchmark Dataset for Challenging Problems in Applied Mathematics. Computing Research Repository, arXiv:2410.09988. — [arXiv:2410.09988](https://arxiv.org/abs/2410.09988) find
  18. Zhewei Kang, Xuandong Zhao, and Dawn Song. 2025. Scalable best-of-n selection for large language models via self-certainty. Preprint, arXiv:2502.18581. find
  19. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. Computing Research Repository, arXiv:2309.06180.
  20. Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024. Gsm-plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers. Computing Research Repository, arXiv:2402.19255.
  21. Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. 2024a. Mathbench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark. Computing Research Repository, arXiv:2405.1220… find
  22. Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. 2024b. Are llms capable of data-based statistical and causal reasoning? Benchmarking Advanced Quantitative Reasoning with Data. Computing Research Repository, arXiv:2402.17644. find
  23. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  24. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and 1 others. 2022. Lila: A Unified Benchmark for Mathematical Reasoning. Computing Research Repository, arXiv:2210.17517.
  25. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with mathvision dataset. Computing Research Repository, arXiv:2402.14804.
  26. Zihao Zhou, Shudong Liu, Maizhen Ning, Wei Liu, Jindong Wang, Derek F. Wong, Xiaowei Huang, Qiufeng Wang, and Kaizhu Huang. 2024. Is your model really a good math reasoner? Evaluating Mathematical Reasoning with Checklist. Computing Research Repository, arXiv:2407.08733. — [arXiv:2407.08733](https://arxiv.org/abs/2407.08733) find
  27. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, and 32 others. 2025. Internvl3: Exploring Advanced Training and Test-…
  28. Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. 2024. Dynamath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. Computing Research Repository, arXiv:2411.00836. find
  29. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023. Scibench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models. Computing Research Repository, arXiv:2307.10635.
  30. Xin Xu, Jiaxin Zhang, Tianhao Chen, Zitong Chao, Jishan Hu, and Can Yang. 2025a. Ugmathbench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models. Computing Research Repository, arXiv:2501.13766. — [arXiv:2501.13766](https://arxiv.org/abs/2501.13766) find
  31. Yunqiu Xu, Linchao Zhu, and Yi Yang. 2025b. Mc-bench: A benchmark for multi-context visual grounding in the era of mllms. In Proceedings of the IEEE/CVF International Conference on Computer Vision. find
  32. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A Massive Multi-Discipline Multimodal U…
  33. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, and 1 others. 2024. Mathverse: Does Your Multi-Modal LLM Truly See the Diagrams in Visual Math Problems? In Proceedings of the European Conference on Computer Vision, pages 169–1…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-mathsight-benchmark-exploring-have-vision/