Elamur Research⌕
☰
home / papers / 2026 / VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning

VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning

Paper · 2026 · Jingkun Ma, Runzhe Zhan, Yang Li et al.

Summary

VisAidMath asks whether large multi-modal models (LMMs) can do more than read a diagram. It tests whether they can devise helpful constructions, such as auxiliary lines, and reason from them. The authors built a benchmark of 1,200 math problems from English and Chinese sources. The problems span exam-level to olympiad-level difficulty, and each comes with a visual aid written as text plus captions, so text-only LLMs can also be evaluated. Models are tested on three tasks: plain General Reasoning (GR), generating their own aid (D-VAR), or using a given aid (I-VAR). Scoring follows a "three-layered funnel" that adds two process-level checks on top of final-answer accuracy (ACCU). Process-Verified Accuracy (PVA) discards correct answers reached through flawed reasoning, and the Solution Process Robustness Score (SPRS) grades the finer quality of each solution. Every model tested, including Doubao-Seed-1.6 and o4-mini, scores much lower on PVA and SPRS than on ACCU. The authors call this gap a "reasoning illusion." The work shows that answer-only metrics overstate visual reasoning ability in current LMMs.

Key points

  • The dataset contains 1,200 problems drawn from 16 sources:
    • Difficulty is tiered by source: Chinese high school entrance exam (Easy), college entrance exam/Gaokao (Medium), and Mathematical Olympiad (Hard).
    • Chinese items were machine-translated with DeepL and Baidu Translate, then refined by human annotators.
  • Coverage spans four branches (plane geometry, solid geometry, analytic geometry, calculus) and six aid types:
    • auxiliary line
    • plane geometry graph
    • plane coordinate
    • three coordinate
    • function graph
    • solid geometry graph
  • Task formats are GR (C, Q → A), D-VAR (C, Q → V_g, A), and I-VAR (C, Q, V_p → A). Text-only variants replace the image with a caption, C_txt.
  • The size of the illusion is measured with two gaps: a Reliability Gap (ACCU − PVA) and a Robustness Gap (ACCU − SPRS).
  • A manual check of 200 correctly answered D-VAR items found that only 3.0% actually used visual aids. Most relied on shortcuts instead: 41.1% used pure arithmetic, and 19.3% used general reasoning that ignored the visuals. Severe visual-aid errors strongly predicted wrong answers. An evaluation platform is available on CodaBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🏢Alibaba DAMO AcademyHangzhou · China39 works · 6,378 citations🎓University of MacauMacao · China33 works · 2,223 citations🏢Alibaba Group (Cayman Islands)George Town · Cayman Islands32 works · 3,389 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksComputational Theory and Mathematics · 252 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Jingkun Ma, Runzhe Zhan, Yang Li, Di Sun, Hou Pong Chan, Lidia S. Chao and Derek F. Wong. 2026. VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1719

@inproceedings{ma2026visaidmath,
  title = {{VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning}},
  author = {Jingkun Ma, Runzhe Zhan, Yang Li, Di Sun, Hou Pong Chan, Lidia S. Chao and Derek F. Wong},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1719},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1719}
}

References

56 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card. find
  2. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3.
  3. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  4. Guanhua Chen, Runzhe Zhan, Derek F. Wong, and Lidia S. Chao. 2023. Multi-level curriculum learning for multi-turn dialogue generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:3958–3967. find
  5. Guanhua Chen, Runzhe Zhan, Derek F. Wong, and Lidia S. Chao. 2024. Dynamic curriculum learning for conversation response selection. Knowl. Based Syst., 293:111687. find
  6. Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. 2022. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746.
  7. Yuri Chervonyi, Trieu H Trinh, Miroslav Olšák, Xiaomeng Yang, Hoang Nguyen, Marcelo Menegali, Junehyuk Jung, Vikas Verma, Quoc V Le, and Thang Luong. 2025. Gold-medalist performance in solving olympiad geometry with alphageometry2. arXiv preprint arXiv:2502.03544. find
  8. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. 2024. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420.
  9. Daocheng Fu, Zijun Chen, Renqiu Xia, Qi Liu, Yuan Feng, Hongbin Zhou, Renrui Zhang, Shiyang Feng, Peng Gao, Junchi Yan, et al. 2025a. Trustgeogen: Scalable and formal-verified data engine for trustworthy multi-modal geometric problem solving. arXiv preprint arXiv:2504.15780. — [arXiv:2504.15780](https://arxiv.org/abs/2504.15780) find
  10. Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Cor-ring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, and Cha Zhang. 2025b. Refocus: Visual editing as a chain of thought for structured image understanding. arXiv preprint arXiv:2501.05452. find
  11. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR. find
  12. Google. 2023. Google gemini pro vision. In Google. Google. find
  13. Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rstar-math: Small LLMs can master math reasoning with self-evolved deep thinking. In Forty-second International Conference on Machine Learning.
  14. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. 2025a. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062. — Seed1.5-VL Technical Report
  15. Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Rui Huang, Haoquan Zhang, Manyuan Zhang, Jiaming Liu, Shanghang Zhang, Peng Gao, et al. 2025b. Can we generate images with cot? let’s verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926. — [arXiv:2501.13926](https://arxiv.org/abs/2501.13926) find
  16. Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14953–14962. find
  17. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  18. Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699. — [arXiv:2211.09699](https://arxiv.org/abs/2211.09699) find
  19. Gueyoung Jung, Matti A Hiltunen, Kaustubh R Joshi, Richard D Schlichting, and Calton Pu. 2010. Mistral: Dynamically managing power, performance, and adaptation cost in cloud infrastructures. In 2010 IEEE 30th International Conference on Distributed Computing Systems, pages 62–73. IEEE. find
  20. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213. larryflynt. 2023. https://github.com/larryflynt/image-concat. In Github. ↗
  21. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843– 3857.
  22. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  23. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165.
  24. Jingkun Ma, Runzhe Zhan, and Derek F. Wong. 2023. Yu sheng: Human-in-loop classical Chinese poetry generation system. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 57–66, Dubrovnik, Croatia. Association fo… find
  25. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.
  26. OpenAI. 2023a. Gpt-3.5-turbo. In OpenAI. find
  27. OpenAI. 2023b. Gpt-4. In OpenAI. OpenAI. find
  28. OpenAI. 2023c. Gpt-4.1. In OpenAI. OpenAI. find
  29. OpenAI. 2023d. Gpt-4v. In OpenAI. OpenAI. find
  30. OpenAI. 2023e. o4-mini. In OpenAI. OpenAI. find
  31. Jaewoo Park, Jungyang Park, Dongju Jang, Jiwan Chung, Byungwoo Yoo, Jaewoo Shin, Seonjoon Park, Taehyeong Kim, and Youngjae Yu. 2025. Explain with visual keypoints like a real mentor! a benchmark for multimodal solution explanation. arXiv preprint arXiv:2504.03197. — [arXiv:2504.03197](https://arxiv.org/abs/2504.03197) find
  32. Can Li, Ying Liu, Ting Zhang, Mei Wang, and Hua Huang. 2025. Visiomath: Benchmarking figure-based mathematical reasoning in lmms. arXiv preprint arXiv:2506.06727. — [arXiv:2506.06727](https://arxiv.org/abs/2506.06727) find
  33. Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. 2024. Multimath: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.00147. — [arXiv:2409.00147](https://arxiv.org/abs/2409.00147) find
  34. Adam Dahlgren Lindström and Savitha Sam Abraham. 2022. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358. — [arXiv:2208.05358](https://arxiv.org/abs/2208.05358) find
  35. Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. 2024. Cogcom: A visual language model with chain-of-manipulations reasoning. arXiv preprint arXiv:2402.04236. — [arXiv:2402.04236](https://arxiv.org/abs/2402.04236) find
  36. Dairu Liu, Ziyue Wang, Minyuan Ruan, Fuwen Luo, Chi Chen, Peng Li, and Yang Liu. 2025. Visual abstract thinking empowers multimodal reasoning. arXiv preprint arXiv:2505.20164. — [arXiv:2505.20164](https://arxiv.org/abs/2505.20164) find
  37. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. find
  38. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. find
  39. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024b. Visual instruction tuning. Advances in neural information processing systems, 36.
  40. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  41. Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024. Math-LLaVA: Bootstrapping mathematical reasoning for multimodal large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4663–4680, Miami, Florid…
  42. Neural Information Processing Systems, 37:90277– 90317. find
  43. Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. 2021. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pages 11251–11262. PMLR. find
  44. Zhaochen Su, Peng Xia, Hangyu Guo, Zhenhua Liu, Yan Ma, Xiaoye Qu, Jiaqi Liu, Yanshu Li, Kaide Zeng, Zhengyuan Yang, et al. 2025. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers. arXiv preprint arXiv:2506.23918. — [arXiv:2506.23918](https://arxiv.org/abs/2506.23918) find
  45. Ruifeng Yuan, Chenghao Xiao, Sicong Leng, Jianyu Wang, Long Li, Weiwen Xu, Hou Pong Chan, Deli Zhao, Tingyang Xu, Zhongyu Wei, et al. 2025. Vl-cogito: Progressive curriculum reinforcement learning for advanced multimodal reasoning. arXiv preprint arXiv:2507.22607. — [arXiv:2507.22607](https://arxiv.org/abs/2507.22607) find
  46. Chameleon Team. 2024. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818.
  47. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015. find
  48. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  49. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653.
  50. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. 2025. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.
  51. Yiming Zhang, Zicheng Zhang, Xinyi Wei, Xiaohong Liu, Guangtao Zhai, and Xiongkuo Min. 2025. Illu-sionbench: A large-scale and comprehensive benchmark for visual illusion understanding in vision-language models. In 2025 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. find
  52. Shichao Weng, Zhiqiang Wang, Yuhua Zhou, Rui Lu, Ting Liu, Zhiyang Teng, Xiaozhang Liu, and Hanmeng Liu. 2025. Geosketch: A neural-symbolic approach to geometric multimodal reasoning with auxiliary line construction and affine transformation. arXiv preprint arXiv:2509.22460. — [arXiv:2509.22460](https://arxiv.org/abs/2509.22460) find
  53. Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. 2025. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1702–1… find
  54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  55. Qinzhuo Wu, Qi Zhang, Jinlan Fu, and Xuan-Jing Huang. 2020. A knowledge-aware sequence-to-tree network for math word problem solving. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 7137– 7146. find
  56. Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. 2024. Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large language models. Advances in Contents find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-visaidmath-benchmarking-visual-aided-mathematical/