Elamur Research⌕
☰
home / papers / 2026 / Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks

Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks

Paper · 2026 · Xinhe Wang, Jin Huang, Xingjian Zhang et al.

Summary

ARC and ARC-AGI are often treated as tests of "fluid" reasoning, and frontier vision-language models (VLMs) score far below humans on them. This is usually read as weak machine reasoning. The authors argue that much of the gap comes from the models' trouble seeing the puzzles, not from failures of rule induction. To test this, they split solving into two stages. First, each image is described in natural language on its own, so no clues from comparing images can leak in. Then a model infers and applies the rule using only those text descriptions. They compare this pipeline with standard end-to-end evaluation on Mini-ARC, ACRE and Bongard-LOGO, and they manually label the cause of each failure. The two-stage setup clearly beats one-stage use of the same VLM, and most errors turn out to be perceptual. This suggests that ARC-style scores mix perception with reasoning and may overstate how poorly models reason. The authors call for evaluation protocols that measure the two separately.

Key points

  • The tests cover three knowledge-light visual abstraction datasets: Mini-ARC, ACRE and Bongard-LOGO. Models studied include o1, Claude-Sonnet-4.5, GPT-4o and LLaVA1.5.
  • A hybrid pipeline used a strong VLM for perception and a weaker one for reasoning. It came close to the strong VLM run end-to-end and far exceeded the weak VLM run end-to-end.
  • Manual inspection of reasoning traces (50 sampled tasks for Mini-ARC and ACRE, 100 for Bongard-LOGO) found that about 80% of failures were perception errors.
  • Perception errors made up this share of all errors, one-stage vs. two-stage:
    • Mini-ARC: 86.4% vs. 59.5%
    • Bongard-LOGO: 65.8% vs. 37.0%
    • ACRE: 76.3% vs. 68.8%
  • On ACRE, using GPT-4o as the perception module removed perception errors entirely, while LLaVA1.5 still had 68.8%. One exception to the overall pattern is o1 on Mini-ARC, which the authors suspect may reflect training-data contamination.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Xinhe Wang, Jin Huang, Xingjian Zhang, Tianhao Wang and Jiaqi W. Ma. 2026. Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.826

@inproceedings{wang2026your,
  title = {{Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks}},
  author = {Xinhe Wang, Jin Huang, Xingjian Zhang, Tianhao Wang and Jiaqi W. Ma},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.826},
  url = {https://doi.org/10.18653/v1/2026.acl-long.826}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. 2025. The surprising effectiveness of test-time training for few-shot learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Lear… find
  2. ARC Prize Foundation. 2025. Arc-agi-1 benchmark. Web page. Accessed 2025-11-22. find
  3. ARC Prize, Inc. 2025. Arc-agi-1 leaderboard. find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Gaurav Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 34 others. 2021. E… — Evaluating Large Language Models Trained on Code
  5. Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. 2025. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831. find
  6. François Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547. find
  7. François Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. 2024. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604. find
  8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  9. Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, and 1 others. 2024. Combining induction and transduction for abstract reasoning. arXiv preprint arXiv:2411.02272. find
  10. Isaac Liao and Albert Gu. 2025. Arc-agi without pre-training. arXiv preprint arXiv:2512.06104. — [arXiv:2512.06104](https://arxiv.org/abs/2512.06104) find
  11. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306.
  12. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2024b. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer.
  13. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  14. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. find
  15. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  16. Daniel Franzen, Jan Disselhoff, and David Hartmann. 2025. The architects - technical report: Arc prize 2025 solution. Accessed: 2026-01-04. find
  17. Melanie Mitchell. 2024. Did openai just solve abstract reasoning? AI: A Guide for Thinking Humans (Substack). Accessed 2025-01-04. find
  18. DeepMind / Google. 2025. Gemini — our most intelligent ai models. Web page. Accessed 2025-11-22. find
  19. Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. 2020. Bongard-logo: A new benchmark for human-level concept learning and reasoning. Advances in Neural Information Processing Systems, 33:16468–16480. find
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  21. Keya Hu, Ali Cy, Linlu Qiu, Xiaoman Delores Ding, Runqian Wang, Yeyin Eva Zhu, Jacob Andreas, and Kaiming He. 2025. Arc is a vision problem! arXiv preprint arXiv:2511.14761. find
  22. Alexia Jolicoeur-Martineau. 2025. Less is more: Recursive reasoning with tiny networks. arXiv preprint arXiv:2510.04871. find
  23. Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. 2022. Playgrounds for abstraction and reasoning. In NeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI). find
  24. OpenAI. 2024. Introducing o3 and o3-mini (announcement video). YouTube video. Accessed 2025-11-22. find
  25. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  26. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326. 18122 10 find
  27. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and 1 others. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: AC…
  28. Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. 2025. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734. find
  29. Zihan Weng, Lucas Gomez, Taylor Whittington Webb, and Pouya Bashivan. 2025. Caption this, reason that: Vlms caught in the middle. arXiv preprint arXiv:2505.21538. — [arXiv:2505.21538](https://arxiv.org/abs/2505.21538) find
  30. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vi…
  31. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, and 1 others. 2025. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lingu… find
  32. Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. 2021. Acre: Abstract causal reasoning beyond covariation. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 10643–10653. 18123 11 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-your-reasoning-benchmark-may-not/