Elamur Research⌕
☰
home / papers / 2026 / MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models

MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models

Paper · 2026 · Yang Shi, Yifeng Xie, Minzhe Guo et al.

Summary

Most vision-language benchmarks score only whether the final answer is correct. They cannot show whether a model actually understands why a line of reasoning breaks down. MMErroR addresses this by testing whether VLMs can spot a flawed multimodal reasoning chain and name the kind of mistake it contains.

The authors took image–question pairs from existing benchmarks and used GPT-5 to plant exactly one error into an otherwise sound Chain-of-Thought. Each injected error belongs to one of four categories. Human experts then screened the candidates, leaving 1,997 items across six domains. Models are tested in two settings:

  • ETC (Error Type Classification): the model is told an error exists and must label its type.
  • EPD (Error Presence Detection): the model must first decide whether any error is present, then label it.

Across 12 VLMs, error diagnosis proved hard. The strongest model, Gemini-3-Pro-Preview, labelled the error type correctly in just 66.65% of cases, well short of human performance. The benchmark matters because it shifts evaluation toward process-level self-checking, which trustworthy multimodal systems need. Since every chain contains an error, the authors present it as a stress test, not a calibration benchmark.

Key points

  • Data sources and scale: items come from MMMU, MathVista, MathVerse, ScienceQA and AI2D, using stratified sampling and complexity filtering. The final set has 1,997 samples covering 24 subdomains within six domains: Data & Analytics, Physics & Engineering, Chemistry & Materials, Earth & Environment, Biology & Healthcare, and Mathematics & Logic.
  • Error types: the four categories are Knowledge Deployment Error (44.07%), Visual Perception Error (31.85%), Question Comprehension Error (12.37%) and Reasoning Error (11.72%).
  • Verification: 20 experts (6 professors and 14 PhD students) spent 23 days reviewing roughly 10,000 candidates. Each sample was checked in three rounds by different reviewers. Items were dropped if the error was incoherent, mislabelled or ambiguous.
  • Error awareness helps correction: on a 200-item subset, telling a model the error type raised its accuracy at correcting the answer. For example, Gemini-3-Pro-Preview rose from 81.0 to 90.5 and GPT-5.2 (xhigh) from 80.0 to 89.5.
  • Stated limitations: the benchmark has no multi-error cascades and no error-free chains. Because the errors were synthesized by a model, they may carry stylistic biases from the generator.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓Hong Kong Baptist UniversityHong Kong · China7 works · 246 citations🎓Guangdong University of TechnologyGuangzhou · China3 works · 112 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksVision & language · 657 works

Cite

Cite as Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu and Zhiqi Huang. 2026. MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2083

@inproceedings{shi2026mmerror,
  title = {{MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models}},
  author = {Yang Shi, Yifeng Xie, Minzhe Guo, Liangsi Lu, Mingxuan Huang, Jingchao Wang, Zhihong Zhu, Boyan Xu and Zhiqi Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2083},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2083}
}

References

41 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716– 23736.
  2. Anthropic. 2025. Claude opus 4.5 system card. https://www.anthropic.com/ claude-opus-4-5-system-card. Accessed: 2026-04-20. ↗
  3. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555–596. find
  4. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. ByteDance Seed Team. 2026. Seed2.0 model card. Official model card for the Seed2.0 series including Pro. find
  7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  8. Rongyao Fang, Aldrich Yu, Chengqi Duan, Linjiang Huang, Shuai Bai, Yuxuan Cai, Kun Wang, Si Liu, Xihui Liu, and Hongsheng Li. 2025. Flux-reason-6m & prism-bench: A million-scale text-to-image reasoning dataset and comprehensive benchmark. arXiv preprint arXiv:2509.09680. find
  9. Google. 2025. Gemini 3 developer guide. https://ai. Ac-google.dev/gemini-api/docs/gemini-3. cessed: 2026-01-01. find
  10. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of…
  11. Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, and 1 others. 2025. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006.
  12. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  13. Shima Imani, Seungwhan Moon, Lambert Mathias, Lu Zhang, and Babak Damavandi. 2026. Trace: A framework for analyzing and enhancing stepwise reasoning in vision-language models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Lo… find
  14. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In European conference on computer vision, pages 235–251. Springer.
  15. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355. 44982 10
  16. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  17. Zhenwei Tang, Difan Jiao, Blair Yang, and Ashton Anderson. 2025. Seam: Semantically equivalent across modalities benchmark for vision-language models. arXiv preprint arXiv:2508.18179. — [arXiv:2508.18179](https://arxiv.org/abs/2508.18179) find
  18. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  19. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  20. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521. find
  21. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, and 1 others. 2026. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276. — Kimi K2.5: Visual Agentic Intelligence
  22. Hongxu Ma, Guanshuo Wang, Fufu Yu, Qiong Jia, and Shouhong Ding. 2025a. Ms-detr: Towards effective video moment retrieval and highlight detection by joint motion-semantic learning. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 4514–4523. find
  23. Hongxu Ma, Chenbo Zhang, Lu Zhang, Jiaogen Zhou, Jihong Guan, and Shuigeng Zhou. 2025b. Fine-grained zero-shot object detection. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 4504–4513. find
  24. Meta. 2025a. Llama-4-maverick-17b-128e-instruct: Model card. https://huggingface.co/meta-llama/ Ac-Llama-4-Maverick-17B-128E-Instruct. cessed: 2026-01-01. ↗
  25. Meta. 2025b. Llama-4-scout-17b-16e-instruct: Model card. https://huggingface.co/meta-llama/ Llama-4-Scout-17B-16E-Instruct. Accessed: 2026-01-01. ↗
  26. Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. 2025. Towards interpreting visual information processing in vision-language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  27. OpenAI. 2025a. Gpt-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf. Accessed: 2026-01-01. ↗
  28. OpenAI. 2025b. Gpt-5.2 model \| openai api. https://developers.openai.com/api/docs/ models/gpt-5.2. Accessed: 2026-04-20. ↗
  29. Qwen Team. 2024. Introducing qwen-vl. https://qwenlm.github.io/blog/qwen-vl/. Accessed: 2026-01-01. ↗
  30. Jiacheng Ruan, Wenzhen Yuan, Xian Gao, Ye Guo, Daoxin Zhang, Zhe Xu, Yao Hu, Ting Liu, and Yuzhuo Fu. 2025. Vlrmbench: A comprehensive and challenging benchmark for vision-language reward models. arXiv preprint arXiv:2503.07478. — [arXiv:2503.07478](https://arxiv.org/abs/2503.07478) find
  31. Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, and 1 others. 2024. Towards generalist biomedical ai. Nejm Ai, 1(3):AIoa2300138. find
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837. xAI. 2025. Grok 4 (api documentation / model…
  33. Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, and 1 others. 2025. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279. — [arXiv:2504.15279](https://arxiv.org/abs/2504.15279) find
  34. Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, and 1 others. 2024. Errorradar: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv preprint arXiv:2410.04509.
  35. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vi…
  36. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, and 1 others. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer. 44983…
  37. Xiaofeng Zhang, Yihao Quan, Chen Shen, Chaochen Gu, Xiaosong Yuan, Shaotian Yan, Jiawei Cao, Hao Cheng, Kaijie Wu, and Jieping Ye. 2025. Shallow focus, deep fixes: Enhancing shallow layers vision attention sinks to alleviate hallucination in lvlms. In Proceedings of the 2025 Conference on Empirical…
  38. Xiaofeng Zhang, Yuanchao Zhu, Chaochen Gu, Jiawei Cao, Hao Cheng, and Kaijie Wu. 2026. What drives attention sinks? a study of massive activations and rotational positional encoding in large vision–language models. Information Processing & Management, 63(2):104431. find
  39. Qiyan Zhao, Xiaofeng Zhang, Yiheng Li, Yun Xing, Xiaosong Yuan, Feilong Tang, Sinan Fan, Xuhang Chen, Da-Han Wang, and Xu-Yao Zhang. 2025. Mca-llava: Manhattan causal attention for reducing hallucination in large vision-language models. In Proceedings of the 33rd ACM International Conference on Mult… find
  40. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Processbench: Identifying process errors in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  41. Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, and 1 others. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR. 44984 12 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-mmerror-benchmark-erroneous-reasoning-vision/