Elamur Research⌕
☰
home / papers / 2026 / Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science

Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science

Paper · 2026 · Sifan Wu, Huan Zhang, Yizhan Li et al.

Summary

Existing materials-science QA benchmarks such as MaScQA and SciQA are mostly text-only. They do not test whether multimodal models can read experimental figures and reason about them at a research level. The authors present MatVQA, a benchmark of 1,672 multiple-choice questions built from recent arXiv materials papers by an automated pipeline called MArxivAgent. The questions cover four structure–property–performance (SPP) task types: quantitative, comparative, causal, and hypothetical variation.

A key step in MArxivAgent is a loop that removes shortcuts. An evaluator agent tries to answer each question from the text alone, then from the text plus the figure caption. If either attempt succeeds, a rewriter revises the question, and a consistency checker makes sure the scientific claim stays intact. The surviving questions can only be solved by close inspection of images such as microscopy or diffraction patterns.

Across 19 open- and closed-source MLLMs, this refinement cut accuracy sharply. Error analysis indicates that missing domain knowledge, not only weak perception, is the main obstacle. The resource matters because it offers a scalable, visually grounded way to evaluate MLLMs for materials discovery.

Key points

  • Dataset composition: 1,672 MCQs drawn from 731 unique images, 120 papers, and 32 research areas.
    • Causal SPP: 950 questions
    • Quantitative SPP: 354 questions
    • Hypothetical SPP: 256 questions
    • Comparative SPP: 112 questions
    • Domain experts reviewed half of all items.
  • Effect of shortcut removal:
    • Removing language shortcuts lowered accuracy by about 12% on average.
    • Removing caption shortcuts led to an extra drop of about 19%.
    • Example: GPT-4o fell from 71.5% to 49.7%, and Claude-3.7-Sonnet from 75.9% to 52.5%.
  • Human baseline: two materials-science experts averaged 28% accuracy on a random sample of 100 questions.
  • Error audit (40 o1 mistakes):
    • Material knowledge misunderstanding: 47.5%
    • Visual perception errors: 30.0%
    • Faulty reasoning judgement: 22.5%
  • Availability: the dataset is released on Hugging Face (trqcbf/matvqa_v2) under a research-only, non-commercial license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksMaterials Science · 31 worksMaterials Chemistry · 31 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Sifan Wu, Huan Zhang, Yizhan Li, Farshid Effaty, Hongyuan Mei, Amirreza Ataei and Bang Liu. 2026. Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1181

@inproceedings{wu2026seeing,
  title = {{Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science}},
  author = {Sifan Wu, Huan Zhang, Yizhan Li, Farshid Effaty, Hongyuan Mei, Amirreza Ataei and Bang Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1181},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1181}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Google AI. Gemini 2.5: Our most intelligent ai model. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/, Mar 2025. ↗
  2. Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. The sciqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7240, 2023. find
  3. Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, et al. Testing and evaluation of health care applications of large language models: A systematic review. JAMA, 333(4):319, 2024. find
  4. Lenz Belzner, Thomas Gabor, and Martin Wirsing. Large language model assisted software engineering: prospects, challenges, and a case study. In International Conference on Bridging the Gap between AI and Reality, pp. 355–374. Springer, 2023. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. James Burgess, Jeffrey J Nirschl, Laura Bravo-Sánchez, Alejandro Lozano, Sanket Rajan Gupte, Jesus G Galaz-Montoya, Yuhui Zhang, Yuchang Su, Disha Bhowmik, Zachary Coman, et al. Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research. In Proceedings of the IEEE/CVF Confer… find
  7. Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding, 2025. URL https://arxiv.org/abs/2501.16411. — [arXiv:2501.16411](https://arxiv.org/abs/2501.16411) ↗
  8. Elizabeth Dinella, Satish Chandra, and Petros Maniatis. Crqbench: A benchmark of code reasoning questions, 2024. URL https://arxiv.org/abs/2408. 08453. ↗
  9. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024. URL https://arxiv.org/abs/2401.03065. ↗
  10. Kehan Guo, Bozhao Nan, Yujun Zhou, Taicheng Guo, Zhichun Guo, Mihir Surve, Zhenwen Liang, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Can llms solve molecule puzzles? a multimodal benchmark for molecular structure elucidation. Advances in Neural Information Processing Systems, 37:134721–13474… find
  11. Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark, 2025. URL https://arxiv.org/abs/2501.05444. — [arXiv:2501.05444](https://arxiv.org/abs/2501.05444) ↗
  12. Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025. find
  13. Kevin Maik Jablonka, Qianxiang Ai, Alexander Al-Feghali, Shruti Badhwar, Joshua D Bocarsly, Andres M Bran, Stefan Bringuier, L Catherine Brinson, Kamal Choudhary, Defne Circi, et al. 14 examples of how llms can transform materials science and chemistry: a reflection on a large language model hackath… find
  14. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
  15. Sung Jae Jung, Hajung Kim, and Kyoung Sang Jang. Llm based biological named entity recognition from scientific literature. In 2024 IEEE International Conference on Big Data and Smart Computing (Big-Comp), pp. 433–435. find
  16. Ehsan Latif, Ramviyas Parasuraman, and Xiaoming Zhai. Physicsassistant: An llm-powered interactive learning robot for physics lab investigations. In 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN), pp. 864–871. find
  17. Yu Song, Santiago Miret, Huan Zhang, and Bang Liu. Honeybee: Progressive instruction finetuning of large language models for materials science. arXiv preprint arXiv:2310.08511, 2023. find
  18. Yizhan Li, Florence Cloutier, Sifan Wu, Ali Parviz, Boris Knyazev, Yan Zhang, Glen Berseth, and Bang Liu. Mˆ 4olgen: Multi-agent, multi-stage molecular generation under precise multi-property constraints. arXiv preprint arXiv:2601.10131, 2026. — [arXiv:2601.10131](https://arxiv.org/abs/2601.10131) find
  19. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images, 2021. find
  20. Alejandro Lozano, Jeffrey Nirschl, James Burgess, Sanket Rajan Gupte, Yuhui Zhang, Alyssa Unell, and Serena Yeung-Levy. Micro-bench: A microscopy benchmark for vision-language understanding. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neur… find
  21. Hongchen Wang, Kangming Li, Scott Ramsay, Yao Fehlis, Edward Kim, and Jason Hattrick-Simpers. Evaluating the performance and robustness of llms in materials science q&a and property predictions. Digital Discovery, 4(6):1612–1624, 2025. find
  22. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255. ↗
  23. Angel Yanguas-Gil, Matthew T. Dearing, Jeffrey W. Elam, Jessica C. Jones, Sungjoon Kim, Adnan Mohammad, Chi Thang Nguyen, and Bratin Sengupta. Benchmarking large language models for materials synthesis: the case of atomic layer deposition, 2024. URL https://arxiv.org/abs/2412.10477. — [arXiv:2412.10477](https://arxiv.org/abs/2412.10477) ↗
  24. Rachel K Luu and Markus J Buehler. Bioinspiredllm: Conversational large language model for the mechanics of biological and bio-inspired materials. Advanced Science, 11(10):2306724, 2024. find
  25. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. pp. 15134–15186, 2025.
  26. Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1–13, 2024. find
  27. Zabir Al Nazi and Wei Peng. Large language models in healthcare and medical domain: A review. In Informatics, volume 11, pp. 57. MDPI, 2024. find
  28. Dung Manh Nguyen, Thang Chau Phan, Nam Le Hai, Tien-Thong Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui. Codemmlu: A multi-task benchmark for assessing code understanding & reasoning capabilities of codellms. In The Thirteenth International Conference on Learning Representations, 2025. xAI. Grok 4… find
  29. Mohd Zaki, NM Anoop Krishnan, et al. Mascqa: investigating materials science knowledge of large language models. Digital Discovery, 3(2):313–327, 2024. find
  30. Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, et al. Chemllm: A chemical large language model. arXiv preprint arXiv:2402.06852, 2024a. find
  31. Huan Zhang, Yu Song, Ziyu Hou, Santiago Miret, and Bang Liu. Honeycomb: A flexible llm-based agent system for materials science. arXiv preprint arXiv:2409.00135, 2024b. find
  32. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, OpenAI. Gpt-4o mini: Advancing cost-efficient intelli-Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does gence, 2024. URL https://openai.com/index/ your multi-modal llm truly see the diagrams in vigpt-4o-mi… ↗
  33. Springer, 2024c. Nima PourNejatian, Anthony B Costa, Cheryl Martin, Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Mona G Flores, Ying Zhang, Tanja Magoc, et al. A Bin Hu, and Huan Zhang. Dynamath: A dynamic study of generative large language model for medical visual benchmark for evaluating mathe… find
  34. Andre Niyongabo Rubungo, Kangming Li, Jason Hattrick-Simpers, and Adji Bousso Dieng. Llm4matbench: benchmarking large language models for materials property prediction. arXiv preprint arXiv:2411.00177, 2024. — [arXiv:2411.00177](https://arxiv.org/abs/2411.00177) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-seeing-beyond-words-matvqa-challenging/