Elamur Research⌕
☰
home / papers / 2026 / How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

Paper · 2026 · Pritam Sil, Durgaprasad Karnam, Vinay Reddy Venumuddala et al.

Summary

The paper asks whether vision–language models can help teachers judge how well students understand STEM concepts. The authors treat answers as evidence of underlying mental models rather than as items to be marked. They introduce MMGrader, which uses concept graphs to analyze handwritten responses that mix text and diagrams. A topic is split into a hierarchy of broader concept areas (BCAs), sub-concept areas (SCAs) and Key Concept Links (KCLs). Instructors link each question to its relevant concept links. A model then assigns a strength score to each link using a fixed scoring scale. Nine VLMs were tested against expert human annotations, mainly on physics questions about vectors. Even the strongest model reached only about 40% accuracy, with a prediction error near 1.1 units, although its score distribution roughly resembled human patterns. The authors argue that more accurate models could let teachers map understanding across a whole class and target weak areas. They conclude that current VLMs are not yet reliable enough for this task without further training on real classroom data.

Key points

  • MMGrader takes three inputs: handwritten multimodal assessments, a concept graph G(V, E) with V = BCA ∪ SCA ∪ KCL, and a question–concept link mapping. It outputs per-link strength scores and requires no extra material from instructors beyond standard curriculum resources.
  • Models were compared using Accuracy, RMSE and EMD. Molmo was best overall; under chain-of-thought (CoT) prompting it scored 34.16% accuracy, 1.27 RMSE and 0.63 EMD.
  • Under CoT, Qwen came second (29.66% accuracy) and appeared to benefit from step-by-step cues. Granite was weakest (0.95% accuracy, 3.93 RMSE).
  • Error analysis found three main failure types:
    • InternLM, Qwen and Pixtral overthought and contradicted themselves, sometimes ignoring the diagram in the question.
    • LLaVa output only 1 or 5, copying the examples in the prompt.
    • Granite described images instead of scoring them.
  • The closed model Gemini reached about 23% accuracy and often broke the required output format by writing scores as \boxed{}. Code is released on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology BombayMumbai · India29 works · 2,104 citations

Also on the byline: Mahindra University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Pritam Sil, Durgaprasad Karnam, Vinay Reddy Venumuddala and Pushpak Bhattacharyya. 2026. How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.50

@inproceedings{sil2026how,
  title = {{How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?}},
  author = {Pritam Sil, Durgaprasad Karnam, Vinay Reddy Venumuddala and Pushpak Bhattacharyya},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.50},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.50}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Google DeepMind. 2025b. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Variant: Gemma-3-12B-IT. find
  2. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hend… — Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  3. Zhiyu Fan, Shin Hwei Tan, and Abhik Roychoudhury. 2023. Concept-based automated grading of cs-1 programming assignments. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, page 199–210, New York, NY, USA. Association for Computing Machinery. find
  4. IBM Research Granite Vision Team. 2025. Granite vision: a lightweight, open-source multimodal model for enterprise intelligence. arXiv preprint arXiv:2502.09927. Model: granite-vision-3.3-2B. — [arXiv:2502.09927](https://arxiv.org/abs/2502.09927) find
  5. David Hestenes. 2009. Modeling theory for math and science education. In Modeling Students’ Mathematical Modeling Competencies: ICTMA 13, pages 13–41. Springer. find
  6. Milo B. Hillegas. 1912. A scale for the measurement of quality in english composition by young people. Teachers College Record, 13(4):1–1. find
  7. Rajat Agarwal, Varun Khurana, Karish Grover, Mukesh Mohania, and Vikram Goyal. 2022. Multi-relational graph transformer for automatic short answer grading. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  8. Durgaprasad Karnam, Harshit Agrawal, Pranay Parte, Saurabh Ranjan, Priyanka Borar, Prasanna Prakash Kurup, Amose Jebin Joel, Pattamadai Sankaran Srinivasan, Uddhav Suryawanshi, Aniket Sule, and Sanjay Chandrasekharan. 2021. Touchy feely vectors: A compensatory design approach to support model-based… find
  9. Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Théophile Gervet, et al. 2024. Pixtral 12b. arXiv preprint. find
  10. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS). Oral presentation.
  11. Google DeepMind. 2025a. Gemini 2.5 flash: Price-performance balanced multimodal thinking model. arXiv preprint arXiv:2507.06261. Also refers to Gemini-2.5 model family including Flash, Flash-Lite, Pro, model card available online.
  12. Nabin Maharjan and Vasile Rus. 2019. A concept map based assessment of free student answers in tutorial dialogues. In Artificial Intelligence in Education, volume 11625 of Lecture Notes in Computer Science, pages 244–257. Springer. find
  13. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 1998. A metric for distributions with applications to image databases. In Proceedings of the Sixth International Conference on Computer Vision (ICCV), pages 59–66. IEEE / Narosa Publishing House. find
  14. Vasile Rus, Mihai Lintean, and Roger Azevedo. 2009. Automatic detection of student mental models during prior knowledge activation in metatutor. In Proceedings of the 2nd International Conference on Educational Data Mining (EDM 2009), pages 161–170, Cordoba, Spain. find
  15. Archana Sahu and Plaban Kumar Bhowmick. 2025. Directed graph-alignment approach for identification of gaps in short answers. arXiv preprint arXiv:2504.04473. — [arXiv:2504.04473](https://arxiv.org/abs/2504.04473) find
  16. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837. Curran Associa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sil-2026-how-effective-are-vlms-assisting/