Elamur Research⌕
☰
home / papers / 2026 / EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions

EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions

Paper · 2026 · Weiyu Sun, Liangliang Chen, Yongnuo Cai et al.

Summary

Multimodal LLMs are increasingly proposed for grading student work. However, it is unclear how accurately they read real university STEM handwriting, where formulas, prose and hand-drawn diagrams are mixed together. Existing benchmarks tend to cover isolated expressions or K–12 material and judge models only by downstream outcomes. That hides recognition mistakes the grading rubric never touches. The authors release EDU-CIRCUIT-HW, a collection of 1,334 authentic homework solutions from an undergraduate circuit analysis course. Every solution carries expert grades, and one subset also has expert-proofread, near-verbatim transcriptions, including textual descriptions of diagrams. Using these references, they compare MLLM transcriptions against ground truth with an LLM-as-a-judge pipeline and sort the errors into a taxonomy. They then trace how those errors affect auto-grading. The models make many hidden recognition errors. These rarely matter for coarse correct/incorrect checks but increasingly distort finer-grained scoring. In a case study, the discovered error patterns are used to flag risky submissions in advance, which makes AI grading more robust with very little human effort.

Key points

  • The data comes from 29 students in a Spring 2025 circuit analysis course at a US public research university. It covers 62 unique problems from the Svoboda and Dorf (2013) textbook, and each solution is graded on five aspects.
  • There are two splits:
    • an observation set of 513 solutions from 11 students, with both verified transcriptions and grades;
    • a test set of 821 solutions from 18 students, with grades only.
    • Student solution accuracy is about 72% in both splits.
  • Gemini-2.5-Pro produces Markdown transcriptions, with equations copied verbatim and diagrams described in prose. An LLM judge then locates the errors, which are grouped into four categories.
  • Case study: only 3.3% of assignments are sent to human graders and the rest go to a GPT-5.1 grader, which improves system reliability.
  • Original problem statements are withheld for copyright reasons. The authors' own reference solutions and grading rubrics are included, and the data is anonymized and IRB-approved.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations🎓Virginia TechBlacksburg · United States36 works · 43,036 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Weiyu Sun, Liangliang Chen, Yongnuo Cai, Huiru Xie, Yi Zeng and Ying Zhang. 2026. EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.751

@inproceedings{sun2026edu,
  title = {{EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions}},
  author = {Weiyu Sun, Liangliang Chen, Yongnuo Cai, Huiru Xie, Yi Zeng and Ying Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.751},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.751}
}

References

29 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sami Baral, Li Lucy, Ryan Knight, Alice Ng, Luca Soldaini, Neil Heffernan, and Kyle Lo. 2025. Drawedumath: Evaluating vision language models with expert-annotated students’ hand-drawn math images. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computationa… find
  2. Adriana Caraeni, Alexander Scarlatos, and Andrew Lan. 2024. Evaluating gpt-4 at grading handwritten solutions in math exams. arXiv preprint arXiv:2411.05231. — [arXiv:2411.05231](https://arxiv.org/abs/2411.05231) find
  3. Liangliang Chen, Zhihao Qin, Yiming Guo, Jacqueline Rohde, and Ying Zhang. 2025a. Benchmarking large language models on homework assessment in circuit analysis. International Journal of Artificial Intelligence in Education, pages 1–62. find
  4. Liangliang Chen, Weiyu Sun, Huiru Xie, Yongnuo Cai, and Ying Zhang. 2025b. Enhancing large language model-based systems for end-to-end circuit analysis problem solving. arXiv preprint arXiv:2512.10159. — [arXiv:2512.10159](https://arxiv.org/abs/2512.10159) find
  5. Liangliang Chen, Huiru Xie, Zhihao Qin, Yiming Guo, Jacqueline Rohde, and Ying Zhang. 2025c. Enhancing large language models for automated homework assessment in undergraduate circuit analysis. In 2025 IEEE Frontiers in Education Conference (FIE), pages 1–9. find
  6. Liangliang Chen, Huiru Xie, Jacqueline Rohde, and Ying Zhang. 2025d. Wip: Large language model-enhanced smart tutor for undergraduate circuit analysis. In 2025 IEEE Frontiers in Education Conference (FIE), pages 1–5. IEEE. find
  7. Zui Chen, Jiaqi Han, Chaofan Yang, and Yi Zhou. 2023. Language model is suitable for correction of handwritten mathematical expressions recognition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4057–4068. find
  8. Philippe Gervais, Anastasiia Fadeeva, and Andrii Maksai. 2025. Mathwriting: A dataset for handwritten mathematical expression recognition. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, page 5459–5469, New York, NY, USA. Association for Computin… find
  9. Harsh Gupta, Luciano Del Corro, Samuel Broscheit, Johannes Hoffart, and Eliot Brenner. 2021. Unsupervised multi-view post-ocr error correction with language models. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 8647–8652. find
  10. Thomas M. Haladyna and Steven M. Downing. 2004. Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1):17–27. find
  11. Karez Hamad and Mehmet Kaya. 2016. A detailed analysis of optical character recognition technology. International Journal of Applied Mathematics Electronics and Computers, pages 244–249. find
  12. Ruslan Khrulev. 2025. Check-mat: Checking handwritten mathematical answers for the russian unified state exam. arXiv preprint arXiv:2507.22958. — [arXiv:2507.22958](https://arxiv.org/abs/2507.22958) find
  13. Gerd Kortemeyer, Julian Nöhl, and Daria Onishchuk. 2024. Grading assistance for a handwritten thermodynamics exam using artificial intelligence: An exploratory study. Physical Review Physics Education Research, 20(2):020144. find
  14. Tianyi Liu, Julia Chatain, Laura Kobel-Keller, Gerd Kortemeyer, Thomas Willwacher, and Mrinmaya Sachan. 2024. Ai-assisted automated short answer grading of handwritten university level mathematics exams. arXiv preprint arXiv:2408.11728. — [arXiv:2408.11728](https://arxiv.org/abs/2408.11728) find
  15. H. Mouchère, C. Viard-Gaudin, R. Zanibbi, and U. Garain. 2014. find
  16. Icfhr 2014 competition on recognition of on-line handwritten mathematical expressions (crohme 2014). In 2014 14th International Conference on Frontiers in Handwriting Recognition, pages 791–796. find
  17. Oikantik Nath, Hanani Bathina, Mohammed Safi Ur Rahman Khan, and Mitesh M Khapra. 2025. Can vision-language models evaluate handwritten math? arXiv preprint arXiv:2501.07244. find
  18. Behnam Parsaeifard, Martin Hlosta, and Per Bergamin. 2025. Automated grading of students’ handwritten graphs: A comparison of meta-learning and vision-large language models. arXiv preprint arXiv:2507.03056. — [arXiv:2507.03056](https://arxiv.org/abs/2507.03056) find
  19. Changjie Wu, Jun Du, Yunqing Li, Jianshu Zhang, Chen Yang, Bo Ren, and Yiqing Hu. 2022. Tdv2: A novel tree-structured decoder for offline mathematical expression recognition. Proceedings of the AAAI Conference on Artificial Intelligence, page 2694–2702. find
  20. Yejing Xie, Harold Mouchère, Foteini Simistira Liwicki, Sumit Rakesh, Rajkumar Saini, Masaki Nakagawa, Cuong Tuan Nguyen, and Thanh-Nghia Truong. 2023. find
  21. Icdar 2023 crohme: Competition on recognition of handwritten mathematical expressions. In Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San José, CA, USA, August 21–26, 2023, Proceedings, Part II, page 553–565, Berlin, Heidelberg. Springer-Verlag. find
  22. Haohang Xu, Chengjie Liu, Qihang Wang, Wenhao Huang, Yongjian Xu, Weiyu Chen, Anlan Peng, Zhijun Li, Bo Li, Lei Qi, and 1 others. 2025. Image2net: datasets, benchmark and hybrid framework to convert analog circuit diagrams into netlists. In 2025 International Symposium of Electronics Design Automati… find
  23. Yoonseok Yang, Minjune Kim, Marlon Rondinelli, and Keren Shao. 2025. Pensieve grader: An ai-powered, ready-to-use platform for effortless handwritten stem grading. arXiv preprint arXiv:2507.01431. — [arXiv:2507.01431](https://arxiv.org/abs/2507.01431) find
  24. John Pavlopoulos, Vasiliki Kougia, Paraskevi Platanou, Holger Essler, and 1 others. 2023. Detecting erroneous handwritten byzantine text recognition. In Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2023). Association for Computational Linguistics. find
  25. Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. 2021. Mathbert: A pre-trained model for mathematical formula understanding. arXiv preprint arXiv:2105.00377. — [arXiv:2105.00377](https://arxiv.org/abs/2105.00377) find
  26. Rohith Reddy Rachala and Mahesh Raveendranatha Panicker. 2022. Hand-drawn electrical circuit recognition using object detection and node recognition. SN Computer Science, 3(3):244. find
  27. Lejla Skelic, Yan Xu, Matthew Cox, Wenjie Lu, Tao Yu, and Ruonan Han. 2025. Circuit: A benchmark for circuit interpretation and reasoning capabilities of llms. arXiv preprint arXiv:2502.07980. — [arXiv:2502.07980](https://arxiv.org/abs/2502.07980) find
  28. Weiyu Sun, Jacqueline Rohde, Liangliang Chen, Yiming Guo, and Ying Zhang. 2025. Data-driven insights into academic success: Analyzing ten years of student academic records in an electrical and computer engineering department. In 2025 ASEE Annual Conference & Exposition. find
  29. James A Svoboda and Richard C Dorf. 2013. Introduction to Electric Circuits (9th Edition). John Wiley & Sons. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-edu-circuit-hw-evaluating-multimodal/