Elamur Research⌕
☰
home / papers / 2026 / EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios

EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios

Paper · 2026 · Bin Xu, Yu Bai, Huashan Sun et al.

Summary

Most LLM benchmarks for education test subject knowledge. They ignore the variety of roles in real learning settings, such as tutors, graders and counselors, and the pedagogical goals behind them. EduBench addresses this with a bilingual (Chinese/English) synthetic benchmark. It has nine scenarios: five serve students (e.g., error correction, emotional support) and four serve teachers (e.g., automatic grading, question generation). Each scenario is varied by subject, school level, difficulty and question type. GPT-4o generated the queries from human-written seed scenarios. Their quality was checked against human ratings on a sample. Model answers are scored with a 12-dimension rubric grouped under Scenario Adaptation, Factual & Reasoning Accuracy, and Pedagogical Application. The authors compared several LLM judges with human annotators, and DeepSeek V3 matched human judgments most closely. They then distilled the best-performing model for each scenario into a 7B model, which reached scores near the 671B DeepSeek V3. This shows that compact education-focused models can be practical.

Key points

  • Dataset size: 18,821 instances (9,141 Chinese, 9,680 English) spread across 4,019 educational contexts in 9 scenarios.
  • Rubric: 3 main aspects, each split into 4 sub-criteria, for 12 dimensions in total.
  • Query quality check: about 500 sampled queries were scored by both humans and LLMs, with agreement measured by MAE.
  • Judge alignment: tested on 198 items; DeepSeek V3 aligned best with human annotators, while GPT-4o was weaker.
  • Distillation: multi-source distillation with roughly 17,000 training samples let a 7B model approach DeepSeek V3 and Qwen Max on the test set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Bin Xu, Yu Bai, Huashan Sun, Yiguan Lin, Siming Liu, Xinyue Liang, Yaolin Li, Zhuangzhi Dong, Jingren Zhang, Yufan Deng, Xinyu Zou, Yang Gao and Heyan Huang. 2026. EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.987

@inproceedings{xu2026edubench,
  title = {{EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios}},
  author = {Bin Xu, Yu Bai, Huashan Sun, Yiguan Lin, Siming Liu, Xinyue Liang, Yaolin Li, Zhuangzhi Dong, Jingren Zhang, Yufan Deng, Xinyu Zou, Yang Gao and Heyan Huang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.987},
  url = {https://doi.org/10.18653/v1/2026.acl-long.987}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  2. Jonathan Berman, Lise McCoy, and Troy Camarata. 2024. LLM-Generated Multiple Choice Practice Quizzes for Pre-Clinical Medical Students; Use and Validity. Physiology, 39(S1):376. find
  3. Yue Cao, Hanqi Jin, Xiaojun Wan, and Zhiwei Yu. 2020. Domain-Adaptive Neural Automated Essay Scoring. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1011–1020, Virtual Event China. ACM. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  5. Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. Large Language Models in Education: Vision and Opportunities. arXiv preprint. Issue: arXiv:2311.13160 arXiv:2311.13160 [cs]. — [arXiv:2311.13160](https://arxiv.org/abs/2311.13160) find
  6. Bilal Ghanem, Lauren Lutz Coleman, Julia Rivard Dexter, Spencer Von Der Ohe, and Alona Fyshe. 2022. Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2131–2146, Dublin, Ireland. Asso… find
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Chieh-Yang Huang, Jing Wei, and Ting-Hao Kenneth Huang. 2024a. Generating Educational Materials with Different Levels of Readability using LLMs. In Proceedings of the Third Workshop on Intelligent and Interactive Writing Assistants, pages 16–22, Honolulu HI USA. ACM. find
  9. Zhenya Huang, Yuting Ning, Longhu Qin, Shiwei Tong, Shangzi Xue, Tong Xiao, Xin Lin, Jiayu Liu, Qi Liu, Enhong Chen, and Shijing Wang. 2024b. EduNLP: Towards a Unified and Modularized Library for Educational Resources. arXiv preprint. Issue: arXiv:2406.01276 arXiv:2406.01276 [cs]. — [arXiv:2406.01276](https://arxiv.org/abs/2406.01276) find
  10. Xin Jia, Wenjie Zhou, Xu Sun, and Yunfang Wu. 2021. EQG-RACE: Examination-Type Question Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):13143–13151. find
  11. Zhuoxuan Jiang, Haoyuan Peng, Shanshan Feng, Fan Li, and Dongsheng Li. 2024. LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought. arXiv preprint. find
  12. Charles Koutcheme, Nicola Dainese, and Arto Hellas. 2024a. Using Program Repair as a Proxy for Language Models’ Feedback Ability in Programming Education. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 165–181, Mexico City, Mexico… find
  13. Charles Koutcheme, Nicola Dainese, Sami Sarsa, Juho Leinonen, Arto Hellas, and Paul Denny. 2024b. Benchmarking Educational Program Repair. arXiv preprint. Issue: arXiv:2405.05347 arXiv:2405.05347 [cs]. — [arXiv:2405.05347](https://arxiv.org/abs/2405.05347) find
  14. Jiawei Li, Yang Gao, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, and 1 others. 2025. Fundamental capabilities and applications of large language models: A survey. ACM Computing Surveys.
  15. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  16. Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495.
  17. Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Sci… find
  18. Julia M. Markel, Steven G. Opferman, James A. Landay, and Chris Piech. 2023. GPTeach: Interactive TA Training with GPT-based Students. In Proceedings of the Tenth ACM Conference on Learning @ Scale, pages 226–236, Copenhagen Denmark. ACM. find
  19. Chee Ng and Yuen Fung. 2024. Educational Personalized Learning Path Planning with Large Language Models. arXiv preprint. Issue: arXiv:2407.11773 arXiv:2407.11773 [cs]. — [arXiv:2407.11773](https://arxiv.org/abs/2407.11773) find
  20. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  21. Zachary A. Pardos and Shreya Bhandari. 2023. Learning gain differences between ChatGPT and human tutor generated algebra hints. arXiv preprint. find
  22. Team Qwen. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  23. Team Qwen. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  24. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv preprint. Issue: arXiv:2311.12022 arXiv:2311.12022 [cs].
  25. Hamdireza Rouzegar and Masoud Makrehchi. 2024. Generative AI for Enhancing Active Learning in Education: A Comparative Study of GPT-3.5 and GPT-4 in Crafting Customized Test Questions. find
  26. Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, and 1 others. 2025. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837.
  27. Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569… find
  28. Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianxiao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, Lei Hou, and Juanzi Li. 2025. Simulating Classroom Education with LLM-Empowered Agents. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of…
  29. Buyuan Zhu, Shiyu Hu, Yiping Ma, Yuanming Zhang, and Kang Hao Cheong. 2025. Edupersona: Benchmarking subjective ability boundaries of virtual student agents. CoRR, abs/2510.04648. — [arXiv:2510.04648](https://arxiv.org/abs/2510.04648) find
  30. Yao Shi, Rongkeng Liang, and Yong Xu. 2025. Educationq: Evaluating llms’ teaching capabilities through multi-agent dialogue framework. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 20…
  31. M.-A.-P. Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, and 78 others. 2025. SuperGPQA: Scaling LLM Evaluation across 28…
  32. Ruiyi Wang, Stephanie Milani, Jamie C Chiu, Jiayin Zhi, Shaun M Eack, Travis Labrum, Samuel M Murphy, Nev Jones, Kate Hardy, Hong Shen, and 1 others. 2024a. Patient-{\\Psi}: Using large language models to simulate patients for training mental health professionals. arXiv preprint arXiv:2405.19660. find
  33. Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. 2024b. Large Language Models for Education: A Survey and Outlook. arXiv preprint. Issue: arXiv:2403.18105 arXiv:2403.18105 [cs] version: 1.
  34. Tianyu Wang, Nianjun Zhou, and Zhixiong Chen. 2024c. Enhancing Computer Programming Education with LLMs: A Study on Effective Prompt Engineering for Python Code Generation. arXiv preprint. find
  35. Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S. Bernstein, and John Mitchell. 2024. Social Skill Training with Large Language Models. arXiv preprint. Issue: arXiv:2404.04204 arXiv:2404.04204 [cs]. find
  36. Author Contributions A.1 • Bin Xu: Conceived and designed the analysis; collected the data; performed the analysis; wrote the paper. • Yu Bai: Developed the idea; conceived and designed the analysis; manage the project; performed the analysis; wrote the paper. • Huashan Sun: Conceived and designed t… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-edubench-comprehensive-benchmarking-dataset-evaluating/