Elamur Research⌕
☰
home / papers / 2025 / EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework

EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework

Paper · 2025 · Yao Shi, Rongkeng Liang and Yong Xu

Summary

LLMs are increasingly used as tutors, but standard benchmarks measure what a model knows rather than how well it teaches. The authors propose EducationQ, a multi-agent dialogue framework built on formative assessment and Vygotsky's learning theory. A teacher agent (the model under test) conducts a multi-round tutoring dialogue with a fixed simulated student agent. An evaluator agent then analyses the dialogue. Teaching quality is measured by the student's improvement between a pre-test and a post-test.

The test set has 1,498 questions taken from GPQA and MMLU-Pro, covering 13 disciplines and 10 difficulty levels. The authors evaluated 14 LLMs from OpenAI, Meta, Google, Anthropic and other developers. Teaching effectiveness did not track model size or general benchmark scores, and some smaller open models outperformed larger commercial ones. Human experts agreed with the automated qualitative analysis in most cases.

The work argues that teaching-oriented LLMs need targeted pedagogical optimization rather than more scale. It also supplies a reusable, scalable evaluation protocol for this purpose.

Key points

  • Dataset: GPQA Diamond (198 questions) plus a newly built MMLU-Pro Stratified subset (1,300 questions). The subset samples 10 questions per subject–difficulty cell, with difficulty set by the accuracy of the top 10 published models. Llama 3.1 70B Instruct as the student reached 47.73% baseline accuracy.
  • Released dialogues: 14,980 five-round teacher–student dialogues, re-annotated and paired with pre-test and post-test results.
  • Top teachers:
    • Llama 3.1 70B Instruct improved student accuracy by 11.01% overall and by up to 24% in single subjects, relying on strong questioning strategies.
    • Gemini 1.5 Pro 002 reached 7.48%, mainly through targeted feedback.
    • OpenAI o1-mini performed best in reasoning-heavy subjects.
  • Human validation: Experts agreed with the evaluator agent in 78% of cases (39/50 dialogue pairs). They rated gain-producing teachers higher (7.38 vs 6.41 out of 10) and found no cases of teachers revealing answers.
  • Limitations:
    • The setup is limited to one-on-one tutoring with a single fixed student model.
    • Only advanced content was tested, with no school-level material.
    • OpenAI models returned empty responses on a Vietnam War question, likely because of content moderation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

Cite

Cite as Yao Shi, Rongkeng Liang and Yong Xu. 2025. EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1576

@inproceedings{shi2025educationq,
  title = {{EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework}},
  author = {Yao Shi, Rongkeng Liang and Yong Xu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1576},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1576}
}

References

48 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Linda Allal and Greta Pelgrims Ducrey. 2000. Assessment of—or in—the zone of proximal development. Learning and Instruction, 10(2):137–152. find
  2. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  3. Yuyan Chen, Chenwei Wu, Songzhou Yan, Panjun Liu, Haoyu Zhou, and Yanghua Xiao. 2024. Dr.academy: A benchmark for evaluating questioning capability in education for large language models. Preprint, arXiv:2408.10947. find
  4. Clayton Cohn, Nicole Hutchins, Tuan Le, and Gautam Biswas. 2024. A chain-of-thought prompting approach with llms for evaluating students’ formative assessment responses in science. Proceedings of the AAAI Conference on Artificial Intelligence, 38(21):23182–23190. find
  5. Richard A. Duschl and Drew H. Gitomer. 1997. Strategies and challenges to changing the focus of assessment and instruction in science classrooms. Educational Assessment, 4(1):37–73. find
  6. Drew H. Gitomer and Richard A. Duschl. 2007. chapter 12 establishing multilevel coherence in assessment. Yearbook of the National Society for the Study of Education, 106(1):288–320. find
  7. Thomas R Guskey. 2005. Formative classroom assessment and benjamin s. bloom: Theory, research, and implications. Online submission. find
  8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  9. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.
  10. Cindy E Hmelo-Silver and Howard S Barrows. 2006. Goals and strategies of a problem-based learning facilitator. Interdisciplinary Journal of Problem-Based Learning, 1(1):4. find
  11. Hyoungwook Jin, Minju Yoo, Jeongeon Park, Yokyung Lee, Xu Wang, and Juho Kim. 2025. Teachtune: Reviewing pedagogical agents against diverse student profiles with simulated students. Preprint, arXiv:2410.04078. — [arXiv:2410.04078](https://arxiv.org/abs/2410.04078) find
  12. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
  13. David R. Krathwohl. 2002. A revision of bloom’s taxonomy: An overview. Theory Into Practice, 41(4):212– 218. find
  14. Zhenwen Liang, Dian Yu, Wenhao Yu, Wenlin Yao, Zhihan Zhang, Xiangliang Zhang, and Dong Yu. 2024. Mathchat: Benchmarking mathematical reasoning and instruction following in multi-turn interactions. Preprint, arXiv:2405.19444. find
  15. Jiawen Liu, Yuanyuan Yao, Pengcheng An, and Qi Wang. 2024. Peergpt: Probing the roles of llm-based peer agents as team moderators and participants in children’s collaborative learning. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’24, New York, NY, USA. A… find
  16. Janet Looney. 2011. Integrating formative and summative assessment: Progress toward a seamless system? Technical Report 58, OECD Publishing, Paris. find
  17. Xinyi Lu and Xu Wang. 2024. Generative students: Using llm-simulated student profiles to support question item evaluation. In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, page 16–27, New York, NY, USA. Association for Computing Machinery. find
  18. Julia M. Markel, Steven G. Opferman, James A. Landay, and Chris Piech. 2023. Gpteach: Interactive ta training with gpt-based students. In Proceedings of the Tenth ACM Conference on Learning @ Scale, L@S ’23, page 226–236, New York, NY, USA. Association for Computing Machinery. find
  19. Cecile Hoareau McGrath, Benoit Guerin, Emma Harte, Michael Frearson, and Catriona Manville. 2015. Learning gain in higher education. RAND Corporation, Santa Monica, CA. find
  20. Bahar Memarian and Tenzin Doleck. 2023. Chatgpt in education: Methods, potentials, and limitations. Computers in Human Behavior: Artificial Humans, 1(2):100022. find
  21. Neil Mercer and Karen Littleton. 2007. Dialogue and the Development of Children’s Thinking: A Sociocultural Approach, 1st edition. Routledge, London. find
  22. Fengchun Miao, Wayne Holmes, Ronghuai Huang, Hui Zhang, et al. 2021. AI and education: A guidance for policymakers. UNESCO Publishing. find
  23. Benjamin D Nye, Dillon Mee, and Mark G Core. 2023. Generative large language models for dialog-based tutoring: An early consideration of opportunities and concerns. In LLM@AIED, pages 78–88. find
  24. Andrew Olney. 2023. Generating multiple choice questions from a textbook: Llms match human performance on most metrics. In Workshop on Empowering Education with LLMs - the Next-Gen Interface and Content Generation at the AIED’23 Conference. find
  25. A. Sullivan Palincsar. 1998. Social constructivist perspectives on teaching and learning. Annual Review of Psychology, 49(Volume 49, 1998):345–375. find
  26. Abelardo Pardo, Jelena Jovanovic, Shane Dawson, Dragan Gašević, and Negin Mirriahi. 2019. Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1):128–138. find
  27. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA. Ass… find
  28. Rajlaxmi Patil, Aditya Ashutosh Kulkarni, Ruturaj Ghatage, Sharvi Endait, Geetanjali Kale, and Raviraj Joshi. 2024. Automated assessment of multimodal answer sheets in the stem domain. Preprint, arXiv:2409.15749. — [arXiv:2409.15749](https://arxiv.org/abs/2409.15749) find
  29. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. Preprint, arXiv:2311.12022.
  30. Maria Araceli Ruiz-Primo and Erin Marie Furtak. 2007. Exploring teachers’ informal formative assessment practices and students’ understanding in the context of scientific inquiry. Journal of Research in Science Teaching, 44(1):57–84. find
  31. Asli Sezen-Barrie and Gregory J. Kelly. 2017. From the teacher’s eyes: facilitating teachers noticings on informal formative assessments (ifas) and exploring the challenges to effective implementation. International Journal of Science Education, 39(2):181–212. find
  32. Kumar Shridhar, Jakub Macina, Mennatallah El-Assady, Tanmay Sinha, Manu Kapur, and Mrinmaya Sachan. 2022. Automatic generation of socratic subquestions for teaching math word problems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4136–4149, Abu Dha… find
  33. U.S. Department of Education, Office of Educational Technology. 2023. Artificial intelligence and future of teaching and learning: Insights and recommendations. Technical report, U.S. Department of Education, Washington, DC. find
  34. L. S. Vygotsky. 1978. Mind in Society: Development of Higher Psychological Processes. Harvard University Press, Cambridge, MA. find
  35. Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. 2024a. Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation. Preprint, arXiv:2402.11443.
  36. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  37. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… find
  38. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. Mmlu-pro: A more robust and challenging multi-task language understanding benchmar…
  39. Rob Wass and Clinton Golding and. 2014. Sharpening a tool for teaching: the zone of proximal development. Teaching in Higher Education, 19(6):671–684. find
  40. J. Wei, M. Bosma, V. Zhao, K. Guu, A.W. Yu, B. Lester, N. Du, A.M. Dai, and Q.V. Le. 2022. Finetuned language models are zero-shot learners. Preprint, arXiv:2109.01652.
  41. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. Large language models are better reasoners with self-verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2550–2575, Singapore. Association for Computat…
  42. William W Wilen. 1987. Questions, questioning techniques, and effective teaching. National Education Association, Washington, DC. find
  43. Dylan William. 2011. What is assessment for learning? Studies in Educational Evaluation, 37(1):3–14. find
  44. David Wood, Jerome S. Bruner, and Gail Ross. 1976. The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2):89–100. find
  45. Songlin Xu and Xinyu Zhang. 2023. Leveraging generative artificial intelligence to simulate student learning behavior. Preprint, arXiv:2310.19206. — [arXiv:2310.19206](https://arxiv.org/abs/2310.19206) find
  46. Siyu Zha, Yujia Liu, Chengbo Zheng, Jiaqi XU, Fuze Yu, Jiangtao Gong, and Yingqing XU. 2024. Mentigo: An intelligent agent for mentoring students in the creative problem solving process. Preprint, arXiv:2409.14228. — [arXiv:2409.14228](https://arxiv.org/abs/2409.14228) find
  47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.
  48. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. Preprint, arXiv:2311.07911.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2025-educationq-evaluating-llms-teaching-capabilities/