Elamur Research⌕
☰
home / papers / 2026 / SEFL: A Framework for Generating Synthetic Educational Assignment Feedback with LLM Agents

SEFL: A Framework for Generating Synthetic Educational Assignment Feedback with LLM Agents

Summary

Giving students good feedback on assignments takes a lot of time and money. Collecting real student work and teacher comments to train automated systems also raises privacy and consent issues. This paper presents SEFL (Synthetic Educational Feedback Loops), which avoids real student data entirely. Two Llama-3.1-70B agents act as teacher and student. The teacher writes an assignment based on a Fineweb-Edu text. The student answers it while deliberately making errors. The teacher then critiques each error and suggests how to fix it, keeping comments short on purpose. The resulting assignment–feedback pairs are used to fine-tune smaller, cheaper LLMs that universities could run on modest hardware. Three LLM judges and three human experts compared 900 outputs. They preferred the SEFL-tuned models over the same models without tuning and over the Book2Dial baseline. Larger tuned models gave more actionable and consistent feedback, and the human raters largely agreed with each other. The work suggests synthetic data can support scalable, privacy-friendly feedback tools.

Key points

  • The released dataset has 19,841 synthetic assignment–feedback pairs, and all models, code and data are open-sourced on GitHub and HuggingFace.
  • In a pilot of 5,000 generations per model, Llama-3.1-70B produced 2,513 valid outputs, compared with 454 for Qwen2.5-72B. "Valid" meant correct JSON in which each feedback point matched an error.
    • BERTScore between errors and feedback was high for both models: 0.877 for Llama-3.1-70B and 0.919 for Qwen2.5-72B.
    • Llama-3.1-70B was therefore chosen to generate the data.
  • Each agent's turn is capped at under 170 subword tokens, because students tend to prefer brief comments.
  • A manual check of 226 sampled instances found 428 intentional student errors, which the authors grouped into seven categories.
  • In evaluation, SEFL-tuned models won against their untuned versions and against Book2Dial, the latter according to all LLM judges.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3gqx9z5n3zsu

@inproceedings{zhang2026sefl,
  title = {{SEFL: A Framework for Generating Synthetic Educational Assignment Feedback with LLM Agents}},
  author = {Zhang, Mike and Dilling, Amalie Pernille and Gondelman, Léon and Lyngdorf, Niels Erik Ruan and Lindsay, Euan D. and Bjerva, Johannes},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3gqx9z5n3zsu},
  url = {https://doi.org/10.63317/3gqx9z5n3zsu}
}

References

63 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laila Alrajhi, Ahmed Alamri, Filipe Dwan Pereira, and Alexandra I Cristea. 2021. Urgency analysis of learners’ comments: An automated intervention priority model for mooc. In Intelligent Tutoring Systems: 17th International Conference, ITS 2021, Virtual Event, June 7–11, 2021, Proceedings 17, pages… find
  2. Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E Mete, Eda Okur, Sidney K D’Mello, and Asli Arslan Esme. 2019. Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. In Proceedings of the 2019 chi conference on human factors in computing… find
  3. AtlaAI. 2025. Judge arena. https://huggingface.co/spaces/AtlaAI/ judge-arena. [Online; accessed 8-April-2025]. ↗
  4. Elisabeth Bauer, Martin Greisel, Ilia Kuznetsov, Markus Berndt, Ingo Kollar, Markus Dresel, Martin R Fischer, and Frank Fischer. 2023. Using natural language processing to support peer-feedback in the age of artificial intelligence: A cross-disciplinary framework and a research agenda. British Journ… find
  5. Anthony Botelho, Sami Baral, John A Erickson, Priyanka Benachamardi, and Neil T Heffernan. 2023. Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics. Journal of computer assisted learning, 39(3):823–840. find
  6. David Carless, Diane Salter, Min Yang, and Joy Lam. 2011. Developing sustainable feedback practices. Studies in higher education, 36(4):395–407. find
  7. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 361–374, Nusa Dua, Bali. Ass… find
  8. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  9. Grainne Conole and Martin Oliver. 2006. Contemporary perspectives in e-learning research. Routledge London. find
  10. Jane Costello and Daph Crane. 2013. Technologies for learner-centered feedback. Open Praxis, 5(3):217–225. find
  11. Dorottya Demszky and Heather Hill. 2023. The NCTE transcripts: A dataset of elementary math classroom transcripts. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 528–538, Toronto, Canada. Association for Computational Linguistics. find
  12. Jeanette Falk, Yiyi Chen, Janet Rafner, Mike Zhang, Johannes Bjerva, and Alexander Nolte. 2025. How do hackathons foster creativity? towards ai collaborative evaluation of creativity at scale. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25. Association for C… find
  13. Peter Ferguson. 2011. Student perceptions of quality feedback in teacher education. Assessment & evaluation in higher education, 36(1):51–62. find
  14. Christian Fischer, Zachary A Pardos, Ryan Shaun Baker, Joseph Jay Williams, Padhraic Smyth, Renzhe Yu, Stefan Slater, Rachel Baker, and Mark Warschauer. 2020. Mining big data in education: Affordances and challenges. Review of Research in Education, 44(1):130–160. find
  15. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. find our card
  16. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. A survey on llm-as-a-judge. find
  17. Camelia Guerraoui, Paul Reisert, Naoya Inoue, Farjana Sultana Mim, Keshav Singh, Jungmin Choi, Irfan Robbani, Shoichi Naito, Wenzhi Wang, and Kentaro Inui. 2023. Teach me how to argue: A survey on NLP feedback systems in argumentation. In Proceedings of the 10th Workshop on Argument Mining, pages 19… find
  18. John Hattie. 2008. Visible learning: A synthesis of over 800 meta-analyses relating to achievement. routledge. find
  19. Fan Huang, Haewoon Kwak, Kunwoo Park, and Jisun An. 2024. ChatGPT rates natural language explanation quality like humans: But on which scales? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3111–3132… find
  20. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. find our card
  21. Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. In IJCAI, volume 19, pages 6300–6308. find
  22. Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. 2022. Generating information-seeking conversations from unlabeled documents. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2362–2378, Abu Dhabi, United Arab Emirates. Association for Computat… find
  23. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535. find our card
  24. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. find
  25. Soonwoo Kwon, Sojung Kim, Minju Park, Seunghyun Lee, and Kyuseok Kim. 2024. Biped: Pedagogically informed tutoring system for esl education. arXiv preprint arXiv:2406.03486. find
  26. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. find
  27. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  28. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991– 52008. find our card
  29. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. 2024. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196. find
  30. Euan D Lindsay, Mike Zhang, Aditya Johri, and Johannes Bjerva. 2024. The responsible development of automated student feedback with generative ai. find
  31. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  32. Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. 2024b. Socraticlm: exploring socratic personalized teaching with large language models. Advances in Neural Information Processing Systems, 37:85693–85721. find
  33. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin… find
  34. Zhengyuan Liu, Stella Xin Yin, Geyu Lin, and Nancy F Chen. 2024c. Personality-aware student simulation for conversational intelligent tutoring systems. arXiv preprint arXiv:2404.06762. find
  35. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  36. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. Fineweb-edu. find
  37. Inderjeet Jayakumar Nair, Jiaye Tan, Xiaotian Su, Anne Gere, Xu Wang, and Lu Wang. 2024. Closing the loop: Learning to generate writing feedback via language model simulated student revisions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16636–1665… find
  38. Ben Naismith, Phoebe Mulcaire, and Jill Burstein. 2023. Automated evaluation of written discourse coherence using GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394–403, Toronto, Canada. Association for Computational Lingui… find our card
  39. David Nicol. 2007. E-assessment by design: using multiple-choice tests to good effect. Journal of Further and higher Education, 31(1):53–64. find
  40. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741. find
  41. Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55(3):2495–2527. find
  42. Donya Rooein and Dirk Hovy. 2024. Conversations as a source for teaching scientific concepts at different education levels. arXiv preprint arXiv:2404.10475. — [arXiv:2404.10475](https://arxiv.org/abs/2404.10475) find
  43. Alexis Ross and Jacob Andreas. 2024. Toward in-context teaching: Adapting examples to students’ misconceptions. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13283–13310, Bangkok, Thailand. Association for Computational Ling… find
  44. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076. — [arXiv:2310.10076](https://arxiv.org/abs/2310.10076) find
  45. Baruch B Schwarz, Naomi Prusak, Osama Swidan, Adva Livny, Kobi Gal, and Avi Segal. 2018. Orchestrating the emergence of conceptual learning: A case study in a geometry class. International Journal of Computer-Supported Collaborative Learning, 13:189–211. find
  46. Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard G Baraniuk. 2024. Pedagogical alignment of large language models. arXiv preprint arXiv:2402.05000. find
  47. Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024. Exploring LLM prompting strategies for joint essay scoring and feedback generation. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 283–298, Mexico City, Mexi… find
  48. Abhijit Suresh, Jennifer Jacobs, Charis Harty, Margaret Perkoff, James H. Martin, and Tamara Sumner. 2022. The TalkMoves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pag… find
  49. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. find
  50. Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588. find our card
  51. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796. find our card
  52. Junling Wang, Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, and Mrinmaya Sachan. 2024a. Book2Dial: Generating teacher student interactions from textbooks for cost-effective development of educational chatbots. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9707–97… find
  53. Rose Wang and Dorottya Demszky. 2024. Edu-ConvoKit: An open-source library for education conversation data. In Proceedings of the 2024 find
  54. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 61–69, Mexico City, Mexico. Association for Computational Linguistics. find
  55. Rose E Wang, Ana T Ribeiro, Carly D Robinson, Susanna Loeb, and Dora Demszky. 2024b. Tutor copilot: A human-ai approach for scaling real-time expertise. arXiv preprint arXiv:2410.03017. find
  56. Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S Yu, and Qingsong Wen. 2024c. Large language models for education: A survey and outlook. arXiv preprint arXiv:2403.18105. find
  57. Grant Wiggins. 2012. Seven keys to effective feedback. Feedback, 70(1):10–16. find
  58. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155. find our card
  59. Joy Yun, Yann Hicke, Mariah Olson, and Dorottya Demszky. 2024. Enhancing tutoring effectiveness through automated feedback: Preliminary findings from a pilot randomized controlled trial on sat tutoring. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 422–426. find
  60. Mike Zhang, Euan Lindsay, Maj-Britt Quitzau, and Johannes Bjerva. 2025. Scaling course evaluations with large language models: Semester-level digestible student feedback for program leaders. In Proceedings of the 53rd Annual Conference of the European Society for Engineering Education (SEFI 2025). find
  61. Mike Zhang, Euan Lindsay, Frederik Bode Thor-bensen, Danny Bøgsted Poulsen, and Johannes Bjerva. 2024. Leveraging large language models for actionable course evaluation student feedback to lecturers. In Proceedings of the 52nd Annual Conference of the European Society for Engineering Education (SEFI… find
  62. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  63. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-sefl-framework-generating-synthetic-educational/