Elamur Research⌕
☰
home / papers / 2026 / The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance

The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance

Paper · 2026 · Tsvetomila Mihaylova, Evanfiya Logacheva, Arto Hellas et al.

Summary

Novice programmers often struggle with compiler errors and failed tests, and instructor feedback does not scale to large courses. This study asks whether the structure of LLM-generated hints changes how students work toward a correct solution. Over two months, an online introductory Web Development course randomly assigned one of three GPT-4o feedback formats to each student–exercise pair. A control group saw only error output. The formats were General (up to three issues, each with explanation and next-step hints), Single-Issue (one problem with guidance), and Diagnostic (up to three issues described, with no fix advice). The authors used mixed-effects models with student and assignment random effects to examine time to solution and number of attempts. They also compared students by self-reported experience and course content by whether it came before or after the model's knowledge cutoff. All LLM feedback types were linked to faster solutions than the baseline, and the lighter, less guided formats were slightly more efficient. Variation between individual students outweighed differences by experience level. Guidance-heavy hints were less consistent on newer material. These findings support adaptive, content-grounded feedback design.

Key points

  • The dataset has 3153 solved assignment–user pairs: 1228 No Feedback, 665 General, 639 Single-Issue and 621 Diagnostic. Students were split into 111 novices (under 3 years of experience) and 38 advanced students.
  • The authors applied several filters to the data:
    • only consenting students were included;
    • pairs with any submission over 2000 characters were excluded;
    • only sequences solved within 30 minutes were kept, covering roughly 95% of solved tasks.
  • Course Part 2 (Client-Side, "New Content") was released after GPT-4o's training cutoff, while Part 3 (Server-Side, "Old Content") was not. Hints on the new material were of lower quality.
  • The hint designs draw on cognitive load concerns and the expertise reversal effect. Fewer issues or less guidance produced more efficient short-term problem solving.
  • Students could rate each message as Good or Poor, but votes were sparse. Messages rated Poor revealed hallucinations. The authors propose retrieval of course materials, including compiler errors in prompts, and automated fact-checking as remedies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🎓New York UniversityNew York · United States201 works · 100,180 citations🎓University of PittsburghPittsburgh · United States59 works · 22,324 citations🎓Aalto UniversityEspoo · Finland13 works · 648 citations🎓North Carolina State UniversityRaleigh · United States13 works · 406 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Science Applications · 102 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Tsvetomila Mihaylova, Evanfiya Logacheva, Arto Hellas, Jing Fan, Francisco Castro, Bita Akram, Narges Norouzi, Peter Brusilovsky and Juho Leinonen. 2026. The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.42

@inproceedings{mihaylova2026effects,
  title = {{The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance}},
  author = {Tsvetomila Mihaylova, Evanfiya Logacheva, Arto Hellas, Jing Fan, Francisco Castro, Bita Akram, Narges Norouzi, Peter Brusilovsky and Juho Leinonen},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.42},
  url = {https://doi.org/10.18653/v1/2026.bea-1.42}
}

References

20 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Juliana Barros, Laura O. Moraes, Fernanda Oliveira, and Carla A. D. M. Delgado. 2025. Large language models generating feedback for students of introductory programming courses. In 26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 2, volume 15877 of Lecture Notes… find
  2. Brett A. Becker. 2016. An effective approach to enhancing compiler error messages. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE ’16, page 126–131, New York, NY, USA. Association for Computing Machinery. find
  3. Slava Kalyuga. 2007. Expertise reversal effect and its implications for learner-tailored instruction. Educational psychology review, 19(4):509–539. find
  4. Slava Kalyuga. 2009. The expertise reversal effect. In Managing cognitive load in adaptive multimedia learning, pages 58–80. IGI Global Scientific Publishing. find
  5. Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises. ACM Trans. Comput. Educ., 19(1). find
  6. Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, Syed Ashraf, and Paul Denny. 2025. Evaluating language models for generating and judging programming feedback. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, pages 624–630. find
  7. Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny. 2024. Open source language models can provide feedback: Evaluating llms’ ability to help students using gpt-4-asa-judge. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V.… find
  8. Oka Kurniawan, Christopher M Poskitt, Ismam Al Hoque, Norman Tiong Seng Lee, Cyrille Jégourel, and Nachamma Sockalingam. 2023. How helpful do novice programmers find the feedback of an automated repair tool? In 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (… find
  9. Juho Leinonen, Paul Denny, and Jacqueline Whalley. 2022. A comparison of immediate and scheduled feedback in introductory programming projects. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1, SIGCSE 2022, page 885–891, New York, NY, USA. Association for C… find
  10. Skipper Seabold and Josef Perktold. 2010. statsmodels: Econometric and statistical modeling with python. In Proceedings of the 9th Python in Science Conference, pages 92–96. find
  11. Raj Shrestha, Juho Leinonen, Albina Zavgorodniaia, Arto Hellas, and John Edwards. 2022. Pausing while programming: Insights from keystroke analysis. in 2022 ieee/acm 44th international conference on software engineering: Software engineering education and training (icse-seet). find
  12. Alexandra Vassar, Jake Renzella, Emily Ross, and Andrew Taylor. 2024. Fine-tuning large language models for better programming error explanations. In Proceedings of the 24th Koli Calling International Conference on Computing Education Research, Koli Calling ’24, New York, NY, USA. Association for Co… find
  13. Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A Becker. 2023. Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pages 563–569. find
  14. Olga Viberg, Jacqueline Wong, Yael Feldman-Maggor, Nora Dunder, and Carrie Demmans Epp. 2025. Chatting with code: Exploring llms as learning partners in programming education. In 26th International Conference on Artificial Intelligence in Education, AIED 2025, Part 6, volume 15882 of Lecture Notes i… find
  15. Mary J Lindstrom and Douglas M Bates. 1988. Newton—raphson and em algorithms for linear mixed-effects models for repeated-measures data. Journal of the American Statistical Association, 83(404):1014–1022. find
  16. Ruiwei Xiao, Xinying Hou, and John Stamper. 2024. Exploring how multiple levels of gpt-generated programming hints support or disappoint novices. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–10. find
  17. Marcus Messer, Neil C. C. Brown, Michael Kölling, and Miaojing Shi. 2024. Automated grading and feedback tools for programming education: A systematic review. ACM Trans. Comput. Educ., 24(1). find
  18. A Moritz Mueller, Corinna List, and Michael Kipp. 2025. The power of context: An llm-based programming tutor with focused and proactive feedback. In Proceedings of the 6th European Conference on Software Engineering Education, ECSEE ’25, page 1–10, New York, NY, USA. Association for Computing Machin… find
  19. Maciej Pankiewicz and Ryan Baker. 2023. Large language models (gpt) for automating feedback on programming assignments. In 31st International Conference on Computers in Education (ICCE’2023), pages 68–77. find
  20. Tung Phung, Heeryung Choi, Mengyan Wu, Adish Singla, and Christopher Brooks. 2025. Plan more, Prompts and Examples of Feedback Messages Figure 4 shows the prompts used for the three feedback types. For choosing these prompts, we experimented with several different variants, and ended up with this fo… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mihaylova-2026-effects-structured-llm-generated-feedback/