Elamur Research⌕
☰
home / papers / 2022 / Using Large Language Models to Enhance Programming Error Messages

Using Large Language Models to Enhance Programming Error Messages

Paper · 2022 · Juho Leinonen, Arto Hellas, Sami Sarsa et al.

Summary

Novice programmers often struggle with error messages from compilers and interpreters because these messages assume prior expertise. Decades of manual rewording efforts have shown limited, labor-intensive gains. This study asks whether OpenAI Codex (model code-davinci-002) can automatically explain such errors and propose fixes.

The authors selected nine Python 3.6 error messages that earlier readability research had rated as especially confusing. For each message, they wrote three programs that trigger it, ranging from a trivial snippet to code using libraries such as PyGame, pandas and scikit-learn. After comparing five zero-shot prompts, they generated 81 outputs at temperatures 0 and 0.7. Two experienced instructors then judged every output on seven yes/no criteria.

The outputs were usually readable, but their accuracy was weak. Fewer than half of all inputs received a correct explanation, and only about a third received a correct fix. The authors conclude that LLM-enhanced messages show promise as a learning scaffold. However, they are not yet reliable enough for unsupervised classroom use, and the results serve as a baseline for future models.

Key points

  • Data: 9 confusing Python 3.6 error messages × 3 author-written programs of increasing complexity × 3 generations (one at temperature 0, two at 0.7) gave 81 Codex outputs.
  • Prompt selection: Five prompts were tested. The chosen one, a request for a plain-English explanation plus a fix, produced the fewest empty answers (4 of 81, versus up to 27 for the worst prompt).
  • Rating method: Two raters scored 162 ratings in total, with Cohen's kappa = 0.83.
  • Explanation results:
    • 88% of outputs were comprehensible.
    • 84% contained an explanation, but only 57% of those were correct (48% of all inputs).
  • Fix results and caveats:
    • 70% of outputs proposed a fix, of which 47% were correct (33% overall).
    • Temperature 0 produced better results.
    • Limitations include non-student code, single-error snippets and zero-shot prompting only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College DublinDublin · Ireland17 works · 1,888 citations🎓Aalto UniversityEspoo · Finland17 works · 1,208 citations🎓University of AucklandAuckland · New Zealand17 works · 1,404 citations

Also on the byline: Abilene Christian University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather and Brett A. Becker. 2022. Using Large Language Models to Enhance Programming Error Messages. arXiv:2210.11630.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3545945.3569770 · arXiv 2210.11630

@misc{leinonen2022using,
  title = {{Using Large Language Models to Enhance Programming Error Messages}},
  author = {Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather and Brett A. Becker},
  year = {2022},
  doi = {10.1145/3545945.3569770},
  eprint = {2210.11630},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3545945.3569770}
}

References

35 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Toufique Ahmed, Noah Rose Ledesma, and Premkumar Devanbu. 2021. SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnostics. arXiv preprint arXiv:2104.14671 (2021). — [arXiv:2104.14671](https://arxiv.org/abs/2104.14671) find
  2. Umair Z. Ahmed, Pawan Kumar, Amey Karkare, Purushottam Kar, Sumit Gulwani, and A.; et al Ahmed, U.; Kumar, P.; Karkare. 2018. Compilation Error Repair: For the Student Programs, From the Student Programs. In ICSE-SEET 2018 : 2018 ACM/IEEE 40th International Conference on Software Engineering : Softw… — [doi:10.1145/3183377.3183383](https://doi.org/10.1145/3183377.3183383) find
  3. Titus Barik. 2018. Error Messages as Rational Reconstructions. Ph. D. Dissertation. North Carolina State University, Raleigh. https://repository.lib.ncsu.edu/handle/1840.20/35439 ↗
  4. Titus Barik, Justin Smith, Kevin Lubick, Elisabeth Holmes, Jing Feng, Emerson Murphy-Hill, and Chris Parnin. 2017. Do Developers Read Compiler Error Messages?. In Proceedings of the 39th International Conference on Software Engineering (Buenos Aires, Argentina) (ICSE ’17). IEEE Press, Piscataway, NJ… — [doi:10.1109/icse.2017.59](https://doi.org/10.1109/icse.2017.59) find
  5. Brett A. Becker. 2021. What Does Saying That ‘Programming is Hard’ Really Say, and About Whom? Commun. ACM 64, 8 (jul 2021), 27–29. https://doi.org/10.1145/3469115 — [doi:10.1145/3469115](https://doi.org/10.1145/3469115) ↗
  6. Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, Janice L. Pearce, and James Prather. 2019. Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Mes… — [doi:10.1145/3344429.3372508](https://doi.org/10.1145/3344429.3372508) find
  7. Brett A. Becker, Paul Denny, James Prather, Raymond Pettit, Robert Nix, and Catherine Mooney. 2021. Towards Assessing the Readability of Programming Error Messages. In Australasian Computing Education Conference (Virtual, SA, Australia) (ACE ’21). Association for Computing Machinery, New York, NY,US… — [doi:10.1145/3441636.3442320](https://doi.org/10.1145/3441636.3442320) find
  8. Brett A. Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, and Catherine Mooney. 2016. Effective Compiler Error Message Enhancement for Novice Programming Students. Computer Science Education 26, 2-3 (2016), 148–175. https://doi.org/10.1080/08993408.2016.1225464 — [doi:10.1080/08993408.2016.1225464](https://doi.org/10.1080/08993408.2016.1225464) ↗
  9. Brett A. Becker, Kyle Goslin, and Graham Glanville. 2018. The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (Baltimore, Maryland, USA) (SIGCSE ’18). Association for Computing Machinery, N… — [doi:10.1145/3159450.3159461](https://doi.org/10.1145/3159450.3159461) find
  10. Brett A Becker and Catherine Mooney. 2016. Categorizing Compiler e Error Messages with Principal Component Analysis. In 12th China-Europe International Symposium on Software Engineering Education (CEISEE 2016), Shenyang, China, 28-29 May 2016. find
  11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-shot Learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021).
  13. Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing Syntax Error Messages Appears Ineffectual. In Proceedings of the 19th Conference on Innovation and Technology in Computer Science Education (Uppsala, Sweden) (ITiCSE ’14). Association for Computing Machinery, New York, NY,USA, 273… — [doi:10.1145/2591708.2591748](https://doi.org/10.1145/2591708.2591748) find
  14. Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. Understanding the Syntax Barrier for Novices. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education (Darmstadt, Germany) (ITiCSE ’11). Association for Computing Machiner… — [doi:10.1145/1999747.1999807](https://doi.org/10.1145/1999747.1999807) find
  15. Paul Denny, James Prather, and Brett A Becker. 2020. Error Message Readability and Novice Debugging Performance. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 480–486. find
  16. Paul Denny, James Prather, Brett A. Becker, Catherine Mooney, John Homer, Zachary C Albrecht, and Garrett B. Powell. 2021. On Designing Programming Error Messages for Novices: Readability and Its Constituent Factors. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yo… — [doi:10.1145/3411764.3445696](https://doi.org/10.1145/3411764.3445696) find
  17. James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference. 10–19. find
  18. Rahul Gupta, Aditya Kanade, and Shirish Shevade. 2019. Deep Reinforcement Learning for Syntactic Error Repair in Student Programs. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 930–937. find
  19. Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fixing common C Language Errors by Deep Learning. In Thirty-First AAAI conference on artificial intelligence. find
  20. Slava Kalyuga. 2009. The Expertise Reversal Effect. In Managing cognitive load in adaptive multimedia learning. IGI Global, 58–80. find
  21. Ioannis Karvelas, Annie Li, and Brett A. Becker. 2020. The Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE ’20). Association for Computing Mach… — [doi:10.1145/3328778.3366882](https://doi.org/10.1145/3328778.3366882) find
  22. Tobias Kohn. 2019. The Error Behind The Message: Finding the Cause of Error Messages in Python. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY,USA, 524–530. https://doi.org/10.114… — [doi:10.1145/3287324.3287381](https://doi.org/10.1145/3287324.3287381) find
  23. J Richard Landis and Gary G Koch. 1977. The Measurement of Observer Agreement for Categorical Data. biometrics (1977), 159–174. find
  24. Hang Li. 2022. Language Models: Past, Present, and Future. Commun. ACM 65, 7 (jun 2022), 56–63. https://doi.org/10.1145/3490443 — [doi:10.1145/3490443](https://doi.org/10.1145/3490443) ↗
  25. David Liu and Andrew Petersen. 2019. Static Analyses in Python Programming Courses. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 666–671. find
  26. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv preprint arXiv:2107.13586 (2021).
  27. Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating Diverse Code Explanations using the GPT-3 Large Language Model. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2. 37–39. find
  28. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768. find
  29. Raymond S. Pettit, John Homer, and Roger Gee. 2017. Do Enhanced Compiler Error Messages Help Students? Results Inconclusive.. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (Seattle, Washington, USA) (SIGCSE ’17). Association for Computing Machinery, New York… — [doi:10.1145/3017680.3017768](https://doi.org/10.1145/3017680.3017768) find
  30. James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive Difficulties Faced by Novice programmers in Automated Assessment Tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research. 41–50. find
  31. James Prather, Raymond Pettit, Kayla Holcomb McMurry, Alani Peters, John Homer, Nevan Simone, and Maxine Cohen. 2017. On Novices’ Interaction with Compiler Error Messages: A Human Factors Approach. In Proceedings of the 2017 ACM Conference on International Computing Education Research (Tacoma, Washi… — [doi:10.1145/3105726.3106169](https://doi.org/10.1145/3105726.3106169) find
  32. Saul Rosen, Robert A. Spurgeon, and Joel K. Donnelly. 1965. PUFFT—The Purdue University Fast FORTRAN Translator. Commun. ACM 8, 11 (nov 1965), 661–666. https://doi.org/10.1145/365660.365671 — [doi:10.1145/365660.365671](https://doi.org/10.1145/365660.365671) ↗
  33. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research V. 1. 27–43. find
  34. Andreas Stefik and Susanna Siebert. 2013. An Empirical Investigation into Programming Language Syntax. ACM Transactions on Computing Education 13, 4 (2013), 1–40. https://doi.org/10.1145/2534973 — [doi:10.1145/2534973](https://doi.org/10.1145/2534973) ↗
  35. Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/leinonen-2022-large-language-models-enhance-programming/