Elamur Research⌕
☰
home / papers / 2023 / Automated distractor and feedback generation for math multiple-choice questions via in-context learning

Automated distractor and feedback generation for math multiple-choice questions via in-context learning

Paper · 2023 · McNichols

Summary

Writing good multiple-choice questions (MCQs) for math is slow, manual work. Teachers must invent wrong options (distractors) that reflect real student misconceptions, and they must write a feedback message for each one. McNichols and co-authors from UMass Amherst and Eedi define two generation tasks: producing distractors from a question stem (with the key and its explanation as optional inputs), and producing feedback for a given distractor. Their baseline is few-shot in-context learning with large language models. Example questions are retrieved by kNN over SBERT (mpnet) embeddings of the full MCQ text. They test this on a real-world set of math MCQs that includes student response statistics. For feedback they also propose two reference-free metrics in which an LLM stands in for a human judge:

  • Answer Adjustment: whether the feedback lets the LLM fix the wrong answer.
  • Distractor Prediction: whether the feedback lets the LLM recover the distractor.

Similarity-based example retrieval clearly beats random and zero-shot prompting. However, absolute scores stay modest, so both tasks remain open problems. The authors note that many generated distractors that miss the ground truth are still reasonable, which suggests these tools could assist teachers.

Key points

  • Distractor generation: kNN prompting reached 10.06% exact, 71.02% partial and 38.16% proportional match with the human-written distractors. Zero-shot GPT-4 reached only 44.52% partial match.
  • Effect of same-topic examples: excluding questions from the same topic (kNN all ¬T) lowered partial match to 55.83%, so retrieving similar items is central to the gains.
  • Feedback, reference-based metrics: kNN one led by a wide margin (BLEU 33.70, ROUGE-L 42.28, METEOR 43.64). The authors attribute much of this to the model copying the style of near-duplicate human messages.
  • Feedback, reference-free metrics: methods were much closer together. Human feedback scored 49.00 Adj. / 24.73 Dist. Pred., kNN one 46.64 / 18.26, and zero-shot 47.70 / 20.49, although zero-shot output often leaked the answer itself.
  • Venue and future work: the paper appeared at the NeurIPS'23 GAIED workshop. Planned work includes fine-tuning smaller models, encodings that are aware of student errors, and human evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States122 works · 69,574 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as McNichols. 2023. Automated distractor and feedback generation for math multiple-choice questions via in-context learning. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2308.03234

@misc{mcnichols2023automated,
  title = {{Automated distractor and feedback generation for math multiple-choice questions via in-context learning}},
  author = {McNichols},
  year = {2023},
  eprint = {2308.03234},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.03234}
}

References

42 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthony J. Nitko. Educational assessment of students. Prentice-Hall, Iowa, USA, 1996. find
  2. Peter Airasian. Classroom assessment: Concepts and applications. McGraw-Hill, Ohio, USA, 2001. find
  3. Tom Kubiszyn and Gary Borich. Educational testing and measurement. John Wiley & Sons, New Jersey, USA, 2016. find
  4. Kim Kelly, Neil Heffernan, Sidney D’Mello, Namais Jeffrey, and Amber C. Strain. Adding teacher-created motivational video to an its. In Proceedings of 26th Florida Artificial Intelligence Research Society Conference, pages 503–508, 2013. find
  5. Yuni Susanti, Takenobu Tokunaga, Hitoshi Nishikawa, and Hiroyuki Obari. Automatic distractor generation for multiple-choice english vocabulary questions. Research and practice in technology enhanced learning, 13:1–16, 2018. find
  6. Katherine Stasaski and Marti A Hearst. Multiple choice question generation utilizing an ontology. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 303–312, 2017. find
  7. Zhaopeng Qiu, Xian Wu, and Wei Fan. Automatic distractor generation for multiple choice questions in standard tests. arXiv preprint arXiv:2011.13100, 2020. find
  8. Pengju Shuai, Li Li, Sishun Liu, and Jun Shen. Qdg: A unified model for automatic question-distractor pairs generation. Applied Intelligence, 53(7):8275–8285, 2023. find
  9. Jiayuan Xie, Ningxin Peng, Yi Cai, Tao Wang, and Qingbao Huang. Diverse distractor generation for constructing high-quality multiple choice questions. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:280–291, 2021. find
  10. Yifan Gao, Lidong Bing, Piji Li, Irwin King, and Michael R Lyu. Generating distractors for reading comprehension questions from real examinations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6423–6430, 2019. find
  11. Dmytro Kalpakchi and Johan Boye. Bert-based distractor generation for swedish reading comprehension questions using a small-scale dataset. arXiv preprint arXiv:2108.03973, 2021. find
  12. Shang-Hsuan Chiang, Ssu-Cheng Wang, and Yao-Chung Fan. Cdgp: Automatic cloze distractor generation based on pre-trained language model. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5835–5840, 2022. find
  13. Ricardo Rodriguez-Torrealba, Eva Garcia-Lopez, and Antonio Garcia-Cabot. End-to-end generation of multiple-choice questions using text-to-text transfer transformer models. Expert Systems with Applications, 208:118258, 2022. find
  14. Ana Paula Tomás and José Paulo Leal. Automatic generation and delivery of multiple-choice math quizzes. In Principles and Practice of Constraint Programming: 19th International Conference, CP 2013, Uppsala, Sweden, September 16-20, 2013. Proceedings 19, pages 848– 863. Springer, 2013. find
  15. Neisarg Dave, Riley Bakes, Barton Pursel, and C Lee Giles. Math multiple choice question solving and distractor generation with attentional gru networks. International Educational Data Mining Society, 2021. find
  16. Su-Youn Yoon, Ching-Ni Hsieh, Klaus Zechner, Matthew Mulholland, Yuan Wang, and Nitin Madnani. Toward automated content feedback generation for non-native spontaneous speech. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 306–315, Flor… find
  17. Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy, Verena Möller, Florian Nuxoll, and Detmar Meurers. Generating feedback for English foreign language exercises. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 127–136, New Orleans, Louisian…
  18. Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. Automated feedback generation for introductory programming assignments. SIGPLAN Not., 48(6):15–26, jun 2013. find
  19. Sumit Gulwani, Ivan Radiček, and Florian Zuleger. Feedback generation for performance problems in introductory programming assignments. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, page 41–51, New York, NY, USA, 2014. Association f… find
  20. Dowon Song, Woosuk Lee, and Hakjoo Oh. Context-aware and data-driven feedback generation for programming assignments. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, page 328–340, New Yo… find
  21. Marcus Messer. Grading programming assignments with an automated grading and feedback assistant. In Maria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, and Vania Dimitrova, editors, Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industr… find
  22. Hemilis Joyse Barbosa Rocha, Patrícia Cabral de Azevedo Restelli Tedesco, and Evandro de Barros Costa. A context-aware approach to personalized feedback for novice programmers. In Maria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, and Vania Dimitrova, editors, Artificial Intelligence in… find
  23. Qinjin Jia, Mitchell Young, Yunkai Xiao, Jialin Cui, Chengyuan Liu, Parvez Rashid, and Edward Gehringer. Insta-reviewer: A data-driven approach for generating instant feedback on students’ project reports. International Educational Data Mining Society, 2022. find
  24. Thanaporn Patikorn and Neil T Heffernan. Effectiveness of crowd-sourcing on-demand assistance from teachers in online learning platforms. In Proceedings of the Seventh ACM Conference on Learning@ Scale, pages 115–124, 2020. find
  25. Kenneth R. Koedinger and Albert Corbett. Cognitive tutors: Technology bringing learning sciences to the classroom. 2006. find
  26. Ran Liu, Rony Patel, and Kenneth R. Koedinger. Modeling common misconceptions in learning process data. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, pages 369–377, 2016. find
  27. Eleanor O’Rourke, Eric Butler, Armando D. Tolentino, and Zoran Popovic’. Automatic generation of problems and explanations for an intelligent algebra tutor. In Proceedings of the Artificial Intelligence in Education: 20th International Conference, pages 383–395, 2019. find
  28. Steven Ritter, John R. Anderson, Kenneth R. Koedinger, and Albert Corbett. Cognitive tutor: Applied research in mathematics education. Psychonomic bulletin & review, 14:249–255, April 2007. find
  29. Andrew S. Lan, Divyanshu Vats, Andrew E. Waters, and Richard G. Baraniuk. Mathematical language processing: Automatic grading and feedback for open response mathematical questions. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale, L@S ’15, page 167–176, New York, NY, USA, 2015.… find
  30. Mengxue Zhang, Zichao Wang, Richard Baraniuk, and Andrew Lan. Math operation embeddings for open-ended solution analysis and feedback, 2021. find
  31. Renah Razzaq, Korinn S. Ostrow, and Neil T. Heffernan. Effect of immediate feedback on math achievement at the high school level. In Ig Ibert Bittencourt, Mutlu Cukurova, Kasia Muldner, Rose Luckin, and Eva Millán, editors, Artificial Intelligence in Education, pages 263–267, Cham, 2020. Springer In… find
  32. Ekaterina Kochmar, Dung Do Vu, Robert Belfer, Varun Gupta, Iulian Vlad Serban, and Joelle Pineau. Automated personalized feedback improves learning gains in an intelligent tutoring system. In Ig Ibert Bittencourt, Mutlu Cukurova, Kasia Muldner, Rose Luckin, and Eva Millán, editors, Artificial Intell…
  33. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  34. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3. abs/2101.06804, 2021.
  35. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
  36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. Bleu: a method for automatic evaluation of machine translation. pages 311–318, 2002.
  37. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  38. Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan, June 2005…
  39. Roman Rietsche, Andrew Caines, Cornelius Schramm, Dominik Pfütze, and Paula Buttery. The specificity and helpfulness of peer-to-peer feedback in higher education. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 107–117, Seattle, Wa… find
  40. Qinjin Jia, Yupeng Cao, and Edward Gehringer. Starting from “zero”: An incremental zero-shot learning approach for assessing peer feedback comments. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 46–50, Seattle, Washington, July 2… find
  41. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  42. OpenAI. Introducing chatgpt, 2022. Method kNN none kNN key kNN all kNN one k no yes yes no Encode e k d no no no no yes no no yes k no yes yes no Prompt e k F (f ) no none no none yes all no one D(d) all all all one *Table 3: Encoding strategies for retrieval and prompt formats used in kNN-based met… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mcnichols-2023-automated-distractor-feedback-generation-math/