Elamur Research⌕
☰
home / papers / 2024 / Improving the Validity of Automatically Generated Feedback via Reinforcement Learning

Improving the Validity of Automatically Generated Feedback via Reinforcement Learning

Paper · 2024 · Alexander Scarlatos, Digory Smith, Simon Woodhead et al.

Summary

LLM-written feedback on students' wrong answers in math must be mathematically sound and must also follow good tutoring practice. Judging both properties automatically is hard, and text-overlap metrics measure them poorly. The authors build a framework for both generating and evaluating such feedback for incorrect options in multiple-choice math questions, using teacher-written messages as gold data.

They define a five-part binary rubric and have GPT-4 apply it through zero-shot chain-of-thought prompting. They then expand the dataset with feedback produced by several prompting strategies, score everything with GPT-4, and turn the scores into preference pairs. These pairs are used to fine-tune Llama 2 7B with DPO, an offline RL method that avoids calling GPT-4 repeatedly during training.

The fine-tuned model produces feedback that is significantly more correct and better aligned than baselines. On alignment metrics, it comes close to human and GPT-4 feedback despite being a far smaller open model. This suggests that affordable, open models can deliver pedagogically sound feedback in tutoring systems.

Key points

  • Rubric: five binary labels: Correct, Revealing (does not give away the answer), Suggestion, Diagnostic, and Positive. Any incorrect feedback scores 0. Otherwise, each satisfied label adds 0.2 to the score.
  • Augmented data: the gold feedback is combined with three LLM-generated variants: few-shot with random examples (D_R), few-shot with the most similar examples (D_S), and zero-shot (D_Z).
  • Annotator validation: on 80 sampled messages, GPT-4 averaged 76% accuracy against human labels. Its rubric scores correlated with human scores at Pearson 0.56, compared with 0.40 for both ROUGE-L and BERTScore. Agreement between the two human annotators (Cohen's kappa) ranged from 0.29 (POS.) to 0.53 (COR., DIA.).
  • GPT-4 weaknesses: it tends to accept persuasive but wrong feedback, especially when checking requires calculation. It also hallucinates diagnostic content, producing false positives on that label.
  • Availability: published at AIED 2024, with code at github.com/umass-ml4ed/feedback-gen-dpo.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States122 works · 69,574 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Alexander Scarlatos, Digory Smith, Simon Woodhead and Andrew Lan. 2024. Improving the Validity of Automatically Generated Feedback via Reinforcement Learning. Lecture Notes in Computer Science.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1007/978-3-031-64302-6_20 · arXiv 2403.01304

@inproceedings{scarlatos2024improving,
  title = {{Improving the Validity of Automatically Generated Feedback via Reinforcement Learning}},
  author = {Alexander Scarlatos, Digory Smith, Simon Woodhead and Andrew Lan},
  booktitle = {Lecture Notes in Computer Science},
  year = {2024},
  doi = {10.1007/978-3-031-64302-6\_20},
  eprint = {2403.01304},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1007/978-3-031-64302-6_20}
}

References

7 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Al-Hossami, E., Bunescu, R., Teehan, R., Powell, L., Mahajan, K., Dorodchi, M.: Socratic questioning of novice debuggers: A benchmark dataset and preliminary evaluations. In: Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications (BEA2023@ACL). pp. 709–726 (Jul…
  2. Improving the Validity of Automatically Generated Feedback 2. Boaler, J.: Ability and mathematics: The mindset revolution that is reshaping education. The Forum 55, 143–152 (2013) 3. Botelho, A., Baral, S., Erickson, J.A., Benachamardi, P., Heffernan, N.T.: Leveraging natural language processing to… find
  3. International Educational Data Mining Society (2022) 11. Kakarla, S., Thomas, D., Lin, J., Gupta, S., Koedinger, K.R.: Using large language models to assess tutors’ performance in reacting to students making math errors. arXiv preprint arXiv:2401.03238 (2024) 12. Kochmar, E., Vu, D.D., Belfer, R., G… — Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System
  4. A. Scarlatos et al. 20. McNichols, H., Feng, W., Lee, J., Scarlatos, A., Smith, D., Woodhead, S., Lan, A.: Automated distractor and feedback generation for math multiple-choice questions via in-context learning. NeurIPS’23 Workshop on Generative AI for Education (2023) 21. McNichols, H., Zhang, M.,…
  5. Springer (2020) 26. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (11 2019) 27. Robinson, J.D., Chuang, C.Y., Sra, S., Je… — Reflexion: Language Agents with Verbal Reinforcement Learning
  6. International Educational Data Mining Society (2022) find
  7. Improving the Validity of Automatically Generated Feedback 37. Zhang, M., Wang, Z., Baraniuk, R., Lan, A.: Math operation embeddings for open-ended solution analysis and feedback. International Educational Data Mining Society (2021) 38. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Be… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/scarlatos-2024-improving-validity-automatically-generated-feedback/