Elamur Research⌕
☰
home / papers / 2024 / Evaluating and Optimizing Educational Content with Large Language Model Judgments

Evaluating and Optimizing Educational Content with Large Language Model Judgments

Paper · 2024 · Joy He-Yueya, Noah D. Goodman and Emma Brunskill

Summary

Testing whether teaching materials actually help learners usually requires slow, costly studies with real students. Earlier efforts to have language models role-play students struggled because the simulated knowledge levels drifted before and after instruction. The authors instead cast the model as a pedagogical expert. In their Simulated Expert Evaluation (SEE), an evaluator LM receives a student persona describing prior skills, a piece of instruction, and a series of post-test questions. It then estimates the chance that the student solves each item, and these estimates are aggregated across personas. With GPT-3.5 in this role, the judgments recover a pair of well-known findings from educational psychology, namely the Variability Effect and the Expertise Reversal Effect. The authors then build an optimization loop for math word problem worksheets. GPT-4 drafts new worksheets from earlier worksheet–score pairs, and GPT-3.5's predicted post-test scores act as the reward. Teachers' pairwise comparisons of the resulting worksheets correlate with the model's predictions. This suggests LMs could serve as inexpensive first-pass evaluators that help plan human experiments, though they cannot replace testing with actual students.

Key points

  • SEE feeds the evaluator a persona, an instruction and test items one after another. It averages the per-item success probabilities into a predicted score. The prompts are released at github.com/StanfordAI4HI/ed-expert-simulator.
  • GPT-3.5 judgments replicate the Expertise Reversal Effect, where the benefit of instruction depends on the learner's prior knowledge, and the Variability Effect.
  • GPT-3.5 served as the evaluator because it is cheaper for long prompts. GPT-4 served as the optimizer, generating worksheets for an 8th-grade student working on algebra word problems.
  • In the human study, 56 teachers rated worksheet pairs on a slider. Their preference scores correlated with LM-predicted post-test scores (Pearson r = 0.661, p < 0.05).
  • The paper notes several limitations:
    • Teachers often could not separate worksheets that the LM scored differently, especially in the 60–90 predicted-score range.
    • The models may have seen the replicated studies during training.
    • Only mathematics and only GPT-3.5/GPT-4 were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Joy He-Yueya, Noah D. Goodman and Emma Brunskill. 2024. Evaluating and Optimizing Educational Content with Large Language Model Judgments. arXiv:2403.02795.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.02795 · arXiv 2403.02795

@misc{heyueya2024evaluating,
  title = {{Evaluating and Optimizing Educational Content with Large Language Model Judgments}},
  author = {Joy He-Yueya, Noah D. Goodman and Emma Brunskill},
  year = {2024},
  doi = {10.48550/arxiv.2403.02795},
  eprint = {2403.02795},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.02795}
}

References

51 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. G. V. Aher, R. I. Arriaga, and A. T. Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337–371. PMLR, 2023. find
  3. J. R. Anderson, A. T. Corbett, K. R. Koedinger, and R. Pelletier. Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2):167–207, 1995. find
  4. L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351, 2023.
  5. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
  6. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  7. P. Denny, H. Khosravi, A. Hellas, J. Leinonen, and S. Sarsa. Can we trust ai-generated educational content? comparative analysis of human and ai-generated learning resources. arXiv preprint arXiv:2306.10509, 2023. — [arXiv:2306.10509](https://arxiv.org/abs/2306.10509) find
  8. J. E. Greer and G. I. McCalla. Student modelling: the key to individualized knowledge-based instruction, volume 125. Springer Science & Business Media, 2013. find
  9. J. He-Yueya, G. Poesia, R. E. Wang, and N. D. Goodman. Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102, 2023.
  10. J. J. Horton. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023. find
  11. H. Jin, S. Lee, H. Shin, and J. Kim. ” teach ai how to code”: Using large language models as teachable agents for programming education. arXiv preprint arXiv:2309.14534, 2023. — [arXiv:2309.14534](https://arxiv.org/abs/2309.14534) find
  12. S. Jinxin, Z. Jiabao, W. Yilei, W. Xingjiao, L. Jiawen, and H. Liang. Cgmi: Configurable general multi-agent interaction framework. arXiv preprint arXiv:2308.12503, 2023. — [arXiv:2308.12503](https://arxiv.org/abs/2308.12503) find
  13. S. Kalyuga. Expertise reversal effect and its implications for learner-tailored instruction. Educational psychology review, 19:509–539, 2007. find
  14. S. Kalyuga, P. Ayres, P. Chandler, and J. Sweller. The expertise reversal effect. Educational Psychologist, 38(1):23–31, 2003. find
  15. S. Kalyuga and J. Sweller. Measuring knowledge to optimize cognitive load factors during instruction. Journal of educational psychology, 96(3):558, 2004. find
  16. P. K. Komoski. Eric/avcr annual review paper: An imbalance of product quantity and instructional quality: The imperative of empiricism. AV Communication Review, 22(4):357–386, 1974. find
  17. H. Kumar, D. M. Rothschild, D. G. Goldstein, and J. M. Hofman. Math education with large language models: Peril or promise? Available at SSRN 4641653, 2023. find
  18. R. Liu, H. Yen, R. Marjieh, T. L. Griffiths, and R. Krishna. Improving interpersonal communication by simulating audiences with language models. arXiv preprint arXiv:2311.00687, 2023. find
  19. Q. Ma, H. Shen, K. Koedinger, and T. Wu. Hypocompass: Large-language-model-based tutor for hypothesis construction in debugging for novices. arXiv preprint arXiv:2310.05292, 2023. — [arXiv:2310.05292](https://arxiv.org/abs/2310.05292) find
  20. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. find
  21. J. M. Markel, S. G. Opferman, J. A. Landay, and C. Piech. Gpteach: Interactive ta training with gpt based students. 2023. find
  22. N. Matsuda, W. W. Cohen, J. Sewall, G. Lacerda, and K. R. Koedinger. Evaluating a simulated student using real students data for training and testing. In International Conference on User Modeling, pages 107–116. Springer, 2007. find
  23. N. Matsuda, V. Keiser, R. Raizada, A. Tu, G. Stylianides, W. W. Cohen, and K. R. Koedinger. Learning by teaching simstudent: Technical accomplishments and an initial use with students. In Intelligent Tutoring Systems: 10th International Conference, ITS 2010, Pittsburgh, PA, USA, June 14-18, 2010, Pr… find
  24. J. S. Mertz. Using a simulated student for instructional design. International Journal of Artificial Intelligence in Education (IJAIED), 8:116–141, 1997. find
  25. M. H. Nguyen, S. Tschiatschek, and A. Singla. Large language models for in-context student modeling: Synthesizing student’s behavior in visual programming from one-shot observation. arXiv preprint arXiv:2310.10690, 2023. — [arXiv:2310.10690](https://arxiv.org/abs/2310.10690) find
  26. N. Nieveen and E. Folmer. Formative evaluation in educational design research. Design Research, 153(1):152–169, 2013. find
  27. S. Ohlsson, A. M. Ernst, and E. Rees. The cognitive complexity of learning and doing arithmetic. Journal for Research in Mathematics Education, 23(5):441–467, 1992. find
  28. A. Opedal, A. Stolfo, H. Shirakami, Y. Jiao, R. Cotterell, B. Schölkopf, A. Saparov, and M. Sachan. Do language models exhibit the same cognitive biases in problem solving as human learners? arXiv preprint arXiv:2401.18070, 2024. — [arXiv:2401.18070](https://arxiv.org/abs/2401.18070) find
  29. F. G. Paas and J. J. Van Merriënboer. Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of educational psychology, 86(1):122, 1994. find
  30. M. Pankiewicz and R. S. Baker. Large language models (gpt) for automating feedback on programming assignments. arXiv preprint arXiv:2307.00150, 2023. find
  31. Z. A. Pardos and S. Bhandari. Learning gain differences between chatgpt and human tutor generated algebra hints. arXiv preprint arXiv:2302.06871, 2023.
  32. J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22, 2023. find
  33. J. S. Park, L. Popowski, C. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18, 2022.
  34. T. Phung, V.-A. Pădurean, A. Singh, C. Brooks, J. Cambronero, S. Gulwani, A. Singla, and G. Soares. Automating human tutor-style programming feedback: Leveraging gpt-4 tutor model for hint generation and gpt-3.5 student model for hint validation. arXiv preprint arXiv:2310.03780, 2023. find
  35. E. Prihar, M. Lee, M. Hopman, A. T. Kalai, S. Vempala, A. Wang, G. Wickline, A. Murray, and N. Heffernan. Comparing different approaches to generating mathematics explanations using large language models. In International Conference on Artificial Intelligence in Education, pages 290–295. Springer, 2… find
  36. R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023.
  37. A. V. Riabko and T. A. Vakaliuk. Physics on autopilot: exploring the use of an ai assistant for independent problem-solving practice. Educational Technology Quarterly, 2024. find
  38. S. Sarsa, P. Denny, A. Hellas, and J. Leinonen. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1, pages 27–43, 2022. find
  39. R. Schmucker, M. Xia, A. Azaria, and T. Mitchell. Ruffle&riley: Towards the automated induction of conversational tutoring systems. arXiv preprint arXiv:2310.01420, 2023. — [arXiv:2310.01420](https://arxiv.org/abs/2310.01420) find
  40. O. Shaikh, V. Chai, M. J. Gelfand, D. Yang, and M. S. Bernstein. Rehearsal: Simulating conflict to teach conflict resolution. arXiv preprint arXiv:2309.12309, 2023. find
  41. P. L. Smith and T. J. Ragan. Instructional design. John Wiley & Sons, 2004. find
  42. J. Sweller, J. J. van Merriënboer, and F. Paas. Cognitive architecture and instructional design: 20 years later. Educational psychology review, 31:261–292, 2019. find
  43. J. E. Tuovinen and J. Sweller. A comparison of cognitive load associated with discovery learning and worked examples. Journal of educational psychology, 91(2):334, 1999. find
  44. K. VanLehn, S. Ohlsson, and R. Nason. Applications of simulated students: An exploration. Journal of artificial intelligence in education, 5:135–135, 1994. find
  45. R. E. Wang and D. Demszky. Is chatgpt a good teacher coach? measuring zero-shot performance for scoring and providing actionable insights on classroom instruction. arXiv preprint arXiv:2306.03090, 2023.
  46. R. E. Wang, Q. Zhang, C. Robinson, S. Loeb, and D. Demszky. Step-by-step remediation of students’ mathematical mistakes. arXiv preprint arXiv:2310.10648, 2023. find
  47. Z. Wang, J. Valdez, D. Basu Mallick, and R. G. Baraniuk. Towards human-like educational question generation with large language models. In International conference on artificial intelligence in education, pages 153–166. Springer, 2022. find
  48. D. D. Williams, J. B. South, S. C. Yanchar, B. G. Wilson, and S. Allen. How do instructional designers evaluate? a qualitative study of evaluation in practice. Educational Technology Research and Development, 59:885–907, 2011. find
  49. C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  50. E. Zelikman, E. Lorch, L. Mackey, and A. T. Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304, 2023. — [arXiv:2310.02304](https://arxiv.org/abs/2310.02304) find
  51. E. Zelikman, W. A. Ma, J. E. Tran, D. Yang, J. D. Yeatman, and N. Haber. Generating and evaluating tests for k-12 students with language model simulations: A case study on sentence reading efficiency. arXiv preprint arXiv:2310.06837, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-yueya-2024-evaluating-optimizing-educational-content-large/