Elamur Research⌕
☰
home / papers / 2024 / Pedagogical Alignment of Large Language Models

Pedagogical Alignment of Large Language Models

Paper · 2024 · Shashank Sonkar, Kangqi Ni, Sapana Chaudhary et al.

Summary

Chat-based LLMs used as tutors tend to give students the final answer right away. Good teaching instead breaks a problem into steps and offers hints. The authors call the goal of training models toward this tutoring behaviour "pedagogical alignment." They treat it as a learning from human preferences (LHP) problem rather than relying only on supervised fine-tuning (SFT).

Real preference data for tutoring is scarce, so they build a large synthetic dataset by reusing the CLASS framework. CLASS has GPT-4 produce tutoring dialogues annotated with structured fields: response evaluation, chosen action and subproblem state. The authors use these fields to pair pedagogically appropriate tutor turns with weaker ones. With this data they train Llama, Mistral and Zephyr using DPO, IPO and KTO and compare the results with SFT. They also propose perplexity-based metrics that measure whether a model favours scaffolded hints or direct solutions.

LHP-trained models clearly beat SFT and stay aligned over long conversations. This offers a cheaper, annotation-free route to tutoring-oriented LLMs.

Key points

  • Compared with SFT, the LHP methods raised pedagogical alignment accuracy by 13.1% for Llama, 8.7% for Mistral and 50.0% for Zephyr.
  • The preference data comes from the GPT-4-based CLASS pipeline and needs no manual annotation. Three agents produce it: GPT-student, GPT-tutor and SFT-tutor. Each tutor reply is a JSON record with evaluation, action, subproblem state and response fields.
  • KTO with beta = 0.9 performed best overall:
    • Llama: 78.1% accuracy, F1 0.57
    • Zephyr: 75.7% accuracy, F1 0.55
    • Mistral: 74.7% accuracy, F1 0.51
  • DPO worked best at low beta (0.1). IPO was the least sensitive to the choice of beta.
  • In later conversation rounds, SFT models lost alignment (Mistral 63–69%, Zephyr 11–30%), while all LHP variants stayed above 70%. Perplexity analysis showed that base models lean toward direct answers, SFT corrects this only partly, and LHP strongly shifts models toward giving guidance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Texas A&M UniversityCollege Station · United States32 works · 2,823 citations🎓Rice UniversityHouston · United States15 works · 2,036 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Shashank Sonkar, Kangqi Ni, Sapana Chaudhary and Richard Baraniuk. 2024. Pedagogical Alignment of Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.797 · arXiv 2402.05000

@inproceedings{sonkar2024pedagogical,
  title = {{Pedagogical Alignment of Large Language Models}},
  author = {Shashank Sonkar, Kangqi Ni, Sapana Chaudhary and Richard Baraniuk},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.797},
  eprint = {2402.05000},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.797}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. find
  2. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036.
  3. Richard Paul and Linda Elder. 1995. Critical Thinking: How to Prepare Students for a Rapidly Changing World. Foundation for Critical Thinking. find
  4. John D. Bransford, Ann L. Brown, and Rodney R. Cock-ing, editors. 2000. How People Learn: Brain, Mind, Experience, and School: Expanded Edition. National Academy Press. find
  5. Michael Prince. 2004. Does active learning work? a review of the research. Journal of Engineering Education, 93(3):223–231. find
  6. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712.
  7. Michelene TH Chi. 2009. Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in cognitive science, 1(1):73–105. find
  8. Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. arXiv preprint arXiv:1706.03741. find
  9. Mary Ann Clark, Matthew Douglas, and Jung Choi. 2018. Biology 2e. OpenStax. find
  10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  11. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. find
  12. John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research, 77(1):81– 112. find
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  14. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. find
  15. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
  16. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  17. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  18. Shashank Sonkar, Naiming Liu, Debshila Mallick, and Richard Baraniuk. 2023. CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1941–1961.
  19. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. find
  20. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. 2020. TRL: Transformer Reinforcement Learning. find
  21. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
  22. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sonkar-2024-pedagogical-alignment-large-language-models/