Elamur Research⌕
☰
home / papers / 2024 / Towards responsible development of generative AI for education: An evaluation-driven approach

Towards responsible development of generative AI for education: An evaluation-driven approach

Paper · 2024 · Jurenka

Summary

Generative AI is often promoted as a way to give every student a personal tutor, but general-purpose chat models are tuned for "helpfulness." In practice, that can mean handing out answers instead of supporting learning. The authors, a large Google DeepMind/Google team working with Arizona State University and others, argue that progress is held back by two things: pedagogical intuitions are hard to express as prompts, and there is no shared way to evaluate tutoring quality.

They worked with learners and educators, drawing on learning-science literature, to build supervised fine-tuning data mixtures for 1:1 conversational tutoring. Fine-tuning Gemini 1.0 on these data produced LearnLM-Tutor. They also designed seven complementary benchmarks covering quantitative and qualitative, human and automatic evaluation, organised by an explicit evaluation taxonomy.

Across most measured pedagogical dimensions, educators and learners preferred LearnLM-Tutor over Gemini 1.0 with a strong external tutoring prompt. The work matters mainly as a proposed starting point for common pedagogical evaluation standards in AI and EdTech.

Key points

  • Model: LearnLM-Tutor is a text-only tutor built on Gemini 1.0 through SFT. Its final version, labelled M4, is compared against a prompt-tuned Gemini 1.0 baseline.
  • Seven benchmarks:
    • subjective learner feedback
    • turn-level pedagogy
    • conversation-level pedagogy
    • side-by-side pedagogy
    • language-model-based evaluations
    • automatic pedagogy scores
    • interviews with ASU students
  • Real-world deployment: the tutor was used in ASU Study Hall, including an Introduction to Programming course, with staged roll-out and ongoing monitoring.
  • Anthropomorphism safety checks (automatic), from the first model version (M0) to M4:
    • "pretends to be human" fell from 0.62 to 0.00
    • "pretends to be creator" fell from 0.61 to 0.07
    • "pretends to have UI control" fell from 0.35 to 0.01
  • Limitations: the authors note that good-pedagogy demonstrations are costly to collect, AI critics can be unreliable, and human ratings are slow. They suggest RLHF and continued benchmark iteration as future directions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations🎓University of OxfordOxford · United Kingdom138 works · 161,448 citations🎓Arizona State UniversityTempe · United States58 works · 5,334 citations🎓Lund UniversityLund · Sweden13 works · 1,317 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksMedicine · 288 worksHealth Informatics · 227 works

Cite

Cite as Jurenka. 2024. Towards responsible development of generative AI for education: An evaluation-driven approach. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2407.12687

@misc{jurenka2024responsible,
  title = {{Towards responsible development of generative AI for education: An evaluation-driven approach}},
  author = {Jurenka},
  year = {2024},
  eprint = {2407.12687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.12687}
}

References

217 references, 46 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jeremy Roschelle, Elizabeth A. McLaughlin, and Kenneth R. Koedinger. Beyond benchmarks: Responsible AI in education needs learning sciences. Communications of the ACM, 68(12): 37–39, 2025. find
  2. Ethan Mollick and Lilach Mollick. Assigning AI: Seven approaches for students, with prompts. arXiv preprint arXiv:2306.10052, 2023. — [arXiv:2306.10052](https://arxiv.org/abs/2306.10052) find
  3. SGD4 United Nations. UN Sustainable Development Goal 4: Quality Education, 2024. URL https://www.globalgoals.org/goals/4-quality-education . ↗
  4. Eric A Hanushek and Ludger Woessmann. Education and economic growth. Economics of education, 60(67):1, 2010. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 47 find
  5. Cristina Iannelli and Lindsay Paterson. Does education promote social mobility?, volume 35. Citeseer, 2005. [1] Joao Pedro Wagner De Azevedo, F. Halsey Rogers, Sanna Ellinore Carroll, Marie-Helene Cloutier, Borhene Chakroun, Gwang-Chol Chang, Suguru Mizunoya, Nicolas Jean Reuge, Matt Brossard, and J… find
  6. Jacob Bryant, Felipe Child, Jose Espinosa, Emma Dorn, Stephen Hall, Dirk Schmautzer, Topsy Kola-Oyeneyin, Cheryl Lim, Frédéric Panier, Jimmy Sarakatsannis, Seckin Ungur, and Bart Woord. How COVID-19 caused a global learning crisis, 2022. URL https://www.mckinsey.com/industries/education/our-insights… ↗
  7. Cecilia Ka Yuk Chan and Katherine KW Lee. The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and millennial generation teachers? Smart Learning Environments, 10(1):60, 2023. find
  8. Emma URL Whitford. ChatGPT and AI will fuel new EdTech boom, 2023. https://www.forbes.com/sites/emmawhitford/2023/01/18/ chatgpt-and-ai-will-fuel-new-edtech-boom/ . ↗
  9. Stefan Bauschard and Sabba Quidwai. From insight to implementation: How to create your AI school guidance. SSRN, 2024. find
  10. Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  11. Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf . ↗
  12. AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md . ↗
  13. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  14. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. find
  15. UK (AI) Department for Education. in education, 2023. Generative artificial intelligence URL https://www.gov.uk/government/ publications/generative-artificial-intelligence-in-education/ generative-artificial-intelligence-ai-in-education . ↗
  16. Wayne Holmes and Kaśka Porayska-Pomsta. The Ethics of Artificial Intelligence in education: Practices, challenges, and debates. Taylor & Francis, 2022. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 48 find
  17. Wayne Holmes, Jen Persson, Irene-Angelica Chounta, Barbara Wasson, and Vania Dimitrova. Artificial intelligence and education. a critical view through the lens of human rights, democracy and the rule of law, 2022. URL https://rm.coe.int/artificial-intelligence-and-education-a-critical-view-through-t… ↗
  18. Fengchun Miao, Wayne Holmes, Ronghuai Huang, Hui Zhang, et al. AI and education: A guidance for policymakers. UNESCO Publishing, 2021. find
  19. Andy Nguyen, Ha Ngan Ngo, Yvonne Hong, Belle Dang, and Bich-Phuong Thi Nguyen. Ethical principles for artificial intelligence in education. Education and Information Technologies, 28 (4):4221–4241, 2023. find
  20. René F Kizilcec. To advance ai use in education, focus on understanding educators. International Journal of Artificial Intelligence in Education, 34(1):12–19, 2024. find
  21. Dina Foster, Caitlin McLemore, Brandon Olszewski, Ali Chaudhry, Ekaterina Cooper, Laurie Forcier, and Rose Luckin. EdTech quality frameworks and standards review, 2023. URL https://assets.publishing.service.gov.uk/media/6579d0ac0467eb001355f761/ EdTech_quality_frameworks_and_standards_review.pdf . ↗
  22. The Open Innovation Team and Department for Education. Generative AI in education: Educator and expert views, 2024. URL https://assets.publishing.service.gov.uk/media/65b8cd41b5cb6e000d8bb74e/DfE_GenAI_in_education_-_Educator_ and_expert_views_report.pdf . ↗
  23. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 214–… find
  24. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
  25. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  26. Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, et al. The ethics of advanced AI assistants. arXiv preprint arXiv:2404.16244, 2024. — [arXiv:2404.16244](https://arxiv.org/abs/2404.16244) find
  27. Kenneth R Koedinger, Julie L Booth, and David Klahr. Instructional complexity and the science to constrain it. Science, 342(6161):935–937, 2013. find
  28. Sherry R Arnstein. A ladder of citizen participation. Journal of the American Institute of planners, 35(4):216–224, 1969. find
  29. Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. Power to the people? Opportunities and challenges for participatory AI. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization… find
  30. Alessandra Tombazzi, Joanna Choukeir, Natalie Lai, and Google DeepMind. AI and the future of learning, 2023. URL https://www.thersa.org/design-for-life-our-mission/ hubs/cities-of-learning/ai-future-learning-deepmind-roundtable . [1] Niels Pinkwart. Another 25 years of AIED? Challenges and opportuni… ↗
  31. Henry Sanoff. Community participation methods in design and planning. John Wiley & Sons, 1999. find
  32. Jasmin Rubinovitz. How it’s made - exploring AI x learning through Shiff Bot, an AI experiment powered by the Gemini API, 2024. URL https://shiffbot.withgoogle.com/ . ↗
  33. Holtzblatt Karen and Jones Sandra. Contextual inquiry: A participatory technique for system design. In Participatory design, pages 177–210. CRC Press, 2017. find
  34. Chadia Abras, Diane Maloney-Krichmar, Jenny Preece, et al. User-centered design. Bainbridge, W. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publications, 37(4): 445–456, 2004. find
  35. Junlei Li and David Klahr. Cognitive research and elementary science instruction: From the laboratory, to the classroom, and back. Journal of Science Education and Technology, 14(2), 2005. find
  36. David Klahr. What do we mean? On the importance of not abandoning scientific rigor when talking about science education. Proceedings of the National Academy of Sciences, 110 (supplement_3):14075–14080, 2013. find
  37. Amy Ogan. Designing culturally-relevant educational technology at a global scale, 2023. URL https://learnlab.org/learning-science-and-engineering-seminar/ . ↗
  38. Audrey Watters. Teaching machines: The history of personalized learning. MIT Press, 2023. find
  39. Edward Fry. Teaching machine dichotomy: Skinner vs. Pressey. Psychological Reports, 6(1): 11–14, 1960. find
  40. Jaime R Carbonell. AI in CAI: An artificial-intelligence approach to computer-assisted instruction. IEEE transactions on man-machine systems, 11(4):190–202, 1970. find
  41. Jack A Chambers and Jerry W Sprecher. Computer-assisted instruction: Its use in the classroom. (No Title), 1983. find
  42. John R Anderson, C Franklin Boyle, and Brian J Reiser. Intelligent tutoring systems. Science, 228(4698):456–462, 1985. find
  43. Vincent Aleven, Bruce Mclaren, Jonathan Sewall, and Kenneth Koedinger. A new paradigm for intelligent tutoring systems: Example-tracing tutors. I. J. Artificial Intelligence in Education, 19: 105–154, 01 2009. find
  44. Vincent Aleven, Bruce M McLaren, and Jonathan Sewall. Scaling up programming by demonstration for intelligent tutoring systems development: An open-access web site for middle school mathematics learning. IEEE transactions on learning technologies, 2(2):64–78, 2009. find
  45. John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2):167–207, 1995. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 50 find
  46. Kenneth R Koedinger, Albert Corbett, et al. Cognitive tutors: Technology bringing learning sciences to the classroom. na, 2006. [1] Miami-Dade county public schools. Evaluation of the cognitive tutor Algebra I program, 2001. find
  47. A Mitrovic. Learning SQL with a computerised tutor. In 29th ACM SIGCSE Technical Symposium. Atlanta, 1998. find
  48. Bruce M McLaren, Sung-Joo Lim, France Gagnon, David Yaron, and Kenneth R Koedinger. Studying the effects of personalized language and worked examples in the context of a web-based intelligent tutor. In Intelligent Tutoring Systems: 8th International Conference, ITS 2006, Jhongli, Taiwan, June 26-30,… find
  49. CR Beal, J Beck, and B Woolf. Impact of intelligent computer instruction on girls’ math self concept and beliefs in the value of math. In Poster presented at the annual meeting of the American Educational Research Association, San Diego, 1998. find
  50. Silvia Schiaffino, Patricio Garcia, and Analia Amandi. eTeacher: Providing personalized assistance to e-learning students. Computers & Education, 51(4):1744–1754, 2008. find
  51. Aytürk Keleş, Rahim Ocak, Ali Keleş, and Aslan Gülcü. ZOSMAT: Web-based intelligent tutoring system for teaching–learning process. Expert Systems with Applications, 36(2):1229–1239, 2009. find
  52. Bruce Cheung, Lucas Hui, J Zhang, and Siu-Ming Yiu. SmartTutor: An intelligent tutoring system in web-based adult education. Journal of Systems and Software, 68(1):11–25, 2003. find
  53. Arthur C Graesser, Katja Wiemer-Hastings, Peter Wiemer-Hastings, Roger Kreuz, Tutoring Research Group, et al. AutoTutor: A simulation of a human tutor. Cognitive Systems Research, 1 (1):35–51, 1999. find
  54. Erica Melis and Jörg Siekmann. ActiveMath: An intelligent tutoring system for mathematics. In International Conference on Artificial Intelligence and Soft Computing, pages 91–101. Springer, 2004. find
  55. Arthur C Graesser, Kurt VanLehn, Carolyn P Rosé, Pamela W Jordan, and Derek Harter. Intelligent tutoring systems with conversational dialogue. AI magazine, 22(4):39–39, 2001. find
  56. Benjamin Clément, Hélène Sauzéon, Didier Roy, and Pierre-Yves Oudeyer. Improved performances and motivation in intelligent tutoring systems: Combining machine learning and learner choice. arXiv preprint arXiv:2402.01669, 2024. — [arXiv:2402.01669](https://arxiv.org/abs/2402.01669) find
  57. Adolphe Maxime, Marion Pech, Masataka Sawayama, Denis Maurel, Alexandra Delmas, Pierre-Yves Oudeyer, and Hélène Sauzeon. Exploring the potential of artificial intelligence in individualized cognitive training: A systematic review. PsyArXiv, 2023. doi: 10.31234/osf.io/2wg59. URL osf.io/preprints/psya… — [doi:10.31234/osf.io/2wg59](https://doi.org/10.31234/osf.io/2wg59) find
  58. Cécile Mazon, Benjamin Clément, Didier Roy, Pierre-Yves Oudeyer, and Hélène Sauzéon. Pilot study of an intervention based on an intelligent tutoring system (ITS) for instructing mathematical skills of students with ASD and/or ID. Education and Information Technologies, 28(8):9325–9354, 2023. find
  59. Jeremy Rochelle, Robert Murphy, Mingyu Feng, and Marianne Bakia. How big is that? Reporting the effect size and cost of ASSISTments in the Maine homework efficacy study. Maine Education Policy Research Institute, 2017. Towards Responsible Development of Generative AI for Education: An Evaluation-Dri… find
  60. John F Pane, Daniel F McCaffrey, Mary Ellen Slaughter, Jennifer L Steele, and Gina S Ikemoto. An experiment to evaluate the efficacy of cognitive tutor geometry. Journal of Research on Educational Effectiveness, 3(3):254–281, 2010. find
  61. Kjetil Egelandsdal, Maria Smith, Cecilie Johanne Slokvik Hansen, Ingunn Johanne Ness, and Barbara Wasson. Adaptiv læring i matematikk: Empirisk rapport om multi smart øving i grunnskolen. SLATE Research Report, 2019. find
  62. Chronis Kynigos. Adaptive learning in mathematics: Situating multi smart øving in the landscape of digital technologies for mathematics education. SLATE Research Report, 2019. find
  63. Kurt VanLehn. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational psychologist, 46(4):197–221, 2011. find
  64. Benjamin D Nye, Arthur C Graesser, and Xiangen Hu. AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24(4):427–469, 2014. find
  65. Vincent Aleven, Bruce M Mclaren, Jonathan Sewall, and Kenneth R Koedinger. A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2):105–154, 2009. find
  66. Kurt VanLehn, Pamela W Jordan, Carolyn P Rosé, Dumisizwe Bhembe, Michael Böttner, Andy Gaydos, Maxim Makatchev, Umarani Pappuswamy, Michael Ringenberg, Antonio Roque, et al. The architecture of Why2-Atlas: A coach for qualitative physics essay writing. In International conference on intelligent tuto… find
  67. Shu-Hsien Liao. Expert system methodologies and applications—a decade review from 1995 to 2004. Expert systems with applications, 28(1):93–103, 2005. find
  68. Hyacinth S Nwana. Intelligent tutoring systems: An overview. Artificial Intelligence Review, 4 (4):251–277, 1990. find
  69. S Mcroy and R Freedman. What is an intelligent tutoring system. Intelligence, 11(3):15–16, 2000. find
  70. Roger Nkambou, Riichiro Mizoguchi, and Jacqueline Bourdeau. Advances in intelligent tutoring systems, volume 308. Springer Science & Business Media, 2010. find
  71. Gary Marcus. The next decade in AI: Four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020.
  72. Irina Higgins, Antonia Creswell, and Sebastien Racaniere. Pay attention to what you need: Do structural priors still matter in the age of billion parameter models?, 2021. URL https://neurips.cc/virtual/2021/tutorial/21891 . ↗
  73. Huw C Davies, Rebecca Eynon, and Cory Salveson. The mobilisation of AI in education: A Bourdieusean field analysis. Sociology, 55(3):539–560, 2021. find
  74. Anthony Seldon, Oladimeji Abidoye, and Timothy Metcalf. The Fourth Education Revolution Reconsidered: Will Artificial Intelligence Enrich Or Diminish Humanity? Legend Press Ltd, 2020. find
  75. Brett Becker. Artificial intelligence in education: What is it, where is it now, where is it going. Ireland’s Yearbook of Education, 2018:42–46, 2017. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 52 find
  76. Olaf Zawacki-Richter, Victoria I Marín, Melissa Bond, and Franziska Gouverneur. Systematic review of research on artificial intelligence applications in higher education–where are the educators? International Journal of Educational Technology in Higher Education, 16(1):1–27, 2019. find
  77. Tuomi Ilkka. The impact of artificial intelligence on learning, teaching, and education. European Union, 2018. find
  78. James A Kulik and John D Fletcher. Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of educational research, 86(1):42–78, 2016. find
  79. Sebastian Wollny, Jan Schneider, Daniele Di Mitri, Joshua Weidlich, Marc Rittberger, and Hendrik Drachsler. Are we there yet? - a systematic literature review on chatbots in education. Frontiers in artificial intelligence, 4:654924, 2021. find
  80. Chinedu Wilfred Okonkwo and Abejide Ade-Ibijola. Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2:100033, 2021. find
  81. Arif Iqbal, Reinhard Oppermann, Ashok Patel, and Kinshuk. A classification of evaluation methods for intelligent tutoring systems. Software-Ergonomie’99: Design von Informationswelten, pages 169–181, 1999. find
  82. Julika Siemer and Marios C Angelides. A comprehensive method for the evaluation of complete intelligent tutoring systems. Decision support systems, 22(1):85–102, 1998. find
  83. Mary A Mark, Jim E Greer, et al. Evaluation methodologies for intelligent tutoring systems. Journal of Artificial Intelligence in Education, 4:129–129, 1993. find
  84. Martha C Polson and J Jeffrey Richardson. Foundations of intelligent tutoring systems. Psychology Press, 2013. find
  85. Tanya Nazaretsky, Mutlu Cukurova, and Giora Alexandron. An instrument for measuring teachers’ trust in AI-based educational technology. In LAK22: 12th international learning analytics and knowledge conference, pages 56–66, 2022. find
  86. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  87. Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019. find
  88. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  89. Justin Vasselli, Christopher Vasselli, Adam Nohejl, and Taro Watanabe. NAISTeacher: A prompt and rerank approach to generating teacher utterances in educational dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 772–784, 20… find
  90. Rania Abdelghani, Hélène Sauzéon, and Pierre-Yves Oudeyer. Generative AI in the classroom: Can students remain active learners? arXiv preprint arXiv:2310.03192, 2023. — [arXiv:2310.03192](https://arxiv.org/abs/2310.03192) find
  91. Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. Evaluating language models for mathematics through interactions. arXiv preprint arXiv:2306.01694, 2023. Towards Responsible Development of…
  92. Changyoon Lee, Junho Myung, Jieun Han, Jiho Jin, and Alice Oh. Learning from teaching assistants to program with subgoals: Exploring the potential for AI teaching assistants. arXiv preprint arXiv:2309.10419, 2023. — [arXiv:2309.10419](https://arxiv.org/abs/2309.10419) find
  93. Yu Li, Shang Qu, Jili Shen, Shangchao Min, and Zhou Yu. Curriculum-driven Edubot: A framework for developing language learning chatbots through synthesizing conversational data. arXiv preprint arXiv:2309.16804, 2023. find
  94. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. MathDial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. arXiv preprint arXiv:2305.14536, 2023.
  95. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and individual differences, 103…
  96. Rose E Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. Step-by-step remediation of students’ mathematical mistakes. arXiv preprint arXiv:2310.10648, 2023. find
  97. Blake Castleman and Mehmet Kerem Turkcan. Examining the influence of varied levels of domain knowledge base inclusion in GPT-based intelligent tutors. arXiv preprint arXiv:2309.12367, 2023. — [arXiv:2309.12367](https://arxiv.org/abs/2309.12367) find
  98. Anaïs Tack and Chris Piech. The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. arXiv preprint arXiv:2205.07540, 2022.
  99. Anaïs Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw, and Chris Piech. The BEA 2023 shared task on generating AI teacher responses in educational dialogues. arXiv preprint arXiv:2306.06941, 2023.
  100. Yann Hicke, Abhishek Masand, Wentao Guo, and Tushaar Gangavarapu. Assessing the efficacy of large language models in generating accurate teacher responses. arXiv preprint arXiv:2307.04274, 2023. find
  101. Rania Abdelghani, Yen-Hsiang Wang, Xingdi Yuan, Tong Wang, Pauline Lucas, Hélène Sauzéon, and Pierre-Yves Oudeyer. GPT-3-driven pedagogical agents for training children’s curious question-asking skills. International Journal of Artificial Intelligence in Education, pages 1–36, 2023. find
  102. Harsh Kumar, David M Rothschild, Daniel G Goldstein, and Jake M Hofman. Math education with large language models: Peril or promise? Available at SSRN 4641653, 2023. find
  103. Erfan Al-Hossami, Razvan Bunescu, Justin Smith, and Ryan Teehan. Can language models employ the Socratic method? Experiments with code debugging. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pages 53–59, 2024.
  104. Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodríguez Fanlo, Simon Frieder, Simon Machado, et al. Language models as science tutors. arXiv preprint arXiv:2402.11111, 2024.
  105. Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J Malan. Teaching CS50 with AI: Leveraging generative artificial intelligence in computer Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 54 science education. In Proceedi… find
  106. Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. Opportunities and challenges in neural dialog tutoring. arXiv preprint arXiv:2301.09919, 2023.
  107. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  108. Paul Denny, Sumit Gulwani, Neil T Heffernan, Tanja Käser, Steven Moore, Anna N Rafferty, and Adish Singla. Generative AI for education (GAIED): Advances, opportunities, and challenges. arXiv preprint arXiv:2402.01580, 2024. — [arXiv:2402.01580](https://arxiv.org/abs/2402.01580) find
  109. Ethan R Mollick and Lilach Mollick. Instructors as innovators: A future-focused approach to new AI learning opportunities, with prompts. With Prompts (April 22, 2024), 2024. find
  110. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… — PaLM 2 Technical Report
  111. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  112. Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibau… — Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  113. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. find
  114. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583–589, 2021.
  115. Katherine Stasaski, Kimberly Kao, and Marti A Hearst. CIMA: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–64, 2020.
  116. Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. The teacher-student chatroom corpus. arXiv preprint arXiv:2011.07109, 2020.
  117. Abhijit Suresh, Jennifer Jacobs, Margaret Perkoff, James H Martin, and Tamara Sumner. Fine-tuning transformers with additional context to classify discursive moves in mathematics classrooms. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications, 2022.
  118. Dorottya Demszky and Heather Hill. The NCTE transcripts: A dataset of elementary math classroom transcripts. arXiv preprint arXiv:2211.11772, 2022.
  119. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  120. Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Y Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. Dialog inpainting: Turning documents into dialogs. In International conference on machine learning, pages 4558–4586. PMLR, 2022. find
  121. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  122. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. NeurIPS, 2021.
  123. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 56 find
  124. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [1] Samuel Gehman, Suchin Gururangan, Maarten S…
  125. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021.
  126. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  127. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675, 2019. find
  128. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  129. Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. Dialogue response ranking training with large-scale human feedback data. arXiv preprint arXiv:2009.06978, 2020. find
  130. Judith D Wilson. A Socratic approach to helping novice programmers debug programs. ACM SIGCSE Bulletin, 19(1):179–182, 1987. find
  131. Alexis Baladón, Ignacio Sastre, Luis Chiruzzo, and Aiala Rosá. RETUYT-InCo at BEA 2023 shared task: Tuning open-source LLMs for generating teacher responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 756–765, 2023. find
  132. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022.
  133. Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746, 2022. find
  134. Hua Shen and Tongshuang Wu. Parachute: Evaluating interactive human-LM co-writing systems. arXiv preprint arXiv:2303.06333, 2023. — [arXiv:2303.06333](https://arxiv.org/abs/2303.06333) find
  135. Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. Measuring conversational uptake: A case study on student-teacher interactions. arXiv preprint arXiv:2106.03873, 2021.
  136. Michelene TH Chi and Ruth Wylie. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational psychologist, 49(4):219–243, 2014.
  137. Kurt VanLehn, Stephanie Siler, Charles Murray, and William B Baggett. What makes a tutorial event effective? In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society. Citeseer, 1999. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approa… find
  138. Yana Weinstein, Megan Sumeracki, and Oliver Caviglioli. Understanding How We Learn: A Visual Guide. Routledge, 2019. [1] Barbara A. Oakley, Beth Rogowsky, and Terrence J. Sejnowski. Uncommon sense teaching: Practical insights in brain science to help students learn. Perigee Books, 2021. find
  139. Richard E. Mayer. Multimedia learning. Cambridge University Press, 2022. find
  140. Stanislas Dehaene. How we learn why brains learn better than any machine ... for now. Penguin Books, 2021. find
  141. Richard K. Cohen. The metacognitive student: How to teach academic, social, and emotional intelligence in every content area. Hawker Brownlow Education, 2022. find
  142. Emily R Lai. Metacognition: A literature review. Always learning: Pearson research report, 24: 1–40, 2011. find
  143. John M Keller. Development and use of the ARCS model of instructional design. Journal of instructional development, 10(3):2–10, 1987. find
  144. Erika A Patall, Harris Cooper, and Jorgianne Civey Robinson. The effects of choice on intrinsic motivation and related outcomes: A meta-analysis of research findings. Psychological bulletin, 134(2):270, 2008. find
  145. 2017. URL https://tech.ed.gov/files/2017/01/NETP17.pdf . ↗
  146. Peter C Brown, Henry L Roediger III, and Mark A McDaniel. Make it stick: The science of successful learning. Harvard University Press, 2014. find
  147. Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39):19251–19257, 2019. find
  148. William Agnew, A Stevie Bergman, Jennifer Chien, Mark Díaz, Seliem El-Sayed, Jaylen Pittman, Shakir Mohamed, and Kevin R McKee. The illusion of artificial inclusion. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024. find
  149. Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020. find
  150. David N Chin. Empirical evaluation of user models and user-adapted systems. User modeling and user-adapted interaction, 11:181–194, 2001. find
  151. George EP Box. Science and statistics. Journal of the American Statistical Association, 71(356): 791–799, 1976. find
  152. Kevin R McKee. Human participants in AI research: Ethics and transparency in practice. arXiv preprint arXiv:2311.01254, 2023. — [arXiv:2311.01254](https://arxiv.org/abs/2311.01254) find
  153. DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502–14515, 2021. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 58 find
  154. Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. CritiqueLLM: Scaling LLM-as-critic for effective and explainable evaluation of large language model generation. arXiv preprint arXiv:2311.18702, 2023.
  155. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685, 2023.
  156. Michael Quinn Patton. Qualitative research & evaluation methods: Integrating theory and practice. Sage Publications, 2014. find
  157. B. S. Bloom, M. B. Engelhart, E. J. Furst, W. H. Hill, and D. R. Krathwohl. Taxonomy of educational objectives. The classification of educational goals. Handbook 1: Cognitive domain. Longmans Green, New York, 1956. find
  158. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  159. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
  160. Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. GeomVerse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023. — [arXiv:2312.12241](https://arxiv.org/abs/2312.12241) find
  161. Kristen DiCerbo. Implementation of AI tools in education at scale, 2023. URL https://neurips.cc/virtual/2023/81332 . ↗
  162. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023.
  163. Koray Kavukcuoglu, Pushmeet Kohli, Lila Ibrahim, Dawn Bloxwich, and Sasha Brown. How our principles helped define AlphaFold’s release, 2022. URL https://deepmind.google/ discover/blog/how-our-principles-helped-define-alphafolds-release/ . ↗
  164. Google. AI at Google: Our principles, 2023. URL https://ai.google/responsibility/ principles/ . ↗
  165. Wiebe Bijker, T Hughes, and Trevor Pinch. The social construction of technology systems. Massachusetts Institute of Technology, 1987. find
  166. Deborah G Johnson and Jameson M Wetmore. Technology and society: Building our sociotechnical future. MIT press, 2021. find
  167. UNESCO. World teachers’ day: UNESCO sounds the alarm on the global teacher shortage crisis, 2022. URL https://www.unesco.org/en/articles/ world-teachers-day-unesco-sounds-alarm-global-teacher-shortage-crisis . Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Appro… ↗
  168. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023.
  169. Peter Henderson, Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, and Prateek Mittal. Safety risks from customizing foundation models via fine-tuning, 2024. find
  170. Google. 2021 AI principles progress update, 2021. URL https://ai.google/static/ documents/ai-principles-2021-progress-update.pdf . ↗
  171. Nicholas Epley, Adam Waytz, and John T Cacioppo. On seeing human: A three-factor theory of anthropomorphism. Psychological review, 114(4):864, 2007. find
  172. Eileen Roesler, Dietrich Manzey, and Linda Onnasch. A meta-analysis on the effectiveness of anthropomorphism in human-robot interaction. Science Robotics, 6(58):eabj5425, 2021. find
  173. Andrew Gambino, Jesse Fox, and Rabindra A Ratan. Building a stronger CASA: Extending the computers are social actors paradigm. Human-Machine Communication, 1:71–85, 2020. find
  174. Katja Wagner, Frederic Nimmermann, and Hanna Schramm-Klein. Is it human? The role of anthropomorphism as a driver for the successful acceptance of digital voice assistants. In Hawaii International Conference on System Sciences, 2019. find
  175. Abbe Don, Susan Brennan, Brenda Laurel, and Ben Shneiderman. Anthropomorphism: From ELIZA to Terminator 2. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 67–70, 1992. find
  176. Arleen Salles, Kathinka Evers, and Michele Farisco. Anthropomorphism in AI. AJOB neuroscience, 11(2):88–95, 2020. find
  177. Gavin Abercrombie, Amanda Cercas Curry, Tanvi Dinkar, and Zeerak Talat. Mirages: On anthropomorphism in dialogue systems. arXiv preprint arXiv:2305.09800, 2023. find
  178. Alexandra D Kaplan, Theresa T Kessler, J Christopher Brill, and Peter A Hancock. Trust in artificial intelligence: Meta-analytic findings. Human factors, 65(2):337–359, 2023. find
  179. Markus Blut, Cheng Wang, Nancy V Wünderlich, and Christian Brock. Understanding anthropomorphism in service provision: A meta-analysis of physical robots, chatbots, and other AI. Journal of the Academy of Marketing Science, 49:632–658, 2021. find
  180. Xinge Li and Yongjun Sung. Anthropomorphism brings us closer: The mediating role of psychological distance in User–AI assistant interactions. Computers in Human Behavior, 118: 106680, 2021. find
  181. Corina Pelau, Dan-Cristian Dabija, and Irina Ene. What makes an AI device human-like? The role of interaction quality, empathy and perceived psychological anthropomorphic characteristics in the acceptance of artificial intelligence in the service industry. Computers in Human Behavior, 122:106855, 20… find
  182. Jenny Van Doorn, Martin Mende, Stephanie M Noble, John Hulland, Amy L Ostrom, Dhruv Grewal, and J Andrew Petersen. Domo arigato Mr. Roboto: Emergence of automated social presence in organizational frontlines and customers’ service experiences. Journal of service research, 20(1):43–58, 2017. Towards… find
  183. Ben Sheehan, Hyun Seung Jin, and Udo Gottlieb. Customer service chatbots: Anthropomorphism and adoption. Journal of Business Research, 115:14–24, 2020. find
  184. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. find
  185. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find
  186. Norman Jouppi, Doe Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63:67–78, 06 2020. doi: 10.1145/3360307. — [doi:10.1145/3360307](https://doi.org/10.1145/3360307) find
  187. Norman P. Jouppi, George Kurian, Sheng Li, Peter C. Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiaoping Zhou, Zongwei Zhou, and David A. Patterson. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware su… — TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
  188. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax . ↗
  189. Jeff Dean. tecture, 2021. Introducing Pathways: A next-generation AI archi-URL https://blog.google/technology/ai/ introducing-pathways-next-generation-ai-architecture/ . ↗
  190. Virginia Braun and Victoria Clarke. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101, 2006. find
  191. Wayne Holmes, Stamatina Anastopoulou, Heike Schaumburg, and Manolis Mavrikis. Technology-enhanced personalised learning: Untangling the evidence, September 2018. URL http://www.studie-personalisiertes-lernen.de/en/ . ↗
  192. Greg Thompson and Ian Cook. The logic of data-sense: Thinking through learning personali-sation. In The education assemblage, pages 81–95. Routledge, 2020. find
  193. Lev S Vygotsky. Mind in society: The development of higher psychological processes, volume 86. Harvard university press, 1978. find
  194. Vincent Aleven, Ido Roll, Bruce M McLaren, and Kenneth R Koedinger. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26:205–223, 2016. find
  195. Sidney D’Mello and Art Graesser. Dynamics of affective states during complex learning. Learning and Instruction, 22(2):145–157, 2012. find
  196. Ran Zhao, Alexandros Papangelis, and Justine Cassell. Towards a dyadic computational model of rapport management for human-virtual agent interaction. In Intelligent Virtual Agents: 14th Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 61 International Con… find
  197. Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1): 973–1018, 2023. find
  198. Carole R Beal, Ivon M Arroyo, Paul R Cohen, and Beverly P Woolf. Evaluation of AnimalWatch: An intelligent tutoring system for arithmetic and fractions. Journal of Interactive Online Learning, 9(1), 2010. find
  199. Janice D Gobert, Raha Moussavi, Haiying Li, Michael Sao Pedro, and Rachel Dickler. Real-time scaffolding of students’ online data interpretation during inquiry with Inq-ITS using educational data mining. Cyber-physical laboratories in engineering and science education, pages 191–217, 2018. find
  200. Michael Mendicino, Leena Razzaq, and Neil T Heffernan. A comparison of traditional homework to computer-supported homework. Journal of Research on Technology in Education, 41(3): 331–359, 2009. find
  201. Kurt VanLehn, Collin Lynch, Kay Schulze, Joel A Shapiro, Robert Shelby, Linwood Taylor, Don Treacy, Anders Weinstein, and Mary Wintersgill. The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence in Education, 15(3):147–204, 2005. find
  202. Wenting Ma, Olusola O Adesope, John C Nesbit, and Qing Liu. Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of educational psychology, 106(4):901, 2014. find
  203. Arthur C Graesser, Natalie K Person, and Joseph P Magliano. Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied cognitive psychology, 9(6):495–522, 1995. find
  204. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023.
  205. George Loewenstein. The psychology of curiosity: A review and reinterpretation. Psychological bulletin, 116(1):75, 1994. find
  206. Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794, 2023. — [arXiv:2309.06794](https://arxiv.org/abs/2309.06794) find
  207. Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024.
  208. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. find
  209. Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological bulletin, 52(4):281, 1955. find
  210. Lee Anna Clark and David Watson. Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 1995. Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach 62 find
  211. Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. FaithDial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473–1490, 2022. [1] Inigo Casanueva, Ivan Vulić, Georgios P Spithourak…
  212. Eyal Peer, David Rothschild, Andrew Gordon, Zak Evernden, and Ekaterina Damer. Data quality of platforms and panels for online behavioral research. Behavior research methods, pages 1–20, 2021. find
  213. Klaus Krippendorff. Content analysis: An introduction to its methodology. Sage publications, 2018. find
  214. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022.
  215. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — Language Models are Few-Shot Learners
  216. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903, 2022. URL https://api.semanticscholar.org/CorpusID: 246411621 . ↗
  217. Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020. \#\# Acknowledgements This work was done as part of the LearnLM effort, which is a cross-Google project, with members from Google DeepMind (GDM), Google Research (GR),… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jurenka-2024-responsible-development-generative-ai-education/