Elamur Research⌕
☰
home / papers / 2023 / Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions

Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions

Paper · 2023 · Naiming Liu, Shashank Sonkar, Zichao Wang et al.

Summary

Standard math benchmarks for LLMs only check whether a model reaches the right answer. This paper asks whether models understand mathematical concepts well enough to reason about wrong answers. It uses multiple-choice grade-school questions from the Eedi platform, where each distractor is tied to a known student misconception, and defines two tasks:

  • Novice learner: given a misconception, the model must pick the specific incorrect option that a student holding it would choose.
  • Expert tutor: given a wrong answer and a list of candidate misconceptions, the model must identify the one that explains it.

GPT-3.5 and GPT-4 were tested with zero-shot and few-shot prompts that request JSON output. Both models solve the questions themselves with high accuracy, but they perform clearly worse at deliberately making a particular error. They also lose much of their diagnostic accuracy as the number of candidate misconceptions grows. The authors argue that these gaps limit the use of LLMs for realistic student simulation and for intelligent tutoring systems.

Key points

  • Baseline problem solving: GPT-4 reaches 94.8% accuracy when asked to explain before answering, and 77.4% without explanation. GPT-3.5 scores 80.5% and 57.6% respectively.
  • Novice-learner task: GPT-4's best zero-shot result is 61.7%. Few-shot prompting with random examples lowers this to 60.0%. Examples sharing the same misconception raise it to 67.9%, though only 152 items were suitable for that setting.
  • Expert-tutor task: GPT-4 picks the right misconception 91.9% of the time with 4 candidates. Accuracy falls to about 39.8% with 100 candidates.
  • Setup: temperature was 0.9, results are averaged over four independent runs, and model outputs were structured as JSON for automatic scoring.
  • Planned extensions: testing open models such as LLaMA-2 and Vicuna, and improving how LLMs handle misconceptions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rice UniversityHouston · United States11 works · 1,850 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Naiming Liu, Shashank Sonkar, Zichao Wang, Simon Woodhead and Richard G. Baraniuk. 2023. Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions. arXiv:2310.02439.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.02439 · arXiv 2310.02439

@misc{liu2023novice,
  title = {{Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions}},
  author = {Naiming Liu, Shashank Sonkar, Zichao Wang, Simon Woodhead and Richard G. Baraniuk},
  year = {2023},
  doi = {10.48550/arxiv.2310.02439},
  eprint = {2310.02439},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.02439}
}

References

10 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  3. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  4. Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. 2022. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. arXiv preprint arXiv:2204.05660. find
  5. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277.
  6. Shashank Sonkar and Richard G Baraniuk. 2023. Deduction under perturbed evidence: Probing student simulation capabilities of large language models. arXiv preprint arXiv:2305.14507. — [arXiv:2305.14507](https://arxiv.org/abs/2305.14507) find
  7. Shashank Sonkar, MyCo Le, Xinghe Chen, Naiming Liu, Debshila Basu Mallick, and Richard G Baraniuk. 2023a. Code soliloquies for accurate calculations in large language models. arXiv preprint arXiv:2309.12161. find
  8. Shashank Sonkar, Lucy Liu, Debshila Basu Mallick, and Richard G Baraniuk. 2023b. Class meet spock: An education tutoring chatbot based on learning science principles. arXiv preprint arXiv:2305.13272. find
  9. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  10. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-novice-learner-expert-tutor-evaluating/