Large Language Models Are Reasoning Teachers

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.acl-long.830.pdf

Summary

Chain-of-thought (CoT) prompting lets language models solve multi-step reasoning problems, but it only works well in models with hundreds of billions of parameters, which cost too much to run at scale. The authors propose Fine-tune-CoT, which uses a very large model as a "teacher" for much smaller "student" models. A teacher such as GPT-3 175B answers training questions with Zero-shot-CoT prompting. Only rationales that reach the correct final answer are kept, and these become fine-tuning data for the student. The method needs no hand-written rationales or task-specific engineering. The authors also introduce diverse reasoning, which samples several different rationales per question to enlarge the training set. Across 12 tasks and a range of public models, the fine-tuned students far outperform prompt-based baselines. Students as small as 0.3B sometimes beat the 175B teacher. This matters because it suggests CoT reasoning, usually seen as an emergent ability of scale, can be distilled into compact models that are practical to deploy.

Key points

• Pipeline: three steps: (1) generate rationales from the teacher with Zero-shot-CoT ("Let's think step by step"), (2) filter them by whether the final answer is correct, and (3) fine-tune the student on the prompt–rationale–answer samples. Compact special-character delimiters keep sequences short.
• Diverse reasoning: stochastic sampling of multiple rationales per question boosts performance substantially across datasets, at the cost of some extra teacher inference during development and no extra human annotation.
• Scaling: student accuracy rises steadily with more diverse rationales, larger datasets, stronger teachers and larger students.
• Why Zero-shot-CoT for the teacher: Few-shot-CoT is more accurate (e.g., 78.7% → 93.0% on MultiArith), but 8-shot prompts cost about 8× more under per-token pricing. The authors prefer to spend that budget on diverse reasoning.
• Limitations and resources: student rationales can be repetitive or wander off-topic, and results are not state-of-the-art (e.g., on GSM8K). Code and data are available at github.com/itsnamgyu/reasoning-teacher.