Elamur Research
☰
home / papers / 2023 / Teaching Small Language Models to Reason

Teaching Small Language Models to Reason

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.acl-short.151.pdf

Summary

Chain-of-thought (CoT) prompting helps very large language models solve multi-step problems. Models below about 10B parameters don't benefit from it, and it often makes them worse. This short paper tests whether knowledge distillation can pass that skill to smaller models. PaLM 540B and GPT-3 175B act as teachers. They get 8-shot prompts that, unlike the original Wei et al. setup, show the correct answer before the example reasoning. With these prompts they write step-by-step rationales for existing training sets, and any rationale that reaches a wrong answer is discarded. T5 students, from small up to XXL, are then finetuned to produce the rationale and the answer straight from the question, so no prompting is needed at inference. Accuracy rises on arithmetic (GSM8K, MAWPS, ASDiv), on commonsense (StrategyQA) and on the coinflip symbolic task. On GSM8K, T5 XXL climbs from 8.11% to 21.99%. The study shows that cheap, deployable models can pick up part of LLM-style reasoning, and that accuracy can be traded against model size and dataset size.

Key points

  • GSM8K, T5 XXL: 8.11% baseline, 21.99% with the PaLM 540B teacher and 18.42% with GPT-3 175B. An external calculator that recomputes each equation raises the PaLM result to 38.21%, compared with 56.9% for 8-shot PaLM itself. On MAWPS the score goes from 54.15% to 70.41%, or 88.22% with the calculator.
  • Ablation: when the gold answer is in the prompt, the share of correct PaLM rationales on GSM8K rises from 59.98% to 79.37%. Most of this comes from fixing steps that were missing or wrong, not from simply copying the answer.
  • Model size: after CoT finetuning, T5 base (220M parameters, 44× smaller) matches the T5 XXL baseline. With the calculator, even T5 small beats that baseline. Dataset size: training on only 20% of GSM8K (1,067 examples) still gives 11.22%.
  • StrategyQA improves only slightly, from 68.12% to 71.98%. The authors attribute this to small models lacking the needed world knowledge. On this dataset, finetuning on the dataset's own gold rationales works best.
  • Symbolic out-of-distribution tests: coinflip with 3 flips jumps from 13.10% to 86.70%, but with 4 flips it drops slightly, from 73.80% to 70.50%. Last letter concatenation stays at 0% for both methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

220 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 207 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-short.151 · arXiv 2212.08410

@inproceedings{magister2023teaching,
  title = {{Teaching Small Language Models to Reason}},
  author = {Magister, Lucie Charlotte and Mallinson, Jonathan and Adamek, Jakub and Malmi, Eric and Severyn, Aliaksei},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.151},
  eprint = {2212.08410},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-short.151}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. BIG-bench collaboration. 2021. Beyond the imitation game: Measuring and extrapolating the capabilities of language models. In preparation. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  3. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. find
  4. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find our card
  6. Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. 2022. Honest students from untrusted teachers: Learning an interpretable question-answering pipeline from a pretrained language model. arXiv preprint arXiv:2210.02498. — [arXiv:2210.02498](https://arxiv.org/abs/2210.02498) find
  7. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462. find our card
  8. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021a. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361. find
  9. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021b. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361. find
  10. Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7). find our card
  11. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071. find our card
  12. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.11610. find our card
  13. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. ACM Computing Surveys. find our card
  14. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157. find
  15. Sarah Kreps, R Miles McCain, and Miles Brundage. 2022. All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation. Journal of Experimental Political Science, 9(1):104– 117. find
  16. Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. 2022. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726. find
  17. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2021. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772. find
  18. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. arXiv preprint arXiv:2104.13346. find our card
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67. find
  20. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions. arXiv preprint arXiv:2212.00193. find our card
  21. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131. find our card
  22. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. find our card
  23. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. find
  24. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. find our card
  25. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270– 280. find