Elamur Research
☰
home / papers / 2023 / Specializing Smaller Language Models towards Multi-Step Reasoning

Specializing Smaller Language Models towards Multi-Step Reasoning

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2301.12726

Summary

Chain-of-thought math reasoning is usually thought to appear only in models above roughly 100B parameters, which puts it out of reach for most researchers. The authors argue that a small model cannot be good at everything, but it can do well on one task if its limited capacity goes there. To test this, they distill step-by-step solutions to GSM8K problems from GPT-3.5 (code-davinci-002) into FlanT5 and T5 students of up to 11B parameters. Teacher and student tokenizers are aligned with dynamic programming, and the training data mixes in-context and zero-shot prompt formats. Checkpoints are chosen by their scores on held-out math datasets, not on GSM8K validation. Specialized FlanT5 models with 250M, 760M and 3B parameters gain about 10 accuracy points on average across four math benchmarks. The cost is a large loss of general ability on BigBench Hard. After specialization, accuracy grows log-linearly with model size, which challenges the idea that CoT is purely emergent. For teams with limited compute, this shows a workable way to build compact, task-specific reasoners.

Key points

  • Setup: the teacher is code-davinci-002 and the students are FlanT5 and T5. Training uses GSM8K only. MultiArith, ASDiv and SVAMP test out-of-distribution generalization, and BigBench Hard (BBH) measures how much general ability is lost.
  • Trade-off: soon after tuning starts, the models lose all of their BBH chain-of-thought prompting ability, and over time a large part of their answer-only BBH ability. Instruction-tuned FlanT5 ends up stronger than specialized raw T5, although T5 improves more relative to its starting point.
  • Model selection: late in training, GSM8K and out-of-distribution scores move independently. Choosing checkpoints on the combined MultiArith/ASDiv/SVAMP set gives better OOD results and costs less on GSM8K.
  • Data format and distillation: training on in-context examples gives both few-shot and zero-shot skill. Training on zero-shot data only gradually removes in-context learning. Matching the teacher's token distributions (KL, top-5 probabilities) converges faster than training on sampled reasoning paths, with similar final accuracy.
  • Data needs: FlanT5 3B levels off at about 90K examples, while the Base model keeps improving through three epochs of about 130K examples each.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

179 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.12726 · arXiv 2301.12726

@misc{fu2023specializing,
  title = {{Specializing Smaller Language Models towards Multi-Step Reasoning}},
  author = {Fu, Yao and Peng, Hao and Ou, Litu and Sabharwal, Ashish and Khot, Tushar},
  year = {2023},
  doi = {10.48550/arxiv.2301.12726},
  eprint = {2301.12726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.12726}
}

References

30 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. find
  2. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. find our card
  3. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  4. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. find
  5. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. find our card
  6. Fu, Y., Peng, H., and Khot, T. How does GPT obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu’s Notion, Dec 2022. URL https://yaofu.notion.site/ b9a57ac0fcf74f30a1ab9e3e36fa1dc1. ↗
  7. Ho, N., Schmid, L., and Yun, S.-Y. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022. find our card
  8. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. find
  9. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022. find our card
  10. Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022. find our card
  11. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. find
  12. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann. find
  13. Li, S., Chen, J., Shen, Y., Chen, Z., Zhang, X., Li, Z., Wang, H., Qian, J., Peng, B., Mao, Y., et al. Explanations from large language models make small reasoners better. arXiv preprint arXiv:2210.06726, 2022. find
  14. Liu, L., Lewis, P., Riedel, S., and Stenetorp, P. Challenges in generalization in open domain question answering. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 2014–2029, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/20… find
  15. Magister, L. C., Mallinson, J., Adamek, J., Malmi, E., and Severyn, A. Teaching small language models to reason. arXiv preprint arXiv:2212.08410, 2022. find our card
  16. Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791–2809, Seattle, United States, July 2022. Associat… find
  17. Needleman, S. B. and Wunsch, C. D. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443–53, 1970. find
  18. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. find
  19. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020. find
  20. Senin, P. Dynamic time warping algorithm review. 2008. find
  21. Shridhar, K., Stolfo, A., and Sachan, M. Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions. arXiv preprint arXiv:2212.00193, 2022. find our card
  22. Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., and Wang, L. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022. find our card
  23. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. find our card
  24. Tan, X., Ren, Y., He, D., Qin, T., and Liu, T.-Y. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=S1gUsoR9YX. ↗
  25. Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Bahri, D., Schuster, T., Zheng, H. S., Houlsby, N., and Metzler, D. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022. find our card
  26. Thakkar, P. Copilot explorer. thakkarparth007.github.io, 2022. URL https://thakkarparth007. github.io/copilot-explorer/posts/ copilot-internals.html. find
  27. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. find our card
  28. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. find our card
  29. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a. URL https://o… find our card
  30. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022b. find our card