Elamur Research⌕
☰
home / papers / 2025 / Small Models Struggle to Learn from Strong Reasoners

Small Models Struggle to Learn from Strong Reasoners

Paper · 2025 · Yuetai Li, Xiang Yue, Zhangchen Xu et al.

Summary

Distilling reasoning traces from powerful LLMs into compact models is commonly assumed to help more as the teacher gets stronger. This paper tests that assumption on mathematical reasoning. The authors fine-tuned ten Qwen2.5 and Llama 3.x instruct models, ranging from 0.5B to 70B parameters, on MATH problems. Each model was trained on one of two contrasting kinds of solutions: lengthy, reflective chain-of-thought versus concise chain-of-thought, or traces from a large teacher versus a small one. Models with roughly 3B parameters or fewer learned better from the shorter or smaller-teacher data. Only larger students profited from the elaborate traces. The authors call this the Small Model Learnability Gap and attribute it to small students coping poorly with distribution shift away from their own outputs. As a remedy, they propose Mix Distillation, which blends the two data types: Mix-Long combines long and short CoT, and Mix-Large combines large- and small-teacher outputs. Both variants beat single-source training on most metrics. The work matters for anyone building efficient small reasoners, because it shows that teacher strength alone does not guarantee better distillation.

Key points

  • Setup: the 7,500-prompt MATH training set was used. Long CoT came from QwQ-32B-Preview and short CoT from Qwen2.5-32B-Instruct. The large/small teacher pairs were Qwen2.5-72B vs 3B, Llama3.1-70B vs 8B, and Gemma2-27B vs 9B.
  • Evaluation: the reported score is the average over MATH, GSM8K, AMC 2023, AIME 2024 and OlympiadBench (English math), using zero-shot greedy decoding with up to 16k tokens. Training used LLaMA-Factory, with full SFT for smaller models and LoRA for those above 14B.
  • Long-vs-short CoT gap: the gap was negative for small students (Qwen2.5-0.5B −4.7, Qwen2.5-1.5B −7.1, Qwen2.5-3B −3.1). It was positive for larger ones (Qwen2.5-32B +13.7, Llama-3.3-70B +3.8).
  • Mix Distillation results: Qwen2.5-3B-Instruct gained over 8 points on MATH and AMC with Mix-Long, compared with long-CoT-only training. It gained over 7 points on MATH, AIME and AMC with Mix-Large, compared with large-teacher-only data.
  • Limitations: the study covers only math reasoning, not code, instruction following or multimodal tasks. Models and data are released at huggingface.co/UWNSL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓University of WashingtonSeattle · United States402 works · 212,762 citations🎓Western Washington UniversityBellingham · United States2 works · 144 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian and Radha Poovendran. 2025. Small Models Struggle to Learn from Strong Reasoners. Findings of the Association for Computational Linguistics: ACL 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.1301 · arXiv 2502.12143

@inproceedings{li2025small,
  title = {{Small Models Struggle to Learn from Strong Reasoners}},
  author = {Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian and Radha Poovendran},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.1301},
  eprint = {2502.12143},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.1301}
}

References

59 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. Preprint, arXiv:2306.13649.
  2. Anthropic. 2023. Introducing claude. find
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794–4802. find
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  6. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, and et al. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  8. Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. Preprint, arXiv:2304.06767.
  9. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819. find
  10. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. 2023. Reinforced self-training (rest) for language modeling. Preprint, a…
  11. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. Pre…
  12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.
  13. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015a. Distilling the knowledge in a neural network. Preprint, arXiv:1503.02531.
  14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015b. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  15. Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. Preprint, arXiv:2212.10071.
  16. Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. 2025. Advancing language model reasoning through reinforcement learning and inference scaling. Preprint, arXiv:2501.11651. find
  17. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  18. Tingfeng Hui, Lulu Zhao, Guanting Dong, Yaqi Zhang, Hua Zhou, and Sen Su. 2024. Smaller language models are better instruction evolvers. Preprint, arXiv:2412.11231. — [arXiv:2412.11231](https://arxiv.org/abs/2412.11231) find
  19. Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. 2020. Lessons learned from the chameleon testbed. In Proceedings of the 2020 US… find
  20. Gyeongman Kim, Doohyuk Jang, and Eunho Yang. 2024a. Promptkd: Distilling student-friendly knowledge for generative language models via prompt tuning. Preprint, arXiv:2402.12842. find
  21. Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, and Graham Neubig. 2024b. Evaluating language models as synthetic data generators. Preprint, arXiv:2412.03679.
  22. Kimi Team. 2025. Kimi k1.5: Scaling reinforcement learning with llms. Preprint, arXiv:2501.12599.
  23. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smi… — Tulu 3: Pushing Frontiers in Open Language Model Post-Training
  24. Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. [https://github.com/project-numina/ aimo-progress-prize](… find
  25. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023. The unlocking spell on base llms: Rethinking alignment via in-context learning. Preprint, arXiv:2312.01552.
  26. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. Preprint, arXiv:2301.13688.
  27. Meta. 2024a. Llama-3.2-3b. https://huggingface.co/meta-llama/Llama-3.2-3B. ↗
  28. Meta. 2024b. Meet llama 3.1. https://llama.meta.com. ↗
  29. Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. 2024. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. Preprint, arXi…
  30. Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2019. Improved knowledge distillation via teacher assistant. Preprint, arXiv:1902.03393.
  31. NovaSky. 2025. Sky-T1: Train your own o1 preview model within $450. Accessed: 2025-01-09. find
  32. OpenAI. 2023. Gpt-4 technical report.
  33. OpenAI. 2024. Learning to reason with llms. find
  34. Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. 2025. Tinyzero. https://github.com/Jiayi-Pan/TinyZero. Accessed: 2025-01-24. ↗
  35. Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. 2023. Openwebmath: An open dataset of high-quality mathematical web text. Preprint, arXiv:2310.06786.
  36. Phuc Phan, Hieu Tran, and Long Phan. 2024. Distillation contrastive decoding: Improving llms reasoning with contrastive decoding and distillation. arXiv preprint arXiv:2402.14874.
  37. Qwen. 2024a. Qwen2.5: A party of foundation models. find
  38. Qwen. 2024b. Qwq: Reflect deeply on the boundaries of the unknown. 2024b. find
  39. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  40. Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simp… — Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
  41. Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. 2024. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. Preprint, arXiv:2407.13690.
  42. Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. 2024. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. Preprint, arXiv:2410.01560.
  43. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprin…
  44. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  45. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. Preprint, arXiv… find
  46. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  47. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024a. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. Preprint, arXiv:2406.08464.
  48. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. 2024b. Stronger models are not stronger teachers for instruction tuning. Preprint, arXiv:2411.07133. — [arXiv:2411.07133](https://arxiv.org/abs/2411.07133) find
  49. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. P…
  50. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. Demystifying long chain-of-thought reasoning in llms. Preprint, arXiv:2502.03373.
  51. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. Metamath: Bootstrap your own mathematical questions for large language models. Preprint, arXiv:2309.12284.
  52. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. Preprint, arXiv:2308.01825.
  53. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mammoth: Building math generalist models through hybrid instruction tuning. Preprint, arXiv:2309.05653.
  54. Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. 2024. Mammoth2: Scaling instructions from the web. Preprint, arXiv:2405.03548.
  55. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Preprint, arXiv:2203.14465.
  56. Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 2025. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/ simplerl-reason. Notion Blog. ↗
  57. Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, and Dawei Song. 2023. Lifting the curse of capacity gap in distilling language models. Preprint, arXiv:2305.12129. find
  58. Dylan Zhang, Qirun Dai, and Hao Peng. 2025. The best instruction-tuning data are those that fit. Preprint, arXiv:2502.04194. — [arXiv:2502.04194](https://arxiv.org/abs/2502.04194) find
  59. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-small-models-struggle-learn-strong/