Elamur Research⌕
☰
home / papers / 2025 / Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement

Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement

Paper · 2025 · Xuechen Zhang, Zijian Huang, Chenshun Ni et al.

Summary

Small language models distilled from strong reasoners through supervised fine-tuning (SFT) gain reasoning ability, but they tend to waste tokens. The authors show that such models do not recognize when to stop thinking. Their traces become long and repetitive, and this is most visible on questions they answer incorrectly. The paper studies three ways to fix this. The first is a training-free intervention called temperature scaling (TS), which adjusts how likely the model is to sample the end-of-sequence token and so controls where reasoning stops. The second is TLDR, a variant of GRPO that adds a length penalty to the reward. The third is a multi-level version of TLDR, in which the user picks a short, moderate or long budget through the prompt. Evaluated on MATH500, AMC, AIME24 and OlympiadBench, TS gives better accuracy-length trade-offs than s1's budget forcing. TLDR cuts token usage by roughly half with little or no accuracy loss compared with SFT baselines. The work argues that explicitly controlling stopping is key to making small reasoners cheap to run.

Key points

  • Diagnosis: SFT-distilled models (DeepSeek-R1-Distill, s1 family) generate redundant, repetitive traces. Their accuracy can even fluctuate non-monotonically as the test-time budget grows. Wrong answers are noticeably longer than correct ones.
  • TLDR training setup:
    • Base models: DeepSeek-R1-Distill-Qwen-1.5B and 7B.
    • Data: the same training mix as DeepScaleR-1.5B-Preview (MATH, AIME, AMC, STILL, OlympiadBench).
    • Penalty hyperparameters: γ=0.1, β=0.3.
  • Length levels: prompt tags SHORT, MODERATE and LONG correspond to one quarter, one half and the full maximum response length.
  • Example results:
    • Qwen2.5-Math-1.5B rises from 23.4 to 71.4 accuracy, while its average length drops from about 1977 to about 496 tokens.
    • DeepSeek-R1-Distill-Qwen-1.5B keeps accuracy (80.6 vs 80.4) while shrinking from about 5770 to about 1105 tokens.
  • Comparisons: TLDR outperforms prompting, budget forcing and s1 on both accuracy and efficiency. It roughly matches L1 under a maximum-length constraint despite starting from a weaker base model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Xuechen Zhang, Zijian Huang, Chenshun Ni, Ziyang Xiong, Jiasi Chen and Samet Oymak. 2025. Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement. arXiv:2505.07961.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.07961 · arXiv 2505.07961

@misc{zhang2025making,
  title = {{Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement}},
  author = {Xuechen Zhang, Zijian Huang, Chenshun Ni, Ziyang Xiong, Jiasi Chen and Samet Oymak},
  year = {2025},
  doi = {10.48550/arxiv.2505.07961},
  eprint = {2505.07961},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.07961}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697, 2025.
  2. Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning. arXiv preprint arXiv:2411.07279, 2024.
  3. Bradley Butcher, Michael O’Keefe, and James Titchener. Precise length control in large language models. arXiv preprint arXiv:2412.11937, 2024. — [arXiv:2412.11937](https://arxiv.org/abs/2412.11937) find
  4. Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023.
  5. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. find
  6. Google. Gemini 2.0 flash thinking mode (gemini-2.0f lash-thinking-exp-1219), 2024. https://cloud.google.com/vertex-ai/generative-ai/docs/thinking. ↗
  7. Halil Alperen Gozeten, M Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi, Marco Mondelli, and Samet Oymak. Test-time training provably improves transformers as in-context learners. arXiv preprint arXiv:2503.11842, 2025. — [arXiv:2503.11842](https://arxiv.org/abs/2503.11842) find
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  9. Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. arXiv preprint arXiv:2404.10136, 2024. — [arXiv:2404.10136](https://arxiv.org/abs/2404.10136) find
  10. Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296, 2025.
  11. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  12. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274– 19286. PMLR, 2023. find
  13. Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Autobalance: Optimized loss functions for imbalanced data. Advances in Neural Information Processing Systems, 34:3163–3177, 2021. find
  14. Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143, 2025.
  15. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. — Let's Verify Step by Step
  16. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100, 2024. find
  17. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/ DeepScaleR-Surpassing-O1-P… ↗
  18. Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. ICLR, 2021. find
  19. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
  20. OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ ∼ . ↗
  21. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
  22. Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502.12118, 2025.
  23. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. find
  24. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
  25. Yuhui Xu, Hanze Dong, Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. arXiv preprint arXiv:2505.05315, 2025. — [arXiv:2505.05315](https://arxiv.org/abs/2505.05315) find
  26. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. find
  27. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024.
  28. Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test-time compute for llm reasoning. arXiv preprint arXiv:2502.18080, 2025.
  29. Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. Following length constraints in instructions. arXiv preprint arXiv:2406.17744, 2024.
  30. Xuechen Zhang, Xiangyu Chang, Mingchen Li, Amit Roy-Chowdhury, Jiasi Chen, and Samet Oymak. Selective attention: Enhancing transformer through principled context control. Advances in Neural Information Processing Systems, 37:11061–11086, 2024. find
  31. Xuechen Zhang, Zijian Huang, Ege Onur Taga, Carlee Joe-Wong, Samet Oymak, and Jiasi Chen. Efficient contextual LLM cascades through budget-constrained policy learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. find
  32. Xuechen Zhang, Mingchen Li, Jiasi Chen, Christos Thrampoulidis, and Samet Oymak. Class-attribute priors: adapting optimization to heterogeneity and fairness objective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16890–16898, 2024. *Figure 10: The length distrib… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-making-small-language-models-efficient/