Elamur Research⌕
☰
home / papers / 2025 / TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation

TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation

Paper · 2025 · Lin Sun, Guangxiang Zhao, Xiaoqi Jian et al.

Summary

Compact reasoning models are usually built by fine-tuning a smaller model on a mix of reasoning traces from a much larger teacher. Choosing that mix is slow trial and error, and training on several domains at once can make their gradients clash, so a gain in one area often costs another. The authors propose Branch-Merge distillation to avoid this. In the Branch phase, separate copies of DeepSeek-R1-Distill-Qwen-32B are fine-tuned with SFT on math, coding and science data, producing three domain experts. In the Merge phase, the experts are combined with Arcee Fusion. This method adopts a parameter from the incoming model only when its KL-based importance score passes a threshold set from the median and interquartile range of those scores. The merged model, TinyR1-32B-Preview, beats its backbone on all three benchmark domains and nearly matches DeepSeek-R1 on AIME 2024. Merging costs a small fraction of retraining on mixed data. The work matters because it gives a cheap, fully open route to strong mid-sized reasoning models that can run locally and can serve as a starting point for RL.

Key points

  • Training data:
    • Math: 58k samples selected from 94k NuminaMath1.5 questions, paired with OpenR1 solutions.
    • Coding: 20k trajectories filtered from OpenThoughts.
    • Science: 8.6k DeepSeek-R1 CoT trajectories generated from seeds in S1, S1k and the science subset of OpenThoughts.
  • Results (pass@1) versus the backbone:
    • AIME 2024: 78.1 vs. 72.6 (DeepSeek-R1 scores 79.8).
    • LiveCodeBench 24.08–25.02: 61.6 vs. 57.2.
    • GPQA-Diamond: 65.0 vs. 62.1.
    • Trade-off: answers are longer than DeepSeek-R1's, by roughly 19–62% more tokens depending on the domain.
  • Ablation:
    • A single model trained on the combined data scored 75.3 / 61.0 / 65.7 (math / coding / science).
    • Its coding score fell below the coding expert's 63.4, showing the domain "seesaw" effect that merging largely avoids.
    • The order in which experts are merged changed results only slightly.
  • Cost:
    • Merging took about 4 GPU hours, versus about 740 for mixed-data SFT (roughly 0.5% of the compute).
    • The authors estimate full reproduction at 744 H800 GPU hours, or about $1500.
  • Openness and limitations:
    • The model is released on Hugging Face as qihoo360/TinyR1-32B-Preview.
    • Training data, training scripts, merging code and evaluation code are on GitHub (Qihoo360/TinyR1-32B-Preview).
    • Codeforces and SWE-bench were not evaluated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Qilong Shi, Change Jia, Aomufei Yuan, Yuxuan Tian, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Sai-er Hu, Zihan Jiang, Junting Zhou, Wenrui Liu, Xusen Xiao, Bin Cui, Tong Yang and Xiangzheng Zhang. 2025. TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation. arXiv:2503.04872.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.04872 · arXiv 2503.04872

@misc{sun2025tinyr1,
  title = {{TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillation}},
  author = {Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Qilong Shi, Change Jia, Aomufei Yuan, Yuxuan Tian, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Sai-er Hu, Zihan Jiang, Junting Zhou, Wenrui Liu, Xusen Xiao, Bin Cui, Tong Yang and Xiangzheng Zhang},
  year = {2025},
  doi = {10.48550/arxiv.2503.04872},
  eprint = {2503.04872},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.04872}
}

References

70 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. 2021. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pages 769–779. PMLR. find
  2. Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  3. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. Preprint, arXiv:2401.01335. find
  4. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  5. Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044. — [arXiv:2204.03044](https://arxiv.org/abs/2204.03044) find
  6. Wojciech Marian Czarnecki, Simon Osindero, Razvan Pascanu, and Max Jaderberg. 2019. A deep neural network’s loss surface contains every low-dimensional pattern. arXiv preprint arXiv:1912.07559. — [arXiv:1912.07559](https://arxiv.org/abs/1912.07559) find
  7. Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. 2023. Model merging by uncertainty-based gradient matching. arXiv preprint arXiv:2310.12808. — [arXiv:2310.12808](https://arxiv.org/abs/2310.12808) find
  8. MohammadReza Davari and Eugene Belilovsky. 2024. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision, pages 270–287. Springer. find
  9. J. Dean and et al. 2018. Scaling neural networks with efficient memory and batching. In ICML. find
  10. Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. 2024. Della-merging: Reducing interference in model merging through magnitude-based sampling. arXiv preprint arXiv:2406.11617. — [arXiv:2406.11617](https://arxiv.org/abs/2406.11617) find
  11. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  12. Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. 2018. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309–1318. PMLR. find
  13. Gintare Karolina Dziugaite and Daniel M Roy. 2017. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008. — [arXiv:1703.11008](https://arxiv.org/abs/1703.11008) find
  14. Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. 2021. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296. — [arXiv:2110.06296](https://arxiv.org/abs/2110.06296) find
  15. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988, Online. Association for Computational Lin…
  16. Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. 2020. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Sys… find
  17. Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. 2019. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757. — [arXiv:1912.02757](https://arxiv.org/abs/1912.02757) find
  18. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259–3269. PMLR. find
  19. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. 2018. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31. find
  20. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee‘s MergeKit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:…
  21. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819. find
  22. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2019. AutoSeM: Automatic task selection and mixing in multi-task learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers… find
  23. M. R. Henry, A. Y. Shi, and et al. 2019. Memory-efficient batching for neural networks. In NeurIPS. find
  24. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Preprint, arXiv:1503.02531.
  25. Sepp Hochreiter and Jürgen Schmidhuber. 1994. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7. find
  26. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Flat minima. Neural computation, 9(1):1–42. find
  27. Chengming Hu, Xuan Li, Dan Liu, Haolun Wu, Xi Chen, Ju Wang, and Xue Liu. 2023. Teacher-student architecture for knowledge distillation: A survey. Preprint, arXiv:2308.04268. — [arXiv:2308.04268](https://arxiv.org/abs/2308.04268) find
  28. Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. 2022. In-context learning distillation: Transferring few-shot learning ability of pre-trained language models. Preprint, arXiv:2212.10670. — [arXiv:2212.10670](https://arxiv.org/abs/2212.10670) find
  29. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022a. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089.
  30. Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022b. Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262–29277. find
  31. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407.
  32. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. Preprint, arXiv:2406.15513.
  33. Li Jiang, Yusen Wu, Junwu Xiong, Jingqing Ruan, Yichuan Ding, Qingpei Guo, Zujie Wen, Jun Zhou, and Xiaotie Deng. 2024. Hummer: Towards limited competitive preference dataset. Preprint, arXiv:2405.11647. — [arXiv:2405.11647](https://arxiv.org/abs/2405.11647) find
  34. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. Preprint, arXiv:1909.10351.
  35. Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. 2022. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577–16595. find
  36. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836. find
  37. Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. 2019. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems, 32. find
  38. Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31. find
  39. Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. [https://huggingface.co/AI-MO/NuminaMath -1.5](https://gi… find
  40. Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. 2022a. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306. find
  41. Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. 2022b. Explanations from large language models make small reasoners better. Preprint, arXiv:2210.06726. find
  42. L. Liu and A. Yao. 2019. Meta-learning for low-shot neural architecture search. In ICLR. find
  43. Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. 2023. Mechanistic mode connectivity. In International Conference on Machine Learning, pages 22965–23004. PMLR. find
  44. Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703– 17716.
  45. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. Preprint, arXiv:2501.19393.
  46. Victor Mustar. 2025. Tinyr1-32b-preview, the new local king? Accessed: 2025-03-05. find
  47. Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2020. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512–523. find
  48. Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. 2025. Userrl: Training interactive user-centric agent via reinforcement learning. Preprint, arXiv:2509.19736. — [arXiv:2509.19736](https://arxiv.org/abs/2509.19736) find
  49. A. Radford and J. W. 2021. Improving generalization via scalable learning and domain adaptation. In NeurIPS. find
  50. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  51. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and 1 others. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: AC…
  52. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  53. K. Tay, H. Dehghani, L. R. Jones, S. J. Koyejo, and A. S. S. Ahmed. 2020. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 53(4):1– 38.
  54. Open R1 Team. 2025a. Open r1. https://huggingf ace.co/open-r1. find
  55. OpenThoughts Team. 2025b. https://open-thoughts.ai. ↗
  56. Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. 2024. Fusechat: Knowledge fusion of chat models. arXiv preprint arXiv:2408.07990.
  57. Chenyang Wang, Liang Wen, Shousheng Jia, Xiangzheng Zhang, and Liang Xu. 2025. Light-if: Endowing llms with generalizable reasoning via preview and self-checking for complex instruction following. arXiv preprint arXiv:2508.03178. — [arXiv:2508.03178](https://arxiv.org/abs/2508.03178) find
  58. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. Preprint, arXiv:2212.10560. find
  59. Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. 2021. Learning neural network subspaces. In International Conference on Machine Learning, pages 11217–11227. PMLR. find
  60. Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and 1 others. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time…
  61. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. Preprint, arXiv:2304.12244.
  62. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. Preprint, arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) find
  63. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093–7115.
  64. Chuanpeng Yang, Wang Lu, Yao Zhu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. 2024. Survey on knowledge distillation for large language models: Methods, evaluation, and application. Preprint, arXiv:2407.01885. — [arXiv:2407.01885](https://arxiv.org/abs/2407.01885) find
  65. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Preprint, arXiv:2001.06782.
  66. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. Preprint, arXiv:2401.10020. find
  67. L. A. Zamir and E. Arbeláez. 2018. Taskonomy: Disentangling task transfer learning. In CVPR, pages 2647–2656. find
  68. Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Sciinstruct: a self-reflective instruction annotated dataset for training scientific language models. Preprint, arXiv:2401.07950. find
  69. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
  70. Haosheng Zou, Xiaowei Lv, Shousheng Jia, and Xiangzheng Zhang. 2024. 360-llama-factory. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2025-tinyr1-32b-preview-boosting-accuracy/