Elamur Research⌕
☰
home / papers / 2025 / Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models

Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models

Paper · 2025 · Huifeng Yin, Yu Zhao, Minghao Wu et al.

Summary

Small models are often taught to reason by fine-tuning them on long chain-of-thought (CoT) traces from large reasoning models such as DeepSeek-R1. This is cheap but flawed. The authors show that the student models copy the style of reasoning without its logic. They loop, repeat text, and keep second-guessing until they produce no final answer. The paper calls this "formalistic long-time thinking" and traces it to learning difficulty and inherited bias.

To avoid it, the team skips LRM traces entirely. They generate tree-structured CoT data from scratch with Monte Carlo Tree Search. Each node has a fixed role, and general-purpose LLMs write its content. On top of this data they apply CoT-aware post-training: balancing thought lengths, fine-grained DPO variants, and a combined DPO+SFT loss.

They evaluate on math (GSM8K, MATH, AIME), multilingual instruction following (Multi-IF), and planning (Blocksworld). The resulting models beat standard distilled baselines, mostly because far fewer outputs end without an answer. The work offers a practical way to build compact reasoners without relying on a huge teacher.

Key points

  • Node roles. The search tree uses seven predefined thought-node types: Sub-Task, Thinking, Reflection, Hypothesis, Double-Check, Reclarify, and Answer. Correctness checks at Answer nodes, scored by rules, provide rewards that are backpropagated through the tree.
  • Multi-model coordination. Qwen2.5-72B-Instruct writes the Thinking steps, while a different model, Llama3.1-70B-Instruct, handles Reflection. When an error is detected, the search backtracks to an earlier node so that the second model can correct it.
  • Post-training methods. Fine-grained DPO covers conservative DPO (cDPO) and a mask-based DPO. In the joint objective L = L_DPO + α·L_SFT, the setting α = 1 worked best. Smaller weights still caused forgetting, and larger ones weakened preference alignment.
  • Benchmarks. Multi-IF covers eight languages; results are reported for Chinese, English, and the remaining languages separately.
  • MCTS at inference (MATH). The best model reaches Test@1 of 51.0, compared with 47.0 for Llama-3.1-8B-Instruct. Adding MCTS decoding lifts Test@32 from 79.2 to 82.8. Code and project page: github.com/AIDC-AI/Marco-o1.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEfficient models & compression · 242 works

Cite

Cite as Huifeng Yin, Yu Zhao, Minghao Wu, Xuanfan Ni, Bo Zeng, Hao Wang, Tianqi Shi, Liangying Shao, Chenyang Lyu, Longyue Wang, Weihua Luo and Kaifu Zhang. 2025. Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models. arXiv:2503.01461.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.01461 · arXiv 2503.01461

@misc{yin2025marco,
  title = {{Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models}},
  author = {Huifeng Yin, Yu Zhao, Minghao Wu, Xuanfan Ni, Bo Zeng, Hao Wang, Tianqi Shi, Liangying Shao, Chenyang Lyu, Longyue Wang, Weihua Luo and Kaifu Zhang},
  year = {2025},
  doi = {10.48550/arxiv.2503.01461},
  eprint = {2503.01461},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.01461}
}

References

38 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43… find
  2. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187.
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021a. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021b. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  5. DeepSeek-AI, Daya Guo, and et al. 2025. Deepseekr1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  6. Heshan Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, and Tianyi Chen. 2025. Mitigating forgetting in llm supervised fine-tuning and preference learning. Preprint, arXiv:2410.15483. — [arXiv:2410.15483](https://arxiv.org/abs/2410.15483) find
  7. Heshan Devaka Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, and Tianyi Chen. 2024. Mitigating forgetting in LLM supervised fine-tuning and preference learning. CoRR, abs/2410.15483. — [arXiv:2410.15483](https://arxiv.org/abs/2410.15483) find
  8. Y. Fu, L. Zhang, and M. Liu. 2023a. Specializing large language models for reasoning tasks. International Journal of AI, 56:134–145. find
  9. Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023b. Specializing smaller language models towards multi-step reasoning. Preprint, arXiv:2301.12726.
  10. Z. Guo, T. Shi, and K. Zhang. 2025. Deepseek-r1: Scaling test-time compute for enhanced reasoning. Machine Learning Journal, 72:25–45. find
  11. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. 2024. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410.15553. — [arXiv:2410.15553](https://arxiv.org/abs/2410.15553) find
  12. Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. 2024a. O1 replication journey – part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? Preprint, arXiv:2411.16489. find
  13. Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. 2024b. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489. find
  14. Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. 2025a. Small models struggle to learn from strong reasoners. Preprint, arXiv:2502.12143. find
  15. Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, and Min Zhang. 2025b. Pe… — [arXiv:2505.04921](https://arxiv.org/abs/2505.04921) find
  16. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  17. Sinuo Liu, Chenyang Lyu, Minghao Wu, Longyue Wang, Weihua Luo, Kaifu Zhang, and Zifu Shang. 2025. New trends for modern machine translation with large reasoning models. arXiv preprint arXiv:2503.10351.
  18. Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, and Xunliang Cai. 2024. Length desensitization in direct preference optimization. Preprint, arXiv:2409.06411. — [arXiv:2409.06411](https://arxiv.org/abs/2409.06411) find
  19. Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. 2024. Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence. Preprint, arXiv:2406.10957. find
  20. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024. Improve mathematical reasoning in language models by automated process supervision. Preprint, arXiv:2406.06592.
  21. Eric Mitchell. 2023. A note on dpo with noisy preferences & relationship to ipo. find
  22. OpenAI. 2024. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/. [Accessed 19-09-2024]. ↗
  23. Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. Preprint, arXiv:2408.06195. find
  24. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 2024. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982. find
  25. Qwen Team. 2024. Qwq: Reflect deeply on the boundaries of the unknown. find
  26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.
  27. Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. Preprint, arXiv:2404.12253. find
  28. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Neural Information Processing Systems.
  29. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. 2024a. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Preprint, arXiv:2312.08935.
  30. Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025a. Thoughts are all over the place: On the underthinking of o1-like llms. Preprint, arXiv:2501.18585. find
  31. Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. 2025b. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585. find
  32. Zhichao Wang, Bin Bi, Zixu Zhu, Xiang-Bo Mao, Jun Wang, and Shiyu Wang. 2024b. UFT: unifying fine-tuning of SFT and RLHF/DPO/UNA through a generalized implicit reward function. CoRR, abs/2410.21438. — [arXiv:2410.21438](https://arxiv.org/abs/2410.21438) find
  33. Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, Yefeng Liu, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. Marco-Bench-MIF: On multilingual instruction-following capability of large language models.…
  34. Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Rest-mcts*: Llm self-training via process reward guided tree search. Preprint, arXiv:2406.03816.
  35. Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024a. Marco-o1: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411.14405. find
  36. Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024b. Marco-o1: Towards open reasoning models for open-ended solutions. Preprint, arXiv:2411.14405. find
  37. Tianyang Zhong, Zhengliang Liu, and et al. 2024a. Evaluation of openai o1: Opportunities and challenges of agi. Preprint, arXiv:2409.18486.
  38. Y. Zhong, R. Li, and X. Wang. 2024b. Evaluation of large reasoning models. Journal of AI Research, 64:1–15. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yin-2025-marco-o1-v2-widening/