Elamur Research⌕
☰
home / papers / 2026 / Incentivizing Strong Reasoning from Weak Supervision

Incentivizing Strong Reasoning from Weak Supervision

Paper · 2026 · Yige Yuan, Teng Xiao, Shuchang Tao et al.

Summary

Improving LLM reasoning usually requires either reinforcement learning with verifiable rewards, which is compute-intensive, or SFT on long chain-of-thought (CoT) data from very strong teachers, which is costly to obtain. The authors ask whether a much weaker model can serve as the teacher instead. They propose W2SR (Weak-to-Strong Reasoning), in which a large student is fine-tuned with plain SFT on long CoT traces from a smaller or less accurate "Reasoner" teacher, even when those traces contain errors. They test variants that keep all teacher traces, only traces with correct answers, or only traces with incorrect answers. Students include Qwen2.5-Math-7B, Qwen2.5-14B and Qwen2.5-32B, evaluated on five reasoning benchmarks against GRPO and untuned baselines.

Weakly supervised students beat their teachers and recover most of the gains RL provides, sometimes exceeding it, at a small fraction of the training cost. What matters most is whether the teacher produces well-structured reasoning, not its size or final-answer accuracy. This makes reasoning enhancement cheaper and lets domain experts use small local teachers.

Key points

  • Teachers of 0.5B–1.5B parameters lift 7B–32B students to as much as 94.34% of the improvement obtained with RL, including +18.8 Pass@1 on MATH.
  • Teacher traces with wrong final answers still teach well. In one configuration, Pass@1 was 50.28% using only incorrect traces, 50.02% using all traces, and 53.53% using only correct ones.
  • Larger teachers give diminishing returns. For a Qwen2.5-32B student on math, moving the teacher from 1.5B to 7B adds 11.4%, but going from 7B to 32B adds only 0.4%.
  • With a Qwen2.5-Math-7B student and a Qwen2.5-1.5B-Reasoner teacher, the W2SR-P variant trains about 25× faster than GRPO and scores 2.92% higher.
  • In a case study, the W2SR student kept the teacher's solution structure but fixed its arithmetic errors. Its response length grew from 307 tokens (base student) to 984, close to the RL-trained student's 1038.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,762 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations🏢Alibaba GroupHangzhou · China201 works · 17,172 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🔬Institute of Computing TechnologyBeijing · China57 works · 3,782 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Yige Yuan, Teng Xiao, Shuchang Tao, Xue Wang, Jinyang Gao, Bolin Ding and Bingbing Xu. 2026. Incentivizing Strong Reasoning from Weak Supervision. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.336

@inproceedings{yuan2026incentivizing,
  title = {{Incentivizing Strong Reasoning from Weak Supervision}},
  author = {Yige Yuan, Teng Xiao, Shuchang Tao, Xue Wang, Jinyang Gao, Bolin Ding and Bingbing Xu},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.336},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.336}
}

References

45 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Art of Problem Solving Foundation. 2023. Amc23 — 2023 american mathematics competitions test set. https://github.com/QwenLM/Qwen2. 5-Math/tree/main/evaluation/data/amc23. 40 problems drawn from the 2023 AMC 12 contests. ↗
  2. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, and et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning.
  3. Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and Pengfei Liu. 2023. Generative ai for math: Abel. https://github.com/ GAIR-NLP/abel. ↗
  4. Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, and Jun Wang. 2024. Large language models are neurosymbolic reasoners. In Proceedings of the AAAI conference on artificial intelligence, pages 17985–17993. find
  5. Kaituo Feng, Changsheng Li, Xiaolu Zhang, Jun Zhou, Ye Yuan, and Guoren Wang. 2024. Keypoint-based progressive chain-of-thought distillation for llms. arXiv preprint arXiv:2405.16064. — [arXiv:2405.16064](https://arxiv.org/abs/2405.16064) find
  6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  7. Yue Guo and Yi Yang. 2024a. Improving weak-to-strong generalization with reliability-aware alignment. arXiv preprint arXiv:2406.19032. — [arXiv:2406.19032](https://arxiv.org/abs/2406.19032) find
  8. Yue Guo and Yi Yang. 2024b. Improving weak-to-strong generalization with reliability-aware alignment. arXiv preprint arXiv:2406.19032. — [arXiv:2406.19032](https://arxiv.org/abs/2406.19032) find
  9. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In…
  10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Aaron Jaech,… — OpenAI o1 System Card
  11. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515.
  12. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, and et al. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  13. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, and 1 others. 2024. T\\" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124.
  14. Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. 2024. Theoretical analysis of weak-to-strong generalization. Advances in neural information processing systems, 37:46837–46880. find
  15. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871.
  16. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022a. Solving quantitative reasoning problems with language models. arXiv preprint a…
  17. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022b. Solving quantitative reasoning problems with language models. pages 3843–3857.
  18. Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023. Symbolic chain-of-thought distillation: Small models can also "think" step-by-step. arXiv preprint arXiv:2306.14050.
  19. Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886. find
  20. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  21. Yuejiang Liu and Alexandre Alahi. 2024. Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture of experts. arXiv preprint arXiv:2402.15505. — [arXiv:2402.15505](https://arxiv.org/abs/2402.15505) find
  22. Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, and Bo Zheng. 2025. Deconstructing long chain-of-thought: A structured reasoning optimization framework for long cot distillation. arXiv preprint arXiv:2503.16385. find
  23. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393.
  24. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  25. Zhihong Shao, Peiyi Wang, Qihao Zhu, and et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  26. Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. 2025. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074.
  27. Leitian Tao and Yixuan Li. 2024. Your weak llm is secretly a strong teacher for alignment. arXiv preprint arXiv:2409.08813. — [arXiv:2409.08813](https://arxiv.org/abs/2409.08813) find
  28. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. — Kimi k1.5: Scaling Reinforcement Learning with LLMs
  29. Jason Wei, Xuezhi Wang, Schuurmans, and et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  30. Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. 2024a. How to leverage demonstration data in alignment for large language model? a self-imitation learning perspective. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Associa… find
  31. Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, and et al. 2025a. SimPER: A minimalist approach to preference alignment without hyperparameters. In International Conference on Learning Representations. find
  32. Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, and Vasant G Honavar. 2025b. On a connection between imitation learning and RLHF. In International Conference on Learning Representations. find
  33. Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. 2024b. Cal-dpo: Calibrated direct preference optimization for language model alignment. In Advances in Neural Information Processing Systems. find
  34. An Yang, Baosong Yang, Beichen Zhang, and et al. 2024a. Qwen2.5 technical report. find
  35. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024b. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
  36. Yuqing Yang, Yan Ma, and Pengfei Liu. 2024c. Weak-to-strong reasoning. arXiv preprint arXiv:2407.13647. find
  37. Ruimeng Ye, Yang Xiao, and Bo Hui. 2024. Weak-to-strong generalization beyond accuracy: a pilot study in safety, toxicity, and legal reasoning. arXiv preprint arXiv:2410.12621. find
  38. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387.
  39. Edward Yeo, Yuxuan Tong, Morry Niu, and et al. 2025. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373.
  40. Yige Yuan, Teng Xiao, Li Yunfan, Xu Bingbing, Shuchang Tao, Yunqi Qiu, Huawei Shen, and Xueqi Cheng. Inference-time alignment in continuous space. In ICLR 2025 Workshop on Bidirectional Human-AI Alignment. find
  41. Yige Yuan, Bingbing Xu, Hexiang Tan, Fei Sun, Teng Xiao, Wei Li, Huawei Shen, and Xueqi Cheng. 2024. Fact-level confidence calibration and self-correction. arXiv preprint arXiv:2411.13343. — [arXiv:2411.13343](https://arxiv.org/abs/2411.13343) find
  42. Yang Yue, Zhiqi Chen, Rui Lu, Zhao, and et al. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837.
  43. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.
  44. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…
  45. Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. 2024. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? In Advances in Neural Information Processing Systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2026-incentivizing-strong-reasoning-weak-supervision/