Elamur Research⌕
☰
home / papers / 2024 / What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning

What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning

Paper · 2024 · Yiran Ma, Zui Chen, Tianqiao Liu et al.

Summary

Step-level reward models (SRMs) built from Monte Carlo Tree Search (MCTS) preference data reliably improve LLM mathematical reasoning, but it has been unclear what signal these models actually learn. The authors model each solution step as an MDP. A natural-language "thought" is the action, and the math expression that executes it is the state transition. Preferences are collected with MCTS, with Llama-3-8B-Instruct acting as both agent and world model.

They then train SRMs that see different parts of each step:

  • full context,
  • math expressions only (MO-SRM),
  • only the current step's math (SSMO-SRM),
  • only the next thought (NT-SRM).

The models are tested as step scorers during greedy and beam search on GSM8K and MATH. Dropping the natural-language thoughts barely changes SRM effectiveness. Math-only models with access to the whole history beat single-step ones, which shows they judge logical coherence across steps and do more than check arithmetic. NT-SRMs, in contrast, train poorly and score badly. The practical implication is that cheaper, leaner reward models could focus only on the mathematical part of each step.

Key points

  • Four SRM input variants (Full-Context, MO-SRM, SSMO-SRM, NT-SRM) were trained on MCTS-collected step preferences. Base models included DeepSeek-Math-7B-Base, Qwen2-7B, Llemma-7B and Llama-3-8B.
  • With DeepSeek-Math-7B-Base, MO-SRM improves accuracy by +7.35% on GSM8K and +8.48% on MATH. SSMO-SRM gains only +3.64% and +6.30%. For Qwen2-7B the gains are +5.31%/+3.94% versus +3.18%/+1.92%.
  • NT-SRMs, which score only the natural-language next thought, struggle consistently during training and perform poorly as scoring functions.
  • An MO-SRM trained on 7B/8B-generated data still lifts Llama-3-70B-Instruct from 90.37 to 92.95 on GSM8K and from 48.48 to 54.12 on MATH.
  • The choice of base model matters in ways that do not track math ability alone: Llama-3-8B underperforms the other bases as an SRM. Larger beam size and candidate count help only slightly before plateauing (GSM8K 85.82 → 88.17).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China236 works · 14,072 citations🎓University of RochesterRochester · United States46 works · 7,948 citations🎓ShanghaiTech UniversityShanghai · China16 works · 1,991 citations🎓Jinan UniversityGuangzhou · China8 works · 107 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu and Weiqi Luo. 2024. What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning. arXiv:2412.15904.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.15904 · arXiv 2412.15904

@misc{ma2024what,
  title = {{What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning}},
  author = {Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu and Weiqi Luo},
  year = {2024},
  doi = {10.48550/arxiv.2412.15904},
  eprint = {2412.15904},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.15904}
}

References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Azerbayev, Z.; Schoelkopf, H.; Paster, K.; Dos Santos, M.; McAleer, S. M.; Jiang, A. Q.; Deng, J.; Biderman, S.; and Welleck, S. 2023. Llemma: An Open Language Model for Mathematics. In The Twelfth International Conference on Learning Representations.
  2. Besta, M.; Blach, N.; Kubicek, A.; Gerstenberger, R.; Podstawski, M.; Gianinazzi, L.; Gajda, J.; Lehmann, T.; Niewiadomski, H.; Nyczyk, P.; et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume… find
  3. Chen, G.; Liao, M.; Li, C.; and Fan, K. 2024a. AlphaMath Almost Zero: process Supervision without process. arXiv preprint arXiv:2405.03553.
  4. Chen, G.; Liao, M.; Li, C.; and Fan, K. 2024b. Step-level Value Preference Optimization for Mathematical Reasoning. In Conference on Empirical Methods in Natural Language Processing. find
  5. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  6. Ding, R.; Zhang, C.; Wang, L.; Xu, Y.; Ma, M.; Zhang, W.; Qin, S.; Rajmohan, S.; Lin, Q.; and Zhang, D. 2023. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254. find
  7. Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; et al. 2024. The Llama 3 Herd of Models.
  8. Fedorenko, E.; Piantadosi, S. T.; and Gibson, E. A. 2024. Language is primarily a tool for communication rather than thought. Nature, 630(8017): 575–586. find
  9. Feng, X.; Wan, Z.; Wen, M.; Wen, Y.; Zhang, W.; and Wang, J. 2023. Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training. In NeurIPS 2023 Foundation Models for Decision Making Workshop. find
  10. Hao, S.; Gu, Y.; Luo, H.; Liu, T.; Shao, X.; Wang, X.; Xie, S.; Ma, H.; Samavedhi, A.; Gao, Q.; et al. 2024. LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models. In ICLR 2024 Workshop on Large Language Model (LLM) Agents. find
  11. Hao, S.; Gu, Y.; Ma, H.; Hong, J.; Wang, Z.; Wang, D.; and Hu, Z. 2023. Reasoning with Language Model is Planning with World Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 8154–8173. find
  12. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  13. Li, X.; Bai, Y.; Guo, T.; Liu, Z.; Huang, Y.; Zhao, X.; Xia, F.; Luo, W.; and Weng, J. 2024. Enhancing Length Generalization for Attention Based Knowledge Tracing Models with Linear Biases. In 33rd International Joint Conference on Artificial Intelligence, IJCAI 2024, 5918–5926. International Joint… find
  14. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations.
  15. Ma, Q.; Zhou, H.; Liu, T.; Yuan, J.; Liu, P.; You, Y.; and Yang, H. 2023. Let’s reward step by step: Step-Level reward model as the Navigators for Reasoning. arXiv preprint arXiv:2310.10080.
  16. Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Zhang, M.; Li, Y.; Wu, Y.; and Guo, D. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  17. Wang, P.; Li, L.; Shao, Z.; Xu, R.; Dai, D.; Li, Y.; Chen, D.; Wu, Y.; and Sui, Z. 2024. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9426– 9439.
  18. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q. V.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.
  19. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837.
  20. Xie, Y.; Goyal, A.; Zheng, W.; Kan, M.-Y.; Lillicrap, T. P.; Kawaguchi, K.; and Shieh, M. 2024. Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning. arXiv preprint arXiv:2405.00451. find
  21. Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al. 2024. Qwen2 Technical Report. find
  22. CoRR. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. find
  23. Zhan, B.; Guo, T.; Li, X.; Hou, M.; Liang, Q.; Gao, B.; Luo, W.; and Liu, Z. 2024. Knowledge tracing as language processing: A large-scale autoregressive paradigm. In International Conference on Artificial Intelligence in Education, 177–191. find
  24. Springer. Zhang, D.; Li, J.; Huang, X.; Zhou, D.; Li, Y.; and Ouyang, W. 2024. Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B. arXiv preprint arXiv:2406.07394. find
  25. Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
  26. Zheng, Y.; Li, X.; Huang, Y.; Liang, Q.; Guo, T.; Hou, M.; Gao, B.; Tian, M.; Liu, Z.; and Luo, W. 2024. Automatic Lesson Plan Generation via Large Language Models with Self-critique Prompting. In International Conference on Artificial Intelligence in Education, 163–178. Springer. Zhou, D.; Schärli…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2024-step-level-reward-models-rewarding/