Elamur Research⌕
☰
home / papers / 2023 / OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning

OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning

Paper · 2023 · Fei Yu, Anningzhe Gao and Benyou Wang

Summary

LLMs solving multi-step math problems often make an early mistake that carries through to a wrong final answer. Guided decoding tries to prevent this by scoring partial solutions step by step and keeping only the most promising candidates. The authors argue that a partial path should be judged by its chance of reaching a correct answer (a value, in planning terms) rather than by whether the steps so far are correct (a reward). They show theoretically that a verifier trained with outcome supervision, where only the final answer's correctness labels every step, already estimates this value. Building on that, they propose OVM, a value model trained this way and used for beam-style search during generation. It needs no costly step-level annotations and no simulation at inference time. On GSM8K and Game of 24, OVM beats or matches process-supervised and complete-path verification baselines while sampling fewer paths. This suggests that cheap outcome labels can power effective planning for LLM reasoning.

Key points

  • On GSM8K, OVM-7B reaches 84.7% accuracy, the best reported result among LLMs up to 13B parameters. It uses no extra datasets, GPT-4 or code execution.
  • On Game of 24, OVM-7B achieves a 78.7% success rate while visiting only 20 nodes per step. Greedy decoding reaches 11%, and majority voting over 100 paths reaches 11.7%.
  • The theory shows that the loss-minimizing output of an outcome-supervised model for a partial path equals the expected probability of a correct final answer, which is exactly the value.
  • Process labels (logical correctness) and outcome-oriented PRM-O labels agree 98.6% of the time on GSM8K but only 56.9% on Game of 24. This explains why the process-supervised PRM is weaker on Game of 24.
  • Stated limitations:
    • the extra model increases memory use and slows inference;
    • outcome supervision may require larger training data;
    • generalization was not tested.
  • Code is available at github.com/FreedomIntelligence/OVM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China159 works · 16,154 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China56 works · 3,212 citations🎓Shenzhen Research Institute of Big DataShenzhen · China20 works · 2,016 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Fei Yu, Anningzhe Gao and Benyou Wang. 2023. OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning. arXiv:2311.09724.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.09724 · arXiv 2311.09724

@misc{yu2023ovm,
  title = {{OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning}},
  author = {Fei Yu, Anningzhe Gao and Benyou Wang},
  year = {2023},
  doi = {10.48550/arxiv.2311.09724},
  eprint = {2311.09724},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.09724}
}

References

29 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, ab…
  2. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
  3. Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  4. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. abs/2307.08691.
  5. CoRR, Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS. find
  6. Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. 2023. Alphazero-like tree-search can guide large language model decoding and training. CoRR, abs/2309.17179. find
  7. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Tora: A tool-integrated reasoning agent for mathematical problem solving. CoRR, abs/2309.17452.
  8. Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. CoRR, abs/2305.14992. find
  9. Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. CoRR, abs/2210.11610.
  10. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350. find
  11. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pa… find
  12. Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press. find
  13. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics…
  14. Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023. Discriminator-guided multi-step reasoning with language models. CoRR, abs/2305.14934. — [arXiv:2305.14934](https://arxiv.org/abs/2305.14934) find
  15. Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. CoRR, abs/2211.14275.
  16. Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. 2023a. Query and response augmentation cannot help out-of-domain math reasoning generalization. CoRR, abs/2310.05506. find
  17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS.
  18. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. CoRR, abs/2206.02336. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) find
  19. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022. Large language models are better reasoners with self-verification. CoRR, abs/2212.09561.
  20. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023b. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Cana…
  21. Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023. Decomposition enhances reasoning via self-evaluation guided decoding. CoRR, abs/2305.00633.
  22. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. CoRR, abs/2305.20050.
  23. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.09583.
  24. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. CoRR, abs/2305.10601. find
  25. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. CoRR, abs/2309.12284.
  26. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. CoRR, abs/2308.01825.
  27. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. CoRR, abs/2308.07921.
  28. Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023. Solving math word problems via cooperative reasoning induced language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa…
  29. Notation q Q S s i a m y S (1:t) S (1:t) K b n Φ f (1:t) v θ PRM ORM OVM find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2023-ovm-outcome-supervised-value-models/