Elamur Research⌕
☰
home / papers / 2023 / MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

Paper · 2023 · Zhongshen Zeng, Pengguang Chen, Shu Liu et al.

Summary

Math word-problem benchmarks such as GSM8K are close to saturation. Leading LLMs exceed 80% accuracy, scoring checks only the final answer, and test-set contamination may inflate results. The authors propose testing models as graders instead of solvers, a setup they call meta-reasoning. They built MR-GSM8K from three GSM8K-derived question variants: the original problems, code-oriented "Program of Thought" versions, and reversed-reasoning versions in which one input is hidden. Step-by-step solutions were sampled from MetaMath-7B so that roughly half are correct. Expert annotators then recorded three things for each solution: whether its reasoning is genuinely sound, where the first faulty step occurs, and why it is wrong. Evaluated models must reproduce these judgments, which are combined into a new metric, MR-Score. Open and commercial models that look alike on GSM8K diverge sharply on MR-GSM8K, revealing weaknesses that answer-only tests miss. The recipe can be applied to other existing benchmarks, giving harder evaluation of the reasoning process that is more resistant to contamination.

Key points

  • Models receive a question-solution pair and must judge correctness, locate the first error step, and explain the error. These outputs feed the MR-Score metric.
  • Deepseek-v2 and Claude3-Sonnet nearly match GPT-4 on GSM8K, yet trail it by more than 20 absolute points on MR-GSM8K. Across all tested models, the gaps reach more than tenfold.
  • Math-specialized fine-tuned models (e.g., MetaMath, WizardMath) generalize poorly to the grading task, whether given instructions or few-shot examples. The small Phi-3-Mini outperforms models about twenty times its size.
  • The three-shot prompts contained two correct solutions and one incorrect solution. This setting raised true positive rates and lowered true negative rates, so models appear swayed by the label mix in the examples rather than by reasoning. An ablation on the number of correct examples supports this explanation.
  • Case studies expose several recurring failures: the reversal curse, confused handling of quantity units (weak ontological understanding), and insensitivity to arithmetic mistakes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China174 works · 21,355 citations🎓Chinese University of Hong KongHong Kong · China176 works · 17,536 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang and Jiaya Jia. 2023. MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation. arXiv:2312.17080.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.17080 · arXiv 2312.17080

@misc{zeng2023mr,
  title = {{MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation}},
  author = {Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang and Jiaya Jia},
  year = {2023},
  doi = {10.48550/arxiv.2312.17080},
  eprint = {2312.17080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.17080}
}

References

40 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Introducing claude, 2023. introducing-claude. find
  2. Konstantine Arkoudas. GPT-4 can’t reason. CoRR, abs/2308.03762, 2023. doi: 10.48550/ARXIV.2308.03762. URL https://doi.org/10.48550/arXiv.2308.03762. ↗
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  4. Simone Balloccu, Patr’icia Schmidtov’a, Mateusz Lango, and Ondvrej Duvsek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:267499939. ↗
  5. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". ArXiv, abs/2309.12288, 2023. URL https://api.semanticscholar.org/CorpusID: 262083829. ↗
  6. Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". CoRR, abs/2309.12288, 2023. doi: 10.48550/ARXIV.2309.12288. URL https://doi.org/10.48550/arXiv.2309.12288. ↗
  7. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li,… find
  8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/23…
  9. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv, abs/2211.12588, 2022. URL https://api.semanticscholar.org/CorpusID: 253801709. ↗
  10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://www.anthropic.com/index/ ↗
  11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305.14314, 2023. URL https://api.semanticscholar.org/CorpusID:258841328. ↗
  12. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaïd Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality.…
  13. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. ArXiv, abs/2309.17452, 2023. URL https://api.semanticscholar.org/CorpusID: 263310365. ↗
  14. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… find
  15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
  16. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. ArXiv, abs/2310.01798, 2023. URL https://api.semanticscholar.org/CorpusID: 263609132. ↗
  17. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In North American Chapter of the Association for Computational Linguistics, 2016. URL https://api.semanticscholar.org/CorpusID: 2228719. ↗
  18. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. ArXiv, abs/2305.20050, 2023. URL https://api.semanticscholar.org/CorpusID: 258987659. ↗
  19. Xiao Liu, Xuanyu Lei, Sheng-Ping Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese alignment of large language models. ArXiv, abs/2311.18743,…
  20. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023.
  21. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. ArXiv, abs/2210.07128, 2022. URL https://api.semanticscholar.org/CorpusID:252873120. ↗
  22. Brian W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et biophysica acta, 405 2:442–51, 1975. URL https://api.semanticscholar.org/CorpusID:44596673. ↗
  23. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Annual Meeting of the Association for Computational Linguistics, 2020. URL https://api.semanticscholar.org/CorpusID: 220047831. ↗
  24. OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL https://openai.com/blog/chatgpt. ↗
  25. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. semanticscholar.org/CorpusID:257532815.
  26. Keiran Paster. Testing language models on a held-out high school national finals exam. https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_exam, 2023. ↗
  27. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. ArXiv, abs/2009.03393, 2020. URL https://api.semanticscholar.org/CorpusID: 221535103. ↗
  28. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: 10.48550/ARXIV.2402.03300. URL https://doi.org/10.48550/arXiv.2402.03300. ↗
  29. Mike Sharples, David C. Hogg, Chris Hutchinson, Steve Torrance, and David J. Young. Computers and thought: A practical introduction to artificial intelligence. In Proceedings of the Conference on Innovative Applications of Artificial Intelligence, 1989. URL https://api.semanticscholar.org/CorpusID:6… ↗
  30. Christian Szegedy. A promising path towards autoformalization and general artificial intelligence. In International Conference on Intelligent Computer Mathematics, 2020. URL https://api.semanticscholar.org/CorpusID:220729524. ↗
  31. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. ↗
  32. Vernon Toh, Ratish Puduppully, and Nancy F. Chen. Veritymath: Advancing mathematical reasoning by self-verification through unit consistency. ArXiv, abs/2311.07172, 2023. URL https://api.semanticscholar.org/CorpusID:265149887. — [arXiv:2311.07172](https://arxiv.org/abs/2311.07172) ↗
  33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903, 2022. URL https://api.semanticscholar.org/CorpusID: 246411621. ↗
  35. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples. ArXiv, abs/2311.04850, 2023. URL https://api.semanticscholar.org/CorpusID: 265050721. ↗
  36. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601, 2023. URL https://api.semanticscholar.org/CorpusID: 258762525. ↗
  37. An-Zi Yen and Wei-Ling Hsu. Three questions concerning the use of large language models to facilitate mathematics learning. ArXiv, abs/2310.13615, 2023. URL https://api.semanticscholar.org/CorpusID:264405766. ↗
  38. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023. URL https://api.
  39. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv, abs/2309.05653, 2023. URL https://api.semanticscholar.org/CorpusID: 261696697. ↗
  40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685, 2023. URL https://api.semanticscholar.or… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2023-mr-gsm8k-meta-reasoning-benchmark/