Elamur Research⌕
☰
home / papers / 2024 / CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities

CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities

Paper · 2024 · Yujun Mao, Yoon Kim and Yilun Zhou

Summary

Most assessments of LLM mathematical reasoning only check whether the final answer is correct. That leaves two questions open: can models benefit from side information, and is their reasoning actually sound? To address both, the authors built CHAMP (Concept and Hint-Annotated Math Problems). It contains 270 competition-style problems drawn from Engel's Problem-Solving Strategies, rewritten so that each has a checkable answer. Every problem comes with a detailed step-wise solution and annotations of two kinds:

  • Concepts: general theorems or formulas.
  • Hints: problem-specific tricks.

Using 17 prompt designs, the authors test ten proprietary and open models. The prompts vary the extra context, for example by supplying hints, misleading concepts or related problems. The authors also manually judged 540 model-written solutions and marked the first wrong step in each flawed one.

The benchmark proves demanding. The strongest model reaches 58.1% in the standard setting and 67.0% in the best configuration, and gains from concepts and hints differ across models. Correct final answers frequently rest on faulty reasoning, and most models perform poorly when asked to verify solutions. CHAMP therefore enables much finer diagnosis of math reasoning than answer-only accuracy.

Key points

  • The 270 problems in CHAMP span five areas, with 80 on number theory, 50 each on polynomials, sequences and inequalities, and 40 on combinatorics. Solutions average 6.0 steps, with about 1.4 concepts and 1.7 hints per problem.
  • The evaluated models are GPT-3.5, GPT-4, GPT-4 Turbo, PaLM 2 Medium, Llama 2 7B/70B, Llama 3 8B/70B, Mistral 7B and Mixtral 8x22B.
  • The 540 solutions annotated for first wrong step (FWS) come from GPT-3.5, GPT-4, GPT-4 Turbo and PaLM 2 Medium. All curation was done by the authors, unlike the crowdsourced PRM800K and FELM.
  • Verification is weak overall:
    • Even with the reference solution and C&H, GPT-4 Turbo locates the FWS in only about 33.8% of wrong solutions.
    • On error-free reference solutions, GPT-4 Turbo still reports a mistake in 161 of 270 cases.
  • Final answers were graded automatically by GPT-4, which matched manual grading 97% of the time. Problems were rewritten from their originals to reduce contamination effects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States274 works · 82,013 citations🏢SalesforceSan Francisco · United States63 works · 18,348 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States80 works · 32,402 citations🎓Boston UniversityBoston · United States35 works · 4,458 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksMathematics · 233 worksComputational Theory and Mathematics · 496 worksApplied Mathematics · 73 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Yujun Mao, Yoon Kim and Yilun Zhou. 2024. CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-acl.785 · arXiv 2401.06961

@inproceedings{mao2024champ,
  title = {{CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities}},
  author = {Yujun Mao, Yoon Kim and Yilun Zhou},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.785},
  eprint = {2401.06961},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.785}
}

References

24 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meta AI. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3. ↗
  2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403.
  3. Shiqi Chen, Yiran Zhao, Jinghan Zhang, I Chern, Siyang Gao, Pengfei Liu, Junxian He, et al. 2023. Felm: Benchmarking factuality evaluation of large language models. arXiv preprint arXiv:2310.00741.
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  5. Arthur Engel. 2008. Problem-solving strategies. Springer Science & Business Media. find
  6. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR. find
  7. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  8. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  9. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  10. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  11. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  12. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.
  13. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  14. Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. 2023. Inverse scaling: When bigger isn’t better. arXiv preprint arXiv:2306.09479.
  15. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824– 24837.
  16. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837.
  17. OpenAI. 2023. Gpt-4 technical report. ArXiv.
  18. Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. 2023. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.08559. find
  19. Mistral AI Team. 2024. Cheaper, better, faster, stronger. https://mistral.ai/news/mixtral-8x22b/. ↗
  20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  21. Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, et al. 2023. The generative ai paradox:" what it can create, it may not understand". arXiv preprint arXiv:2311.00059.
  22. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477. find
  23. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR.
  24. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. 2023. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mao-2024-champ-competition-level-dataset-fine/