Elamur Research⌕
☰
home / papers / 2023 / ARB: Advanced Reasoning Benchmark for Large Language Models

ARB: Advanced Reasoning Benchmark for Large Language Models

Paper · 2023 · Tomohiro Sawada, Daniel Paleka, Alexander Havrilla et al.

Summary

Many popular LLM benchmarks are saturating. Models like GPT-4 now score highly on them even though they still lack expert-level competence. ARB (Advanced Reasoning Benchmark) offers a harder test built from graduate-level and professional problems in mathematics, physics, chemistry, biology and law. The sources were written for specialists: Harvard PhD comprehensive exams in mathematics, university and contest problem books, PhD qualifying question collections in physics, McGraw-Hill MCAT practice tests and an older bar exam practice set. Several were chosen because they rarely appear online, which lowers the risk that models saw them during training. Questions are multiple choice, short answer or open response. Recent models, including GPT-4 and Claude, solve well under half of the more demanding quantitative tasks. Grading symbolic answers and multi-step reasoning is difficult, so the authors also propose a rubric method. GPT-4 drafts a scoring rubric from a reference solution and then grades solutions against it. Human checks show these scores align fairly well with manual grading. ARB thus provides a tougher target for tracking progress and a step toward cheaper automated evaluation.

Key points

  • The mathematics portion contains 52 numerical, 34 symbolic and 19 proof-like problems. The physics portion mixes numerical and symbolic items, some of which include figures.
  • High-school contest math was left out on purpose because benchmarks like MATH already cover it. The Putnam and Brayman sources come with official solutions, which are included in the dataset.
  • Human raters scored the GPT-4 rubrics on a 1–5 scale. Coverage of key steps was high (about 3.9–4.4), but point allocation across steps was weaker. GPT-3.5-turbo tended to concentrate points on one or two steps.
  • On a 10-point scale, Pearson correlation between human and GPT-4 rubric scores was 0.91 for symbolic physics, 0.78 for symbolic math and 0.82 for proof-like problems. The main failure was GPT-4 giving partial credit to attempts that fell entirely outside the rubric.
  • Rubric grading suggests GPT-4 recalls physics concepts well but makes mistakes in algebraic manipulation. It is weakest at planning solutions for proof-style math.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🎓Georgia Institute of TechnologyAtlanta · United States152 works · 27,763 citations🎓ETH ZurichZurich · Switzerland84 works · 9,181 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada65 works · 14,425 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta and Aran Komatsuzaki. 2023. ARB: Advanced Reasoning Benchmark for Large Language Models. arXiv:2307.13692.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.13692 · arXiv 2307.13692

@misc{sawada2023arb,
  title = {{ARB: Advanced Reasoning Benchmark for Large Language Models}},
  author = {Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta and Aran Komatsuzaki},
  year = {2023},
  doi = {10.48550/arxiv.2307.13692},
  eprint = {2307.13692},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.13692}
}

References

72 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022. find
  2. Daman Arora, Himanshu Gaurav Singh, and Mausam. Have LLMs advanced enough? A challenging problem solving benchmark for Large Language Models, 2023.
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — Constitutional AI: Harmlessness from AI Feedback
  4. Barbri. Barbri Practice Questions: Multistate Testing Practice Questions. Thomson/Bar/Bri, 2007. ISBN 9780314174017. find
  5. Michael Bommarito II and Daniel Martin Katz. GPT takes the bar exam. arXiv preprint arXiv:2212.14402, 2022. find
  6. Samuel R. Bowman. The dangers of underclaiming: Reasons for caution when reporting how NLP systems fail, 2021. find
  7. Volodymyr Brayman and A. G. Kukush. Undergraduate Mathematics Competitions (1995-2016): Taras Shevchenko National University of Kyiv. find
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  9. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712, 2023.
  10. Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws, 2023. find
  11. Candice McCloskey Campbell, Shaun Murphree, Jennifer M. Warner, Amy B. Wachholz, Kathy A. Zahler, and George J. Hademenos. McGraw-Hill Education 3 MCAT Practice Tests, Third Edition. McGraw-Hill Education, Jan 2017. ISBN 1259859622. find
  12. Bryan Caplan. GPT retakes my midterm and gets an A, 2023. URL https://betonit.substack.com/ p/gpt-retakes-my-midterm-and-gets-an. ↗
  13. Cheng-Han Chiang and Hung-yi Lee. Can Large Language Models be an alternative to human evaluations? arXiv e-prints, art. arXiv:2305.01937, may 2023. doi: 10.48550/arXiv.2305.01937. find
  14. François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. find
  15. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  16. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  18. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as you desire. arXiv e-prints, art. arXiv:2302.04166, feb 2023. doi: 10.48550/arXiv.2302.04166. find
  19. Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance, 2023. — Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  20. Răzvan Gelca and Titu Andreescu. Putnam and beyond. Springer, 2017. find
  21. Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models are not abstract reasoners, 2023. find
  22. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies, 2021. find
  23. Zoubin Ghahramani. Introducing PaLM 2, 2023. URL https://blog.google/technology/ai/ google-palm-2-ai-large-language-model. ↗
  24. Department of Mathematics Harvard University. Qualifying examination for fall 2021, Aug 2021. URL https://www.math.harvard.edu/media/quals-F21_with_solutions.pdf. ↗
  25. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2020. URL https://arxiv.org/abs/2009. 03300. ↗
  26. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. CoRR, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874. ↗
  27. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  28. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
  29. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the bar exam. Available at SSRN 4389233, 2023. find
  30. Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. arXiv e-prints, art. arXiv:2302.14520, feb 2023. doi: 10.48550/arXiv.2302.14520. find
  31. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023.
  32. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152–1157, San Diego,… find
  33. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health,…
  34. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets, 2023.
  35. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022. URL https://arxiv…
  36. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier, 2023. find
  37. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  38. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023. — Let's Verify Step by Step
  39. Swee Cheng Lim, Choy Heng Lai, and Leong Chuan Kwek. Problems and solutions on optics. World Scientific, 2019. find
  40. Yung-kuo Lim. Problems and solutions on thermodynamics and Statistical Mechanics. World Scientific, 1996. find
  41. Yung-kuo Lim. Problems and solutions in quantum mechanics: Major, American universities ph. D. qualifying questions and, solutions. World Scientific, 1998. find
  42. Yung-kuo Lim. Problems and solutions on electromagnetism. World Scientific Pub. Co, 2007. find
  43. Yung-kuo Lim and Yuan-qi Qiang. Problems and solutions on Mechanics. World Scientif., 2001. find
  44. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning, to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver, Can… find
  45. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human alignment. arXiv e-prints, art. arXiv:2303.16634, mar 2023. doi: 10.48550/ arXiv.2303.16634.
  46. Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fab… — [doi:10.7717/peerj-cs.103](https://doi.org/10.7717/peerj-cs.103) find
  47. Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984, Online, jul 2020. Association for Computational Linguistics. do… find
  48. Shakked Noy and Whitney Zhang. Experimental evidence on the productivity effects of generative artificial intelligence. Available at SSRN 4375283, 2023. find
  49. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.
  50. OpenAI. GPT-4 technical report, 2023.
  51. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems?, 2021.
  52. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251, 2022.
  53. Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. arXiv e-prints, art. arXiv:2204.01075, April 2022. doi: 10.48550/arXiv. 2204.01075. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) find
  54. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  55. Subhro Roy and Dan Roth. Solving general arithmetic word problems, 2016.
  56. Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu, and Lakshmivihari Mareedu. An independent evaluation of ChatGPT on mathematical word problems (MWP). arXiv preprint arXiv:2302.13814, 2023.
  57. Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever Hans or Neural Theory of Mind? Stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763, 2023. find
  58. Paulo N de Souza and Jorge N. Silva. Berkeley problems in Mathematics. Springer New York, 2008. find
  59. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.…
  60. Jacob Steinhardt. AI forecasting: One year in, 2022. URL https://bounded-regret.ghost.io/ ai-forecasting-one-year-in/. ↗
  61. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. CoRR, abs/1811.00937, 2018. URL http://arxiv.org/ abs/1811.00937. ↗
  62. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change), 2023. find
  63. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems, 2019a.
  64. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019b. In the Proceedings of ICLR.
  65. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.
  66. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022a.
  67. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, 2022b. URL https://arxiv.org/abs/2201.11903. ↗
  68. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering, 2018.
  69. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. find
  70. Ping Yu, Tianlu Wang, Olga Golovneva, Badr Alkhamissy, Gargi Ghosh, Mona Diab, and Asli Celikyilmaz. ALERT: Adapting language models to reasoning tasks, 2022. find
  71. Sarah Zhang, Reece Shuttleworth, Derek Austin, Yann Hicke, Leonard Tang, Sathwik Karnik, Darnell Granberry, and Iddo Drori. A dataset and benchmark for automatically answering and generating machine learning final exams. arXiv preprint arXiv:2206.05442, 2022. — [arXiv:2206.05442](https://arxiv.org/abs/2206.05442) find
  72. Hefei Zhongguo-Kexue-Jishu-Daxue. Major American universities Ph. D. qualifying questions and solutions. 5. Problems and solutions on thermodynamics and statistical mechanics. World Scientific, 1990. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/