Elamur Research⌕
☰
home / papers / 2024 / Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

Paper · 2024 · Seungpil Lee, Woochang Sim, Donghyeon Shin et al.

Summary

Most ways of testing LLM reasoning only check whether the final answer is correct, so they reveal little about how the model reached it. This paper proposes a process-oriented evaluation based on the Language of Thought Hypothesis (LoTH), which describes human thinking through three properties: Logical Coherence, Compositionality and Productivity. The authors adapt these ideas from psychology for computer science and test each one on the Abstraction and Reasoning Corpus (ARC). ARC is a visual grid-puzzle benchmark that needs little prior knowledge, is still hard to solve, and is easy to modify.

Each property gets its own experiment:

  • Coherence: models face many variants of tasks they already solved, and the authors check whether correct answers come from sound reasoning.
  • Compositionality: models must choose and chain step-by-step DSL functions.
  • Productivity: models must generate new tasks that follow given rules.

The models show basic image understanding and handle simple object manipulations. However, they fall clearly short of humans on all three properties. The study matters because it offers a framework for judging how models reason, not just what scores they get.

Key points

  • An ARC task has 2–5 demonstration pairs and a test input. Grids range from 1×1 to 30×30 and use up to 10 colors.
  • Reported reference accuracies on ARC: about 80% for humans, 40–55% for the best specialised systems, and roughly 10–20% for LLMs such as GPT-4 and PaLM.
  • For Logical Coherence, each solved ARC task was expanded with 100 similar test examples. The authors also flagged cases where right answers came from flawed reasoning.
  • LLMs struggled to select DSL functions even when given extra context, and their performance dropped as task complexity grew.
  • When asked to generate new ARC tasks, LLMs mostly imitated patterns they had seen instead of applying the abstract rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Gwangju Institute of Science and Technology.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim and Sundong Kim. 2024. Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus. arXiv:2403.11793.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.11793 · arXiv 2403.11793

@misc{lee2024reasoning,
  title = {{Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus}},
  author = {Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim and Sundong Kim},
  year = {2024},
  doi = {10.48550/arxiv.2403.11793},
  eprint = {2403.11793},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.11793}
}

References

79 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel Acquaviva, Yewen Pu, Marta Kryven, Theodoros Sechopoulos, Catherine Wong, Gabrielle Ecanow, Maxwell Nye, Michael Tessler, and Joshua B. Tenenbaum. 2022. Communicating Natural Programs to Humans and Machines. In NeurIPS. find
  2. Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Physics of Language Models: Part 3.2, Knowledge Manipulation. arXiv:2309:14402 (2023). find
  3. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2023. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. arXiv:2308.09687 (2023). find
  4. Or Biran and Courtenay Cotton. 2017. Explanation and Justification in Machine Learning: A Survey. In IJCAI Workshop. find
  5. Alexey Borsky. 2021. ARC-Game. https://github.com/volotat/ARC-Game ↗
  6. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. arXiv:2303.…
  7. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A Survey on Evaluation of Large Language Models. ACM Transactions on Intelligent Systems and Technology…
  8. François Chollet. 2019. On the Measure of Intelligence. arXiv:1911.01547 (2019). find
  9. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In NeurIPS. find
  10. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL.
  11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 (2021).
  12. Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes. In CVPR. 5828–5839. find
  13. Gonçalo Hora de Carvalho, Robert Pollice, and Oscar Knap. 2024. Show, Don’t Tell: Evaluating Large Language Models Beyond Textual Understanding with ChildPlay. arXiv:2407.11068 (2024). — [arXiv:2407.11068](https://arxiv.org/abs/2407.11068) find
  14. Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation. arXiv:2311.04254 (2023). find
  15. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In NAACL.
  16. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and Fate: Limits of Transformers on Composition…
  17. Jerry A Fodor. 1975. The Language of Thought. Vol. 5. Harvard University Press. find
  18. Jerry A Fodor and Zenon W Pylyshyn. 1988. Connectionism and Cognitive Architecture: A Critical Analysis. Cognition 28, 1-2 (1988), 3–71. find
  19. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: Program-Aided Language Models. In ICML. 10764–10799. find
  20. Howard E Gardner. 2011. Frames of Mind: The Theory of Multiple Intelligences. Basic Books. find
  21. Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. 2024. Large Language Models are Not Strong Abstract Reasoners. IJCAI (2024).
  22. Wes Gurnee and Max Tegmark. 2023. Language Models Represent Space and Time. arXiv:2310.02207 (2023). find
  23. Michael Hodel. 2024. Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation. arXiv:2404.07353 (2024). https://github.com/michaelhodel/re-arc?tab=readme-ov-file ↗
  24. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards Reasoning in LLMs: A Survey. In ACL Findings. find
  25. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality Decomposed: How Do Neural Networks Generalise? Journal of Artificial Intelligence Research 67 (2020), 757–795.
  26. Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering. In CVPR. 2758–2766. find
  27. Aysja Johnson, Wai Keen Vong, Brenden M. Lake, and Todd M. Gureckis. 2021. Fast and Flexible: Human Program Induction in Abstract Reasoning Tasks. In CogSci. find
  28. Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. 2022. Playgrounds for Abstraction and Reasoning. In NeurIPS Workshop on nCSI. find
  29. Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. 2024. Generating Images with Multimodal Language Models. In NeurIPS, Vol. 36. find
  30. Lab42. 2024. ARC-PRIZE Competition. https://arcprize.org/ ↗
  31. Brenden Lake and Marco Baroni. 2018. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. In ICML. PMLR, 2873–2882. find
  32. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. TVQA: Localized, Compositional Video Question Answering. In EMNLP.
  33. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving Quantitative Reasoning Problems with Language Models. In NeurIPS.
  34. Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024. Understanding and Patching Compositional Reasoning in LLMs. arXiv:2402.14328 (2024). find
  35. Mushui Liu, Yuhang Ma, Xinfeng Zhang, Yang Zhen, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. 2024. LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation. arXiv:2407.00737 (2024). — [arXiv:2407.00737](https://arxiv.org/abs/2407.00737) find
  36. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-BERT: Enabling Language Representation with Knowledge Graphs. In AAAI. find
  37. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models. In ICLR. find
  38. Teli Ma, Rong Li, and Junwei Liang. 2024. An Examination of the Compositionality of Large Generative Vision-Language Models. In NAACL. find
  39. Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. 2023. SQA3D: Situated Question Answering in 3D Scenes. In ICLR. find
  40. Warren S McCulloch and Walter Pitts. 1943. A Logical Calculus of the Ideas Immanent in Nervous Activity. The Bulletin of Mathematical Biophysics 5 (1943), 115–133. find
  41. Małkiński Mikołaj and Mańd́ziuk Jacek. 2023. A Review of Emerging Research Directions in Abstract Visual Reasoning. Information Fusion 91 (2023), 713–736. find
  42. Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. 2023. Large Language Models as General Pattern Machines. arXiv:2307.04721 (2023). find
  43. Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023. The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain. TMLR (2023). find
  44. Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. 2020. Bongard-Logo: A New Benchmark for Human-Level Concept Learning and Reasoning. In NeurIPS. find
  45. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simense, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Trai… — Training language models to follow instructions with human feedback
  46. Jaehyun Park, Jaegyun Im, Sanha Hwang, Mintaek Lim, Sabina Ualibekova, Sejin Kim, and Sundong Kim. 2023. Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformer. ICML Workshop on ILHF (2023). find
  47. Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. 2024. Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement. In ICLR. find
  48. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In ACL.
  49. Frank Rosenblatt. 1958. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review 65, 6 (1958), 386–408. find
  50. Stuart J Russell and Peter Norvig. 1995. Artificial Intelligence: A Modern Approach. Pearson. find
  51. Donghyeon Shin, Seungpil Lee, Klea Lena Kovacec, and Sundong Kim. 2024. From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions. In EMNLP Findings. find
  52. Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. FLAVA: A Foundational Language and Vision Alignment Model. In CVPR. find
  53. Sania Sinha, Tanawan Premsri, and Parisa Kordjamshidi. 2024. A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices. arXiv:2406.08787 (2024). — [arXiv:2406.08787](https://arxiv.org/abs/2406.08787) find
  54. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In NAACL.
  55. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding Stories in Movies through Question-Answering. In CVPR. 4631–4640. find
  56. Alan Turing. 1950. Computing Machinery and Intelligence. Mind 59, 236 (1950), 433–460. find
  57. Gladys Tyen, Hassan Mansoor, Victor Cărbune, Yuanzhu Peter Chen, and Tony Mak. 2024. LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location. In ACL Findings.
  58. Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2024. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change. In NeurIPS.
  59. Frank van der Velde, Gwendid T van der Voort van der Kleij, and Marc de Kamps. 2004. Lack of Combinatorial Productivity in Language Processing with Simple Recurrent Networks. Connection Science 16, 1 (2004), 21–46. find
  60. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. In ACL.
  61. Haoyu Wang, Mo Yu, Xiaoxiao Guo, Rajarshi Das, Wenhan Xiong, and Tian Gao. 2019. Do Multi-Hop Readers Dream of Reasoning Chains?. In EMNLP Workshop on MRQA. find
  62. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. 2024. Hypothesis Search: Inductive Reasoning with Language Models. In ICLR. find
  63. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR.
  64. Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi, Xin Liu, and Yangqiu Song. 2023. AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph. arXiv:2311.09174 (2023). find
  65. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS.
  66. Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. 2024. The Generative AI Paradox: “What It Can Create, It May Not Understand”. In ICLR. find
  67. Norbert Wiener. 1950. Cybernetics. Bulletin of the American Academy of Arts and Sciences 3, 7 (1950), 2–4. find
  68. Johan Sokrates Wind. 2020. ARC-Solution. https://github.com/top-quarks/ARC-solution ↗
  69. Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 2021. STAR: A Benchmark for Situated Reasoning in Real-World Videos. In NeurIPS. find
  70. Yudong Xu, Elias B. Khalil, and Scott Sanner. 2023. Graphs, Constraints, and Search for the Abstraction and Reasoning Corpus. In AAAI. find
  71. Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B Khalil. 2023. LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations. TMLR (2023).
  72. Cheng Xue, Vimukthini Pinto, Chathura Gamage, Ekaterina Nikonova, Peng Zhang, and Jochen Renz. 2023. Phy-Q as a Measure for Physical Reasoning Intelligence. Nature Machine Intelligence 5, 1 (2023), 83–93. find
  73. Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. 2024. Exploring Compositional Generalization of Large Language Models. In NAACL Workshop. find
  74. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurIPS. find
  75. Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. RAVEN: A Dataset for Relational and Analogical Visual Reasoning. In CVPR. find
  76. Jinghan Zhang, Xiting Wang, Weijieying Ren, Lu Jiang, Dongjie Wang, and Kunpeng Liu. 2024. RATT: A Thought Structure for Coherent and Correct LLM Reasoning. arXiv:2406.02746 (2024). — [arXiv:2406.02746](https://arxiv.org/abs/2406.02746) find
  77. Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. 2024. Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives. In ACL.
  78. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In ICLR.
  79. Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. 2022. Teaching Algorithmic Reasoning via In-Context Learning. arXiv:2211.09066 (2022). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2024-reasoning-abilities-large-language-models/