Elamur Research⌕
☰
home / papers / 2023 / LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations

LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations

Paper · 2023 · Yudong Xu, Wenhao Li, Pashootan Vaezipoor et al.

Summary

The paper asks whether GPT-style LLMs can handle the Abstraction and Reasoning Corpus (ARC), a benchmark of grid puzzles built on human "core knowledge" priors such as objectness, counting and basic geometry. The authors serialized ARC grids as text, using digits or color names with several delimiters. They prompted GPT-3.5 and GPT-4 with plain few-shot prompts and with few-shot prompts that add chain-of-thought examples.

On 50 of the easiest ARC tasks, GPT-4 solved only 13. Their error analysis traced most failures to one cause: objects spread vertically across several text rows are hard for the model to track as single units. To test this, they built 1D-ARC, a new set of single-row tasks, and GPT performed better there. They then fed GPT object-level descriptions extracted by the ARGA solver, which raised GPT-4 to 23/50 and gave near-perfect results on many 1D-ARC task types.

The study suggests that external tools providing structured, object-centric input matter a great deal for LLM reasoning outside natural language.

Key points

  • Baseline: with direct text encodings of 2D grids, GPT-4 solves 13 of the 50 simplest ARC tasks. API queries ran at temperature 0, and the authors compared delimiters (",", "|", none) and prompting strategies.
  • New resource: 1D-ARC, a set of one-dimensional ARC-like tasks that fit on a single line of text. GPT performs better on it than on 2D ARC, but not flawlessly.
  • Object-based encodings derived from ARGA (Abstract Reasoning with Graph Abstractions) lift GPT-4 to 23/50 ARC tasks. They also yield near-perfect scores on many 1D-ARC categories.
  • A similar trend appears with LLAMA-2, on the full ARC training set, and on Mini-ARC.
  • GPT-4V, prompted manually with images, solved only 2 of the 50 tasks. By comparison, LLAMA2-13B solved 6. This points to weak visual reasoning about ARC in multimodal models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner and Elias B. Khalil. 2023. LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations. arXiv:2305.18354.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.18354 · arXiv 2305.18354

@misc{xu2023llms,
  title = {{LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations}},
  author = {Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner and Elias B. Khalil},
  year = {2023},
  doi = {10.48550/arxiv.2305.18354},
  eprint = {2305.18354},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.18354}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sam Acquaviva, Yewen Pu, Marta Kryven, Theodoros Sechopoulos, Catherine Wong, Gabrielle Ecanow, Maxwell Nye, Michael Tessler, and Josh Tenenbaum. Communicating natural programs to humans and machines. Advances in Neural Information Processing Systems, 35:3731–3743, 2022. find
  2. Simon Alford, Anshula Gandhi, Akshay Rangamani, Andrzej Banburski, Tony Wang, Sylee Dandekar, John Chin, Tomaso Poggio, and Peter Chin. Neural-guided, bidirectional program search for abstraction and reasoning. In International Conference on Complex Networks and Their Applications, pages 657–668. Sp… find
  3. Rim Assouel, Pau Rodriguez, Perouz Taslakian, David Vazquez, and Yoshua Bengio. Object-centric compositional imagination for visual abstract reasoning. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022. URL https://openreview.net/forum?id=rCzfIruU5x5. ↗
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  5. Giacomo Camposampiero, Loïc Houmard, Benjamin Estermann, Joël Mathys, and Roger Wattenhofer. Abstract visual reasoning enabled by language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642–2646, 2023. find
  6. François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. find
  7. Alejandro de Miquel Bleier. Arc_kaggle. https://github.com/alejandrodemiquel/ARC_Kaggle, 2020. Accessed: 2022-08-01. ↗
  8. Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint arXiv:2006.08381, 2020. find
  9. Sébastien Ferré. First steps of an approach to the arc challenge based on descriptive grid models and the minimum description length principle. arXiv preprint arXiv:2112.00848, 2021. — [arXiv:2112.00848](https://arxiv.org/abs/2112.00848) find
  10. Raphael Fischer, Matthias Jakobs, Sascha Mücke, and Katharina Morik. Solving abstract reasoning tasks with grammatical evolution. In LWDA, pages 6–10, 2020. find
  11. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. find
  12. Vlad Golubev. Arc-kaggle-3rd-place. https://www.kaggle.com/c/ abstraction-and-reasoning-challenge/discussion/154305, 2020. Accessed: 2022-08-01. ↗
  13. Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022.
  14. Kaggle. Arc-kaggle-main. https://www.kaggle.com/c/abstraction-and-reasoning-challenge, 2020. Accessed: 2022-08-01. ↗
  15. Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for abstraction and reasoning. In NeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI), 2022. URL https://openreview.net/forum?id=F4RNpByoqP. ↗
  16. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?i… ↗
  17. Victor Kolev, Bogdan Georgiev, and Svetlin Penkov. arXiv:2011.09860, 2020. — [arXiv:2011.09860](https://arxiv.org/abs/2011.09860) find
  18. Neural abstract reasoner. arXiv preprint Agnis Liukis. Arc-kaggle-5th-place. https://www.kaggle.com/c/ abstraction-and-reasoning-challenge/discussion/154377, 2020. Accessed: 2022-08-01. ↗
  19. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023.
  20. Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.04721, 2023. find
  21. Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. arXiv preprint arXiv:2305.07141, 2023. find
  22. Victor Vikram Odouard and Melanie Mitchell. Evaluating understanding on conceptual abstraction benchmarks. arXiv preprint arXiv:2206.14187, 2022. — [arXiv:2206.14187](https://arxiv.org/abs/2206.14187) find
  23. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023a.
  24. OpenAI. Gpt-4v(ision) system card. 2023b. URL https://openai.com/research/gpt-4v-system-card. ↗
  25. Andy Penrose. Arc-kaggle-8th-place. https://www.kaggle.com/c/ abstraction-and-reasoning-challenge/discussion/154384, 2020. Accessed: 2022-08-01. ↗
  26. Yewen Pu and Simon Alford. Larc solving with gpt4. https://github.com/evanthebouncy/larc_gpt4, 2023. Accessed: 2023-12-20. ↗
  27. Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597, 2022.
  28. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  29. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. find
  30. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018. find
  31. Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89–96, 2007. top quarks. Arc-solution. https://github.com/top-quarks/ARC-solution, 2020. Accessed: 2022-08-01. ↗
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  33. Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. Hypothesis search: Inductive reasoning with language models. arXiv preprint arXiv:2309.05660, 2023. find
  34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
  35. Yudong Xu, Elias B. Khalil, and Scott Sanner. Graphs, constraints, and search for the abstraction and reasoning corpus. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI-23), Washington D.C., USA, 2023. *Figure 10: 1D ARC: visualizations of selected sample tasks. For each t… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2023-llms-abstraction-reasoning-corpus-successes/