Elamur Research⌕
☰
home / papers / 2023 / Large Language Models Are Not Strong Abstract Reasoners

Large Language Models Are Not Strong Abstract Reasoners

Paper · 2023 · Gael Gendron, Qiming Bao, Michael Witbrock et al.

Summary

The paper asks whether today's LLMs can do genuine abstract reasoning, which means inferring a general rule from a handful of examples and applying it to new cases. The authors treat this as a test of broad generalisation rather than memorisation. They assemble a benchmark that combines existing text tasks with vision datasets (ARC, ACRE, RAVEN) rewritten into textual and symbolic form. Tasks are framed either as Open QA or as MCQA. Several kinds of models are evaluated on it: text-completion models from GPT-2 to GPT-4, LLaMA-family models, and QA engines. The evaluation also tests chain-of-thought prompting, code generation and fine-tuning. Performance is weak across the board, and techniques that help on ordinary NLP tasks bring little gain here. The authors locate the bottleneck in spotting new abstract patterns, not in understanding the prompt or producing valid code. Fine-tuned models fit the training distribution but fail to generalise to new combinations of rules. The work matters because it exposes a gap between fluent language ability and robust reasoning. The authors suggest causal reasoning and program induction as directions for improvement.

Key points

  • The benchmark covers ARC T, BIG-Bench-F, Evals-S, PVR, ACRE T, Evals-P and RAVEN T. ACRE T and RAVEN T come in both Text and Symbolic versions, and the code and data are released at github.com/Strong-AI-Lab/Logical-and-abstract-reasoning.
  • The models tested include GPT-2, Text-Davinci-3, GPT-3.5-Turbo, GPT-4, LLaMA-7B, LLaMA2-7B, Alpaca, Alpaca-LoRA and Zephyr-7B-β. The authors also introduce their own fine-tuned variants: LLaMA(2)-7B-AR-LoRA, RoBERTa-AR and MERIt-AR.
  • In ARC T, models receive raw integer grids sized between 8×8 and 30×30. Roughly 12% of ARC tasks are considered impossible to describe in natural language.
  • Fine-tuning LLaMA2 on RAVEN T-mcqa raises in-distribution accuracy from about 0.13 to as high as 0.977. On compositional o.o.d. splits (-Four, -In-Center), accuracy falls to around 0.44–0.77, and to as low as 0.06–0.09 on symbolic -In-Center.
  • A per-path analysis of ACRE T (Direct, Indirect, Backward-blocking, Screening-off) reveals that the GPT models become overly tuned to backward-blocking examples in the text version, a pattern that does not appear in the symbolic version. The authors read this as evidence that natural language adds distracting cues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AucklandAuckland · New Zealand11 works · 888 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Gael Gendron, Qiming Bao, Michael Witbrock and Gillian Dobbie. 2023. Large Language Models Are Not Strong Abstract Reasoners. arXiv:2305.19555.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.19555 · arXiv 2305.19555

@misc{gendron2023large,
  title = {{Large Language Models Are Not Strong Abstract Reasoners}},
  author = {Gael Gendron, Qiming Bao, Michael Witbrock and Gillian Dobbie},
  year = {2023},
  doi = {10.48550/arxiv.2305.19555},
  eprint = {2305.19555},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.19555}
}

References

47 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel Acquaviva et al. “Communicating Natural Programs to Humans and Machines”. In: CoRR abs/2106.07824 (2021). arXiv: 2106.07824. URL : https://arxiv.org/abs/2106.07824. ↗
  2. Qiming Bao et al. “Contrastive Learning with Logic-driven Data Augmentation for Logical Reasoning over Text”. In: CoRR abs/2305.12599 (2023). DOI : 10.48550/arXiv.2305.12599. arXiv: 2305.12599. URL : https://doi.org/10.48550/arXiv.2305.12599. — [doi:10.48550/arxiv.2305.12599](https://doi.org/10.48550/arxiv.2305.12599) ↗
  3. Qiming Bao et al. “Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation”. In: Proceedings of the 16th International Workshop on Neural-Symbolic Learning and Reasoning as part of the 2nd International Joint Conference on Learning & Reasoning (… find
  4. Emily M. Bender et al. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” In: FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021. Ed. by Madeleine Clare Elish, William Isaac, and Richard S. Zemel. ACM,… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
  5. Rishi Bommasani et al. “On the Opportunities and Risks of Foundation Models”. In: CoRR abs/2108.07258 (2021). arXiv: 2108.07258. URL : https://arxiv.org/abs/2108.07258. ↗
  6. Grady Booch et al. “Thinking Fast and Slow in AI”. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, V… find
  7. Tom B. Brown et al. “Language Models are Few-Shot Learners”. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ed. by Hugo Larochelle et al. 2020. URL : https://proceedings.neurips.cc… ↗
  8. Sébastien Bubeck et al. “Sparks of Artificial General Intelligence: Early experiments with GPT-4”. In: CoRR abs/2303.12712 (2023). DOI : 10.48550/arXiv.2303.12712. arXiv: 2303.12712. URL : https://doi.org/10.48550/arXiv.2303.12712. ↗
  9. Patricia A Carpenter, Marcel A Just, and Peter Shell. “What one intelligence test measures: a theoretical account of the processing in the Raven Progressive Matrices Test.” In: Psychological review 97.3 (1990), p. 404. find
  10. Mark Chen et al. “Evaluating Large Language Models Trained on Code”. In: CoRR abs/2107.03374 (2021). arXiv: 2107.03374. URL : https://arxiv.org/abs/2107.03374. ↗
  11. Wenhu Chen et al. “Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks”. In: CoRR abs/2211.12588 (2022). DOI : 10.48550/arXiv.2211.12588. arXiv: 2211.12588. URL : https://doi.org/10.48550/arXiv.2211.12588. ↗
  12. François Chollet. “On the Measure of Intelligence”. In: CoRR abs/1911.01547 (2019). arXiv: 1911.01547. URL : http://arxiv.org/abs/1911.01547. ↗
  13. Aakanksha Chowdhery et al. “PaLM: Scaling Language Modeling with Pathways”. In: CoRR abs/2204.02311 (2022). DOI : 10.48550/arXiv.2204.02311. arXiv: 2204.02311. URL : https://doi.org/10.48550/arXiv.2204.02311. ↗
  14. Peter Clark, Oyvind Tafjord, and Kyle Richardson. “Transformers as Soft Reasoners over Language”. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020. Ed. by Christian Bessiere. ijcai.org, 2020, pp. 3882–3890. DOI : 10.24963/ijcai.2020/537. URL :…
  15. Jacob Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7,…
  16. Computational Linguistics, 2019, pp. 4171–4186. DOI : 10.18653/v1/n19-1423. URL : https://doi.org/10.18653/v1/n19-1423. Kevin Ellis et al. “DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning”. In: CoRR abs/2006.08381 (2020). arXiv: 2006.08381. URL :… — BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding ↗
  17. Long Ouyang et al. “Training language models to follow instructions with human feedback”. In: NeurIPS. 2022. URL : http://papers.nips.cc/paper%5C_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstra ↗
  18. Linlu Qiu et al. “Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement”. In: CoRR abs/2310.08559 (2023). DOI : 10.48550/ARXIV.2310.08559. arXiv: 2310.08559. URL : https://doi.org/10.48550/arXiv.2310.08559. ↗
  19. Alec Radford et al. “Language models are unsupervised multitask learners”. In: OpenAI blog 1.8 (2019), p. 9. find
  20. Colin Raffel et al. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”. In: J. Mach. Learn. Res. 21 (2020), 140:1–140:67. URL : http://jmlr.org/papers/v21/20-074.html. ↗
  21. John C Raven. “Raven standard progressive matrices”. In: Journal of Cognition and Development (1938). find
  22. Joshua Stewart Rule. “The child as hacker: building more human-like models of learning”. PhD thesis. Massachusetts Institute of Technology, 2020. find
  23. Adam Santoro et al. “Measuring abstract reasoning in neural networks”. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018. Ed. by Jennifer G. Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Resear… find
  24. Bernhard Schölkopf et al. “Toward Causal Representation Learning”. In: Proc. IEEE 109.5 (2021), pp. 612–634. DOI : 10.1109/JPROC.2021.3058954. URL : https://doi.org/10.1109/JPROC.2021.3058954. — [doi:10.1109/jproc.2021.3058954](https://doi.org/10.1109/jproc.2021.3058954) ↗
  25. Aarohi Srivastava et al. “Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models”. In: CoRR abs/2206.04615 (2022). DOI : 10.48550/arXiv.2206.04615. arXiv: 2206.04615. URL : https://doi.org/10.48550/arXiv.2206.04615. ↗
  26. Alessandro Stolfo et al. “A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models”. In: CoRR abs/2210.12023 (2022). DOI : 10.48550/arXiv.2210.12023. arXiv: 2210.12023. URL : https://doi.org/10.48550/arXiv.2210.12023. ↗
  27. Alon Talmor et al. “Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge”. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ed. by Hugo Laroc… find
  28. Rohan Taori et al. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. 2023. ↗
  29. Kushal Tirumala et al. “Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models”. In: NeurIPS. 2022. URL : http://papers.nips.cc/paper%5C_files/paper/2022/hash/fa0509f4dab6807e2cb465715bf2d249-Abstra ↗
  30. Hugo Touvron et al. “Llama 2: Open Foundation and Fine-Tuned Chat Models”. In: CoRR abs/2307.09288 (2023). DOI : 10.48550/ARXIV.2307.09288. arXiv: 2307.09288. URL : https://doi.org/10.48550/arXiv.2307.09288. ↗
  31. Hugo Touvron et al. “LLaMA: Open and Efficient Foundation Language Models”. In: CoRR abs/2302.13971 (2023). DOI : 10.48550/arXiv.2302.13971. arXiv: 2302.13971. URL : https://doi.org/10.48550/arXiv.2302.13971. ↗
  32. Lewis Tunstall et al. The Alignment Handbook. https://github.com/huggingface/alignment-handbook. 2023. ↗
  33. Lewis Tunstall et al. “Zephyr: Direct Distillation of LM Alignment”. In: CoRR abs/2310.16944 (2023). DOI : 10.48550/ARXIV.2310.16944. arXiv: 2310.16944. URL : https://doi.org/10.48550/arXiv.2310.16944. ↗
  34. Petar Velickovic et al. “The CLRS Algorithmic Reasoning Benchmark”. In: International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. Ed. by Kamalika Chaudhuri et al. Vol. 162. Proceedings of Machine Learning Research. PMLR, 2022, pp. 22084–22102. URL : https://… find
  35. Ruocheng Wang et al. “Hypothesis Search: Inductive Reasoning with Language Models”. In: CoRR abs/2309.05660 (2023). DOI : 10.48550/ARXIV.2309.05660. arXiv: 2309.05660. URL : https://doi.org/10.48550/arXiv.2309.05660. ↗
  36. Siyuan Wang et al. “Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text”. In: Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022. Ed. by Smaranda Muresan, Preslav Nakov, and Aline Villavicencio. Association for Computa… find
  37. Yizhong Wang et al. “Self-Instruct: Aligning Language Model with Self Generated Instructions”. In: CoRR abs/2212.10560 (2022). DOI : 10.48550/arXiv.2212.10560. arXiv: 2212.10560. URL : https://doi.org/10.48550/arXiv.2212.10560. ↗
  38. Jason Wei et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. In: NeurIPS. 2022. URL : http://papers.nips.cc/paper%5C_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstra ↗
  39. Moritz Willig et al. “Can Foundation Models Talk Causality?” In: CoRR abs/2206.10591 (2022). DOI : 10.48550/arXiv.2206.10591. arXiv: 2206.10591. URL : https://doi.org/10.48550/arXiv.2206.10591. — [doi:10.48550/arxiv.2206.10591](https://doi.org/10.48550/arxiv.2206.10591) ↗
  40. Frank F. Xu et al. “A systematic evaluation of large language models of code”. In: MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego, CA, USA, 13 June 2022. Ed. by Swarat Chaudhuri and Charles Sutton. ACM, 2022, pp. 1–10. DOI : 10.1145/3520312.3534862. URL : h… find
  41. Zonglin Yang et al. “Language Models as Inductive Reasoners”. In: CoRR abs/2212.10923 (2022). DOI : 10.48550/arXiv.2212.10923. arXiv: 2212.10923. URL : https://doi.org/10.48550/arXiv.2212.10923. ↗
  42. Nathan Young et al. “AbductionRules: Training Transformers to Explain Unexpected Inputs”. In: Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022. Ed. by Smaranda Muresan, Preslav Nakov, and Aline Villavicencio. Association for Computational Linguist… find
  43. Weihao Yu et al. “ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning”. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL : https://openreview.net/forum?id=HJgJtT4tvB. ↗
  44. Chi Zhang et al. “ACRE: Abstract Causal REasoning Beyond Covariation”. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, 2021, pp. 10643–10653. DOI : 10.1109/CVPR46437.2021.01050. URL : https://openaccess.thecvf.c… — [doi:10.1109/cvpr46437.2021.01050](https://doi.org/10.1109/cvpr46437.2021.01050) ↗
  45. Chi Zhang et al. “RAVEN: A Dataset for Relational and Analogical Visual REasoNing”. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019, pp. 5317–5327. DOI : 10.1109/CVPR.2019.00546. URL : http://op… find
  46. Chiyuan Zhang et al. “Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization”. In: CoRR abs/2107.12580 (2021). arXiv: 2107.12580. URL : https://arxiv.org/abs/2107.12580. ↗
  47. Jiayao Zhang et al. “ROCK: Causal Inference Principles for Reasoning about Commonsense Causality”. In: International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. Ed. by Kamalika Chaudhuri et al. Vol. 162. Proceedings of Machine Learning Research. PMLR, 2022,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gendron-2023-large-language-models-strong-abstract/