Elamur Research⌕
☰
home / papers / 2023 / Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond

Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond

Paper · 2023 · Fangzhi Xu, Qika Lin, Jiawei Han et al.

Summary

The paper asks whether current LLMs reason logically or mostly rely on surface patterns and memorized benchmark content. The authors argue that earlier evaluations were too narrow: they used few models, scored only accuracy, skipped error analysis, and relied on test data mixed with world knowledge. To address this, they collect fifteen logical reasoning datasets, grouped into deductive, inductive, abductive and mixed-form settings. They test seven models under several prompting setups: three earlier systems (text-davinci-003, ChatGPT, BARD) and four newer ones (LLaMA3.1-Chat, Mistral-Instruct-v0.3, Claude-3, GPT-4). Beyond answer accuracy, they judge the quality of generated explanations and trace failures to specific error categories. To separate reasoning from background knowledge, they release NeuLR, a benchmark built from content-neutral items. The findings feed a six-dimension scheme and per-model ability maps for each reasoning setting, which show where LLMs succeed and fail. The work matters because it gives a reusable protocol, usable with human annotation or automatically, for checking how models reason rather than only their final answers.

Key points

  • Coverage: 15 datasets in 4 reasoning settings (deductive, inductive, abductive, mixed-form), with 7 LLMs compared across zero-shot and few-shot prompting.
  • Metrics: answers are scored for correctness, and explanations are rated for correctness, completeness and redundancy, combining objective and subjective views.
  • Error analysis: faulty cases are sorted into five error types along two axes, evidence selection and the reasoning process itself.
  • NeuLR: a new benchmark of 3,000 content-neutral samples spanning deductive, inductive and abductive reasoning. It is designed to limit knowledge bias and training-data overlap.
  • Evaluation scheme: six properties (Correct, Rigorous, Self-aware, Active, Oriented, No hallucination), visualized as ability maps per model and per setting to guide future research.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore73 works · 24,390 citations🎓Xi'an Jiaotong UniversityXi'an · China7 works · 35,747 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 3 Language models & pretraining · 1,343 works Reasoning · 510 works Evaluation & benchmarks · 685 works

Cite

Cite as Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu and Erik Cambria. 2023. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond. arXiv:2306.09841.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.09841 · arXiv 2306.09841

@misc{xu2023are,
  title = {{Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond}},
  author = {Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu and Erik Cambria},
  year = {2023},
  doi = {10.48550/arxiv.2306.09841},
  eprint = {2306.09841},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.09841}
}

References

50 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Antoniou and A. Bikakis, “Dr-prolog: A system for defeasible reasoning with rules and ontologies on the semantic web,” IEEE Transactions on Knowledge and Data Engineering (TKDE), vol. 19, no. 2, pp. 233–245, 2007. T. Lukasiewicz, “A novel combination of answer set programming with description log… find
  2. Z. Yang, X. Du, R. Mao, J. Ni, and E. Cambria, “Logical reasoning over natural language as knowledge representation: A survey,” CoRR, vol. abs/2303.12023, 2023. find
  3. B. Peng, C. Li, P. He, M. Galley, and J. Gao, “Instruction tuning with GPT-4,” CoRR, vol. abs/2304.03277, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2304.03277 ↗ our card
  4. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nie, and J. Wen, “A survey of large language models,” CoRR, vol. abs/2303.18223, 2023. our card
  5. M. M. Amin, E. Cambria, and B. W. Schuller, “Will affective computing emerge from foundation models and general artificial intelligence? A first evaluation of chatgpt,” IEEE Intelligent Systems, vol. 38, no. 2, pp. 15–23, 2023. find
  6. F. Jiao, Y. Guo, X. Song, and L. Nie, “Merit: Meta-path guided contrastive learning for logical reasoning,” in Findings of the Association for Computational Linguistics (Findings of ACL), 2022, pp. 3496–3509. find
  7. F. Xu, Q. Lin, T. Zhao, J. JiaweiHan, and J. Liu, “PathReasoner: Modeling reasoning path with equivalent extension for logical question answering,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for C… find
  8. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V. Do, Y. Xu, and P. Fung, “A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” CoRR, vol. abs/2302.04023, 2023. our card
  9. N. Bian, X. Han, L. Sun, H. Lin, Y. Lu, and B. He, “Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models,” CoRR, vol. abs/2303.16421, 2023. our card
  10. S. Imani, L. Du, and H. Shrivastava, “Mathprompter: Mathematical reasoning using large language models,” CoRR, vol. abs/2303.05398, 2023. our card
  11. P. A. Flach and A. C. Kakas, “Abductive and inductive reasoning: background and issues,” Abduction and induction: Essays on their relation and integration, pp. 1–27, 2000. find
  12. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback… our card
  13. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022. our card
  14. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen et al., “Palm 2 technical report,” arXiv preprint arXiv:2305.10403, 2023. our card
  15. C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is chatgpt a general-purpose natural language processing task solver?” CoRR, vol. abs/2302.06476, 2023. our card
  16. S. Tu, C. Li, J. Yu, X. Wang, L. Hou, and J. Li, “Chatlog: Recording and analyzing chatgpt across time,” CoRR, vol. abs/2304.14106, 2023. — [arXiv:2304.14106](https://arxiv.org/abs/2304.14106) find
  17. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, F. Qi, Y. Fu, M. Sun, and J. He, “Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models,” CoRR, vol. abs/2305.08322, 2023. our card
  18. T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. R. McKeown, and T. B. Hashimoto, “Benchmarking large language models for news summarization,” CoRR, vol. abs/2301.13848, 2023. our card
  19. Y. Pan, J. Liu, L. Zhang, T. Zhao, Q. Lin, X. Hu, and Q. Wang, “Inductive relation prediction with logical reasoning using contrastive representations,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, pp. 4261–4274. find
  20. Z. Huang, M. Chiang, and W. Lee, “Line: Logical query reasoning over hierarchical knowledge graphs,” in 28th SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 615–625. find
  21. K. Cheng, J. Liu, W. Wang, and Y. Sun, “Rlogic: Recursive logical rule learning from knowledge graphs,” in 28th SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 179–189. find
  22. P. N. Johnson-Laird, “Deductive reasoning,” Annual review of psychology, vol. 50, no. 1, pp. 109–135, 1999. find
  23. V. Goel, “Anatomy of deductive reasoning,” Trends in cognitive sciences, vol. 11, no. 10, pp. 435–441, 2007. find
  24. E. Heit and C. M. Rotello, “Relations between inductive reasoning and deductive reasoning.” Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 36, no. 3, p. 805, 2010. find
  25. F. Yu, H. Zhang, and B. Wang, “Nature language reasoning, a survey,” arXiv preprint arXiv:2303.14725, 2023. find
  26. J. R. Josephson and S. G. Josephson, Abductive inference: Computation, philosophy, technology. Cambridge University Press, 1996. find
  27. D. Walton, “Abductive, presumptive and plausible arguments,” Informal Logic, vol. 21, no. 2, 2001. find
  28. B. Dalvi, P. Jansen, O. Tafjord, Z. Xie, H. Smith, L. Pipatanangkura, and P. Clark, “Explaining answers with entailment trees,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 7358–7370. our card
  29. S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell, L. Benson, L. Sun, E. Zubova, Y. Qiao, M. Burtell, D. Peng, J. Fan, Y. Liu, B. Wong, M. Sailor, A. Ni, L. Nan, J. Kasai, T. Yu, R. Zhang, S. R. Joty, A. R. Fabbri, W. Kryscinski, X. V. Lin, C. Xiong, and D. Radev, “FOLIO: natural language reasoning… our card
  30. A. Talmor, O. Tafjord, P. Clark, Y. Goldberg, and J. Berant, “Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge,” in Advances in Neural Information Processing Systems (NeurIPS), 2020. find
  31. K. Sinha, S. Sodhani, J. Dong, J. Pineau, and W. L. Hamilton, “CLUTRR: A diagnostic benchmark for inductive reasoning from text,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP… find
  32. W. Yu, Z. Jiang, Y. Dong, and J. Feng, “Reclor: A reading comprehension dataset requiring logical reasoning,” in 8th International Conference on Learning Representations (ICLR), 2020. our card
  33. J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang, “Logiqa: A challenge dataset for machine reading comprehension with logical reasoning,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 3622–3628. our card
  34. H. Liu, R. Ning, Z. Teng, J. Liu, Q. Zhou, and Y. Zhang, “Evaluating the logical reasoning ability of chatgpt and GPT-4,” CoRR, vol. abs/2304.03439, 2023. our card
  35. J. Weston, A. Bordes, S. Chopra, and T. Mikolov, “Towards ai-complete question answering: A set of prerequisite toy tasks,” in 4th International Conference on Learning Representations (ICLR), 2016. our card
  36. P. Clark, O. Tafjord, and K. Richardson, “Transformers as soft reasoners over language,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), 2020, pp. 3882–3890. our card
  37. C. Bhagavatula, R. L. Bras, C. Malaviya, K. Sakaguchi, A. Holtzman, H. Rashkin, D. Downey, W. Yih, and Y. Choi, “Abductive commonsense reasoning,” in 8th International Conference on Learning Representations (ICLR), 2020. our card
  38. N. Young, Q. Bao, J. Bensemann, and M. Witbrock, “Abductionrules: Training transformers to explain unexpected inputs,” in Findings of the Association for Computational Linguistics, 2022, pp. 218–227. find
  39. O. Tafjord, B. Dalvi, and P. Clark, “Proofwriter: Generating implications, proofs, and abductive statements over natural language,” in Findings of the Association for Computational Linguistics, 2021, pp. 3621–3634. our card
  40. P. Minervini, S. Riedel, P. Stenetorp, E. Grefenstette, and T. Rocktäschel, “Learning reasoning strategies in end-to-end differentiable proving,” in Proceedings of the 37th International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 119, 2020, pp. 6938–6… find
  41. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.00234, 2022. our card
  42. Z. Wu, Y. Wang, J. Ye, and L. Kong, “Self-adaptive in-context learning,” arXiv preprint arXiv:2212.10375, 2022. find
  43. W. Chen, X. Ma, X. Wang, and W. W. Cohen, “Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks,” arXiv preprint arXiv:2211.12588, 2022. find
  44. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, “Pal: Program-aided language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 10 764–10 799. find
  45. F. Xu, Z. Wu, Q. Sun, S. Ren, F. Yuan, S. Yuan, Q. Lin, Y. Qiao, and J. Liu, “Symbol-LLM: Towards foundational symbol-centric interface for large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: A… find
  46. L. Pan, A. Albalak, X. Wang, and W. Wang, “Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 3806–3824. our card
  47. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in NeurIPS, 2022. our card
  48. I. Drori, S. Zhang, R. Shuttleworth, L. Tang, A. Lu, E. Ke, K. Liu, L. Chen, S. Tran, N. Cheng et al., “A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level,” Proceedings of the National Academy of Sciences (PNAS), vol. 1… find
  49. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Computing Surveys, vol. 55, no. 12, pp. 248:1–248:38, 2023. our card
  50. E. Cambria, Q. Liu, S. Decherchi, F. Xing, and K. Kwok, “Sentic-Net 7: A commonsense-based neurosymbolic AI framework for explainable sentiment analysis,” in LREC, 2022, pp. 3829–3839. A PPENDIX find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/