Elamur Research⌕
☰
home / papers / 2025 / Logical Reasoning in Large Language Models: A Survey

Logical Reasoning in Large Language Models: A Survey

Paper · 2025 · Hanmeng Liu, Zhizhang Fu, Mengru Ding et al.

Summary

This survey by Liu et al. (Westlake University and partners) examines how well large language models perform rigorous logical inference. The authors argue that earlier reviews blurred formal logic with generic prompting heuristics such as Chain-of-Thought. They frame the field around four paradigms: deductive, inductive, abductive and analogical reasoning. The literature is organised into three areas: tasks and benchmarks (NLI, machine reading comprehension, unified test suites), evaluation studies for each paradigm, and enhancement methods. Enhancement methods fall into four groups:

  • data-centric work, using expert-curated, synthetic or LLM-distilled training sets;
  • model-centric work, covering instruction fine-tuning, reinforcement learning (as in DeepSeek-R1) and inference-time decoding;
  • use of external knowledge;
  • neuro-symbolic pipelines, where natural language is converted into formal logic and passed to solvers or LLM-driven resolvers.

The main conclusion is that models still reason inconsistently. They fail under rephrasings or adversarial perturbations, symbolic hybrids scale poorly, and accuracy-based metrics on multiple-choice benchmarks do not capture consistency or soundness. The paper is useful as a map of resources and open problems for anyone building or evaluating reasoning-capable LLMs.

Key points

  • Benchmarks are classified by source into rule-based, expert-designed and exam-based types. Examples include LogiQA (Zh/En, 15,937 items), ReClor (6,138), AR-LSAT (2,064), ConTRoL (8,325, built from recruitment exams) and FOLIO (1,351, expert-written first-order logic).
  • The unified suites covered are GLoRE (17 tasks), LogiGLUE (24 tasks) and LogiTorch (16 tasks).
  • The training resources reviewed are LogiCoT, LogicPro, PODA, P-FOLIO, LeanDojo, Symbol-LLM, RuleTaker and FLD×2.
  • Neuro-symbolic systems discussed include LINC (NL→FOL plus an external theorem prover), Logic-LM, LOGICLLAMA, CLOVER, SymbCoT and Aristotle. Aristotle is built from three modules: a Logical Decomposer, a Logical Search Router and a Logical Resolver.
  • Proposed future directions include:
    • differentiable theorem provers and other hybrid neural-symbolic architectures;
    • stress tests using negated premises or swapped quantifiers;
    • multimodal grounding of inference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Westlake UniversityHangzhou · China60 works · 8,829 citations🎓Zhejiang Normal UniversityJinhua · China6 works · 142 citations🎓Hainan UniversityHaikou · China4 works · 27 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu and Yue Zhang. 2025. Logical Reasoning in Large Language Models: A Survey. arXiv:2502.09100.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.09100 · arXiv 2502.09100

@misc{liu2025logical,
  title = {{Logical Reasoning in Large Language Models: A Survey}},
  author = {Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu and Yue Zhang},
  year = {2025},
  doi = {10.48550/arxiv.2502.09100},
  eprint = {2502.09100},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.09100}
}

References

48 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrew M Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A Chi, et al. Lingoly: A benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages. arXiv preprint arXiv:2406.06196, 2024. [Bowen et al., 2024] Chen Bowen, Rune Sætre, and Yusuke Miyao. A comprehens…
  2. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. 2021.
  3. Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, et al. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 2947–2962, 2023. find
  4. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4, 2023.
  5. Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. Logicot: Logical chain-of-thought instruction tuning. In Proc. of EMNLP Findings, pages 2908–2921, 2023. find
  6. Hanmeng liu, Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, and Yue Zhang. Glore: Evaluating logical reasoning of large language models, 2023. find
  7. Hanmeng Liu, Zhiyang Teng, Chaoli Zhang, and Yue Zhang. Logic agent: Enhancing validity with logic rule invocation, 2024. find
  8. Lihui Liu, Zihao Wang, Ruizhong Qiu, Yikun Ban, Eunice Chan, et al. Logic query of thoughts: Guiding large language models to answer complex logic queries with knowledge graphs, 2024. find
  9. Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulić, and Nigel Collier. Aligning with logic: Measuring, evaluating and improving logical consistency in large language models. arXiv preprint arXiv:2410.02205, 2024. [Lu et al., 2021] Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras…
  10. Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, et al. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models, 2024.
  11. Chengcheng Mai, Yuxiang Wang, Ziyu Gong, Hanxiang Wang, and Yihua Huang. Knowra: Knowledge retrieval augmented method for document-level relation extraction with comprehensive reasoning abilities, 2025. [Malon et al., 2024] Christopher Malon, Martin Min, Xiaodan Zhu, et al. Exploring the role of rea… find
  12. J. McCarthy and P.J. Hayes. Some philosophical problems from the standpoint of artificial intelligence. In Readings in Artificial Intelligence, pages 431–450. 1981. find
  13. John McCarthy. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes, 1959. [McCarthy, 1989] John McCarthy. Artificial intelligence, logic and formalizing common sense. Philosophical Logic and Artificial Intelligence, pages 161–190, 1989. […
  14. Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, et al. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Proc. of EMNLP, pages 5153–5176, 2023.
  15. OpenAI. Learning to reason with LLMs. Technical report, 2024. [Ouyang et al., 2023] Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. Fact-driven logical reasoning for machine reading comprehension, 2023. find
  16. Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Proc. of EMNLP Findings, pages 3806–3824, 2023. [Parmar et al., 2023] Mihir Parmar, Neeraj Varshney, Nisarg Patel, Santosh Mashetty, Man Luo,…
  17. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, et al. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proc. of ACL, pages 13679–13707, 2024. [Pereira, 1982] Fernando Carlos Neves Pereira. Logic for natural language analysis…
  18. Molly Petersen and Lonneke van der Plas. Can language models learn analogical reasoning? investigating training objectives and comparisons to human performance. In Proc. of EMNLP, pages 16414–16425, 2023. [Plaat et al., 2024] Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, an…
  19. Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, et al. Relevant or random: Can llms truly perform analogical reasoning?, 2024. find
  20. Kanagasabai Rajaraman, Saravanan Rajamanickam, and Wei Shi. Investigating transformer-guided chaining for interpretable natural logic reasoning. In Proc. of ACL Findings, pages 9240–9253, 2023. find
  21. Samuel Ryb, Mario Giulianelli, Arabella Sinclair, and Raquel Fernández. AnaLog: Testing analytical and deductive logic learnability in language models. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 55–68, 2022. find
  22. Hyun Ryu, Gyeongman Kim, Hyemin S Lee, and Eunho Yang. Divide and translate: Compositional first-order logic translation and verification for complex logical reasoning. arXiv preprint arXiv:2410.08047, 2024. [Saparov et al., 2023] Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Jo…
  23. Sergio Servantez, Joe Barrow, Kristian Hammond, and Rajiv Jain. Chain of logic: Rule-based reasoning with large language models. In Proc. of ACL Findings, pages 2721–2733, 2024. find
  24. Jihao Shi, Xiao Ding, Li Du, Ting Liu, and Bing Qin. Neural natural logic inference for interpretable question answering. In Proc. of EMNLP, pages 3673–3684, 2021. find
  25. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. Empirical Methods of Natural Language Processing (EMNLP), 2019. [Srivastava et al., 2022] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md S…
  26. Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. Logical reasoning with span-level predictions for interpretable and robust NLI models. In Proc. of EMNLP, pages 3809–3823, 2022. find
  27. Michael Sullivan. It is not true that transformers are inductive learners: Probing NLI models with external negation. In Proc. of EACL, pages 1924–1945, 2024.
  28. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, et al. A survey of reasoning with foundation models. arXiv preprint arXiv:2312.11562, 2023. — [arXiv:2312.11562](https://arxiv.org/abs/2312.11562) find
  29. Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, et al. Determlr: Augmenting llm-based logical reasoning from indeterminacy to determinacy. In Proc. of ACL, pages 9828–9862, 2024. find
  30. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Proc. of ACL Findings, pages 3621–3634, 2021.
  31. Ramya Keerthy Thatikonda, Wray Buntine, and Ehsan Shareghi. Assessing the alignment of fol closeness metrics with human judgement. arXiv preprint arXiv:2501.08613, 2025. — [arXiv:2501.08613](https://arxiv.org/abs/2501.08613) find
  32. Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first-order logical reasoning ability through LogicNLI. In Proc. of EMNLP, pages 3738–3747, 2021.
  33. Armin Toroghi, Willis Guo, Ali Pesaranghader, and Scott Sanner. Verifiable, debuggable, and repairable commonsense logical reasoning via llm-based theory resolution. In Proc. of EMNLP, pages 6634–6652, 2024. [Wang et al., 2019] Po-Wei Wang, Priya L. Donti, Bryan Wilder, and Zico Kolter. Satnet: Brid… find
  34. Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, et al. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022. find
  35. Chen Wang, Xudong Li, Haoran Liu, Xinyue Wu, and Wanting He. Efficient logical reasoning in large language models through program-guided learning. Authorea Preprints, 2024. find
  36. Chenxu Wang, Ping Jian, and Zhen Yang. Thought-path contrastive learning via premise-oriented data augmentation for logical reading comprehension. arXiv preprint arXiv:2409.14495, 2024. — [arXiv:2409.14495](https://arxiv.org/abs/2409.14495) find
  37. Thilini Wijesiriwardene, Ruwan Wickramarachchi, Bimal Gajera, Shreeyash Gowaikar, Chandan Gupta, et al. ANA-LOGICAL - a novel benchmark for long text analogy evaluation in large language models. In Proc. of ACL Findings, pages 3534–3549, 2023. find
  38. Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. In Proc. of ICML, 2024.
  39. Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, et al. Beyond chain-of-thought: A survey of chain-of-x paradigms for llms. arXiv preprint arXiv:2404.15676, 2024.
  40. Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? a comprehensive evaluation and beyond, 2023.
  41. Fangzhi Xu, Zhiyong Wu, Qiushi Sun, Siyu Ren, Fei Yuan, et al. Symbol-LLM: Towards foundational symbol-centric interface for large language models. In Proc. of ACL, pages 13091–13116, 2024. find
  42. Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, et al. Aristotle: Mastering logical reasoning with a logic-complete decompose-search-resolve framework. arXiv preprint arXiv:2412.16953, 2024. [Xu et al., 2024c] Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful…
  43. Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, et al. Leandojo: theorem proving with retrieval-augmented language models. In Proc. of ICONIP, 2023.
  44. Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. Harnessing the power of large language models for natural language to first-order logic translation. In Proc. of ACL, pages 6942–6959, 2024. [Yang et al., 2024b] Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, et al.… find
  45. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In Proc. of ICLR, 2020.
  46. Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey. ACM Computing Surveys, pages 1– 39, 2024. [Yuan et al., 2023] Zhangdie Yuan, Songbo Hu, Ivan Vulić, Anna Korhonen, and Zaiqiao Meng. Can pretrained language models (yet) reason deductively? In Proc. of EACL,…
  47. Majd Zayyad and Yossi Adi. Formal language knowledge corpus for retrieval augmented generation, 2024. find
  48. Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, et al. o1-coder: an o1 replication for coding. arXiv preprint arXiv:2412.00154, 2024. [Zhao et al., 2024] Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, et al. Marco-o1: Towards open reasoning models for open-ended solutions. ar…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-logical-reasoning-large-language-models/