Elamur Research⌕
☰
home / papers / 2025 / Empowering LLMs with Logical Reasoning: A Comprehensive Survey

Empowering LLMs with Logical Reasoning: A Comprehensive Survey

Paper · 2025 · Fengxiang Cheng, Haoxuan Li, Fenrong Liu et al.

Summary

This survey looks at why large language models remain weak at logical reasoning. The authors split the problem into two linked parts. The first is logical question answering: solving multi-premise deductive, inductive or abductive problems correctly. The second is logical consistency: avoiding answers that contradict each other across related questions. They attribute the weakness to two causes. Next-token pretraining does not guarantee logical skill, and human-written corpora contain few formal proofs or deductions.

The paper organizes recent work into a taxonomy. For question answering, methods fall into three groups:

  • systems that hand symbolic translations to external solvers,
  • prompting strategies,
  • fine-tuning on data enriched with explicit reasoning.

For consistency, the authors formally define negation, implication, transitivity, factuality and compositional consistency, then map existing fixes onto each type. They also catalogue benchmarks and evaluation metrics. The survey closes with open problems: conditional, modal and higher-order logic, and scalable methods that enforce several consistency types at once. Its value lies in a single structured map of a fragmented field that matters for trustworthy LLM deployment.

Key points

  • Baseline weakness: LLaMA-13B reaches only 33.63% accuracy on FOLIO with 8-shot prompting, barely above the 33.33% chance level for true/false/unknown.
  • Consistency failures: Macaw affirms that a magpie is a bird and that birds have wings, yet denies that magpies have wings (a transitivity failure). LLaMA-2-70B accepts both a statement and its negation (a negation failure).
  • Solver-based methods follow a translate, solve, back-translate pipeline, often with majority voting:
    • SatLM, Logic-LM and LINC are the core examples.
    • CLOVER, VERUS-LM and ∀uto∃val target errors in the NL→SL translation step.
    • The main weakness is information loss during symbolic translation.
  • Named resources covered:
    • Prompt-based: SymbCoT, LoT, Aristotle.
    • Fine-tuning: LoGiPT, LogicLLM, ALT.
    • Benchmarks: LogicNLI, LogiGLUE, LogicBench, Multi-LogiEval, LogicPro, SATBench.
  • Scalability gap: Checking transitivity exhaustively costs O(n³). ZebraLogic shows a "curse of complexity," with accuracy falling as the search space grows. This motivates efficient algorithms that satisfy multiple consistency types together.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang and Zhouchen Lin. 2025. Empowering LLMs with Logical Reasoning: A Comprehensive Survey. arXiv:2502.15652.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.15652 · arXiv 2502.15652

@misc{cheng2025empowering,
  title = {{Empowering LLMs with Logical Reasoning: A Comprehensive Survey}},
  author = {Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert {van Rooij}, Kun Zhang and Zhouchen Lin},
  year = {2025},
  doi = {10.48550/arxiv.2502.15652},
  eprint = {2502.15652},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.15652}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai and Hannaneh Hajishirzi. Logic-guided data augmentation and regularization for consistent question answering. In ACL, 2020. [Bao et al., 2022] Qiming Bao, Alex Yuxuan Peng, et al. Multi-step deductive reasoning over natural language: An empirical study on out-of-distribution generalisatio…
  2. Rushang Karia, Daniel Bramblett, et al. ∀uto∃val: Autonomous assessment of llms in formal synthesis and interpretation tasks. arXiv preprint arXiv:2403.18327, 2024. — [arXiv:2403.18327](https://arxiv.org/abs/2403.18327) find
  3. Nora Kassner, Oyvind Tafjord, et al. BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief. In EMNLP, 2021.
  4. Nora Kassner, Oyvind Tafjord, et al. Language models with rationality. In EMNLP, 2023. find
  5. Long Hei Matthew Lam, Ramya Keerthy Thatikonda, et al. A closer look at tool-based logical reasoning with LLMs: The choice of tool matters. In ALTC, 2024. find
  6. Alina Leidinger, Robert Van Rooij, et al. Are llms classical or nonmonotonic reasoners? lessons from generics. In ACL, 2024. find
  7. Huaxia Li and Miklos A Vasarhelyi. Applying large language models in accounting: A comparative analysis of different methodologies and off-the-shelf examples. JETA, 2024. find
  8. Tao Li, Vivek Gupta, et al. A logic-driven framework for consistency of neural models. In EMNLP, 2019.
  9. Huihan Li, Yuting Ning, et al. In search of the long-tail: Systematic generation of long-tail inferential knowledge via logical rule guided search. In EMNLP, 2024. find
  10. Qingchuan Li, Jiatong Li, et al. Leveraging LLMs for hypothetical deduction in logical inference: A neuro-symbolic approach. arXiv preprint arXiv:2410.21779, 2024. — [arXiv:2410.21779](https://arxiv.org/abs/2410.21779) find
  11. Bill Yuchen Lin, Ronan Le Bras, et al. ZebraLogic: On the scaling limits of LLMs for logical reasoning. In ICML, 2025.
  12. Jian Liu, Leyang Cui, et al. LogiQA: a challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, 2021.
  13. Hanmeng Liu, Yiran Ding, et al. Evaluating the logical reasoning abilities of large reasoning models. arXiv preprint arXiv:2505.11854, 2025. — [arXiv:2505.11854](https://arxiv.org/abs/2505.11854) find
  14. Tongxuan Liu, Wenjiang Xu, et al. Logic-of-thought: Injecting logic into contexts for full reasoning in large language models. In NAACL, 2025. find
  15. Yinhong Liu, Zhijiang Guo, et al. Aligning with logic: Measuring, evaluating and improving logical consistency in large language models. In ICML, 2025. find
  16. Man Luo, Shrinidhi Kumbhar, et al. Towards LogiGLUE: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models. arXiv preprint arXiv:2310.00836, 2023.
  17. Qing Lyu, Shreya Havaldar, et al. Faithful chain-of-thought reasoning. In IJCNLP-AACL, 2023.
  18. Eric Mitchell, Joseph Noh, et al. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In EMNLP, 2022. [Morishita et al., 2024] Terufumi Morishita, Gaku Morio, et al. Enhancing reasoning capabilities of llms via principled synthetic logic corp… find
  19. Zhongsheng Wang, Jiamou Liu, et al. Chatlogic: Integrating logic programming with large language models for multi-step reasoning. In IJCNN, 2024. [Wei et al., 2022] Jason Wei, Xuezhi Wang, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. [Wei et al., 20…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2025-empowering-llms-logical-reasoning-comprehensive/