Elamur Research⌕
☰
home / papers / 2025 / JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

Paper · 2025 · Michael K. Chen, Xikun Zhang and Dacheng Tao

Summary

The authors argue that current deductive reasoning benchmarks for LLMs have three weaknesses. They are simple in either their language or their argument structure. They let models score well by using world knowledge instead of logic. Their error analysis is shallow. To fix this, they built JustLogic, a code-generated dataset. Argument structures are assembled programmatically from seven propositional-logic argument forms, and real sentences sampled from GenericsKB-Best fill them in. Each item gives a paragraph of premises and a statement, and the model must judge the statement as true, false, or uncertain. Because the sentences are combined at random, the resulting arguments rarely match reality. A new test confirmed that prior knowledge barely helps on JustLogic, while it inflates scores on manually curated sets. Most SOTA models fell below the average human score. Only DeepSeek R1 beat it, and no model reached the human ceiling. The benchmark therefore exposes real gaps in LLM logic and supports detailed diagnosis by argument form and reasoning depth.

Key points

  • Prior knowledge test: the accuracy gap |∆| when the premises are removed is only 0.4 points on JustLogic. It is 27.1 on LogiQA 2.0, 6.7 on FOLIO, 3.7 on ProofWriter and 2.0 on CLUTRR.
  • Human vs. model scores: humans averaged 73.0% and the human ceiling is 100%. DeepSeek R1 scored 80.9% and OpenAI o1 72.9%. GPT-4o reached 65.6% with CoT, and random guessing gives 33.3%.
  • Prompting vs. scale: prompting matters more than model size. Llama3-8B with chain-of-thought (57.8%) beat zero-shot Llama3-70B (53.1%).
  • Effect of depth: accuracy generally falls as reasoning depth grows (depths 1–7, grouped into low, medium and high). OpenAI o1-mini matches DeepSeek R1 at low depth but collapses at medium and high depth.
  • Effect of argument form: hypothetical syllogism and constructive dilemma are solved more reliably. Models find reductio ad absurdum, disjunctive syllogism and modus tollens more difficult. Code and data are available at github.com/michaelchen-lab/JustLogic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Michael K. Chen, Xikun Zhang and Dacheng Tao. 2025. JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models. arXiv:2501.14851.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.14851 · arXiv 2501.14851

@misc{chen2025justlogic,
  title = {{JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models}},
  author = {Michael K. Chen, Xikun Zhang and Dacheng Tao},
  year = {2025},
  doi = {10.48550/arxiv.2501.14851},
  eprint = {2501.14851},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.14851}
}

References

36 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nan, L., Kasai, J., Yu, T., Zhang, R., Joty, S., Fabbri, A. R., Kryscinski, W., Lin, X. V., Xiong, C., and Radev, D. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022. URL https://arxiv.org/abs/2209.00840. ↗
  2. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  3. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  4. Amazon. Amazon Mechanical Turk, 2005. URL https://www.mturk.com/. ↗
  5. Beurer-Kellner, L., Fischer, M., and Vechev, M. Guiding LLMs the right way: Fast, non-invasive constrained generation. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning,… find
  6. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024.
  7. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  8. Bhakthavatsalam, S., Anastasiades, C., and Clark, P. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005.00660, 2020. find
  9. Johnson, R. A Logic Book: Fundamentals of Reasoning. Cengage Learning, 5 edition, 2006. find
  10. Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700, 2020.
  11. Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  13. Kincaid, J. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Chief of Naval Technical Training, 1975. find
  14. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  15. Liu, H., Liu, J., Cui, L., Teng, Z., Duan, N., Zhou, M., and Zhang, Y. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023a. find
  16. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022.
  17. Liu, H., Teng, Z., Cui, L., Zhang, C., Zhou, Q., and Zhang, Y. Logicot: Logical chain-of-thought instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2908–2921, 2023b. find
  18. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019.
  19. Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020.
  20. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  21. OpenAI. Learning to Reason with LLMs, 9 2024. URL https://openai.com/index/ learning-to-reason-with-llms/. ↗
  22. Fahlman, S. E. and Hinton, G. E. Connectionist architectures for artificial intelligence. Computer, 20(01):100– 109, 1987. find
  23. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  24. Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Benson, L., Sun, L., Zubova, E., Qiao, Y., Burtell, M., Peng, D., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Saparov, A., Pang, R. Y., Padmakumar, V., Joshi, N., Kazemi, M., Kim, N., and He, H. Testing the general deductive reasoning capac… — Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
  25. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.
  26. Sinha, K., Sodhani, S., Dong, J., Pineau, J., and Hamilton, W. L. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint arXiv:1908.06177, 2019. find
  27. Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S., and Durrett, G. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310.16049, 2023. find
  28. Tafjord, O., Mishra, B. D., and Clark, P. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048, 2020.
  29. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  30. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  31. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  32. Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824, 2024. find
  33. Yadav, V., Bethard, S., and Surdeanu, M. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. arXiv preprint arXiv:1911.07176, 2019. find
  34. Yu, W., Jiang, Z., Dong, Y., and Feng, J. Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.04326, 2020.
  35. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  36. Zhang, H., Li, L. H., Meng, T., Chang, K.-W., and Broeck, G. V. d. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022. — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2025-justlogic-comprehensive-benchmark-evaluating-deductive/