Elamur Research⌕
☰
home / papers / 2023 / Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models

Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models

Paper · 2023 · Man Luo, Shrinidhi Kumbhar, Ming shen et al.

Summary

The paper asks how well language models handle logical reasoning expressed in natural language. Formal knowledge-representation systems needed heavy expert effort and did not scale. The authors first survey the field. They cover three reasoning types (deductive, inductive, abductive), the usual task formats, how datasets are built, and how LLMs are applied. They then release LogiGLUE, a benchmark that puts 24 datasets into a unified format for training and testing.

Using LogiGLUE, they fine-tune Flan-T5 with single-task training, multi-task training and chain-of-thought distillation. The resulting model is called LogiT5. Multi-task training and CoT distillation gave the best results. They also test GPT-4, GPT-3.5-turbo and LLaMA-2-chat with zero-shot CoT prompting on held-out test sets.

All three LLMs show the same ranking: abductive reasoning is easiest, then deductive, with inductive weakest. GPT-4 leads overall. The small fine-tuned LogiT5 matches or beats larger LLMs on some inductive and abductive datasets. The work offers a shared, extensible testbed for measuring and improving reasoning in LMs.

Key points

  • LogiGLUE has 24 datasets grouped as deductive, inductive, abductive and mixed. Examples include bAbi, RuleBERT, PrOntoQA, CLUTTR, AbductionRule, ReClor and LogiQA 2.0.
  • The survey sorts logical-reasoning datasets into four task formats: MCQA, FFQA, fact verification and NLI. It names three ways datasets are built: crowdsourcing, exam extraction and synthetic generation.
  • GPT-4 average scores by reasoning type:
    • abductive: about 98.8
    • mixed: 83.0
    • deductive: 73.6
    • inductive: 31.6
  • LogiT5 has under 1B parameters and clearly improves on Flan-T5. Its deductive average rises from 41.5 to 56.7, and its AbductionRule score jumps from 3.8 to 96.1. However, LLMs beat LogiT5 on ReClor and LogiQA, possibly because these public exam datasets appeared in their training data.
  • On LogiQA, adding 15K CoT examples raised Flan-T5 accuracy from 0.37 to 0.41. This suggests CoT distillation helps more as the amount of data grows.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States128 works · 41,697 citations🎓Arizona State UniversityTempe · United States54 works · 2,356 citations🎓Indian Institute of Technology KharagpurKharagpur · India25 works · 714 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya and Chitta Baral. 2023. Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models. arXiv:2310.00836.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.00836 · arXiv 2310.00836

@misc{luo2023logiglue,
  title = {{Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models}},
  author = {Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya and Chitta Baral},
  year = {2023},
  doi = {10.48550/arxiv.2310.00836},
  eprint = {2310.00836},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.00836}
}

References

91 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anonymous. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Submitted to The Twelfth International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=71kocBuhNO. under review. ↗
  2. Stephen H. Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. J. Mach. Learn. Res., 18(1):3846–3912, jan 2017. ISSN 1532-4435. find
  3. Pratyay Banerjee, Chitta Baral, Man Luo, Arindam Mitra, Kuntal Pal, Tran C Son, and Neeraj Varshney. Can transformers reason about effects of actions? arXiv:2012.09938, 2020. — [arXiv:2012.09938](https://arxiv.org/abs/2012.09938) find
  4. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. In International Conference on Learning Representations, 2019.
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Ruth MJ Byrne, Jonathan St BT Evans, and Stephen E Newstead. Human reasoning: The psychology of deduction. Psychology Press, 2019. find
  7. Chunkit Chan, Xin Liu, Tsz Ho Chan, Jiayang Cheng, Yangqiu Song, Ginny Wong, and Simon See. Self-consistent narrative prompts on abductive natural language inference. arXiv preprint arXiv:2309.08303, 2023. find
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  9. Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Christian Bessiere (ed.), IJCAI, 7 2020. doi: 10.24963/ijcai.2020/537. URL https://doi.org/10.24963/ijcai.2020/537. ↗
  10. Alain Colmerauer and Philippe Roussel. The birth of prolog. In History of programming languages— II. 1996. find
  11. Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, 2022. find
  12. Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051, 2022.
  13. Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: A probabilistic prolog and its application in link discovery. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, pp. 2468–2473, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc. find
  14. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019a.
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, Minneapolis, Minnesota, June 2019b. ACL. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/ N19-1423. ↗
  16. Xuemeng Song Liqiang Nie Fangkai Jiao, Yangyang Guo. Merit: Meta-path guided contrastive learning for logical reasoning. arXiv preprint arXiv:2203.00357, 2022. find
  17. Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. Language models can be logical solvers. arXiv preprint arXiv:2311.06158, 2023. — [arXiv:2311.06158](https://arxiv.org/abs/2311.06158) find
  18. Jörg Frohberg and Frank Binder. Crass: A novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2126–2140, 2022. find
  19. Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. arXiv preprint arXiv:2305.14909, 2023.
  20. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, et al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022.
  21. Simon Jerome Han, Keith J Ransom, Andrew Perfors, and Charles Kemp. Inductive reasoning in humans and large language models. Cognitive Systems Research, pp. 101155, 2023. find
  22. Li Erran Li3 Eric Xing Hanlin Zhang1, Yi-Fan Zhang2. The impact of symbolic representations on in-context learning for few-shot reasoning. arXiv preprint arXiv:2212.08686, 2022. — [arXiv:2212.08686](https://arxiv.org/abs/2212.08686) find
  23. Weinan He, Canming Huang, Yongmei Liu, and Xiaodan Zhu. WinoLogic: A zero-shot logic-based diagnostic dataset for Winograd Schema Challenge. In EMNLP, November 2021. URL https://aclanthology.org/2021.emnlp-main.307. ↗
  24. Evan Heit. What Is Induction and Why Study It?, pp. 1–24. Cambridge University Press, 2007. doi: 10.1017/CBO9780511619304.002. — [doi:10.1017/cbo9780511619304.002](https://doi.org/10.1017/cbo9780511619304.002) find
  25. Chadi Helwe, Chloé Clavel, and Fabian Suchanek. Logitorch: A pytorch-based library for logical reasoning on natural language. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022. find
  26. Jerry R Hobbs, Mark E Stickel, Douglas E Appelt, and Paul Martin. Interpretation as abduction. Artificial intelligence, 63(1-2):69–142, 1993. find
  27. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In EMNLP-IJCNLP, November 2019. doi: 10.18653/v1/D19-1243. URL https://aclanthology.org/D19-1243. ↗
  28. Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva. Abduction-based explanations for machine learning models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1511–1519, 2019. find
  29. Mete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva, and Antoine Bosselut. CRoW: Benchmarking commonsense reasoning in real-world tasks. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9785–9821,… find
  30. Fangkai Jiao, Zhiyang Teng, Shafiq Joty, Bosheng Ding, Aixin Sun, Zhengyuan Liu, and Nancy F Chen. Logicllm: Exploring self-supervised logic-enhanced training for large language models. arXiv preprint arXiv:2305.13718, 2023. find
  31. Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. TaxiNLI: Taking a ride up the NLU hill. In CoNLL, November 2020. doi: 10.18653/v1/2020.conll-1.4. URL https://aclanthology.org/2020.conll-1.4. ↗
  32. Jia Deng Kaiyu Yang and Danqi Chen. Generating natural language proofs with verifier-guided search. arXiv preprint arXiv:2205.12443, 2022. find
  33. Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. Lambada: Backward chaining for automated reasoning in natural language. arXiv preprint arXiv:2212.13894, 2022.
  34. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukosiute, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saura… — Measuring Faithfulness in Chain-of-Thought Reasoning
  35. Joohyung Lee and Man Luo. Strong equivalence for lpmln programs. In 35th International Conference on Logic Programming (ICLP 2019), 2018. find
  36. Joohyung Lee and Yi Wang. Weighted rules under the stable model semantics. In KRR, 2016. find
  37. Vladimir Lifschitz. Answer set programming. Springer Berlin, 2019. find
  38. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. Wanli: Worker and ai collaboration for natural language inference dataset creation. arXiv preprint arXiv:2201.05955, 2022.
  39. Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2947–2962, 2023a. doi: 10.1109/ TASLP.2023.3293046. find
  40. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439, 2023b.
  41. Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. Logicot: Logical chain-of-thought instruction-tuning data collection with gpt-4. arXiv preprint arXiv:2305.12147, 2023c. find
  42. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI, 2020. doi: 10.24963/ijcai.2020/501. URL https://doi.org/10.24963/ijcai.2020/501. ↗
  43. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3622–3628, 2…
  44. Yinhan Liu, Myle Ott, Naman Goyal, and et al. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692, 2019.
  45. Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. arXiv preprint arXiv:2103.13009, 2021. find
  46. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023.
  47. Man Luo, Kazuma Hashimoto, Semih Yavuz, Zhiwei Liu, Chitta Baral, and Yingbo Zhou. Choose your qa model wisely: A systematic study of generative and extractive readers for question answering. Spa-NLP 2022, pp. 7, 2022. find
  48. John McCarthy. Artificial intelligence, logic and formalizing common sense. In Philosophical logic and artificial intelligence. Springer, 1989. find
  49. Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. SPARTQA: A textual question answering benchmark for spatial reasoning. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao…
  50. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470–3487, 2022a. find
  51. Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa…
  52. Se-Young Yun Namgyu Ho, Laura Schmid. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022.
  53. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885–4901, 2020.
  54. Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. In ACL, 2019. find
  55. Theo X. Olausson*, Alex Gu*, Ben Lipkin*, Cedegao E. Zhang*, Armando Solar-Lezama, Joshua B. Tenenbaum, and Roger P. Levy. Linc: A neuro-symbolic approach for logical reasoning by combining language models with first-order logic provers. 2023. find
  56. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295, 2023.
  57. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014, 2023.
  58. Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. In-boxbart: Get instructions into biomedical multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 112–128, 2022. find
  59. Gabriele Paul. Approaches to abductive reasoning: an overview. Artificial intelligence review, 7(2): 109–152, 1993. find
  60. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. Kyle Richardson and Ashish Sabharwal. Pushing the limits of rule reasoning in transformers through natural language satisfiability. arXiv:2112.09054, 2021. find
  61. Matthew Richardson and Pedro Domingos. Markov logic networks. Mach. Learn., 62(1–2):107–136, feb 2006. ISSN 0885-6125. doi: 10.1007/s10994-006-5833-1. URL https://doi.org/10. 1007/s10994-006-5833-1. find
  62. Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. RuleBERT: Teaching soft rules to pre-trained lms. In EMNLP, November 2021. URL https://aclanthology.org/ 2021.emnlp-main.110. ↗
  63. Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. Prover: Proof generation for interpretable reasoning over rules. In EMNLP, 2020.
  64. Soumya Sanyal, Yichong Xu, Shuohang Wang, Ziyi Yang, Reid Pryzant, Wenhao Yu, Chenguang Zhu, and Xiang Ren. Apollo: A simple approach for adaptive pretraining of language models for logical reasoning. arXiv preprint arXiv:2212.09282, 2022. find
  65. Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=qFVVBzXxR2V. ↗
  66. Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=qFVVBzXxR2V. ↗
  67. Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. Testing the general deductive reasoning capacity of large language models using ood examples. arXiv preprint arXiv:2305.15269, 2023.
  68. Bruno Sauce and Louis D Matzel. Inductive reasoning. Encyclopedia of animal cognition and behavior, 6:1–8, 2017. find
  69. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  70. Koustuv Sinha, Shagun Sodhani, Jin Dong, and et al. Clutrr: A diagnostic benchmark for inductive reasoning from text. In EMNLP, 2019. find
  71. Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017. find
  72. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  73. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Resear…
  74. Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023. find
  75. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings-ACL-IJCNLP, August 2021. doi: 10.18653/v1/2021.findings-acl.317. URL https://aclanthology.org/2021. findings-acl.317. ↗
  76. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. Wiqa: A dataset for “what if...” reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language… find
  77. Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, Yue Zhang, et al. Glore: Evaluating logical reasoning of large language models. arXiv preprint arXiv:2310.09107, 2023. — [arXiv:2310.09107](https://arxiv.org/abs/2310.09107) find
  78. Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first-order logical reasoning ability through logicnli. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3738–3747, 2021a.
  79. Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first-order logical reasoning ability through LogicNLI. In EMNLP, November 2021b. URL https://aclanthology.org/2021.emnlp-main.303. ↗
  80. Neeraj Varshney, Mihir Parmar, Nisarg Patel, Divij Handa, Sayantan Sarkar, Man Luo, and Chitta Baral. Can nlp models correctly reason over contexts that break the common assumptions? arXiv preprint arXiv:2305.12096, 2023. — [arXiv:2305.12096](https://arxiv.org/abs/2305.12096) find
  81. Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al. Attention is all you need. In NeurIPS, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/ paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. ↗
  82. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
  83. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  84. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015a.
  85. Jason Weston, Antoine Bordes, and et al. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv:1502.05698, 2015b.
  86. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. CoRR, abs/2307.02477, 2023. doi: 10.48550/arXiv. 2307.02477. URL h… find
  87. Zhilin Yang, Zihang Dai, Yiming Yang, and et al. Carbonell. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf. ↗
  88. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In ICLR, 2019.
  89. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations, 2020.
  90. Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022. — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) find
  91. Dataset Abductive NLI CLUTTR Abduction Rules Adversarial NLI LogiNLI proofWriter RuleBERT Folio LogiQA baBi Electric-bird Winologic Wanli BigBench ProntoQA License Apache-2.0 license Attribution-NonCommercial 4.0 International MIT License Attribution-NonCommercial 4.0 International Creative Commons… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2023-towards-logiglue-brief-survey-benchmark/