Elamur Research⌕
☰
home / papers / 2021 / Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI

Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI

Paper · 2021 · Jidong Tian, Yitian Li, Wenqing Chen et al.

Summary

Pretrained Transformers score well on many NLU benchmarks, but it is unclear whether they actually perform first-order logic (FOL) reasoning or exploit dataset artifacts and commonsense cues. The authors address this with LogicNLI, a semi-automatically generated NLI-style benchmark. Each instance pairs a premise of facts and rules with a statement, and the facts and rules are built from logic templates over names and adjectives. The predicates are deliberately meaningless, so logic is separated from world knowledge. The label set adds a fourth class, Paradox, for cases where both a statement and its negation can be derived. Every instance also includes its proofs and reasoning path. Using this resource, they evaluate BERT, RoBERTa, and XLNet on four aspects:

  • accuracy
  • robustness to irrelevant sentences
  • generalization to more reasoning hops
  • proof-based traceability

RoBERTa performs best, yet none of the models reasons reliably or generalizes as humans do. This matters because it shows that strong NLU scores can mask shallow logic. It also gives a reusable recipe for building disentangled reasoning diagnostics.

Key points

  • LogicNLI has over 30K instances and covers all seven FOL operators: ∧, ∨, ¬, →, ≡, ∀, and ∃. It is compared with LogiQA, ReClor, CLUTRR, LTL, and SoftReasoner, each of which covers fewer operators or lacks natural language or proofs.
  • The task assumes an open world and uses four labels: Entailment, Contradiction, Neutral, and Paradox. Because of Paradox, a model must explore a minimum of two reasoning paths, which lessens its dependence on bias.
  • Separate test sets target each diagnostic aspect. Traceability is measured with P-EM, an exact-match score on proofs.
  • In a variant without Paradox (d-LogicNLI), models become almost untraceable, with a maximum P-EM of 4.1% (XLNet). Their accuracy drops sharply on harder multi-hop tests, with δA→G of 44.0% for BERT, 40.0% for RoBERTa, and 36.8% for XLNet.
  • RoBERTa leads on LogicNLI and XLNet leads on d-LogicNLI. The authors attribute RoBERTa's lead to its larger pretraining corpus, which helps on more complex reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China63 works · 15,047 citations

Also on the byline: State Key Laboratory of Advanced Optical Communication Systems and Networks.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He and Yaohui Jin. 2021. Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.303

@inproceedings{tian2021diagnosing,
  title = {{Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI}},
  author = {Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He and Yaohui Jin},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.303},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.303}
}

References

31 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Md Mosharaf Hossain, Kathleen E. Hamilton, Alexis Palmer, and Eduardo Blanco. 2020b. Predicting the focus of negation: Model and error analysis. In ACL. find
  2. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In NAACL-HLT.
  3. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020c. An analysis of natural language inference benchmarks through the lens of negation. In EMNLP.
  4. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: machine reading comprehension with contextual commonsense reasoning. In EMNLP-IJCNLP, pages 2391–2401.
  5. Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. 2019. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In ACL. find
  6. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models imppressive? learning implicature and presupposition. In ACL.
  7. Shane Storks, Qiaozi Gao, and Joyce Y. Chai. 2019. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. CoRR.
  8. Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In ACL.
  9. Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020. Assessing the benchmarking capacity of machine reading comprehension datasets. In AAAI. find
  10. Aikaterini-Lida Kalouli, Richard S. Crouch, and Valeria de Paiva. 2020. Hy-nli: a hybrid system for natural language inference. In COLING. find
  11. Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In ACL.
  12. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In EMNLP.
  13. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In IJCAI.
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR. find
  15. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In ACL.
  16. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In ACL.
  17. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. Quartz: An open-domain dataset of qualitative relationship questions. In EMNLP-IJCNLP. find
  18. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. olmpics - on what language model pre-training captures. TACL.
  19. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACL-HLT.
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
  21. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS.
  22. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
  23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP.
  24. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT. find
  25. Abhilasha Ravichander, Aakanksha Naik, Carolyn Penstein Rosé, and Eduard H. Hovy. 2019. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In CoNLL.
  26. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
  27. Marco Túlio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In ACL. find
  28. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
  29. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI. find
  30. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le-Bras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. CoRR.
  31. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. In ICLR.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/