Elamur Research⌕
☰
home / papers / 2024 / Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic

Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic

Paper · 2024 · Nathaniel Weir, Kate Sanders, Orion Weller et al.

Summary

Entailment-tree reasoners answer a question by recursively splitting a hypothesis into premises that should jointly support it. They are only trustworthy if every split is actually valid, but no clear standard existed for judging such decompositions. As a result, earlier datasets had noisy labels and verifiers improved little. The authors borrow the Relevance–Acceptability–Sufficiency (RAS) framework from informal logic. Instead of one yes/no entailment label, annotators rate each premise set on separate ordinal facets, and these facets feed a final sufficiency judgment. Using this protocol, they build RDTE, over 1000 expert-labelled decompositions drawn from tree searches on EntailmentBank and HotpotQA. RDTE is about 9% more internally consistent than earlier decompositional entailment sets. Given the same rubric, GPT-4 acts as a teacher that labels large amounts of silver data. Student ChatGPT and RoBERTa verifiers are trained on it and plugged into a new engine, TreeWise. With these verifiers, both answer accuracy and tree quality rise. This shows that a well-defined notion of a valid step pays off downstream.

Key points

  • RDTE (Recognizing Decompositional Textual Entailment) has 1000+ expert annotations, each scored on RAS-inspired ordinal facets that lead to a sufficiency label. Its internal consistency is about +9% over prior decompositional datasets.
  • The judging "audience" follows Manning (2006): an attentive, informed layperson using ordinary world knowledge, not a semanticist.
  • GPT-4, prompted with the human annotation instructions, labelled a released silver set of 24K items per domain. Student ChatGPT and RoBERTa models were distilled from it.
  • TreeWise builds on NELLIE and adds in-context learning, forward chaining, branch consolidation and the distilled verifiers. It was evaluated on 340 EntailmentBank test questions (with WorldTree or Wikipedia as knowledge) and 419 HotpotQA questions converted to multiple choice.
  • With distillation, TreeWise on EBQA/WorldTree rose from 72.8 to 79.2 QA and from 71.6 to 75.2 tree integrity. On HotpotQA, QA went from 46.1 to 51.3. Tree integrity improved for every tree-building method tested.

Interesting ideas

  • I1. Split one fuzzy validity judgment into separate graded facets (is it relevant, is it acceptable, is it sufficient), so annotators and models disagree less and errors can be named.
  • I2. Fix an explicit imagined reader (a careful, informed non-expert) as the standard for what counts as an acceptable inference.
  • I3. Give an LLM teacher exactly the same rubric as human annotators, then distil its labels into small task-specific verifiers.
  • I4. Rate a whole reasoning chain by its weakest step, taking the minimum rather than the average of step scores.
  • I5. Generate several candidate proofs per answer option and pick the answer whose best proof scores highest under the verifier.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): When prag links a clause to a claimed implicature or intent, it could score relevance and sufficiency separately, so the explanation shows which facet failed.
  • I2 (world): Write down which background facts from WordNet/FrameNet/Wikidata such a reader may assume, so inferences in the text world have a stable, documented threshold.
  • I3 (learn): Our LLM teacher tiers could receive the human guideline text verbatim, so that silver labels for frames or answers stay comparable with gold labels.
  • I4 (answer): School-test and word-problem solvers could report the lowest-confidence step as the chain's confidence, flagging answers that are right for wrong reasons.
  • I5 (answer): For multiple-choice tests, build a justification per option over the text world and choose the option with the strongest weakest-link justification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of ArizonaTucson · United States38 works · 10,603 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark and Benjamin Van Durme. 2024. Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.531

@inproceedings{weir2024enhancing,
  title = {{Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic}},
  author = {Nathaniel Weir, Kate Sanders, Orion Weller, Shreya Sharma, Dongwei Jiang, Zhengping Jiang, Bhavana Dalvi Mishra, Oyvind Tafjord, Peter Jansen, Peter Clark and Benjamin {Van Durme}},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.531},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.531}
}

References

44 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabor Angeli and Christopher D. Manning. 2014. NaturalLI: Natural logic inference for common sense reasoning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 534–545. Association for Computational Linguistics.
  2. J Anthony Blair. 2012. Relevance, acceptability and sufficiency today. Groundwork in the Theory of Argumentation: Selected Papers of J. Anthony Blair, pages 87–100. ↗
  3. Johan Bos. 2014. Recognizing textual entailment and computational semantics. In Computing Meaning: Volume 4, pages 89–105. Springer. ↗
  4. Kaj Bostrom, Zayne Sprague, Swarat Chaudhuri, and Greg Durrett. 2022. Natural language deduction through search over statement compositions. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4871–4883. Association for Computational Linguistics. find
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics. ↗
  6. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. Uncertain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779. Association for Computational Linguistics.
  7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. ↗
  8. Peter Clark, Bhavana Dalvi Mishra, and Oyvind Tafjord. 2023. BaRDa: A belief and reasoning dataset that separates factual accuracy and reasoning ability. arXiv preprint arXiv:2312.07527. ↗
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop. find
  10. Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, and Peter Clark. 2021. Explaining answers with entailment trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7358–7370. Association for Computational Linguistics.
  11. Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02922. ↗
  12. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey for in-context learning. ArXiv, abs/2301.00234. ↗
  13. Max Glockner, Ieva Staliunaite, James Thorne, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Ambifc: Fact-checking ambiguous claims with evidence. Transactions of the Association for Computational Linguistics, 12:1–18. find
  14. Leo Groarke. 2022. Informal Logic. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy, winter 2022 edition. Metaphysics Research Lab, Stanford University. find
  15. Reto Gubelmann, Ioannis Katis, Christina Marianne Niklaus, and Siegfried Handschuh. 2023. Capturing the varieties of natural language inference: A systematic survey of existing datasets and two novel benchmarks. Journal of Logic, Language and Information, pages 1–28. ↗
  16. Ruixin Hong, Hongming Zhang, Xintong Yu, and Changshui Zhang. 2022. METGEN: A module-based entailment tree generation framework for answer explanation. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1887–1905. Association for Computational Linguistics. find
  17. Ruixin Hong, Hongming Zhang, Hong Zhao, Dong Yu, and Changshui Zhang. 2023. Faithful question answering with Monte-Carlo planning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3944–3965. Association for Computational Linguistics. find
  18. Harsh Jhamtani and Peter Clark. 2020. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 137–150. Association for Computational Linguistics. find
  19. Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180–7198. Association for Computational Linguistics. ↗
  20. Ralph H. Johnson and J. Anthony Blair. 1977. Logical self-defense. find
  21. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090. ↗
  22. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362. find
  23. Christopher D Manning. 2006. Local textual inference: it’s hard to circumscribe, but you know it when you see it–and nlp needs it. find
  24. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774. ↗
  25. ArXiv, Justin Sulik, Jeroen van Paridon, and Gary Lupyan. 2021. Explanations in the wild. Cognition, 237. ↗
  26. Raquel Mochales Palau and Marie-Francine Moens. 2009. Argumentation mining: the detection, classification and structure of arguments in text. In Proceedings of the 12th International Conference on Artificial Intelligence and Law, ICAIL ’09, page 98–107, New York, NY, USA. Association for Computing Machinery. find
  27. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2022. Entailer: Answering questions with faithful and truthful chains of reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2078–2093. Association for Computational Linguistics. find
  28. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  29. Chenhao Tan. 2022. On the diversity and limits of human explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2173–2188. Association for Computational Linguistics. ↗
  30. Nils Reimers and Iryna Gurevych. 2019a. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. Association for Computational Linguistics.
  31. Marco Valentino, Ian Pratt-Hartmann, and André Freitas. 2021. Do natural language explanations represent valid logical arguments? verifying entailment in explainable NLI gold standards. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 76–86. Association for Computational Linguistics. ↗
  32. Nils Reimers and Iryna Gurevych. 2019b. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  33. Danilo Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henghui Zhu, Xinchi Chen, Peng Xu, Zhiheng Huang, Andrew Arnold, and Dan Roth. 2022. Entailment tree explanations via iterative retrieval-generation reasoner. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 465–475, Seattle, United States. Association for Computational Linguistics. ↗
  34. Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Henghui Zhu, Rui Dong, Deguang Kong, Juliette Burger, Anjelica Ramos, zhiheng huang, William Yang Wang, George Karypis, Bing Xiang, and Dan Roth. 2023. STREET: A MULTI-TASK STRUCTURED REASONING AND EXPLANA-TION BENCHMARK. In The Eleventh International Conference on Learning Representations. find
  35. Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
  36. Kate Sanders, Nathaniel Weir, and Benjamin Van Durme. 2024. Tv-trees: Multimodal entailment trees for neuro-symbolic video reasoning. EMNLP. find
  37. Christian Stab and Iryna Gurevych. 2017. Recognizing insufficiently supported arguments in argumentative essays. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 980–990. Association for Computational Linguistics. find
  38. Nathaniel Weir, Peter Clark, and Benjamin Van Durme. 2024. NELLIE: A neuro-symbolic inference engine for grounded, compositional, and explainable reasoning. In Proceedings of IJCAI. find
  39. Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). ↗
  40. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗
  41. Zhengnan Xie, Sebastian Thiem, Jaycie Martin, Elizabeth Wainwright, Steven Marmorstein, and Peter Jansen. 2020. WorldTree v2: A corpus of science-domain structured explanations and inference patterns supporting multi-hop inference. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5456–5473. European Language Resources Association. find
  42. Kaiyu Yang, Jia Deng, and Danqi Chen. 2022. Generating natural language proofs with verifier-guided search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 89–105. Association for Computational Linguistics. find
  43. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering.
  44. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weir-2024-enhancing-systematic-decompositional-natural-language/