Elamur Research⌕
☰
home / papers / 2023 / Conditional Natural Language Inference

Conditional Natural Language Inference

Paper · 2023 · Youngwoo Kim, Razieh Rahimi and James Allan

Summary

Some sentence pairs look contradictory but actually describe different situations. For example, two biomedical claims may give opposite answers about antihypertensive drugs because they concern different drugs or patient groups. Standard NLI labels such pairs neutral, which lumps them together with unrelated pairs. The authors define Conditional NLI (Cond-NLI), a token-level task: for a claim pair, mark which spans carry the contradictory outcome and which spans express conditions not shared by the other claim.

They build BioClaim by adding token-level annotations to the PCC corpus of potentially contradictory biomedical claims. They also propose PAT (Partial-ATtention), a BERT-base model that:

  • splits the hypothesis into two parts,
  • predicts an NLI label for each part against the premise,
  • combines the two labels into the sentence-level decision.

Training uses only sentence-level NLI data. Token-level attributions then fall out of the partition predictions.

PAT beats feature-attribution explainers and a full cross-encoder at finding condition tokens. It roughly matches InstructGPT/ChatGPT on BioClaim and outperforms ChatGPT on SciEntsBank. This shows that a small, structurally interpretable model can recover fine-grained "under which condition" information.

Key points

  • PCC, the base corpus for BioClaim, contains 24 closed-form research questions and 259 claims. BioClaim adds token-level annotations of contradiction and neutral (condition) spans.
  • On SciEntsBank (partial entailment), PAT reaches a mean macro F1 of 0.789. This is ahead of Co-attention (0.754), word2vec (0.763), exact match (0.759), SLR (0.711), LIME (0.657) and ChatGPT (0.674). Similarity baselines do win on the Unseen Domain split.
  • PAT has about 110M parameters (BERT-base), versus 175B for GPT-3, yet it performs comparably on BioClaim.
  • On condition tokens, PAT is up to 8% more accurate than the strongest NLI-based baseline.
  • PAT slightly trails a full cross-encoder BERT on plain sentence-level NLI. Robustness checks on e-SNLI and MNLIEx show it is competitive with LIME and SE-NLI.

Interesting ideas

  • I1. Have the classifier predict labels for pieces of the input and derive the whole-input label by a fixed combination rule, so explanations come from the architecture rather than post-hoc attribution.
  • I2. During training, choose the split points of the hypothesis at random rather than fixing span boundaries, so the model learns to judge arbitrary fragments.
  • I3. Restricting the split to exactly two parts lets aggregation be a simple matrix product over label distributions, which beat fuzzy-logic aggregation in accuracy.
  • I4. Treat "different condition" as its own relation, distinct from both contradiction and unrelatedness, by marking the tokens that separate the two contexts.
  • I5. Perturbation explainers fail when the neutral content is long, because removing one token barely moves the prediction while removing many pushes inputs out of distribution.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A frame-compatibility checker could score each argument slot of two frames separately and combine them, so it can name the exact role (agent, patient, condition) that clashes.
  • I2 (learn): When training span-level judges on silver data, random subtree or span cuts could make them robust to whatever segmentation the parser later produces.
  • I3 (world): Merging the states asserted by two clauses could use a small fixed label-combination table, keeping the reasoning transparent and auditable.
  • I4 (answer): When readers find conflicting statements in a text, they could report "yes under X, no under Y" instead of picking one or giving up.
  • I5 (learn): Negative-control evaluations of explanations should include long, multi-condition inputs, where deletion-based attribution is known to break down.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States106 works · 66,692 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Youngwoo Kim, Razieh Rahimi and James Allan. 2023. Conditional Natural Language Inference. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.456

@inproceedings{kim2023conditional,
  title = {{Conditional Natural Language Inference}},
  author = {Youngwoo Kim, Razieh Rahimi and James Allan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.456},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.456}
}

References

38 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdulaziz Alamri and Mark Stevenson. 2016. A corpus of potentially contradictory research claims from cardiovascular research abstracts. Journal of biomedical semantics, 7(1):1–9. ↗
  2. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural ence. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, NIPS 2018, Montreal, Canada, December 3-8, 2018. ↗
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  6. Björn Dahlöf, Richard B Devereux, Sverre E Kjeldsen, Stevo Julius, Gareth Beevers, Ulf de Faire, Frej Fyhrquist, Hans Ibsen, Krister Kristiansson, Ole Lederballe-Pedersen, et al. 2002. Cardiovascular morbidity and mortality in the losartan intervention for endpoint reduction in hypertension study (life): a randomised trial against atenolol. The Lancet, 359(9311):995–1003. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458, Online. Association for Computational Linguistics.
  9. Myroslava O Dzikovska, Rodney Nielsen, and Chris Brew. 2012. Towards effective tutorial feedback for explanation questions: A dataset and baselines. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 200–210. find
  10. Myroslava O Dzikovska, Rodney Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa Trang Dang. 2013. Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Second joint conference on lexical and computational semantics (* SEM), Volume 2: Proceedings of the seventh international workshop on semantic evaluation (SemEval 2013), pages 263–274.
  11. Yufei Feng, Xiaoyu Yang, Xiaodan Zhu, and Michael Greenspan. 2022. Neuro-symbolic natural logic with introspective revision for natural language inference. Transactions of the Association for Computational Linguistics, 10:240–256. Paraphrasing, RTE ’07, pages 1–9, Stroudsburg, PA, USA. Association for Computational Linguistics. ↗
  12. Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80–89. IEEE. find
  13. Peter Hase, Harry Xie, and Mohit Bansal. 2021. The out-of-distribution problem in explainability and search methods for feature importance explanations. Advances in Neural Information Processing Systems, 34. ↗
  14. Zhongtao Jiang, Yuanzhe Zhang, Zhao Yang, Jun Zhao, and Kang Liu. 2021. Alignment rationale for natural ence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5372–5387. find
  15. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI. find
  16. Youngwoo Kim, Myungha Jang, and James Allan. 2020. Explaining text matching on neural natural language inference. ACM Transactions on Information Systems (TOIS), 38(4):1–23. ↗
  17. Sawan Kumar and Partha Talukdar. 2020. Nile: Natural language inference with faithful natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8730–8742.
  18. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117.
  19. Omer Levy, Torsten Zesch, Ido Dagan, and Iryna Gurevych. 2013. Recognizing partial textual entailment. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 451–455. find
  20. Tao Li and Vivek Srikumar. 2019. Augmenting neural networks with first-order logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 292–302. ↗
  21. Reza Ghaeini, Xiaoli Z Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural ence. arXiv preprint arXiv:1808.03894. ↗
  22. Shusen Liu, Zhimin Li, Tao Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018. Nlize: A perturbation-driven visual interrogation tool for analyzing and interpreting natural ence models. IEEE transactions on visualization and computer graphics, 25(1):651–660. find
  23. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Yoshio Matsui, Kazuo Eguchi, Seiichi Shibasaki, Joji Ishikawa, Satoshi Hoshide, Thomas G Pickering, Kazuyuki Shimada, Kazuomi Kario, JMS-1 study group, et al. 2008. Effect of doxazosin on the left ventricular structure and function in morning hypertensive patients: the japan morning surge 1 study. Journal of hypertension, 26(7):1463–1471.
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
  25. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797. ↗
  26. OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt. Accessed: 2023-06-23. ↗
  27. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. ↗
  28. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. ↗
  29. Zijun Wu, Atharva Naik, Zi Xuan Zhang, and Lili Mou. 2021. Weakly supervised explainable phrasal reasoning with neural fuzzy logic. arXiv preprint arXiv:2109.08927. ↗
  30. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer. ↗
  31. Xinyan Zhao and VG Vinod Vydiswaran. 2021. Lirex: Augmenting ence with relevant explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14532–14539. find
  32. Xiang Zhou and Mohit Bansal. 2020. Towards robustifying nli models against lexical dataset biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8759– 8771.
  33. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM. ↗
  34. Rodrygo LT Santos, Craig Macdonald, and Iadh Ounis. 2011. Aggregated search result diversification. In Conference on the Theory of Information Retrieval, pages 250–261. Springer. find
  35. Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. 2022. Logical reasoning with span-level predictions for interpretable and robust nli models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3809–3823. find
  36. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In NAACL-HLT.
  37. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Generating token-level explanations for natural language inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 963–969. ↗
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2023-conditional-natural-language-inference/