Elamur Research⌕
☰
home / papers / 2022 / Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences

Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences

Summary

This paper asks whether transformer models fine-tuned for natural language inference actually reason about presuppositions or simply rely on shallow regularities. The authors fine-tune BERT (bert-large-uncased) and RoBERTa (roberta-large) on MNLI, following the ImpPres setup, and evaluate them on the presupposition portion of ImpPres. They run two analyses. The first is contrastive probing. They build modified test sets with simple rules: two produce grammatical variants (dropping the possessed noun, swapping person names) and two corrupt the input (keeping only the first half of the premise, shuffling word order). The second is a manual inspection of 100 random items per sub-dataset to find lexical and structural cues behind correct and incorrect predictions. Grammatical changes that should flip the label to neutral cause substantial accuracy drops. Corrupted inputs often barely hurt performance and sometimes even improve it. The authors conclude that the models exploit task-specific surface cues that fail to generalize, not pragmatic reasoning. This matters for anyone treating NLI benchmark scores as evidence of pragmatic competence.

Key points

  • Models and training: BERT and RoBERTa large models were fine-tuned on MNLI. BERT reached 85.14% dev accuracy, close to earlier reports. Evaluation used the ImpPres presupposition sub-datasets: All N, Both, Change of State, Cleft Existence/Uniqueness, Only, Possessed Definites, and Question.
  • POSS probe: Deleting the possessed noun makes every gold label neutral. A manual check of 100 cases confirmed the rule was always correct. Under this probe, BERT fell from 70.6% to 62.4% accuracy and RoBERTa from 64.8% to 35.2%.
  • Neutral and contradiction errors: On the original Possessed Definites data, BERT's per-class accuracy was about 99% for entailment and contradiction but only 31.5% for neutral. In the contrastive set, 529 of its 941 contradiction predictions stayed contradiction instead of switching to neutral.
  • NAMES probe: Person names were found with spaCy NER and replaced with random names drawn from the 200 most common US names. This sharply reduced BERT's accuracy on several subsets, for example from 70.6% to 26.3% on Possessed Definites Existence.
  • Corrupt-input probes (1ST-HALF, RNDMZ): These often left accuracy nearly unchanged or even raised it. For example, BERT scored 65.8% on Cleft Existence with only the first half of the premise, versus 63.0% on the original. On Possessed Definites Uniqueness, it scored 36.4% with shuffled words, versus 23.3% on the original. Both results point to reliance on cues rather than understanding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kabbara2022investigating,
  title = {{Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences}},
  author = {Kabbara, Jad and Cheung, Jackie Chi Kit},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  doi = {10.48448/03v5-py42},
  url = {https://doi.org/10.48448/03v5-py42}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. find
  2. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. ↗
  3. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  4. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung, volume 23, pages 107–124. ↗ our card
  5. Judith Degen. 2015. Investigating the distribution of some (but not all) implicatures using corpora and web-based methods. Semantics and Pragmatics, 8:11–1. ↗ our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2019. The KnowRef coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3952–3961, Florence, Italy. Association for Computational Linguistics. ↗ our card
  8. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics. ↗ our card
  9. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  10. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗ our card
  11. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceeding of the 2015 International Conference on Learning Representation (ICLR 2015), San Diego, California. ↗
  12. Alice Lai, Yonatan Bisk, and Julia Hockenmaier. 2017. Natural language inference from multiple premises. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 100–109, Taipei, Taiwan. Asian Federation of Natural Language Processing. ↗
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  14. Christopher Manning. 2006. Local textual inference : It’s hard to circumscribe , but you know it when you see it - and nlp needs it. find
  15. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. ↗ our card
  16. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy. Association for Computational Linguistics. find
  17. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗ our card
  18. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguistics. find
  19. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. ↗ our card
  20. Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021. UnNatural Language Inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7329–7346, Online. Association for Computational Linguistics. find
  21. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kabbara-2022-investigating-performance-transformer-based-nli/