Elamur Research⌕
☰
home / papers / 2022 / Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences

Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences

Paper · 2022 · Cheng Luo, Wei Liu, Jieyu Lin et al.

Summary

Transformer models fine-tuned on SNLI and MNLI reach high accuracy, but it is unclear whether they actually reason about sentence meaning or exploit dataset artifacts. Instead of building harder benchmarks, the authors test whether models can handle very plain sentences. They built Simple Pair, a template-generated test set of short N-is-A and subject-verb-object sentences, plus premises made by joining two such clauses. Every premise-hypothesis pair is designed to be neutral. BERT, RoBERTa and DeBERTa (base and large), fine-tuned on MNLI or SNLI, mostly fail, scoring below 35.4% accuracy, which is under three-way chance. The analysis points to two problems:

  • Event coreference bias: the models assume both sentences describe the same event, so they label unrelated sentences as contradiction.
  • Compositional binding failure: in conjoined clauses, they confuse which subject goes with which predicate.

Adding about 1% extra training data built from Simple Pair raises accuracy to roughly 100% while keeping MNLI/SNLI performance. Part of the gain carries over to Extended Pair, a harder set built from real corpus sentences. The broader point is that simple, controlled probes expose gaps that complex benchmarks hide.

Key points

  • Models tested: BERT, RoBERTa and DeBERTa, base and large, built with Huggingface. Each was fine-tuned on MNLI or SNLI and reached over 83.9% on their original test sets.
  • Simple Pair, simple-sentence part: 24,000 N-is-A pairs and 32,000 SVO pairs. Nouns come from five categories: fruits, animals, humans, names and objects. The pairs contain no antonyms or synonyms.
  • Simple Pair, other parts:
    • A conjunction-sentence set with 16,000 pairs for each of four templates.
    • Premise-relevant and premise-irrelevant variants, totalling 12,000 N-is-A pairs and 16,000 SVO pairs.
  • Extended Pair: 6000 extended-simple pairs, made by randomly re-pairing MNLI/SNLI test sentences. Another 6000 extended-conjunction pairs come from joining corpus sentences and breaking subject–predicate links.
  • Validation: humans checked 300 sampled pairs. A few turned out not to be neutral, which the authors judge too few to explain the model failures. Gains on Extended Pair were smaller than on Simple Pair.

Interesting ideas

  • I1. Fixing the gold label of a whole test set to one class (neutral) turns every other prediction into a visible bias, so a single label distribution shows the error pattern without any other analysis.
  • I2. Randomly re-pairing sentences from an existing corpus yields neutral pairs for free, giving a cheap control set drawn from realistic text.
  • I3. Joining a hypothesis with either a premise-matching clause or an unrelated clause forms a minimal contrast, which isolates whether partial overlap alone pushes the model's decision.
  • I4. Swapping predicates between conjoined clauses tests whether a system keeps each subject tied to its own predicate.
  • I5. The coreference bias comes from an annotation convention: SNLI assumed both sentences describe one scene. This shows that "same event" should be an explicit assumption, not a silent default.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): build reader tests where the correct reply is always "not stated in the text," to catch readers that invent facts or contradictions.
  • I2 (learn): shuffle questions across fables to produce negative controls, where any confident answer signals leakage or a shallow shortcut.
  • I3 (learn): use such matched pairs in evaluation to check that answers depend on the relevant clause rather than on lexical overlap.
  • I4 (frames): generate conj sentences with swapped predicates and verify that the extracted who-did-what-to-whom frames follow the UD tree rather than word proximity.
  • I5 (world): merge events from different sentences only when there is evidence, and record that merge decision so it stays explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China46 works · 11,542 citations🎓University of ChicagoChicago · United States28 works · 49,773 citations🔬Zhejiang LabHangzhou · China3 works · 1,587 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Reasoning · 510 works

Cite

Cite as Cheng Luo, Wei Liu, Jieyu Lin, Jiajie Zou, Ming Xiang and Nai Ding. 2022. Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences. Findings of the Association for Computational Linguistics: EMNLP 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.252

@inproceedings{luo2022simple,
  title = {{Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences}},
  author = {Cheng Luo, Wei Liu, Jieyu Lin, Jiajie Zou, Ming Xiang and Nai Ding},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.252},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.252}
}

References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. ↗
  2. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics. ↗
  3. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium. find
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163–173, Online. Association for Computational Linguistics.
  7. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for Computational Linguistics. ↗
  8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  9. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. ↗
  10. Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022. An analysis of negation in natural language understanding corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 716–723, Dublin, Ireland. Association for Computational Linguistics. find
  11. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. ↗
  12. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  13. Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2021. Natural language inference in context-investigating contextual reasoning over long texts. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13388–13396. find
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  15. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. ↗
  16. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  17. Nikita Nangia and Samuel R. Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4566–4575, Florence, Italy. Association for Computational Linguistics. find
  18. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. ↗
  19. Adam Poliak. 2020. A survey on recognizing textual entailment as an NLP evaluation. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 92–109, Online. Association for Computational Linguistics.
  20. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  21. Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361, Hong Kong, China. Association for Computational Linguistics.
  22. Kyle Richardson, Hai Hu, Lawrence Moss, and Ashish Sabharwal. 2020. Probing natural language inference models through semantic fragments. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8713–8721. ↗
  23. Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021. UnNatural Language Inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7329–7346, Online. Association for Computational Linguistics.
  24. Noah A Smith. 2012. Adversarial evaluation for models of natural language. arXiv preprint arXiv:1207.0245. ↗
  25. Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. arXiv preprint arXiv:1904.01172. ↗
  26. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. ↗
  27. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  29. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn systematicity of monotonicity inference in natural language? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6105–6117, Online. Association for Computational Linguistics. ↗
  30. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 250–255, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  31. Aarne Talman and Stergios Chatzikyriakidis. 2019. Testing the generalization power of neural network models across NLI benchmarks. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 85– 94, Florence, Italy. Association for Computational Linguistics. find
  32. Haohan Wang, Da Sun, and Eric P Xing. 2019. What if we simply swap the two text fragments? a straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7136–7143. A Appendices Appendix Table 1: Hyper-parameters for fine-tuning models. The performance of models on MNLI, and SNLI test sets is shown in the last row. Appendix Table 2: Human classification for premise-hypothesis pairs randomly selected from the Simple Pair and Extended Pair sets. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2022-simple-but-challenging-natural-language/