Elamur Research⌕
☰
home / papers / 2018 / Hypothesis Only Baselines in Natural Language Inference

Hypothesis Only Baselines in Natural Language Inference

Paper · 2018 · Adam Poliak, Jason Naradowsky, Aparajita Haldar et al.

Summary

The authors test whether Natural Language Inference (NLI) datasets actually force a model to analyze the relationship between the premise and the hypothesis. To do so, they train a classifier that sees only the hypothesis and ignores the context entirely. It is a modified InferSent: a BiLSTM encoder with max-pooling and an MLP classifier over 300-dimensional GloVe vectors. This "degenerate" model is compared with a majority-class baseline on ten corpora. The corpora are divided into three groups by how they were created: recast (DPR, SPR, FN+), judged (ADD-1, SciTail, SICK, MPE, JOCI), and elicited (SNLI, Multi-NLI). On many of them, the premise-free model clearly outperforms the majority class. According to the authors' analysis, it exploits statistical artifacts in the data, such as characteristic words or the grammaticality of sentences, as well as background knowledge about the world. Two conclusions follow. High accuracy on an NLI benchmark does not by itself prove that a model understands language. The majority baseline is a poor indicator of the true difficulty of the task. The authors therefore recommend including a hypothesis-only baseline in the description of every new NLI corpus.

Key points

  • Ten NLI corpora of varying size are studied: from DPR (~3.4K pairs) to SNLI (~550K) and Multi-NLI (~425K). These corpora have 2 or 3 label classes.
  • On FN+, the premise-free model reaches 76.31% for the NOT-ENTAILED class versus 56.79% for the majority baseline (+34.37%). For ENTAILED, the result barely changes: 44.18% versus 43.20%.
  • The authors manually examined 200 sentences from FN+. Among the hypotheses the model correctly labeled as ENTAILED, 88% are grammatical, whereas among those labeled incorrectly only 38% are. The model thus likely picks up on a correlation between a sentence's grammaticality and its label.
  • On SPR (956 hypotheses per property), the largest gain comes from the property "aware": 88.70% versus 59.94% (+47.99%). The model guesses the answer from the lexical semantics of the argument, for example whether it is sentient; that is, it relies on world knowledge rather than on the context.
  • The code and data are openly available on GitHub (azpoliak/hypothesis-only-NLI). Independently of the authors, Tsuchiya (2018) and Gururangan et al. (2018) obtained similar results on three of these ten corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓Birla Institute of Technology and Science, PilaniPilāni · India16 works · 41,879 citations🎓Birla Institute of Technology and Science, Pilani - Goa CampusSancoale · India4 works · 1,114 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Natural language understanding · 51 works

Cite

Cite as Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-2023 · arXiv 1805.01042

@inproceedings{poliak2018hypothesis,
  title = {{Hypothesis Only Baselines in Natural Language Inference}},
  author = {Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger and Benjamin {Van Durme}},
  booktitle = {Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics},
  year = {2018},
  doi = {10.18653/v1/s18-2023},
  eprint = {1805.01042},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/s18-2023}
}

References

2 references, 1 of them with a card of ours. ↗ leads to the work.

  1. Ion Androutsopoulos and Prodromos Malakasiotis. 2010. A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research, 38:135–187. ↗
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425–2433. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/