Elamur Research⌕
☰
home / papers / 2015 / A large annotated corpus for learning natural language inference

A large annotated corpus for learning natural language inference

Paper · 2015 · Samuel R. Bowman, Gabor Angeli, Christopher Potts et al.

Summary

The paper addresses the task of natural language inference (NLI). The goal is to determine whether the second sentence follows from the first, contradicts it, or is unrelated to it. The authors note that existing corpora for this task suffer from three shortcomings. They are too small for modern models, some of them were generated automatically, and their labels are often ambiguous because of problems with event and entity coreference. To address this, they created the SNLI corpus. About 2.5 thousand workers on Amazon Mechanical Turk were given a caption of a Flickr30k photo without the image itself. For each caption they wrote three new sentences: one definitely true, one possibly true, and one definitely false. All sentences describe the same scene, so coreference problems become far less frequent. On this corpus the authors compared rule-based systems, a lexicalized classifier with many features, and an LSTM-based model. The latter two achieved nearly identical accuracy. The LSTM trained on SNLI was then transferred to SICK, yielding the best result among neural networks at the time. The significance of the work is that it provided, for the first time, a large NLI corpus in which all sentences and labels were produced by humans. Its size is sufficient for training neural network models.

Key points

  • SNLI contains 570,152 sentence pairs. This is two orders of magnitude more than other comparable resources. The labels are entailment, neutral, and contradiction, and the classes are balanced, since each worker wrote one sentence for each label.
  • For 56,941 pairs, four additional independent judgments were collected. In 98% of cases at least three annotators assigned the same label, and in 58% all five agreed.
  • The premise averages 14.1 tokens, the hypothesis 8.3. The corpus contains 37,026 distinct words. About 4 thousand additional pilot pairs from VisualGenome appear only in train.
  • The LSTM's accuracy on the SICK test set depends on training: SICK only — 71.3%, SNLI only — 46.7%, with SNLI→SICK transfer — 80.8%. The best overall result at the time was 84.6% (Lai and Hockenmaier, 2014), and inter-annotator agreement is about 84%.
  • The same transfer to RTE-3 did not yield competitive results: 800 training examples were not enough for the model to adapt to an unfamiliar text genre.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

💡 Concepts 2Key ideas and methods explained on our concept pages.Natural language understanding · 51 worksLinguistic annotation · 26 works

Cite

Cite as Samuel R. Bowman, Gabor Angeli, Christopher Potts and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1075 · arXiv 1508.05326

@inproceedings{bowman2015large,
  title = {{A large annotated corpus for learning natural language inference}},
  author = {Samuel R. Bowman, Gabor Angeli, Christopher Potts and Christopher D. Manning},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1075},
  eprint = {1508.05326},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1075}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proc. EMNLP.
  2. Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015. Recursive neural networks can learn logical semantics. In Proc. of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. find
  3. Timothy Chklovski and Patrick Pantel. 2004. Verb-Ocean: Mining the web for fine-grained semantic verb relations. In Proc. EMNLP. find
  4. Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In Proc. of the HLT-NAACL 2003 Workshop on Text Meaning. find
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. Evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. ↗
  6. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proc. ACL.
  7. W. Nelson Francis and Henry Kucera. 1979. Brown corpus manual. Brown University. find
  8. Yaroslav Fyodorov, Yoad Winter, and Nissim Francez. 2000. A natural logic inference system. In Proc. of the 2nd Workshop on Inference in Computational Semantics. find
  9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proc. of the ACL-PASCAL workshop on textual entailment and paraphrasing.
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  11. Jerrold J. Katz. 1972. Semantic Theory. Harper & Row, New York. find
  12. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proc. ACL.
  13. Alice Lai and Julia Hockenmaier. 2014. Illinois-LH: A denotational and distributional approach to semantics. In Proc. SemEval. find
  14. Hector J. Levesque. 2013. On our best behaviour. In Proc. AAAI. ↗
  15. Omer Levy, Ido Dagan, and Jacob Goldberger. 2014. Focused entailment graphs for open IE propositions. In Proc. CoNLL. find
  16. Bill MacCartney and Christopher D Manning. 2009. An extended model of natural logic. In Proc. of the Eighth International Conference on Computational Semantics. find
  17. Bernardo Magnini, Roberto Zanoli, Ido Dagan, Kathrin Eichler, Günter Neumann, Tae-Gil Noh, Sebastian Pado, Asher Stern, and Omer Levy. 2014. The Excitement Open Platform for textual inferences. Proc. ACL. find
  18. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proc. SemEval.
  19. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014b. A SICK cure for the evaluation of compositional distributional semantic models. In Proc. LREC.
  20. George A Miller. 1995. WordNet: a lexical database for english. Communications of the ACM, 38(11):39–41. ↗
  21. Sebastian Padó, Tae-Gil Noh, Asher Stern, Rui Wang, and Roberto Zanoli. 2014. Design and realization of a modular architecture for textual entailment. Journal of Natural Language Engineering. ↗
  22. Ellie Pavlick, Johan Bos, Malvina Nissim, Charley Beller, Ben Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proc. ACL.
  23. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proc. EMNLP.
  24. Lorien Y Pratt, Jack Mostow, Candace A Kamm, and Ace A Kamm. 1991. Direct transfer of learned information among neural networks. In Proc. AAAI. find
  25. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP.
  26. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR. find
  27. Johan van Benthem. 2008. A brief history of natural logic. In M. Chakraborty, B. Löwe, M. Nath Mitra, and S. Sarukki, editors, Logic, Navya-Nyaya and Applications: Homage to Bimal Matilal. College Publications. find
  28. Sida I. Wang and Christopher D. Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proc. ACL.
  29. Rui Wang and Günter Neumann. 2007. Recognizing textual entailment using sentence similarity based on dependency tree skeletons. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.
  30. Gregory Ward and Betty Birner. 2004. Information structure and non-canonical syntax. In Laurence R. Horn and Gregory Ward, editors, Handbook of Pragmatics, pages 153–174. Blackwell, Oxford. find
  31. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015a. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv:1502.05698. ↗
  32. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015b. Memory networks. In Proc. ICLR. find
  33. Terry Winograd. 1972. Understanding natural language. Cognitive Psychology, 3(1):1–191. ↗
  34. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67– 78. ↗
  35. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/