Elamur Research
☰
home / papers / 2018 / e-SNLI: Natural Language Inference with Natural Language Explanations

e-SNLI: Natural Language Inference with Natural Language Explanations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1812.01193

Summary

The authors want NLI systems that can justify their decisions in plain sentences, and that can also learn from human justifications during training. To support this, they added free-form human explanations to every premise–hypothesis pair in the Stanford Natural Language Inference (SNLI) corpus, producing a resource called e-SNLI. The explanations were collected on Amazon Mechanical Turk. Before writing an explanation, workers highlighted the key words, and label-specific checks in the browser rejected obviously bad submissions. Template-like answers were then detected automatically and sent for reannotation. Building on the InferSent architecture (a BiLSTM encoder with max-pooling plus an LSTM decoder), the authors set up baselines for several tasks: predicting a label together with an explanation, explaining first and then classifying from the explanation alone, learning sentence representations, and transferring to other NLI datasets. A key finding is that a hypothesis-only model can exploit dataset artifacts to guess labels, but those artifacts help far less when it has to produce a correct explanation. The work matters because it supplies a large, reusable benchmark for explainable and knowledge-aware inference models.

Key points

  • e-SNLI is built on SNLI's ~570K pairs. The training set has one explanation per pair, and the validation and test sets have three each. Highlighted words are released alongside, and the data is on GitHub (OanaMariaCamburu/e-SNLI).
  • 6,325 crowd workers took part. After filtering template-like answers by edit distance and reannotating them (11% of the data), a manual check of 1,000 samples found a 9.62% error rate overall. Entailment was the hardest label, at 19.55%.
  • The models follow Conneau et al.'s InferSent: 2048-unit BiLSTM encoders with max-pooling (4096-dimensional sentence vectors), a 3-layer MLP classifier and a one-layer LSTM explanation decoder with a vocabulary of about 12K words.
  • Experiments: PremiseAgnostic, PredictAndExplain, ExplainThenPredict, Represent and Transfer.
  • In out-of-domain transfer, e-InferSent scored 53.54 on SICK-E and 57.16 on MultiNLI, slightly above InferSent trained only on SNLI labels (53.27 / 57.00).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

351 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 333 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1812.01193 · arXiv 1812.01193

@misc{camburu2018e,
  title = {{e-SNLI: Natural Language Inference with Natural Language Explanations}},
  author = {Camburu, Oana-Maria and Rocktäschel, Tim and Lukasiewicz, Thomas and Blunsom, Phil},
  year = {2018},
  doi = {10.48550/arxiv.1812.01193},
  eprint = {1812.01193},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1812.01193}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alvarez-Melis, D. and Jaakkola, T. S. (2017). A causal framework for explaining the predictions of black-box sequence-to-sequence models. CoRR, abs/1707.01943. ↗
  2. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. ↗ our card
  3. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. CoRR, abs/1508.05326. ↗
  4. Chan, W., Jaitly, N., Le, Q. V., and Vinyals, O. (2015). Listen, attend and spell. CoRR, abs/1508.01211. ↗
  5. Chen, Q., Zhu, X., Ling, Z., Inkpen, D., and Wei, S. (2017). Natural language inference with external knowledge. CoRR, abs/1711.04289. ↗
  6. Chen, Q., Zhu, X., Ling, Z., Wei, S., and Jiang, H. (2016). Enhancing and combining sequential and tree LSTM for natural language inference. CoRR, abs/1609.06038. ↗
  7. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. CoRR, abs/1705.02364. ↗ our card
  8. Das, A., Agrawal, H., Zitnick, C. L., Parikh, D., and Batra, D. (2016). Human attention in visual question answering: Do humans and deep networks look at the same regions? CoRR, abs/1606.03556. ↗
  9. Dasgupta, I., Guo, D., Stuhlmüller, A., Gershman, S. J., and Goodman, N. D. (2018). Evaluating compositionality in sentence embeddings. find
  10. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. In Proc. of CVPR. find
  11. Glockner, M., Shwartz, V., and Goldberg, Y. (2018). Breaking NLI systems with sentences that require simple lexical inferences. In Proc. of ACL. find our card
  12. Gong, Y., Luo, H., and Zhang, J. (2017). Natural language inference over interaction space. CoRR, abs/1709.04348. ↗
  13. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., and Smith, N. A. (2018). Annotation artifacts in natural language inference data. In Proc. of NAACL. find
  14. Hill, F., Cho, K., and Korhonen, A. (2016). Learning distributed representations of sentences from unlabelled data. CoRR, abs/1602.03483. ↗ our card
  15. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Comput., 9(8):1735–1780. find
  16. Jansen, P. A., Wainwright, E., Marmorstein, S., and Morrison, C. T. (2018). Worldtree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. CoRR, abs/1802.03052. ↗
  17. Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., and Fidler, S. (2015). Skip-thought vectors. CoRR, abs/1506.06726. ↗
  18. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. (2017). Program induction by rationale generation: Learning to solve and explain algebraic word problems. CoRR, abs/1705.04146. ↗
  19. Liu, P., Qiu, X., and Huang, X. (2016). Modelling interaction of sentence pair with coupled-lstms. CoRR, abs/1605.05573. ↗
  20. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., bernardi, R., and Zamparelli, R. (2014). A sick cure for the evaluation of compositional distributional semantic models. find
  21. Nie, Y. and Bansal, M. (2017). Shortcut-stacked sentence encoders for multi-domain inference. In Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, pages 41–45. Association for Computational Linguistics. find
  22. Parikh, A. P., Täckström, O., Das, D., and Uszkoreit, J. (2016). A decomposable attention model for natural language inference. CoRR, abs/1606.01933. ↗ our card
  23. Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., and Rohrbach, M. (2018). Multimodal explanations: Justifying decisions and pointing to the evidence. CoRR, abs/1802.08129. ↗
  24. Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). "why should I trust you?": Explaining the predictions of any classifier. CoRR, abs/1602.04938. ↗ our card
  25. Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kociský, T., and Blunsom, P. (2015). Reasoning about entailment with neural attention. CoRR, abs/1509.06664. ↗ our card
  26. Williams, A., Nangia, N., and Bowman, S. R. (2017). A broad-coverage challenge corpus for sentence understanding through inference. CoRR, abs/1704.05426. ↗
  27. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A. C., Salakhutdinov, R., Zemel, R. S., and Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. CoRR, abs/1502.03044. ↗ our card