Elamur Research
☰
home / papers / 2015 / Reasoning about Entailment with Neural Attention

Reasoning about Entailment with Neural Attention

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1509.06664

Summary

Recognizing textual entailment (RTE) means deciding whether a premise sentence entails a hypothesis, contradicts it, or is unrelated to it. Earlier strong systems relied on hand-built features, complex NLP pipelines and external resources. End-to-end neural networks had not beaten simple lexical baselines. The authors use the large SNLI corpus to train an LSTM model that does not encode the two sentences separately. Instead, a second LSTM reads the hypothesis starting from the memory state that the first LSTM reached after reading the premise, which they call conditional encoding. They then add attention over the premise's output vectors. One version uses a single attention step. Another uses word-by-word attention, where each hypothesis token softly aligns with premise words. A two-way variant also attends with the sentences swapped. The conditional model already beats a feature-engineered lexicalized classifier. Word-by-word attention reaches 83.5% accuracy, a new state of the art on SNLI for a generic differentiable model. Visualized attention weights show sensible word and phrase alignments. Because the method needs only tokenization, it is not tied to any particular language.

Key points

  • Dataset: Stanford Natural Language Inference (SNLI), with three classes: ENTAILMENT, NEUTRAL, CONTRADICTION. The previous neural baseline (Bowman et al., 2015) scored 77.6%.
  • The conditional-encoding LSTM achieves 80.9%, which is 2.7 points above the lexicalized feature-based classifier.
  • Word-by-word attention adds 2.6 more points and reaches 83.5% accuracy on SNLI.
  • Inputs are fixed word2vec embeddings. About 12.1k SNLI words have no word2vec vector, so their embeddings are trained, giving roughly 3.65M tunable parameters.
  • No POS tags or dependency parses are needed. The authors suggest transfer learning, hierarchical attention for longer texts, and entailment over non-language sequences as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

200 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 184 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1509.06664 · arXiv 1509.06664

@misc{rocktaschel2015reasoning,
  title = {{Reasoning about Entailment with Neural Attention}},
  author = {Rocktaschel, Tim and Grefenstette, Edward and Hermann, Karl Moritz and Kovcisky, Tomavs and Blunsom, Phil},
  year = {2015},
  doi = {10.48550/arxiv.1509.06664},
  eprint = {1509.06664},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1509.06664}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabor Angeli and Christopher D Manning. Naturalli: Natural logic inference for common sense reasoning. In EMNLP, 2014. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. find our card
  3. Islam Beltagy, Stephen Roller, Pengxiang Cheng, Katrin Erk, and Raymond J. Mooney. Representing meaning with a combination of logical form and vectors. arXiv preprint arXiv:1505.06816, 2015. — [arXiv:1505.06816](https://arxiv.org/abs/1505.06816) find
  4. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015. find
  5. Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In NIPS, 2015. find our card
  6. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Lecture Notes in Computer Science, volume 3944, pages 177–190. Springer, 2006. find
  7. Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  8. Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5):602–610, 2005. find
  9. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. find
  10. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In NIPS, 2015. find
  11. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, 2015. find our card
  12. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  13. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. Convolutional neural network architectures for matching natural language sentences. In NIPS, pages 2042–2050, 2014. find
  14. Sergio Jimenez, George Duenas, Julia Baquero, Alexander Gelbukh, Av Juan Dios Bátiz, and Av Mendizábal. Unal-nlp: Combining soft cardinality features for semantic textual similarity, relatedness and entailment. In SemEval 2014, page 732, 2014. find
  15. Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. arXiv preprint arXiv:1503.01007, 2015. find
  16. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. find
  17. Alice Lai and Julia Hockenmaier. Illinois-lh: A denotational and distributional approach to semantics. In SemEval 2014, page 329, 2014. find
  18. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In SemEval 2014, 2014. find our card
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119, 2013. find
  20. Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In NIPS, pages 2204–2212, 2014. find
  21. Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, 2015. find
  22. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS, pages 801–809, 2011. find
  23. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. arXiv preprint arXiv:1503.08895, 2015. find
  24. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112, 2014. find
  25. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as a foreign language. In NIPS, 2014. find
  26. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, 2015. find
  27. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015. find our card
  28. Wenpeng Yin and Hinrich Schütze. Convolutional neural network for paraphrase identification. In NAACL-HLT, pages 901–911, 2015. find
  29. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. find our card
  30. Jiang Zhao, Tian Tian Zhu, and Man Lan. Ecnu: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment. In SemEval 2014, page 271, 2014. find