Elamur Research⌕
☰
home / papers / 2016 / Learning Natural Language Inference with LSTM

Learning Natural Language Inference with LSTM

Summary

The paper addresses natural language inference (NLI): deciding whether a hypothesis sentence follows from, contradicts, or is neutral to a premise. Earlier neural models on the SNLI corpus, including the attention model of Rocktäschel et al. (2016), reduce each sentence to one vector before comparing them. This makes it hard to weigh individual word-level alignments, even though a single mismatched subject can be enough to signal contradiction. Wang and Jiang propose the match-LSTM (mLSTM). Two separate LSTMs encode the premise and the hypothesis. For each hypothesis word, the model builds an attention-weighted summary of the premise and joins it with that word's hidden state. A further LSTM consumes these combined inputs and tracks how well the sentences align so far. Its last hidden state feeds the three-way classifier. On SNLI the model reaches 86.1% accuracy, ahead of previously published systems. Gate analysis suggests the network discards good matches and keeps decisive mismatches. This gives an interpretable argument for sequential word-level matching over comparing whole-sentence embeddings.

Key points

  • Data: SNLI, with roughly 570K premise–hypothesis pairs labelled entailment, contradiction or neutral.
  • Accuracy comparison on SNLI:
    • mLSTM: 86.1%
    • Rocktäschel et al. (2016), word-by-word attention: 83.5%
    • Mou et al. (2015), tree-based CNN: 82.1%
    • Vendrov et al. (2015), Skip-Thought: 81.5%
    • Bowman et al. (2015), lexicalized classifier: 78.2%
    • Bowman et al. (2015), LSTM sentence embeddings: 77.6%
  • Architectural change: the hypothesis LSTM is not initialized with the premise's final cell state. Matching happens only through attention plus the dedicated mLSTM.
  • Average forget-gate values on the test data are lowest for entailment and higher for the other two labels:
    • entailment: 0.446±0.123
    • neutral: 0.507±0.148
    • contradiction: 0.536±0.170
  • Limitation: the model needs large training data. Preliminary runs on the smaller SICK corpus were weak, and the authors suggest adding resources such as the paraphrase database as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1170

@inproceedings{wang2016learning,
  title = {{Learning Natural Language Inference with LSTM}},
  author = {Wang, Shuohang and Jiang, Jing},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1170},
  url = {https://doi.org/10.18653/v1/n16-1170}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, HyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
  3. Peter Clark and Phil Harrison. 2009. An inference-based approach to recognizing entailment. In Proceedings of the Text Analysis Conference. find
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL Recognising Textual Entailment Challenge. In Proceedings of the PASCAL Challenges Workshop on Recognizing Textual Entailment. find
  5. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. find
  6. Morgan & Claypool Publishers. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  7. Oren Glickman, Ido Dagan, and Moshe Koppel. 2005. Web based probabilistic textual entailment. In Proceedings of the PASCAL Challenges Workshop on Recognizing Textual Entailment. find
  8. Alex Graves. 2012. Supervised sequence labelling with recurrent neural networks, volume 385. find
  9. Springer. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory… find our card
  10. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations. find
  11. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems. find
  12. Bill MacCartney, Michel Galley, and Christopher D Manning. 2008. A phrase-based alignment model for natural language inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  13. Bill MacCartney. 2009. Natural Language Inference. Ph.D. thesis, Stanford University. find
  14. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation. Yashar Mehdad, A… find
  15. Lili Mou, Men Rui, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2015. Recognizing entailment and contradiction by tree-based convolution. arXiv preprint arXiv:1512.08422. find our card
  16. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Conference on Empirical Methods in Natural Language Processing. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2016… find
  17. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  18. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2015. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2016-learning-natural-language-inference-lstm/