Elamur Research
☰
home / papers / 2016 / Natural Language Inference by Tree-Based Convolution and Heuristic Matching

Natural Language Inference by Tree-Based Convolution and Heuristic Matching

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-2022.pdf

Summary

The paper addresses natural language inference (NLI): deciding whether a hypothesis sentence follows from a premise, contradicts it, or is unrelated to it. The authors introduce TBCNN-pair, a Siamese-style model in which a single shared tree-based convolutional neural network (TBCNN) encodes both sentences. Subtree feature detectors slide over dependency parses from the Stanford parser. Max pooling and a hidden layer then turn each sentence into a fixed-size vector. The two vectors are combined through inexpensive heuristic matching (concatenation, element-wise product and element-wise difference) and passed to a softmax classifier. On the SNLI corpus, the full model reaches 82.1% test accuracy. This beats all earlier sentence encoding-based systems, including LSTM encoders and a skip-thought-pretrained GRU, and comes close to attention-based LSTMs that cost more to run. Matching is O(1) with respect to sentence length, so sentence embeddings can be computed in advance. This makes the approach useful for retrieval and reranking pipelines where response time matters.

Key points

  • Data: SNLI (Bowman et al., 2015), with three labels (Entailment, Contradiction, Neutral) and a standard split of roughly 550k / 10k / 10k pairs.
  • Setup:
    • All layers are 300-dimensional.
    • Word2vec embeddings were trained on English Wikipedia and fine-tuned during training.
    • Dependency trees are collapsed.
    • Activation is ReLU, optimization is SGD with batch size 50, and dropout above 0.3 hurt accuracy.
  • Ablation of matching heuristics:
    • Concatenation alone reaches 79.3% test accuracy, and product alone only 72.5%.
    • Adding the difference term gives a 1–2% boost, and adding the product gives a further ~0.5%, for 82.1% in total.
  • Baselines:
    • Sentence encoders score lower: LSTM 77.6%, CNN 77.0%, lexicalized features 78.2% and skip-thought GRU 81.4%.
    • The word-by-word attention LSTM scores higher (83.5%), but its matching costs O(n²).
  • Efficiency and code: matching takes only 1.71% of prediction time (single CPU, C++), and the code is publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

93 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 83 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-2022 · arXiv 1512.08422

@inproceedings{mou2016natural,
  title = {{Natural Language Inference by Tree-Based Convolution and Heuristic Matching}},
  author = {Mou, Lili and Men, Rui and Li, Ge and Xu, Yan and Zhang, Lu and Yan, Rui and Jin, Zhi},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-2022},
  eprint = {1512.08422},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-2022}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lara Taylor. 2006. LCCs GISTexter at DUC 2006: Multi-strategy multi-document summarization. In Proceedings of DUC 2006. find
  2. Johan Bos and Katja Markert. 2005. Combining shallow and deep NLP methods for recognizing textual entailment. In Proceedings of the First PASCAL Challenges Workshop on Recognising Textual Entailment, pages 65–68. find
  3. Xiang Li, Lili Mou, Rui Yan, and Ming Zhang. 2016. StalemateBreaker: A proactive content-introducing approach to automatic human-computer conversation. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. find
  5. Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. find
  6. Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a “Siamese” time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669–688. find
  7. Bill MacCartney, Trond Grenager, Marie-Catherine de Marneffe, Daniel Cer, and Christopher D. Manning. 2006. Learning to recognize features of valid textual entailments. In Proceedings of the Human Language Technology Conference of the NAACL, pages 41–48. find
  8. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine learning, pages 160–167. find
  9. Bill MacCartney. 2009. Natural Language Inference. Ph.D. thesis, Stanford University. find
  10. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  11. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the Language Resource and Evaluation Conference, pages 449–454. find our card
  12. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In NAACL-HLT, pages 746– 751. find
  13. Sanda Harabagiu and Andrew Hickl. 2006. Methods for using textual entailment in open-domain question answering. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pages 905–912. find
  14. Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2015. Discriminative neural sentence modeling by tree-based convolution. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2315–2325. find
  15. Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2006. Negation, contrast and contradiction in text processing. In Proceedings of AAAI Conference on Artificial Intelligence, pages 755–762. find
  16. Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. find
  17. Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 17–21. find
  18. Hao Peng, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2015. A comparative study on regularization strategies for embedding-based neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2106–2111. find
  19. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems, pages 2042–2050. find
  20. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedings of the International Conference on Learning Representations. find our card
  21. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 655–665. find our card
  22. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems, pages 801–809. find
  23. Finley Lacatusu, Andrew Hickl, Kirk Roberts, Ying Shi, Jeremy Bensley, Bryan Rink, Patrick Wang, and Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. find our card
  24. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2015. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361. ↗
  25. Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2015. A deep architecture for semantic matching with multiple positional sentence representations. arXiv preprint arXiv:1511.08277. ↗
  26. Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of Conference on Empirical Methods in Natural Language Processing, pages 1785–1794. find
  27. Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2016. Improved relation classification by deep recurrent neural networks with data augmentation. arXiv preprint arXiv:1601.03651. ↗
  28. Rui Yan, Liang Kong, Congrui Huang, Xiaojun Wan, Xiaoming Li, and Yan Zhang. 2011a. Timeline generation through evolutionary trans-temporal summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 433–443. find
  29. Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. 2011b. Evolutionary timeline summarization: A balanced optimization framework via iterative substitution. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 745–754. find
  30. Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval based human-computer conversation system. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  31. Wenpeng Yin and Hinrich Schütze. 2015. Convolutional neural network for paraphrase identification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 901–911. find
  32. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2230–2235. find