Elamur Research
☰
home / papers / 2016 / ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs

ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/Q16-1019.pdf

Summary

Judging how two sentences relate is central to a range of NLP problems, among them answer selection (AS), textual entailment (TE) and the identification of paraphrases (PI). Earlier systems usually had one or more of three limitations: they encoded each sentence independently, they were tuned for a single task, or they relied on hand-crafted features. Yin et al. propose ABCNN, a general architecture that adds attention to a convolutional network so that each sentence's representation is shaped by its partner.

The starting point is BCNN, a Siamese pair of weight-sharing CNNs over word2vec embeddings, with average pooling and a logistic regression output. On top of it, the authors design three attention variants, ABCNN-1, ABCNN-2 and ABCNN-3, the last combining the first two.

Across AS (WikiQA), PI and TE (SICK), the attention models consistently beat the plain BCNN. ABCNN-3 performs best, reaching state of the art on AS and TE and competitive results on PI. The work matters because it is presented as the first NLP study to bring attention into CNNs, showing that attention is not only useful for LSTMs.

Key points

  • The baseline BCNN uses 300-dimensional precomputed word2vec embeddings, convolution with filter width 3, and column-wise average pooling (all-ap). Stacking layers lets attention operate on progressively longer phrases.
  • Results ranked consistently: ABCNN-2 generally beats ABCNN-1, and ABCNN-3 beats both. Adding a second convolution layer gave little gain, which the authors attribute to small training sets.
  • On SICK (TE), ABCNN-3 with two conv layers reached 86.2% accuracy, 1.6 points above the prior best (Lai and Hockenmaier, 2014). Without its 24 linguistic features, it scored 84.6%.
  • Linguistic features helped every task:
    • AS: +0.0321 MAP and +0.0338 MRR
    • PI: +3.8 accuracy and +2.1 F1
    • TE: +1.6 accuracy
  • Attention visualizations suggest the model aligns semantically matching words and phrases across sentences and even picks up some cross-sentence coreference. The code is released on GitHub (yinwenpeng/Answer_Selection).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

124 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00097

@article{yin2016abcnn,
  title = {{ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs}},
  author = {Yin, Wenpeng and Schütze, Hinrich and Xiang, Bing and Zhou, Bowen},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00097},
  url = {https://doi.org/10.1162/tacl_a_00097}
}

References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2015. Multiple object recognition with visual attention. In Proceedings of ICLR. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. find our card
  3. Matthew W. Bilotti, Paul Ogilvie, Jamie Callan, and Eric Nyberg. 2007. Structured retrieval for question answering. In Proceedings of SIGIR, pages 351–358. find
  4. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of EMNLP-CoNLL, pages 546– 556. find
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015a. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP, pages 632–642. find
  6. Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. 2015b. Recursive neural networks can learn logical semantics. In Proceedings of CVSC Workshop, pages 12–21. Jane Bromley, James W. Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993.… find
  7. Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang, Chang Huang, Wei Xu, Deva Ramanan, and Thomas S. Huang. 2015. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In Proceedings of ICCV, pages 2956–2… find
  8. Ming-Wei Chang, Dan Goldwasser, Dan Roth, and Vivek Srikumar. 2010. Discriminative learning over constrained latent representations. In Proceedings of NAACL-HLT, pages 429–437. find
  9. Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. 2015. ABC-CNN: An attention based convolutional neural network for visual question answering. CoRR, abs/1511.05960. — [arXiv:1511.05960](https://arxiv.org/abs/1511.05960) find
  10. Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. End-to-end continuous speech recognition using attention-based recurrent NN: First results. In Proceedings of Deep Learning and Representation Learning Workshop, NIPS. find
  11. Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Proceedings of NIPS, pages 577–585. find our card
  12. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of COLING, pages 350–356. find our card
  13. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121–2159. find
  14. Minwei Feng, Bing Xiang, Michael R. Glass, Lidan Wang, and Bowen Zhou. 2015. Applying deep learning to answer selection: A study and an open task. In Proceedings of IEEE ASRU Workshop, pages 813– 820. find
  15. Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. In Proceedings of ICML, pages 1462–1471. find
  16. Hua He, Kevin Gimpel, and Jimmy Lin. 2015. Multi-perspective sentence similarity modeling with convolutional neural networks. In Proceedings of EMNLP, pages 1576–1586. find
  17. Michael Heilman and Noah A. Smith. 2010. Tree edit models for recognizing textual entailments, paraphrases, and answers to questions. In Proceedings of NAACL-HLT, pages 1011–1019. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find our card
  18. Seunghoon Hong, Junhyuk Oh, Honglak Lee, and Bohyung Han. 2016. Learning transferrable knowledge for semantic segmentation with deep convolutional neural network. In Proceedings of CVPR. find
  19. Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Proceedings of NIPS, pages 2042–2050. find
  20. Yangfeng Ji and Jacob Eisenstein. 2013. Discriminative improvements to distributional sentence similarity. In Proceedings of EMNLP, pages 891–896. Sergio Jimenez, George Dueñas, Julia Baquero, and Alexander Gelbukh. 2014. UNAL-NLP: Combining soft cardinality features for semantic textual similarity… find
  21. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of ACL, pages 655– 665. find our card
  22. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746–1751. find our card
  23. Alice Lai and Julia Hockenmaier. 2014. Illinois-LH: A denotational and distributional approach to semantics. In Proceedings of SemEval, pages 329–334. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IE… find
  24. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of ACL, pages 1106–1115. find
  25. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the ACL Text Summarization Workshop. find
  26. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP, pages 1412–1421. find
  27. Nitin Madnani, Joel Tetreault, and Martin Chodorow. 2012. Re-examining machine translation metrics for paraphrase identification. In Proceedings of NAACL-HLT, pages 182–190. find
  28. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of SemEval, pages 1–… find our card
  29. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014b. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC, pages 216–223. find
  30. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS, pages 3111–3119. find
  31. George A. Miller. 1995. WordNet: A lexical database for english. Commun. ACM, 38(11):39–41. find our card
  32. Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. 2014. Recurrent models of visual attention. In Proceedings of NIPS, pages 2204–2212. find
  33. Dan Moldovan, Christine Clark, Sanda Harabagiu, and Daniel Hodges. 2007. Cogex: A semantically and contextually enriched logic prover for question answering. Journal of Applied Logic, 5(1):49–69. find
  34. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2004. Mapping dependencies trees: An application to question answering. In Proceedings of AI&Math 2004 (Special session: Intelligent Text Processing). Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2016. R… find
  35. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP, pages 379–389. find
  36. Dan Shen and Mirella Lapata. 2007. Using semantic roles to improve question answering. In Proceedings of EMNLP-CoNLL, pages 12–21. find
  37. Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proceedings of NIPS, pages 801–809. find
  38. Ming Tan, Bing Xiang, and Bowen Zhou. 2016. LSTM-based deep learning models for non-factoid answer selection. In Proceedings of ICLR Workshop. find
  39. Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016. A deep architecture for semantic matching with multiple positional sentence representations. In Proceedings of AAAI, pages 2835– 2841. find
  40. Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007. What is the jeopardy model? A quasi-synchronous grammar for QA. In Proceedings of EMNLP-CoNLL, pages 22–32. find
  41. Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. 2015. The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In Proceedings of CVPR, pages 842–850. find
  42. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of ICML, pages 2048–2057. find our card
  43. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of EMNLP, pages 2013–2018. find our card
  44. Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013a. Semi-markov phrase-based monolingual alignment. In Proceedings of EMNLP, pages 590–600. find
  45. Xuchen Yao, Benjamin Van Durme, and Peter Clark. 2013b. Automatic coupling of answer extraction and information retrieval. In Proceedings of ACL, pages 159–165. find
  46. Wen-tau Yih, Ming-Wei Chang, Christopher Meek, and Andrzej Pastusiak. 2013. Question answering using enhanced lexical semantic models. In Proceedings of ACL, pages 1744–1753. Wenpeng Yin and Hinrich Schütze. 2015a. Convolutional neural network for paraphrase identification. In Proceedings of NAACL-… find
  47. Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection. In Proceedings of Deep Learning and Representation Learning Workshop, NIPS. find
  48. Jiang Zhao, Tiantian Zhu, and Man Lan. 2014. ECNU: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment. In Proceedings of SemEval, pages 271–277. find