Elamur Research⌕
☰
home / papers / 2017 / DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

Paper · 2017 · Tao Shen, Tianyi Zhou, Guodong Long et al.

Summary

Sentence encoders in NLP usually rely on RNNs for long-range context or on CNNs for local patterns. Both have drawbacks: RNNs process tokens sequentially, and standard attention ignores word order. The authors propose DiSAN, a lightweight sentence-encoding network that uses only attention. It has two new components. The first is multi-dimensional attention, which gives each token-pair a score per feature rather than a single scalar. The second is directional self-attention, where positional masks restrict attention to forward or backward context and so encode temporal order. A fusion gate mixes each token's input with its attended context. A final multi-dimensional "source2token" attention then pools the sequence into one vector for downstream classifiers or regressors. Without stacked blocks or an encoder-decoder design, DiSAN posts the best SNLI result among sentence-encoding models and strong results on seven other benchmarks. It also needs fewer parameters and less time than the LSTM and tree-based models it beats. This shows that order-aware attention alone can replace recurrence and convolution for general sentence representation.

Key points

  • Improves the previous best sentence-encoding accuracy on SNLI by 1.02%.
  • Reports state-of-the-art test accuracy on SST, MultiNLI, SICK, Customer Review (CR), MPQA, TREC and SUBJ.
  • Architecture: forward and backward masked self-attention blocks with a fusion gate, followed by multi-dimensional source2token attention. No RNN or CNN layers are used.
  • Heatmaps on SNLI sentences show three patterns:
    • Content words get more attention than stop words.
    • Forward and backward blocks focus on different parts of a sentence.
    • Feature-wise vectors separate different senses of the same word (e.g., "glass", "close").
  • Published at AAAI 2018; code and pre-trained models are at github.com/taoshen58/DiSAN.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States402 works · 212,762 citations🎓University of Technology SydneySydney · Australia19 works · 12,908 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Natural language understanding · 52 works

Cite

Cite as Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan and Chengqi Zhang. 2017. DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. arXiv:1709.04696.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.04696 · arXiv 1709.04696

@misc{shen2017disan,
  title = {{DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding}},
  author = {Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan and Chengqi Zhang},
  year = {2017},
  doi = {10.48550/arxiv.1709.04696},
  eprint = {1709.04696},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.04696}
}

References

49 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, D.; Cho, K.; and Bengio, Y. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
  2. Bjerva, J.; Bos, J.; Van der Goot, R.; and Nissim, M. 2014. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In SemEval@ COLING, 642–646.
  3. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In EMNLP.
  4. Bowman, S. R.; Gauthier, J.; Rastogi, A.; Gupta, R.; Manning, C. D.; and Potts, C. 2016. A fast unified model for parsing and sentence understanding. In ACL. find
  5. Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS. find
  6. Clevert, D.-A.; Unterthiner, T.; and Hochreiter, S. 2016. Fast and accurate deep network learning by exponential linear units (elus). In ICLR.
  7. Glorot, X., and Bengio, Y. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 249–256. find
  8. Graves, A.; Jaitly, N.; and Mohamed, A.-r. 2013. Hybrid speech recognition with deep bidirectional lstm. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, 273–278. find
  9. IEEE. Hermann, K. M.; Kocisky, T.; Grefenstette, E.; Espeholt, L.; Kay, W.; Suleyman, M.; and Blunsom, P. 2015. Teaching machines to read and comprehend. In NIPS.
  10. Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  11. Hu, M., and Liu, B. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 168–177. ACM. find
  12. Hu, M.; Peng, Y.; and Qiu, X. 2017. Reinforced mnemonic reader for machine comprehension. arXiv preprint arXiv:1705.02798.
  13. Kalchbrenner, N.; Grefenstette, E.; and Blunsom, P. 2014. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188.
  14. Kim, Y.; Jernite, Y.; Sontag, D.; and Rush, A. M. 2016. Character-aware neural language models. In AAAI.
  15. Kim, Y. 2014. Convolutional neural networks for sentence classification. In EMNLP.
  16. Kingma, D., and Ba, J. 2015. Adam: A method for stochastic optimization. In ICLR.
  17. Kiros, R.; Zhu, Y.; Salakhutdinov, R. R.; Zemel, R.; Urtasun, R.; Torralba, A.; and Fidler, S. 2015. Skip-thought vectors. In NIPS. find
  18. Kokkinos, F., and Potamianos, A. 2017. Structural attention neural networks for improved sentiment analysis. arXiv preprint arXiv:1701.01811. find
  19. Lei, T., and Zhang, Y. 2017. Training rnns as fast as cnns. arXiv preprint arXiv:1709.02755. — [arXiv:1709.02755](https://arxiv.org/abs/1709.02755) find
  20. Lei, T.; Barzilay, R.; and Jaakkola, T. 2015. Molding cnns for text: non-linear, non-consecutive convolutions. In EMNLP. find
  21. Li, X., and Roth, D. 2002. Learning question classifiers. In ACL.
  22. Li, J.; Luong, M.-T.; Jurafsky, D.; and Hovy, E. 2015. When are tree structures necessary for deep learning of representations? arXiv preprint arXiv:1503.00185.
  23. Liu, Y.; Sun, C.; Lin, L.; and Wang, X. 2016. Learning natural language inference using bidirectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090. find
  24. Luong, M.-T.; Pham, H.; and Manning, C. D. 2015. Effective approaches to attention-based neural machine translation. In EMNLP.
  25. Marelli, M.; Menini, S.; Baroni, M.; Bentivogli, L.; Bernardi, R.; and Zamparelli, R. 2014. A sick cure for the evaluation of compositional distributional semantic models. In LREC.
  26. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  27. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS.
  28. Mou, L.; Men, R.; Li, G.; Xu, Y.; Zhang, L.; Yan, R.; and Jin, Z. 2016. Natural language inference by tree-based convolution and heuristic matching. In ACL.
  29. Munkhdalai, T., and Yu, H. 2017a. Neural semantic encoders. In EACL. find
  30. Munkhdalai, T., and Yu, H. 2017b. Neural tree indexers for text understanding. In EACL. find
  31. Pang, B., and Lee, L. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL.
  32. Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP.
  33. Qian, Q.; Huang, M.; and Zhu, X. 2017. Linguistically regularized lstms for sentiment classification. In ACL. find
  34. Rush, A. M.; Chopra, S.; and Weston, J. 2015. A neural attention model for abstractive sentence summarization. In EMNLP.
  35. Seo, M.; Kembhavi, A.; Farhadi, A.; and Hajishirzi, H. 2017. Bidirectional attention flow for machine comprehension. In ICLR.
  36. Shang, L.; Lu, Z.; and Li, H. 2015. Neural responding machine for short-text conversation. In ACL.
  37. Socher, R.; Perelygin, A.; Wu, J. Y.; Chuang, J.; Manning, C. D.; Ng, A. Y.; Potts, C.; et al. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
  38. Socher, R.; Karpathy, A.; Le, Q. V.; Manning, C. D.; and Ng, A. Y. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics 2:207–218.
  39. Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(1):1929–1958.
  40. Sukhbaatar, S.; Weston, J.; Fergus, R.; et al. 2015. End-to-end memory networks. In NIPS. find
  41. Tai, K. S.; Socher, R.; and Manning, C. D. 2015. Improved semantic representations from tree-structured long short-term memory networks. In ACL. find
  42. Teng, Z.; Vo, D.-T.; and Zhang, Y. 2016. Context-sensitive lexicon features for neural sentiment analysis. In EMNLP. find
  43. Vaswani, A.; Shazeer; Noam; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is all you need. In NIPS.
  44. Vendrov, I.; Kiros, R.; Fidler, S.; and Urtasun, R. 2016. Order-embeddings of images and language. In ICLR.
  45. Wiebe, J.; Wilson, T.; and Cardie, C. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation 39(2):165–210. find
  46. Williams, A.; Nangia, N.; and Bowman, S. R. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
  47. Zeiler, M. D. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.
  48. Zhao, H.; Lu, Z.; and Poupart, P. 2015. Self-adaptive hierarchical sentence model. In IJCAI. find
  49. Zhao, J.; Zhu, T.; and Lan, M. 2014. Ecnu: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment. In SemEval@ COLING, 271–277. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2017-disan-directional-self-attention-network/