Elamur Research
☰
home / papers / 2017 / A Structured Self-attentive Sentence Embedding

A Structured Self-attentive Sentence Embedding

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1703.03130

Summary

Most neural sentence encoders reduce a sentence to one vector. They typically take the final RNN state or apply max or average pooling over time steps. This makes it hard to represent several distinct parts of the meaning and reveals little about what was encoded. Lin et al. replace the pooling step with a self-attention layer on top of a bidirectional LSTM. The model outputs a matrix with r rows. Each row is a separate weighted mix of the hidden states, with weights produced by a small two-layer MLP and a softmax over tokens, so different rows can focus on different phrases. No external query is required, which makes attention usable for single-sentence tasks. A Frobenius-norm penalty pushes the rows to be both sharp and mutually distinct. The attention weights also serve as heat maps showing which words each row depends on. The authors tested the model on author profiling (Age), sentiment classification (Yelp) and textual entailment (SNLI) and report gains over other sentence embedding methods on all three. The work is an influential early example of interpretable, multi-hop self-attention for sentence representation.

Key points

  • Architecture: The annotation matrix is computed as A = softmax(W_s2 tanh(W_s1 Hᵀ)), where d_a is a free hyperparameter. The sentence embedding is M = AH, of size r × 2u, with u hidden units per LSTM direction.
  • Regularization: The penalty is P = ‖AAᵀ − I‖²_F. It drives diagonal entries toward 1, so each row stays focused, and off-diagonal entries toward 0, so rows avoid redundancy. It costs about a third of the computation of a KL-divergence penalty, which the authors found unstable.
  • SNLI setup: Premise and hypothesis share one encoder. Their matrices are combined by a "Gated Encoder" inspired by the Factored Gated Autoencoder (Memisevic, 2013), which uses batched products with weight tensors followed by an element-wise product before an MLP classifier.
  • Weight pruning (Appendix A): Pruning exploits the 2-D structure of M and removes (r−1)/r of the hidden-layer connections. Model sizes and accuracies before and after pruning:
    • Yelp: 55.3M → 4.1M parameters, accuracy 64.21% → 63.86%
    • Age: 73.2M → 2.1M parameters, accuracy 80.45% → 77.32%
    • SNLI: 95.0M → 28.6M parameters, accuracy 84.43% → 83.16%
  • Interpretability: The model offers two visualizations. One is a heat map for each row of A. The other sums and renormalizes all rows to show which words the whole embedding emphasizes or ignores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

265 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 255 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.03130 · arXiv 1703.03130

@misc{lin2017structured,
  title = {{A Structured Self-attentive Sentence Embedding}},
  author = {Lin, Zhouhan and Feng, Minwei and Santos, Cicero Nogueira dos and Yu, Mo and Xiang, Bing and Zhou, Bowen and Bengio, Yoshua},
  year = {2017},
  doi = {10.48550/arxiv.1703.03130},
  eprint = {1703.03130},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.03130}
}

References

36 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In Advances in Neural Information Processing Systems, pp. 932–938, 2001. find
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015. find
  3. Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. arXiv preprint arXiv:1603.06021, 2016. find
  4. Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. In Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2016. find our card
  5. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. find
  6. Cicero dos Santos and Maira Gatti. Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 69–78, 2014. find
  7. Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. Attentive pooling networks. arXiv preprint arXiv:1602.03609, 2016. — [arXiv:1602.03609](https://arxiv.org/abs/1602.03609) find
  8. Minwei Feng, Bing Xiang, Michael R. Glass, Lidan Wang, and Bowen Zhou. Applying deep learning to answer selection: a study and an open task. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015, Scottsdale, AZ, USA, December 13-17, 2015, pp. 813–820, 2015. find
  9. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1367– 1377, San Diego, Californ… find our card
  10. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  11. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188, 2014. find our card
  12. Yoon Kim. Convolutional neural networks for sentence classification. arXiv:1408.5882, 2014. find our card
  13. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294–3302, 2015. find
  14. Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, volume 14, pp. 1188–1196, 2014. find our card
  15. Ji Young Lee and Franck Dernoncourt. Sequential short-text classification with recurrent and convolutional neural networks. arXiv preprint arXiv:1603.03827, 2016. find our card
  16. Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. arXiv preprint arXiv:1607.06275, 2016. — [arXiv:1607.06275](https://arxiv.org/abs/1607.06275) find
  17. Wang Ling, Lin Chu-Cheng, Yulia Tsvetkov, and Silvio Amir. Not all contexts are created equal: Better word representations with variable attention. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1367–1372, Lisbon, Portugal, September 2015. Association… find
  18. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. Learning natural language inference using bidirectional LSTM model and inner-attention. CoRR, abs/1605.09090, 2016a. — [arXiv:1605.09090](https://arxiv.org/abs/1605.09090) find
  19. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. Learning natural language inference using bidirectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090, 2016b. — [arXiv:1605.09090](https://arxiv.org/abs/1605.09090) find
  20. Mingbo Ma, Liang Huang, Bing Xiang, and Bowen Zhou. Dependency-based convolutional neural networks for sentence embedding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, volume 2, p… find
  21. Horia Margarit and Raghav Subramaniam. A batch-normalized recurrent network for sentiment classification. In Advances in Neural Information Processing Systems, 2016. find
  22. Roland Memisevic. Learning to relate images. IEEE transactions on pattern analysis and machine intelligence, 35(8):1829–1846, 2013. arXiv preprint Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 20… find
  23. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. Natural language inference by tree-based convolution and heuristic matching. arXiv preprint arXiv:1512.08422, 2015a. find our card
  24. Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. Discriminative neural sentence modeling by tree-based convolution. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2315–2325, Lisbon, Portugal, September 2015b. Association for Computational Ling… find
  25. Tsendsuren Munkhdalai and Hong Yu. Neural tree indexers for text understanding. arXiv preprint arXiv:1607.04492, 2016a. find
  26. Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. arXiv preprint arXiv:1607.04315, 2016b. find
  27. Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(4):694–707… find
  28. Ankur P. Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Proceedings of EMNLP, 2016. find our card
  29. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pp. 1532–43, 2014. find
  30. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 151–161, Edinburgh, Scotland, UK.… find our card
  31. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631… find
  32. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of ACL, pp. 1556–1566, 2015. find
  33. Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. Improved representation learning for question answer matching. In Proceedings of ACL, pp. 464–473, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P16-1044. ↗
  34. Theano Development Team. Theano: A {Python} framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.0, 2016. URL http://arxiv.org/abs/1605.02688. ↗ our card
  35. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361, 2015. find
  36. Wenpeng Yin and Hinrich Schütze. Convolutional neural network for paraphrase identification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 901–911, 2015. find