Elamur Research⌕
☰
home / papers / 2019 / Adaptively Sparse Transformers

Adaptively Sparse Transformers

Summary

Standard Transformers normalize every attention head with softmax, so each head spreads at least some weight over all context tokens. Earlier sparse variants (Sparse Transformer, adaptive span Transformer) restrict heads only to a contiguous window of preceding tokens. The authors swap softmax for α-entmax, a differentiable mapping that can give irrelevant tokens exactly zero weight. Its parameter α ranges from softmax (α = 1) to sparsemax (α = 2). Their main technical step is a closed-form derivative of the entmax output with respect to α. This makes α an ordinary trainable parameter, so each head learns its own shape by gradient descent instead of through a costly per-head search. They tested softmax, fixed 1.5-entmax and adaptive α-entmax Transformers on four machine translation tasks. The sparse models matched or slightly beat softmax accuracy. They were also easier to interpret, with heads that were more diverse and more specialized. Examples include near-deterministic positional heads and a head that joins subwords. This matters because sparsity and adaptivity can be added without any loss in quality.

Key points

  • α-entmax unifies softmax (α = 1) and sparsemax (α = 2). Values in between yield increasingly sparse attention, and the paper adds a closed-form Jacobian with respect to α (Proposition 1, derived via implicit differentiation).
  • Three systems were compared on four MT tasks: softmax, fixed 1.5-entmax (its first use inside a Transformer) and adaptive α-entmax with a separately learned α for each head.
  • Learned α values differ widely across heads. Examples include a previous-position head at α = 1.91 and a nearly dense head at α = 1.05 that captures the noun phrase "Armani Polo". Heads in later layers tend to be denser on average.
  • Sparse heads can pick tokens that are not contiguous and not tied to position. Even a single head (α = 1.33) can vary its sparsity from one input to the next.
  • Code and a pip package are released at github.com/deep-spin/entmax.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

72 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 67 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1223

@inproceedings{correia2019adaptively,
  title = {{Adaptively Sparse Transformers}},
  author = {Correia, Gonçalo M. and Niculae, Vlad and Martins, André F. T.},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1223},
  url = {https://doi.org/10.18653/v1/d19-1223}
}

References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brandon Amos and J. Zico Kolter. 2017. OptNet: Differentiable optimization as a layer in neural networks. In Proc. ICML. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. find our card
  3. Mathieu Blondel, André FT Martins, and Vlad Niculae. 2019. Learning classifiers with Fenchel-Young losses: Generalized entropies, margins, and algorithms. In Proc. AISTATS. find
  4. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proc. Workshop on Statistical Machine Translation. find our card
  5. Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 conference on machine translation. In Proc. WMT. find our card
  6. M Cettolo, M Federico, L Bentivogli, J Niehues, S Stüker, K Sudoh, K Yoshino, and C Federmann. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proc. IWSLT. find our card
  7. Sameen Maruf, André FT Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. preprint arXiv:1903.08788. find our card
  8. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse Transformers. preprint arXiv:1904.10509. find
  9. Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt. ↗
  10. Frank H Clarke. 1990. Optimization and Nonsmooth Analysis. SIAM. find
  11. Vlad Niculae and Mathieu Blondel. 2017. A regularized framework for sparse and structured neural attention. In Proc. NeurIPS. find
  12. Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. 2018. Latent alignment and variational attention. In Proc. NeurIPS. find
  13. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. WMT. find our card
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT. find
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL. find our card
  16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proc. ICML. find our card
  17. Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. 2016. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. preprint arXiv:1607.05447. — [arXiv:1607.05447](https://arxiv.org/abs/1607.05447) find
  18. Michael Held, Philip Wolfe, and Harlan P Crowder. 1974. Validation of subgradient optimization. Mathematical Programming, 6(1):62–88. find
  19. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proc. NAACL-HLT. find our card
  20. Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang, Roman Grundkiewicz, and Anthony Aue. 2018. Marian: Cost-effective high-quality neural machine translation in C++. In Proc. WNMT. find
  21. Jian Li, Zhaopeng Tu, Baosong Yang, Michael R Lyu, and Tong Zhang. 2018. Multi-Head Attention with Disagreement Regularization. In Proc. EMNLP. find
  22. Ben Peters, Vlad Niculae, and André FT Martins. 2019. Sparse sequence-to-sequence models. In Proc. ACL. find
  23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. preprint. find
  24. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in Transformer-based machine translation. In Proc. BlackboxNLP. find our card
  25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. find
  26. Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. 2019. SSN: Learning sparse switchable normalization via SparsestMax. In Proc. CVPR. find
  27. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive Attention Span in Transformers. In Proc. ACL. find our card
  28. Christos Louizos, Max Welling, and Diederik P Kingma. 2018. Learning sparse neural networks through L 0 regularization. Proc. ICLR. find
  29. Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? A targeted evaluation of neural machine translation architectures. In Proc. EMNLP. find
  30. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP. find
  31. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proc. ACL. find our card
  32. Chaitanya Malaviya, Pedro Ferreira, and André FT Martins. 2018. Sparse and constrained attention for neural machine translation. In Proc. ACL. find
  33. Constantino Tsallis. 1988. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52:479–487. find
  34. David Mareček and Rudolf Rosa. 2018. Extracting syntactic trees from Transformer encoder self-attentions. In Proc. BlackboxNLP. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. NeurIPS. find
  36. André FT Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proc. of ICML. find
  37. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proc. ACL. find our card
  38. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proc. ACL. find our card
  39. Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In Proc. ICLR. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/