Elamur Research
☰
home / papers / 2019 / Pay Less Attention with Lightweight and Dynamic Convolutions

Pay Less Attention with Lightweight and Dynamic Convolutions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1901.10430

Summary

Self-attention compares every position with every other position. Its cost therefore grows quadratically with sequence length, and it is not clear that such unrestricted context is needed. Wu et al. test whether much simpler convolutional operators can match it on sequence modeling. They propose two operators. LightConv is a depthwise convolution whose kernel weights are softmax-normalized over the window and shared across groups of channels. DynamicConv builds on LightConv but predicts a fresh kernel at each position, computed only from that position's own representation. This makes its cost linear in sequence length. Both operators replace self-attention inside a Transformer Big encoder-decoder implemented in fairseq. They are evaluated on machine translation, language modeling and abstractive summarization. LightConv already performs close to strong self-attention models, and DynamicConv matches or beats them. On WMT'14 English-German it reaches a new state of the art. The work matters because it shows that content-based attention over the whole sequence is not required for top results, and that a cheaper, faster alternative exists.

Key points

  • DynamicConv scores 29.7 BLEU on WMT'14 English-German, a new state of the art. It ties the best published result on WMT English-French and beats self-attention by 0.8 BLEU on IWSLT German-English.
  • It runs about 20% faster than a highly optimized self-attention baseline.
  • Weight sharing makes the kernels tiny. With d = 1024, k = 7 and H = 16:
    • a standard convolution needs 7,340,032 weights,
    • a depthwise convolution needs 7,168,
    • LightConv needs only 112.
  • On the Billion word benchmark, DynamicConv equals or exceeds self-attention. On CNN-DailyMail summarization it outperforms a strong self-attention model.
  • Kernel normalization is essential for stable training. Without it, training diverges, while softmax gives the best result, 26.9 BLEU on newstest2013. Code and pretrained models are released in fairseq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 138 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1901.10430 · arXiv 1901.10430

@misc{wu2019pay,
  title = {{Pay Less Attention with Lightweight and Dynamic Convolutions}},
  author = {Wu, Felix and Fan, Angela and Baevski, Alexei and Dauphin, Yann N. and Auli, Michael},
  year = {2019},
  doi = {10.48550/arxiv.1901.10430},
  eprint = {1901.10430},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1901.10430}
}

References

63 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation. arxiv, abs/1711.02132, 2017. — [arXiv:1711.02132](https://arxiv.org/abs/1711.02132) find
  2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. abs/1607.06450, 2016. find our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, 2015. find our card
  4. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. Deep communicating agents for abstractive summarization. In Proc. of NAACL, 2018. find
  5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv, 2013. find our card
  6. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. arxiv, abs/1804.09849, 2018. find our card
  7. Yu-hsin Chen, Ignacio Lopez-Moreno, Tara N Sainath, Mirkó Visontai, Raziel Alvarez, and Carolina Parada. Locally-connected and convolutional neural networks for small footprint speaker recognition. In Proc. of Interspeech, 2015. find
  8. Franois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proc. of CVPR, pp. 1800–1807, 2017. find
  9. Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, KyungHyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. arXiv, abs/1506.07503, 2015. find our card
  10. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proc. of ICML, 2017. find our card
  11. Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander M Rush. Latent alignment and variational attention. arXiv, abs/1807.03756, 2018. — [arXiv:1807.03756](https://arxiv.org/abs/1807.03756) find
  12. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proc. of NAACL, 2018. find
  13. Maha Elbayad, Laurent Besacier, and Jakob Verbeek. Pervasive attention: 2d convolutional neural networks for sequence-to-sequence prediction. arXiv, abs/1808.03867, 2018. find
  14. Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. arXiv, abs/1711.05217, 2017. find our card
  15. Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin. A Convolutional Encoder Model for Neural Machine Translation. arXiv, abs/1611.02344, 2016. find
  16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional Sequence to Sequence Learning. In Proc. of ICML, 2017. find our card
  17. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summarization. arXiv, abs/1808.10792, 2018. find our card
  18. Jingjing Gong, Xipeng Qiu, Xinchi Chen, Dong Liang, and Xuanjing Huang. Convolutional interaction network for natural language inference. In Proc. of EMNLP, 2018. find
  19. Edouard Grave, Armand Joulin, Moustapha Cisse, David Grangier, and Herve Jegou. Efficient softmax approximation for gpus. arXiv, abs/1609.04309, 2016. find
  20. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B1l8BtlCb. ↗ our card
  21. Layer normalization. arXiv, Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. Non-autoregressive neural machine translation with enhanced decoder input. AAAI, 2019. find
  22. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, Will Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhiru… find our card
  23. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proc. of CVPR, 2015. find
  24. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. of NIPS, 2015. find our card
  25. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. find our card
  26. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv, abs/1602.02410, 2016. find
  27. Lukasz Kaiser, Aidan N. Gomez, and François Chollet. Depthwise separable convolutions for neural machine translation. arXiv, abs/1706.03059, 2017. — [arXiv:1706.03059](https://arxiv.org/abs/1706.03059) find
  28. Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pp. 2390–2399, 2018. find
  29. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural Machine Translation in Linear Time. arXiv, 2016. find
  30. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR, 2015. find
  31. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(11):2278–2324, 1998. find
  32. Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. arXiv, abs/1802.06901, 2018. find our card
  33. Zhuohan Li, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. Hint-based training for non-autoregressive translation, 2019. URL https://openreview.net/forum?id=r1gGpjActQ. ↗
  34. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out, 2004. find
  35. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv, abs/1801.10198, 2018. find our card
  36. Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv, abs/1608.03983, 2016. find
  37. Minh-Thang Luong, Hieu Pham, and Christopher Manning. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP, 2015. find
  38. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proc. of CONLL, 2016. find
  39. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proc. of WMT, 2018. find our card
  40. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proc. of ICML, 2013. find
  41. Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv, abs/1705.04304, 2017. find our card
  42. Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In Proc. of ICLR Workshop, 2017. find our card
  43. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proc. of EACL 2017, 2017. find our card
  44. Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Proc. of ACL, 2017. find
  45. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016. find
  46. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proc. of NAACL, 2018. find our card
  47. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proc. of ICLR, 2017. find
  48. Dinghan Shen, Martin Renqiang Min, Yitong Li, and Lawrence Carin. Learning context-sensitive convolutional filters for text processing. In Proc. of EMNLP, 2018a. find
  49. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Directional self-attention network for rnn/cnn-free language understanding. arXiv, abs/1709.04696, 2017. find
  50. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self-attention for fast and memory-efficient sequence modeling. arXiv, abs/1804.00857, 2018b. — [arXiv:1804.00857](https://arxiv.org/abs/1804.00857) find
  51. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Fast directional self-attention mechanism. arXiv, abs/1805.00912, 2018c. — [arXiv:1805.00912](https://arxiv.org/abs/1805.00912) find
  52. Laurent Sifre. Rigid-motion scattering for image classification. Ph.D. thesis section 6.2, 2014. find
  53. Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Proc. of NIPS, 2015. find
  54. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013. find
  55. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Networks. In Proc. of NIPS, 2014. find
  56. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. arXiv, abs/1512.00567, 2015. find
  57. Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proc. of CVPR, 2014. find
  58. Gongbo Tang, Mathias Mller, Annette Rios, and Rico Sennrich. Why self-attention? a targeted evaluation of neural machine translation architectures. In Proc. of EMNLP, 2018. find
  59. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proc. of NIPS, 2017. find
  60. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proc. of ICML, 2013. find
  61. Zhengyang Wang and Shuiwang Ji. Smoothed dilated convolutions for improved dense prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2486–2495. ACM, 2018. find
  62. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv, abs/1609.08144, 2016. find
  63. Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average attention network. arXiv, abs/1805.00631, 2018. find