Elamur Research⌕
☰
home / papers / 2022 / The Devil in Linear Transformer

The Devil in Linear Transformer

Paper · 2022 · Zhen Qin, Xiaodong Han, Weixuan Sun et al.

Summary

Kernel-based linear transformers reduce attention cost from quadratic to linear in sequence length, but they usually lose accuracy compared with the standard softmax transformer. The authors trace this gap to two causes. First, their analysis shows that the scaling (row normalization) that linear attention inherits from softmax leaves its gradients without an upper bound. Softmax gradients, by contrast, stay within 1/4. The unbounded gradients make training unstable. Second, linear attention spreads its weights almost evenly over long inputs, a problem they call attention dilution, so local structure is lost. They measure this with a locally accumulated attention score. Their model, TransNormer, removes the scaling and adds a normalization layer after attention instead (NormAttention). It also uses block-local DiagAttention in the lower layers, and the whole model keeps linear complexity. On WikiText-103 language modeling, GLUE classification and the Long-Range Arena benchmark, TransNormer outperforms the vanilla transformer and earlier linear variants while running faster. This makes efficient attention a realistic replacement for softmax attention rather than a compromise.

Key points

  • Gradient analysis: softmax attention gradients are bounded by 1/4. The linear-attention gradient bound scales as 1/(4|s_ik|), so it can grow arbitrarily large (Proposition 3.1). NormAttention works with either LayerNorm or RMSNorm.
  • Architecture: in a 12-layer model, the best setup puts DiagAttention in the first 6 layers and NormAttention in the remaining 6. Using only one attention type, or swapping their order, gives higher validation perplexity in the RoBERTa-style pre-training ablation (3.82/3.81 for the chosen T1/T2 configuration).
  • Ablations: larger DiagAttention blocks help (block size 128 gives ppl 3.72/3.69), but block size 64 was chosen to balance quality against the O(nwd) cost. A GLU feed-forward layer beats a standard FFN (3.82 vs 3.93 for T1). Running both attentions in parallel within a layer doubles compute without a gain.
  • Speed: on an Nvidia A6000 (48 GB) using Skyformer's benchmark settings, TransNormer trains over 300% faster and runs inference 150% faster than FLASH-quad. The vanilla transformer runs out of memory beyond roughly 3k tokens.
  • Results and code: both variants (T1, T2) beat all compared methods overall on Long-Range Arena. Code is published at github.com/OpenNLPLab/Transnormer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Hong KongPok Fu Lam · China89 works · 11,774 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations🎓Australian National UniversityCanberra · Australia23 works · 6,522 citations🏢SensetimeBeijing · China12 works · 2,065 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes and Yiran Zhong. 2022. The Devil in Linear Transformer. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.473

@inproceedings{qin2022devil,
  title = {{The Devil in Linear Transformer}},
  author = {Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes and Yiran Zhong},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.473},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.473}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization.
  2. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  3. Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. 2021. Skyformer: Remodel self-attention with gaussian kernel and nyström method. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual. find
  4. Xuelian Cheng, Huan Xiong, Deng-Ping Fan, Yiran Zhong, Mehrtash Harandi, Tom Drummond, and Zongyuan Ge. 2022a. Implicit motion handling for video camouflaged object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13864–13873. find
  5. Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Tom Drummond, Zhiyong Wang, and Zongyuan Ge. 2022b. Deep laparoscopic stereo matching with transformers. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 464–474. Springer. find
  6. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509.
  7. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794.
  8. Bolin Gao and Lacra Pavel. 2017. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805. — [arXiv:1704.00805](https://arxiv.org/abs/1704.00805) find
  9. Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V Le. 2022. Transformer quality in linear time. arXiv preprint arXiv:2202.10447. find
  10. Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
  11. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR. find
  12. Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451.
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  14. Zexiang Liu, Dong Li, Kaiyue Lu, Zhen Qin, Weixuan Sun, Jiacheng Xu, and Yiran Zhong. 2022. Neural architecture search on efficient transformers and beyond. arXiv preprint arXiv:2207.13955. — [arXiv:2207.13955](https://arxiv.org/abs/2207.13955) find
  15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. 5th International Conference on Learning Representations, ICLR, Toulon, France.
  16. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038.
  17. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2020. Random feature attention. In International Conference on Learning Representations. find
  18. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. 2022. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations. find
  19. Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. find
  20. Jingyu Sun, Guiping Zhong, Dinghao Zhou, Baoxiang Li, and Yiran Zhong. 2022a. Locality matters: A locality-biased linear attention for automatic speech recognition. arXiv preprint arXiv:2203.15609. — [arXiv:2203.15609](https://arxiv.org/abs/2203.15609) find
  21. Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. 2022b. Vicinity vision transformer. arXiv preprint arXiv:2206.10552. — [arXiv:2206.10552](https://arxiv.org/abs/2206.10552) find
  22. Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020a. Sparse sinkhorn attention. In International Conference on Machine Learning, pages 9438–9447. PMLR. find
  23. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. In NeurIPS.
  24. Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada. find
  25. Biao Zhang, Ivan Titov, and Rico Sennrich. 2021. Sparse attention with linear units. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6507–6520, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  26. Lin Zheng, Chong Wang, and Lingpeng Kong. 2022. Linear complexity randomized self-attention mechanism. arXiv preprint arXiv:2204.04667. — [arXiv:2204.04667](https://arxiv.org/abs/2204.04667) find
  27. Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. 2022. Audio-visual segmentation. In European Conference on Computer Vision. find
  28. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020b. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations.
  29. Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021. Long-short transformer: Efficient transformers for language and vision. In Advances in Neural Information Processing Systems. find
  30. Michalis K Titsias. 2016. One-vs-each approximation to softmax for scalable estimation of probabilities. arXiv preprint arXiv:1609.07410. — [arXiv:1609.07410](https://arxiv.org/abs/1609.07410) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qin-2022-devil-linear-transformer/