Elamur Research⌕
☰
home / papers / 2020 / Understanding the Difficulty of Training Transformers

Understanding the Difficulty of Training Transformers

Paper · 2020 · Liyuan Liu, Xiaodong Liu, Jianfeng Gao et al.

Summary

Transformers are hard to train. They often need warmup, carefully tuned adaptive optimizers and learning-rate schedules, and the original Post-LN layout tends to diverge as depth grows. This paper looks for the cause, using both theory and experiments. The authors first show that gradient vanishing is not the main culprit. It affects only Post-LN decoders, and removing it does not stop divergence. Instead, they point to an amplification effect: each Post-LN layer relies heavily on its residual branch, so small parameter updates turn into large shifts in the model output. Pre-LN relies less on residual branches, which makes it more stable but often leaves it with weaker final models. Based on this, they propose Admin (Adaptive model initialization). Admin limits residual-branch dependency early in training and lets it grow later, without adding hyper-parameters. On IWSLT'14 De-En, WMT'14 En-De and WMT'14 En-Fr, Admin trains more stably, converges faster and reaches better BLEU. This makes very deep Post-LN Transformers practical to train.

Key points

  • In a 15-setting hyper-parameter grid without warmup, Pre-LN converged every time and Post-LN diverged in 7 cases. In 7 of the 8 runs where Post-LN converged, it beat Pre-LN.
  • In 18-layer decoders, replacing the Post-LN layout with Pre-LN removes the vanishing-gradient problem, yet training still diverges.
  • According to the paper's analysis, output change under parameter perturbation grows roughly as O(N) with depth for Post-LN and O(log N) for Pre-LN. Admin keeps the change near the Pre-LN level.
  • Admin trained a 72-layer Transformer on WMT'14 En-Fr to 43.80 BLEU.
  • ReZero, FixUp and LookLinear all diverged on 18-layer models. ReZero on Transformer-small lost 1–2 BLEU on IWSLT'14 De-En. Code is available in the GitHub repo "Transforemr-Clinic".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen and Jiawei Han. 2020. Understanding the Difficulty of Training Transformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.463 · arXiv 2004.08249

@inproceedings{liu2020understanding,
  title = {{Understanding the Difficulty of Training Transformers}},
  author = {Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen and Jiawei Han},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.463},
  eprint = {2004.08249},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.463}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. ArXiv, abs/1607.06450.
  2. Thomas C. Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian J. McAuley. 2020. Rezero is all you need: Fast convergence at large depth. ArXiv, abs/2003.04887. find
  3. Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In ICLR. find
  4. David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. 2017a. The shattered gradients problem: If resnets are the answer, then what is the question? In ICML. find
  5. David Balduzzi, Marcus Frean, Lennox Leary, J P Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. 2017b. The shattered gradients problem: If resnets are the answer, then what is the question? In ICML. find
  6. Yoshua Bengio, Patrice Y. Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks. find
  7. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Workshop on Statistical Machine Translation.
  8. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th iwslt evaluation campaign, iwslt 2014. In International Workshop on Spoken Language Translation, Hanoi, Vietnam.
  9. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Michael Schuster, Zhi-Feng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In ACL.
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
  11. Boris Hanin and David Rolnick. 2018. How to start training: The effect of initialization and architecture. In NeurIPS. find
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV. find
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. find
  14. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music transformer: Generating music with long-term structure. In ICLR. find
  15. Hao Li, Zheng Xu, Gavin Taylor, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In NeurIPS. find
  16. Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020a. On the variance of the adaptive learning rate and beyond. In ICLR. find
  17. Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. 2020b. Very deep transformers for neural machine translation. ArXiv, abs/2008.07772. — [arXiv:2008.07772](https://arxiv.org/abs/2008.07772) find
  18. Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2020. Understanding and improving transformer from a multi-particle dynamic system point of view. In ICLR Workshop DeepDiffEq. find
  19. Dmytro Mishkin and Juan E. Sala Matas. 2015. All you need is a good init. In ICLR. find
  20. Toan Q. Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. In IWSLT. find
  21. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT Demonstrations.
  22. Chengyu Dong, Liyuan Liu, Zichao Li, and Jingbo Shang. 2020. Towards adaptive residual network training: A neural-ode perspective. In ICML. find
  23. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In ICML. find
  24. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS. find
  25. Martin Popel and Ondrej Bojar. 2018. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110:43 – 70.
  26. Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch sgd: Training imagenet in 1 hour. ArXiv, abs/1706.02677. find
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683.
  28. Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. 2019. Stand-alone self-attention in vision models. In NeurIPS. find
  29. Andrew M Saxe, James L McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ArXiv, abs/1312.6120. find
  30. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2020-understanding-difficulty-training-transformers/