Elamur Research⌕
☰
home / papers / 2020 / ReZero is All You Need: Fast Convergence at Large Depth

ReZero is All You Need: Fast Convergence at Large Depth

Paper · 2020 · Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao et al.

Summary

Deep neural networks are hard to train because signals and gradients vanish or explode as they pass through many layers. The authors propose ReZero, a minimal change to the architecture. The output of each residual block is multiplied by its own scalar parameter α, which is initialized to zero. As a result, the network acts as the identity map at the start of training and automatically satisfies dynamical isometry. Earlier theory links this property to a network's ability to train efficiently. During optimization, the α parameters gradually reach the values they need. The method was tested on fully connected networks, on ResNet with the CIFAR-10 dataset, and on a Transformer for language modeling on enwiki8. ReZero makes it possible to train networks with thousands of layers and Transformers with more than 100 layers. It needs no LayerNorm, no warm-up and no auxiliary losses. Standard-depth models also converge noticeably faster. In practice, the method takes only a few lines of code, works with any architecture that has residual connections, and lowers the compute cost of training.

Key points

  • Layer update rule: x_{i+1} = x_i + α_i·F(x_i), where α_i is learned and initialized to 0. The code is published in the repository github.com/majumderb/rezero.
  • A 12-layer Transformer on enwiki8 reaches 1.2 BPB in 8,800 iterations. Post-Norm with warm-up needs 13,690 iterations to get there, so ReZero is about 1.56 times faster. Without warm-up, Post-Norm diverges. ReZero with α = 1 at initialization needs 14,506 iterations. The final quality (1.17 BPB) matches that of the standard Transformer.
  • A standard 64-layer Transformer diverges even with warm-up, while a ReZero model of the same depth reaches 1.11 BPB. A 128-layer version (101M parameters) reaches 1.08 BPB. This is close to the Character Transformer (64 layers, 219M parameters, 1.06 BPB), which needs auxiliary losses to get there.
  • ReZero made it possible to train a fully connected network with 10,000 layers. ResNet on CIFAR-10 reaches 85% accuracy 32% faster.
  • Early in training the α weights grow quickly and then slowly decrease. By the end, their mean is about 1/L: 0.0898 for 12 layers and 0.0185 for 64 layers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California San DiegoSan Diego · United States101 works · 54,571 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 works

Cite

Cite as Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell and Julian McAuley. 2020. ReZero is All You Need: Fast Convergence at Large Depth. arXiv:2003.04887.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.04887 · arXiv 2003.04887

@misc{bachlechner2020rezero,
  title = {{ReZero is All You Need: Fast Convergence at Large Depth}},
  author = {Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell and Julian McAuley},
  year = {2020},
  doi = {10.48550/arxiv.2003.04887},
  eprint = {2003.04887},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.04887}
}

References

44 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553), 2015. find
  2. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
  3. Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In NIPS, 2017. find
  4. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. find
  5. Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In NIPS, pages 3360–3368, 2016. find
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. find
  7. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.
  8. Dmytro Mishkin and Jiri Matas. All you need is a good init. arXiv preprint arXiv:1511.06422, 2015. — [arXiv:1511.06422](https://arxiv.org/abs/1511.06422) find
  9. Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. arXiv preprint arXiv:1806.05393, 2018. — [arXiv:1806.05393](https://arxiv.org/abs/1806.05393) find
  10. Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321, 2019. find
  11. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. find
  12. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  13. Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NIPS, 2017. find
  14. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In AAAI, volume 33, 2019.
  15. Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014. find
  16. Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. arXiv preprint arXiv:1802.09979, 2018. — [arXiv:1802.09979](https://arxiv.org/abs/1802.09979) find
  17. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In NIPS, 2010. find
  18. Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
  19. Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint arXiv:1611.01232, 2016. — [arXiv:1611.01232](https://arxiv.org/abs/1611.01232) find
  20. Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In NIPS, 2017. find
  21. Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S Schoenholz, Ed H Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of lstms and grus. arXiv preprint arXiv:1901.08987, 2019. — [arXiv:1901.08987](https://arxiv.org/abs/1901.08987) find
  22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision. Springer, 2016. find
  23. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. find
  24. Moritz Hardt and Tengyu Ma. arXiv:1611.04231, 2016. — [arXiv:1611.04231](https://arxiv.org/abs/1611.04231) find
  25. Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, 2019. find
  26. Soham De and Samuel L Smith. Batch normalization biases deep residual networks towards shallow paths. arXiv preprint arXiv:2002.10444, 2020. — [arXiv:2002.10444](https://arxiv.org/abs/2002.10444) find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
  28. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul), 2011. find
  29. Pedro HP Savarese, Leonardo O Mazza, and Daniel R Figueiredo. Learning identity mappings with residual gates. arXiv preprint arXiv:1611.01260, 2016. — [arXiv:1611.01260](https://arxiv.org/abs/1611.01260) find
  30. Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 11006, page 1100612. International Society for Optics and Photonics, 2019. find
  31. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL, 2019.
  32. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
  33. Matt Mahoney. Large text compression benchmark, 2009. find
  34. Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. CoRR, abs/1910.05895, 2019. Identity matters in deep learning. arXiv preprint
  35. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019.
  36. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. arXiv preprint arXiv:2002.04745, 2020. find
  37. Microsoft. Turing-nlg: A 17-billion-parameter language model, 2020. find
  38. Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu. Efficient training of BERT by progressively stacking. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, volume 97 of Proceedings of Machine Learning Research, 2019. find
  39. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  40. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August…
  41. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. abs/1907.10597, 2019.
  42. Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016. find
  43. Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing BERT pre-training time from 3 days to 76 minutes. CoRR, abs/1904.00962, 2019. find
  44. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bachlechner-2020-rezero-all-need-fast-convergence/