Elamur Research
☰
home / papers / 2016 / Layer Normalization

Layer Normalization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1607.06450

Summary

Deep neural networks take a long time to train. Batch normalization is a popular way to speed up training, but it has limitations. It depends on the mini-batch size, computes differently during training and testing, and is poorly suited to recurrent networks. The authors propose layer normalization. The mean and variance are computed over all neurons of a layer for each individual example, not over the examples in a batch. After normalization, each neuron has its own learnable gain and bias. The method works the same way during training and inference and is suitable even for a batch of a single example. In RNNs, the statistics are computed separately at each time step, while the parameters are shared across all steps. The authors also give a theoretical comparison of the transformations to which layer, batch and weight normalization are invariant. In experiments on recurrent models, DRAW, handwriting generation and MNIST, the method noticeably speeds up convergence. The largest gain is in RNNs with long sequences and small batches. This technique later became a standard part of modern architectures, including the Transformer, so it matters for NLP projects.

Key points

  • DRAW (64 glimpses, 256 LSTM units) on binarized MNIST: with layer normalization the model converges almost twice as fast. After 200 epochs it reaches 82.09 nats, compared with 82.36 for the baseline model.
  • IAM-OnDB: 12179 handwritten lines from 221 writers, with an average length of about 700 steps. The model has 3 layers of 400 LSTM cells and a mini-batch of 8. With layer normalization it reaches a similar log likelihood, but much faster.
  • Permutation invariant MNIST (784-1000-1000-10), batch sizes 128 and 4: layer normalization is robust to the batch size and converges faster than batch normalization. The final softmax layer is not normalized.
  • Invariance (Table 1): layer normalization leaves the output unchanged when the whole weight matrix is scaled or shifted, or when a single example is scaled. Scaling a single weight vector does change the output.
  • Convolutional networks: in preliminary experiments, layer normalization speeds up training compared with a network without normalization, but batch normalization performs better. This is because neurons near the image boundaries have different statistics from the rest of the layer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

1145 papers of our library cite it (of the 140673 whose bibliographies we read); 77 of them have a page here, 1068 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1607.06450 · arXiv 1607.06450

@misc{ba2016layer,
  title = {{Layer Normalization}},
  author = {Ba, Jimmy Lei and Kiros, Jamie Ryan and Hinton, Geoffrey E.},
  year = {2016},
  doi = {10.48550/arxiv.1607.06450},
  eprint = {1607.06450},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1607.06450}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. find
  2. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE, 2012. find
  3. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In NIPS, 2012. find
  4. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015. find
  5. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014. find
  6. César Laurent, Gabriel Pereyra, Philémon Brakel, Ying Zhang, and Yoshua Bengio. Batch normalized recurrent neural networks. arXiv preprint arXiv:1510.01378, 2015. — [arXiv:1510.01378](https://arxiv.org/abs/1510.01378) find
  7. Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. find
  8. Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. Recurrent batch normalization. arXiv preprint arXiv:1603.09025, 2016. — [arXiv:1603.09025](https://arxiv.org/abs/1603.09025) find
  9. Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016. — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) find
  10. Behnam Neyshabur, Ruslan R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, pages 2413–2421, 2015. find
  11. Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 1998. find
  12. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. ICLR, 2016. find
  13. The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. Theano: A python framework for fast computation of mathematical expressions. arXiv preprint arXiv:1605.02688… find our card
  14. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014. find our card
  15. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP, 2014. find
  16. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015. find our card
  17. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014. find our card
  18. D. Kingma and J. L. Ba. Adam: a method for stochastic optimization. ICLR, 2014. arXiv:1412.6980. find
  19. Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings. CVPR, 2016. find
  20. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, 2015. find our card
  21. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In NIPS, 2015. find
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. find
  23. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, 2015. find our card
  24. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. SemEval-2014, 2014. find our card
  25. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115–124, 2005. find
  26. Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004. find
  27. Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL, 2004. find
  28. Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 2005. find
  29. K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: a recurrent neural network for image generation. arXiv:1502.04623, 2015. find
  30. Hugo Larochelle and Iain Murray. The neural autoregressive distribution estimator. In AISTATS, volume 6, page 622, 2011. find
  31. Marcus Liwicki and Horst Bunke. Iam-ondb-an on-line english sentence database acquired from handwritten text on a whiteboard. In ICDAR, 2005. find
  32. Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find