Elamur Research
☰
home / papers / 2015 / Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.07289

Summary

The paper proposes the exponential linear unit (ELU), an activation function for deep networks designed to train faster and generalize better than ReLU-style units. For positive inputs, ELU acts as the identity, which avoids vanishing gradients. For negative inputs, it follows an exponential curve that levels off at a fixed negative value. The negative outputs pull average activations toward zero, much as batch normalization does but more cheaply. The saturation gives a stable, noise-insensitive "off" state that leaky ReLU and PReLU lack. The authors support this with a theoretical analysis based on the unit natural gradient. They show that non-zero mean activations cause a bias shift in the next layer, and that reducing this shift brings ordinary gradient descent closer to Fisher-optimal updates. Experiments on MNIST, CIFAR-10, CIFAR-100 and ImageNet show quicker convergence and better accuracy than ReLU and LReLU in networks deeper than five layers. These results made ELU a simple drop-in option for training deep models.

Key points

  • The theory section contains two results. Theorem 1 expresses the unit natural gradient as an affine correction of the standard gradient. Theorem 2 interprets the bias-shift correction as moving incoming mean activations toward zero.
  • On CIFAR-100, ELU networks reached the best published result at the time, without multi-view testing or model averaging. They also beat ReLU networks that used batch normalization, while batch normalization gave ELU networks no extra gain.
  • On CIFAR-10, ELU results ranked among the top 10 reported. On ImageNet, a single-model, single-crop ELU network learned faster than an identical ReLU network and reached under 10% classification error.
  • An appendix experiment on MNIST used 5 hidden layers of 256 units, trained for 200 epochs at learning rate 0.01. The per-epoch median activations varied much more under ReLU than under ELU, which suggests ReLU networks keep compensating for bias shift.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

129 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 122 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.07289 · arXiv 1511.07289

@misc{clevert2015fast,
  title = {{Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)}},
  author = {Clevert, Djork-Arné and Unterthiner, Thomas and Hochreiter, Sepp},
  year = {2015},
  doi = {10.48550/arxiv.1511.07289},
  eprint = {1511.07289},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.07289}
}

References

35 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amari, S.-I. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998. find
  2. Clevert, D.-A., Unterthiner, T., Mayr, A., and Hochreiter, S. Rectified factor networks. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 28. Curran Associates, Inc., 2015. find
  3. Desjardins, G., Simonyan, K., Pascanu, R., and Kavukcuoglu, K. Natural neural networks. CoRR, abs/1507.00210, 2015. URL http://arxiv.org/abs/1507.00210. — [arXiv:1507.00210](https://arxiv.org/abs/1507.00210) ↗
  4. Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. In Gordon, G., Dunson, D., and Dudk, M. (eds.), JMLR W&CP: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2011), volume 15, pp. 315–323, 2011. find
  5. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. Maxout networks. ArXiv e-prints, 2013. find
  6. Graham, Benjamin. Fractional max-pooling. CoRR, abs/1412.6071, 2014. URL http://arxiv.org/abs/1412.6071. — [arXiv:1412.6071](https://arxiv.org/abs/1412.6071) ↗
  7. Grosse, R. and Salakhudinov, R. Scaling up natural gradient by sparsely factorizing the inverse Fisher matrix. Journal of Machine Learning Research, 37:2304–2313, 2015. URL http://jmlr.org/proceedings/papers/v37/grosse15.pdf. Proceedings of the 32nd International Conference on Machine Learning (ICML… ↗
  8. He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision (ICCV), 2015. relu elu relu elu Top 1 Test Error [%] find
  9. Updates (1e3) Updates (1e3) (c) Top-1 test error Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(2):107–116, 1998. find
  10. Hochreiter, S. and Schmidhuber, J. Feature extraction through LOCOCODE. Neural Computation, 11(3): 679–714, 1999. find
  11. Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In Kremer and Kolen (eds.), A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001. find
  12. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Journal of Machine Learning Research, 37:448–456, 2015. URL http://jmlr.org/proceedings/papers/v37/ioffe15.pdf. Proceedings of the 32nd International Conference on Machine Learning… ↗
  13. Jia, Yangqing. Learning Semantic Image Representations at a Large Scale. PhD thesis, EECS Department, University of California, Berkeley, May 2014. URL http://www.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-93.html. ↗
  14. Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc., 2012. find
  15. Kurita, T. Iterative weighted least squares algorithms for neural networks classifiers. In Proceedings of the Third Workshop on Algorithmic Learning Theory (ALT92), volume 743 of Lecture Notes in Computer Science, pp. 77–86. Springer, 1993. find
  16. LeCun, Y., Kanter, I., and Solla, S. A. Eigenvalues of covariance matrices: Application to neural-network learning. Physical Review Letters, 66(18):2396–2399, 1991. find
  17. LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Orr, G. B. and Müller, K.-R. (eds.), Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pp. 9–50. Springer, 1998. find
  18. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick W., Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. In AISTATS, 2015. find
  19. LeRoux, N., Manzagol, P.-A., and Bengio, Y. Topmoumoute online natural gradient algorithm. In Platt, J. C., Koller, D., Singer, Y., and Roweis, S. T. (eds.), Advances in Neural Information Processing Systems 20 (NIPS), pp. 849–856, 2008. find
  20. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. CoRR, abs/1312.4400, 2013. URL http: //arxiv.org/abs/1312.4400. — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) find
  21. Maas, A. L., Hannun, A. Y., and Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International Conference on Machine Learning (ICML13), 2013. find
  22. Martens, J. Deep learning via Hessian-free optimization. In Fürnkranz, J. and Joachims, T. (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML10), pp. 735–742, 2010. find
  23. Mayr, A., Klambauer, G., Unterthiner, T., and Hochreiter, S. DeepTox: Toxicity prediction using deep learning. Front. Environ. Sci., 3(80), 2015. doi: 10.3389/fenvs.2015.00080. URL http://journal. frontiersin.org/article/10.3389/fenvs.2015.00080. — [doi:10.3389/fenvs.2015.00080](https://doi.org/10.3389/fenvs.2015.00080) find
  24. Nair, V. and Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In Fürnkranz, J. and Joachims, T. (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML10), pp. 807–814, 2010. find
  25. Olivier, Y. Riemannian metrics for neural networks i: feedforward networks. CoRR, abs/1303.0818, 2013. URL http://arxiv.org/abs/1303.0818. — [arXiv:1303.0818](https://arxiv.org/abs/1303.0818) ↗
  26. Pascanu, R. and Bengio, Y. Revisiting natural gradient for deep networks. In International Conference on Learning Representations 2014, 2014. URL http://arxiv.org/abs/1301.3584. arXiv:1301.3584. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) ↗
  27. Raiko, T., Valpola, H., and LeCun, Y. Deep learning made easier by linear transformations in perceptrons. In Lawrence, N. D. and Girolami, M. A. (eds.), Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS12), volume 22, pp. 924–932, 2012. find
  28. Schraudolph, N. N. Centering neural network gradient factor. In Orr, G. B. and Müller, K.-R. (eds.), Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pp. 207–226. Springer, 1998. find
  29. Schraudolph, Nicol N. A Fast, Compact Approximation of the Exponential Function. Neural Computation, 11: 853–862, 1999. find
  30. Springenberg, Jost Tobias, Dosovitskiy, Alexey, Brox, Thomas, and Riedmiller, Martin A. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014. URL http://arxiv.org/abs/1412.6806. — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) ↗
  31. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmidhuber, Jürgen. Training very deep networks. CoRR, abs/1507.06228, 2015. URL http://arxiv.org/abs/1507.06228. — [arXiv:1507.06228](https://arxiv.org/abs/1507.06228) ↗
  32. Unterthiner, T., Mayr, A., Klambauer, G., and Hochreiter, S. Toxicity prediction using deep learning. CoRR, abs/1503.01445, 2015. URL http://arxiv.org/abs/1503.01445. — [arXiv:1503.01445](https://arxiv.org/abs/1503.01445) ↗
  33. Vinyals, O. and Povey, D. Krylov subspace descent for deep learning. In AISTATS, 2012. URL http: //arxiv.org/pdf/1111.4259v1. arXiv:1111.4259. — [arXiv:1111.4259](https://arxiv.org/abs/1111.4259) find
  34. Xu, B., Wang, N., Chen, T., and Li, M. Empirical evaluation of rectified activations in convolutional network. CoRR, abs/1505.00853, 2015. URL http://arxiv.org/abs/1505.00853. — [arXiv:1505.00853](https://arxiv.org/abs/1505.00853) ↗
  35. Yang, H. H. and Amari, S.-I. Complexity issues in natural gradient descent method for training multilayer perceptrons. Neural Computation, 10(8), 1998. find