Elamur Research⌕
☰
home / papers / 2014 / Adam: A Method for Stochastic Optimization

Adam: A Method for Stochastic Optimization

Paper · 2014 · Diederik P. Kingma and Jimmy Ba

Summary

Training large models on noisy, minibatch-based objectives needs a first-order optimizer that is cheap, light on memory, and robust to gradient scale, sparse gradients and objectives that drift over time. Kingma and Ba propose Adam (adaptive moment estimation). For each parameter it keeps two exponential moving averages: one of the gradient and one of its elementwise square. The update divides the first by the square root of the second, so every parameter gets its own effective learning rate. Both averages start at zero and are therefore biased toward zero early in training. The authors derive a simple fix: divide each average by one minus its decay rate raised to the step count. They show that the effective step is roughly capped by the stepsize hyperparameter. It also stays the same when gradients are rescaled and shrinks on its own when the gradient direction is uncertain. In the online convex setting they prove an O(√T) regret bound, comparable to the best known results. Empirically, Adam compares favorably with other stochastic optimizers. The method combines strengths of AdaGrad and RMSProp, and a variant called AdaMax uses the infinity norm.

Key points

  • Suggested defaults for the tested problems are α = 0.001, β1 = 0.9, β2 = 0.999 and ε = 10^-8. All operations are element-wise.
  • Bias correction divides m_t by (1 − β1^t) and v_t by (1 − β2^t). Without it, early steps would be much too large, especially for sparse gradients where β2 must be close to 1.
  • The effective step |Δt| is approximately bounded by α. A looser bound of α·(1 − β1)/√(1 − β2) applies only in the extreme sparse case.
  • The convergence analysis follows Zinkevich's online learning framework. It assumes bounded gradients and bounded parameter distances, a learning rate α_t = α/√t, and β1,t = β1·λ^(t−1) with λ typically close to 1 (e.g. 1 − 10^-8). The resulting regret is O(√T).
  • The design aims to merge AdaGrad (good with sparse gradients) and RMSProp (good for on-line, non-stationary settings). A cheaper ordering of the computation folds the bias corrections into a per-step stepsize.

Interesting ideas

  • I1. Any running average that starts from zero is biased low at the beginning, and dividing by one minus the decay rate to the power of the step count removes this cheaply and exactly under stationarity.
  • I2. The ratio of mean gradient to its root-mean-square works like a signal-to-noise measure, so updates automatically become small where evidence is inconsistent, giving annealing for free.
  • I3. Normalizing by the second moment makes the step size independent of gradient scale, so the single knob α acts as a trust-region radius whose order of magnitude can be chosen in advance.
  • I4. Scoring an online learner by regret, meaning its cumulative loss minus that of the best fixed choice in hindsight, measures how fast adaptation pays off rather than only the final quality.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Running estimates of witness-parser agreement or LLM-teacher accuracy could get this correction, so early silver-data batches are not wrongly judged as low-agreement.
  • I2 (tree): The perceptron parser could scale each feature's update by how consistent its update sign has been, damping features that flip between correct and wrong attachments.
  • I3 (tree): Training gparse with this property means one stepsize setting can transfer across treebanks and loss mixtures without retuning for each UD dialect conversion.
  • I4 (learn): Self-induced repair rules learned online could be judged against the best fixed rule set chosen afterwards, exposing rules that only help after they are frozen.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🎓University of TorontoToronto · Canada96 works · 142,007 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksMathematics · 177 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 worksNumerical Analysis · 3 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Machine learning · 51 works

Cite

Cite as Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1412.6980 · arXiv 1412.6980

@misc{kingma2014adam,
  title = {{Adam: A Method for Stochastic Optimization}},
  author = {Diederik P. Kingma and Jimmy Ba},
  year = {2014},
  doi = {10.48550/arxiv.1412.6980},
  eprint = {1412.6980},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1412.6980}
}

References

22 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amari, Shun-Ichi. Natural gradient works efficiently in learning. Neural computation, 10(2):251–276, 1998. find
  2. Deng, Li, Li, Jinyu, Huang, Jui-Ting, Yao, Kaisheng, Yu, Dong, Seide, Frank, Seltzer, Michael, Zweig, Geoff, He, Xiaodong, Williams, Jason, et al. Recent advances in deep learning for speech research at microsoft. ICASSP 2013, 2013. find
  3. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. find
  4. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  5. Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6645–6649. find
  6. Hinton, G.E. and Salakhutdinov, R.R. Reducing the dimensionality of data with neural networks. Science, 313 (5786):504–507, 2006. find
  7. Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Mag… find
  8. Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012b. find
  9. Kingma, Diederik P and Welling, Max. Auto-Encoding Variational Bayes. In The 2nd International Conference on Learning Representations (ICLR), 2013.
  10. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
  11. Maas, Andrew L, Daly, Raymond E, Pham, Peter T, Huang, Dan, Ng, Andrew Y, and Potts, Christopher. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pp. 142–150. Association f…
  12. Moulines, Eric and Bach, Francis R. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems, pp. 451–459, 2011. find
  13. Pascanu, Razvan and Bengio, Yoshua. arXiv:1301.3584, 2013. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) find
  14. Revisiting natural gradient for deep networks. Polyak, Boris T and Juditsky, Anatoli B. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992. arXiv preprint Roux, Nicolas L and Fitzgibbon, Andrew W. A fast natural newton method. In Proc… find
  15. Ruppert, David. Efficient estimations from a slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988. find
  16. Schaul, Tom, Zhang, Sixin, and LeCun, Yann. No more pesky learning rates. arXiv preprint arXiv:1206.1106, 2012. — [arXiv:1206.1106](https://arxiv.org/abs/1206.1106) find
  17. Sohl-Dickstein, Jascha, Poole, Ben, and Ganguli, Surya. Fast large-scale optimization by unifying stochastic gradient and quasi-newton methods. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 604–612, 2014. find
  18. Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 1139–1147, 2013. find
  19. Tieleman, T. and Hinton, G. Lecture 6.5 - RMSProp, COURSERA: Neural Networks for Machine Learning. Technical report, 2012. find
  20. Wang, Sida and Manning, Christopher. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 118–126, 2013. find
  21. Zeiler, Matthew D. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
  22. Zinkevich, Martin. Online convex programming and generalized infinitesimal gradient ascent. 2003. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/