ADADELTA: An Adaptive Learning Rate Method

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1212.5701

Summary

Plain stochastic gradient descent needs a hand-tuned global learning rate. ADAGRAD adapts the rate for each parameter, but it still has a global step size, and its sum of squared gradients keeps growing, so steps eventually shrink to almost nothing. Zeiler's ADADELTA fixes both problems with two changes. First, it replaces ADAGRAD's full history of squared gradients with an exponentially decaying average, so the denominator reflects only recent gradients. Second, it scales each step by a decaying RMS of earlier parameter updates. This gives updates the same units as the parameters, like a diagonal Newton-style step, and removes the global learning rate. It costs very little more computation than SGD. The author tests it on MNIST digit classification on a single machine and on a large speech-recognition network trained on Google's distributed system. ADADELTA converges quickly and beats ADAGRAD on speech frame accuracy, using the same two hyperparameters throughout. Because it largely removes learning-rate tuning, it became a widely used optimizer for deep networks.

Key points

• The update keeps two running averages with decay ρ, one for squared gradients and one for squared updates. Each step is the gradient times the ratio RMS[Δx]ₜ₋₁ / RMS[g]ₜ, and a small ε stabilizes both RMS terms.
• The same settings (ρ = 0.95, ε = 1e-6) were used for MNIST and speech without retuning.
• The speech model had 4 hidden layers of 2560 units each, using logistic or ReLU activations. Its input was 26 frames of 40 log-energy filter-bank features, and its output was 8,000 senone labels. It was trained on several hundred hours of US English from Voice Search, Voice IME and read speech.
• With 100 replicas, ADADELTA was ahead of ADAGRAD for the whole run. With 200 replicas (ReLU), the gradients were much noisier, but it still quickly reached the same frame accuracy as ADAGRAD and momentum.
• The MNIST analysis shows three things. Lower layers get larger effective steps, which offsets vanishing gradients. Late in training, the step sizes approach 1 while the actual updates shrink, which works like built-in annealing. Well-tuned momentum can still pull slightly ahead late in training.