Elamur Research
☰
home / papers / 2018 / Averaging Weights Leads to Wider Optima and Better Generalization

Averaging Weights Leads to Wider Optima and Better Generalization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1803.05407

Summary

Deep networks are usually trained with SGD and a decaying learning rate, and the final iterate tends to sit near the edge of a broad region of good solutions rather than in its middle. Izmailov et al. propose Stochastic Weight Averaging (SWA). SGD runs with a cyclical or relatively high constant learning rate, which keeps it moving through a low-loss region, and the weights visited along the way are combined with equal weights into one model. The averaged point lands closer to the centre of a flatter area of the training loss. Its training loss is often slightly higher than SGD's, but its test error is lower. SWA can be seen as a single-network approximation of Fast Geometric Ensembling (FGE), so it avoids the k-fold inference cost of an ensemble. It was tested on CIFAR-10, CIFAR-100 and ImageNet with VGG-16, Preactivation ResNet-164, Wide ResNet-28-10, Shake-Shake, PyramidNet, ResNet and DenseNet models. SWA consistently beat standard training at almost no extra compute. This makes it a cheap, drop-in way to improve generalization.

Key points

  • ImageNet: 10 extra SWA epochs on pretrained torchvision models raised top-1 accuracy:
    • ResNet-50: 76.15 → 76.97
    • ResNet-152: 78.31 → 78.94
    • DenseNet-161: 77.65 → 78.44
  • CIFAR: typical gains were about 0.5% on CIFAR-10 and 0.75–1.5% on CIFAR-100. A single SWA model matched or beat FGE ensembles of 6–12 networks.
  • Training from scratch with a fixed learning rate: Wide ResNet-28-10 was trained on CIFAR-100 with a fixed learning rate of 0.05 for 300 epochs, averaging weights from epoch 140 onward. The averaged model reached 81.7% accuracy, while the individual averaged snapshots stayed around 65%.
  • Loss geometry: along the line from the SGD solution to the SWA solution, the loss rises sharply on the SGD side, while SWA sits in a flatter area.
  • Code: the authors' implementation is available at github.com/timgaripov/swa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.05407 · arXiv 1803.05407

@misc{izmailov2018averaging,
  title = {{Averaging Weights Leads to Wider Optima and Better Generalization}},
  author = {Izmailov, Pavel and Podoprikhin, Dmitrii and Garipov, Timur and Vetrov, Dmitry and Wilson, Andrew Gordon},
  year = {2018},
  doi = {10.48550/arxiv.1803.05407},
  eprint = {1803.05407},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.05407}
}

References

23 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Chaudhari, Anna Choromanska, S. Soatto, Yann Le-Cun, C. Baldassi, C. Borgs, J. Chayes, Levent Sagun, and R. Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In International Conference on Learning Representations (ICLR), 2017. find
  2. Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pages 192–204, 2015. find
  3. Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019–1028, 2017. find
  4. Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference on Machine Learning, pages 1308–1317, 2018. find
  5. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. arXiv preprint arXiv:1802.10026, 2018. — [arXiv:1802.10026](https://arxiv.org/abs/1802.10026) find
  6. Xavier Gastaldi. Shake-shake regularization. preprint arXiv:1705.07485, 2017. arXiv Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4): 2341–2368, 2013. — [arXiv:1705.07485](https://arxiv.org/abs/1705.07485) find
  7. Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. International Conference on Learning Representations, 2015. find
  8. Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. arXiv preprint arXiv:1610.02915, 2016. — [arXiv:1610.02915](https://arxiv.org/abs/1610.02915) find
  9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. find
  10. Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997. find
  11. Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, volume 1, page 3, 2017. find
  12. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456, 2015. find
  13. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. International Conference on Learning Representations, 2017. find
  14. Ilya Loshchilov and Frank Hutter. Sgdr: stochastic gradient descent with restarts. International Conference on Learning Representations, 2017. find
  15. Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873–4907, 2017. find
  16. Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variance networks: When expectation does not meet your expectations. arXiv preprint arXiv:1803.03764, 2018. — [arXiv:1803.03764](https://arxiv.org/abs/1803.03764) find
  17. Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992. find
  18. David Ruppert. Efficient estimations from a slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988. find
  19. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211– 252, 2012. find our card
  20. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. find our card
  21. Leslie N Smith and Nicholay Topin. Exploring loss function topology with cyclical learning rates. arXiv preprint arXiv:1702.04283, 2017. — [arXiv:1702.04283](https://arxiv.org/abs/1702.04283) find
  22. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15 (1):1929–1958, 2014. find
  23. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. — [arXiv:1605.07146](https://arxiv.org/abs/1605.07146) find