Elamur Research
☰
home / papers / 2017 / mixup: Beyond Empirical Risk Minimization

mixup: Beyond Empirical Risk Minimization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1710.09412

Summary

Deep networks trained under Empirical Risk Minimization (ERM) can memorize their training sets, even with random labels, and react unpredictably to inputs slightly outside the training distribution, such as adversarial examples. Standard data augmentation helps, but it depends on the dataset, needs expert-designed transformations, and never relates examples from different classes. The authors propose mixup, a data-agnostic form of Vicinal Risk Minimization. Each virtual training sample is a weighted blend of two random training inputs, and its target is the same blend of their one-hot labels. The mixing weight λ is drawn from Beta(α, α). This pushes the model toward linear behaviour between training points. It takes only a few lines of code and adds almost no compute. Experiments cover ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI tabular data. mixup consistently lowers error for state-of-the-art architectures. It also reduces memorization of corrupted labels, improves adversarial robustness and stabilizes GAN training. Its simplicity and generality make it a broadly applicable regularizer.

Key points

  • Method: λ ~ Beta(α, α), and ERM is recovered as α → 0. α = 0.2 was used for ResNet and α = 0.4 for ResNeXt on ImageNet-2012.
  • ImageNet-2012, 200 epochs: top-1 error fell from 23.6 to 22.1 for ResNet-50 and from 21.3 to 20.1 for ResNeXt-101 32*4d.
  • CIFAR test error: DenseNet-BC-190 dropped from 3.7 to 2.7 on CIFAR-10 and from 19.0 to 16.8 on CIFAR-100. PreAct ResNet-18 dropped from 25.6 to 21.1 on CIFAR-100.
  • Design choices:
    • Mixing three or more examples (Dirichlet weights) gave no extra gain.
    • Mixing only same-label pairs lost the benefit.
    • Shuffling a single minibatch worked as well as using two data loaders.
  • Code: CIFAR-10 replication code is available at github.com/facebookresearch/mixup-cifar10 (PyTorch).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

318 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 307 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1710.09412 · arXiv 1710.09412

@misc{zhang2017mixup,
  title = {{mixup: Beyond Empirical Risk Minimization}},
  author = {Zhang, Hongyi and Cisse, Moustapha and Dauphin, Yann N. and Lopez-Paz, David},
  year = {2017},
  doi = {10.48550/arxiv.1710.09412},
  eprint = {1710.09412},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1710.09412}
}

References

46 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, et al. Deep speech 2: End-to-end speech recognition in English and Mandarin. In ICML, 2016. find
  2. D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. A closer look at memorization in deep networks. ICML, 2017. find
  3. P. Bartlett, D. J. Foster, and M. Telgarsky. Spectrally-normalized margin bounds for neural networks. NIPS, 2017. find
  4. O. Chapelle, J. Weston, L. Bottou, and V. Vapnik. Vicinal risk minimization. NIPS, 2000. find
  5. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321–357, 2002. find our card
  6. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv, 2013. find our card
  7. M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. ICML, 2017. find
  8. W. M. Czarnecki, S. Osindero, M. Jaderberg, G. Świrszcz, and R. Pascanu. Sobolev training for neural networks. NIPS, 2017. find
  9. T. DeVries and G. W. Taylor. Dataset augmentation in feature space. ICLR Workshops, 2017. find
  10. H. Drucker and Y. Le Cun. Improving generalization performance using double backpropagation. IEEE Transactions on Neural Networks, 3(6):991–997, 1992. find
  11. I. Goodfellow. Tutorial: Generative adversarial networks. NIPS, 2016. find
  12. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. NIPS, 2014. find
  13. I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015. find our card
  14. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv, 2017. find
  15. A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In ICASSP. IEEE, 2013. find
  16. I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of Wasserstein GANs. NIPS, 2017. find
  17. N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension bounds for piecewise linear neural networks. JMLR, 2017. find
  18. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. ECCV, 2016. find
  19. M. Hein and M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. NIPS, 2017. find
  20. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. find
  21. G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. CVPR, 2017. find
  22. D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015. find
  23. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. NIPS, 2012. find
  24. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of IEEE, 2001. find
  25. M. Lichman. UCI machine learning repository, 2013. find
  26. K. Liu, 2017. URL https://github.com/kuangliu/pytorch-cifar. ↗
  27. G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton. Regularizing neural networks by penalizing confident output distributions. ICLR Workshops, 2017. find our card
  28. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet large scale visual recognition challenge. IJCV, 2015. find our card
  29. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016. find
  30. P. Simard, Y. LeCun, J. Denker, and B. Victorri. Transformation invariance in pattern recognitiontangent distance and tangent propagation. Neural networks: tricks of the trade, 1998. find
  31. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015. find our card
  32. J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. ICLR Workshops, 2015. find
  33. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. find
  34. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. ICLR, 2014. find our card
  35. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. find
  36. V. N. Vapnik. Statistical learning theory. J. Wiley, 1998. find
  37. V. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 1971. find
  38. A. Veit, 2017. URL https://github.com/andreasveit. ↗
  39. Warden, 2017. URL https://research.googleblog.com/2017/08/launching-speech-commands-dataset.html. ↗
  40. S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. CVPR, 2016. find
  41. S. Zagoruyko and N. Komodakis. Wide residual networks. BMVC, 2016a. find
  42. S. Zagoruyko and N. Komodakis, wide-residual-networks. 2016b. find
  43. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017. find
  44. C. Zhang, 2017. URL https://github.com/pluskid/fitting-random-labels. ↗
  45. Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang. Random erasing data augmentation. arXiv, 2017. find
  46. URL https://github.com/szagoruyko/ ↗