Elamur Research
☰
home / papers / 2016 / Improved Techniques for Training GANs

Improved Techniques for Training GANs

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1606.03498

Summary

Training a GAN means looking for a Nash equilibrium between generator and discriminator. Ordinary gradient descent offers no guarantee of finding one, so training often oscillates or the generator collapses onto a single output. Salimans et al. (OpenAI) propose a set of heuristic fixes:

  • Feature matching: the generator matches the average intermediate discriminator features of real data.
  • Minibatch discrimination: the discriminator compares samples within a batch to detect collapse.
  • Historical averaging: parameters are penalized for drifting from their running mean.
  • One-sided label smoothing.
  • Virtual batch normalization: a fixed reference batch is used for normalization.

The authors also propose two ways to measure sample quality: a human "visual Turing test" on Amazon Mechanical Turk and an automatic metric, the Inception score. They evaluate on MNIST, CIFAR-10, SVHN and 128×128 ImageNet (ILSVRC2012). The approach gives state-of-the-art semi-supervised classification on the three smaller datasets. Humans cannot tell MNIST samples from real digits. The ImageNet samples show recognizable animal parts, though with broken anatomy. The work matters because these practical tricks made earlier untrainable GANs workable and gave the field a standard evaluation metric. Code is released at github.com/openai/improved_gan.

Key points

  • Semi-supervised CIFAR-10 (4000 labels): the model reaches 18.63% test error. Ladder network gets 20.40% and CatGAN 19.58%. An ensemble of 10 models lowers this to 15.59%.
  • SVHN (1000 labels): a single model reaches 8.11% error, and the ensemble reaches 5.88%.
  • MTurk test: annotators correctly labeled 78.7% of real-vs-generated CIFAR-10 images, a 21.3% human error rate. Accuracy fell to 71.4% when only samples in the top 1% by Inception score were shown.
  • Inception score on CIFAR-10: the full method scores 8.09 against 11.24 for real data. Removing minibatch features causes the largest drop in the ablations, down to 3.87.
  • Which technique to use: feature matching works best for semi-supervised classification. Minibatch discrimination produces the most visually convincing samples.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

189 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 180 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1606.03498 · arXiv 1606.03498

@misc{salimans2016improved,
  title = {{Improved Techniques for Training GANs}},
  author = {Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi},
  year = {2016},
  doi = {10.48550/arxiv.1606.03498},
  eprint = {1606.03498},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.03498}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. In NIPS, 2014. find
  2. Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. arXiv preprint arXiv:1506.05751, 2015. — [arXiv:1506.05751](https://arxiv.org/abs/1506.05751) find
  3. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. find our card
  4. Ian J Goodfellow. On distinguishability criteria for estimating generative models. arXiv:1412.6515, 2014. — [arXiv:1412.6515](https://arxiv.org/abs/1412.6515) find
  5. Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic. Generating images with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016. — [arXiv:1602.05110](https://arxiv.org/abs/1602.05110) find
  6. Donggeun Yoo, Namil Kim, Sunggyun Park, Anthony S Paek, and In So Kweon. Pixel-level domain transfer. arXiv preprint arXiv:1603.07442, 2016. — [arXiv:1603.07442](https://arxiv.org/abs/1603.07442) find
  7. Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic learning theory, pages 63–77. Springer, 2005. find
  8. Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of conditional dependence. In NIPS, volume 20, pages 489–496, 2007. find
  9. Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In Algorithmic learning theory, pages 13–31. Springer, 2007. find
  10. Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment matching networks. CoRR, abs/1502.02761, 2015. — [arXiv:1502.02761](https://arxiv.org/abs/1502.02761) find
  11. Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint arXiv:1505.03906, 2015. — [arXiv:1505.03906](https://arxiv.org/abs/1505.03906) find
  12. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. find
  13. Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, et al. Towards principled unsupervised learning. arXiv preprint arXiv:1511.06440, 2015. — [arXiv:1511.06440](https://arxiv.org/abs/1511.06440) find
  14. Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390, 2015. — [arXiv:1511.06390](https://arxiv.org/abs/1511.06390) find
  15. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. 2016. MIT Press. find
  16. George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374–376, 1951. find
  17. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. ArXiv e-prints, December 2015. find
  18. David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press. find
  19. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015. find
  20. Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016. — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) find
  21. Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Neural Information Processing Systems, 2014. arXiv preprint find
  22. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing by virtual adversarial examples. arXiv preprint arXiv:1507.00677, 2015. — [arXiv:1507.00677](https://arxiv.org/abs/1507.00677) find
  23. Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv:1602.05473, 2016. — [arXiv:1602.05473](https://arxiv.org/abs/1602.05473) find
  24. Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, 2015. find
  25. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, et al. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. find our card
  26. Martı́n Abadi, Ashish Agarwal, Paul Barham, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org. find