Elamur Research⌕
☰
home / papers / 2014 / Generative Adversarial Nets

Generative Adversarial Nets

Paper · 2014 · Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza et al.

Summary

Deep generative models have lagged behind discriminative ones. Likelihood-based training usually involves intractable quantities such as partition functions, approximate posterior inference or Markov chain sampling. The authors replace this with an adversarial setup. A generator network turns random noise into data samples. A discriminator network learns to separate real training examples from generated ones, and the generator is trained to fool it. Both are multilayer perceptrons trained together with backpropagation and dropout, and drawing a sample needs only a forward pass. Training is framed as a two-player minimax game. In the idealised infinite-capacity setting, the paper proves that the best discriminator for a fixed generator equals p_data/(p_data+p_g), and that the game's global optimum occurs exactly when the generator reproduces the data distribution. Experiments on MNIST, TFD and CIFAR-10 produce samples the authors describe as competitive with earlier generative models, assessed visually and with Parzen-window density estimates. The approach matters because it frees generative modelling from Markov chains and inference, opening a broad and flexible model family.

Key points

  • The value function is E[log D(x)] over real data plus E[log(1 − D(G(z)))] over noise. D tries to maximise it and G tries to minimise it.
  • Theory: the criterion's global minimum is −log 4, reached if and only if p_g = p_data. At that point D outputs 1/2 everywhere.
  • Algorithm 1 alternates k discriminator updates with one generator update using minibatch SGD. The experiments used k = 1 (the cheapest option) and momentum.
  • Datasets: MNIST, TFD and CIFAR-10. For CIFAR-10 the paper shows both a fully connected model and a convolutional discriminator with a "deconvolutional" generator. Code and hyperparameters are published at github.com/goodfeli/adversarial (built on Pylearn2/Theano).
  • Known weaknesses: p_g(x) has no explicit form, and G and D must stay in balance. Over-training G risks "the Helvetica scenario", where many noise inputs collapse onto the same output.

Interesting ideas

  • I1. Learning a distribution through a learned critic: a trainable classifier serves as the loss for a generator, avoiding any explicit density. This is useful whenever samples are easy to produce but likelihoods are hard to write down.
  • I2. Non-saturating generator objective: G maximises log D(G(z)) instead of minimising log(1 − D(G(z))). The fixed point stays the same, but early gradients become much stronger. Swapping a saturating loss for an equivalent but better-conditioned one is a trick that transfers to other objectives.
  • I3. Partial inner-loop optimisation: the inner player gets only a few steps per outer step rather than being solved to convergence. This keeps it near-optimal cheaply, much like persistent chains in SML/PCD, and suits any nested optimisation problem.
  • I4. Sanity check against memorisation: each generated sample is displayed next to its nearest training example. It is a simple, reusable way to show that a generative model is not just copying its data.
  • I5. Conditioning by extra input: feeding a condition c into both G and D yields a model of p(x | c). This hints at controllable generation, for example by class or attribute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🎓Polytechnique MontréalMontreal · Canada11 works · 35,142 citations🎓École PolytechniquePalaiseau · France15 works · 6,688 citations🎓Indian Institute of Technology DelhiNew Delhi · India24 works · 6,933 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio. 2014. Generative Adversarial Nets. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1406.2661

@misc{goodfellow2014generative,
  title = {{Generative Adversarial Nets}},
  author = {Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville and Yoshua Bengio},
  year = {2014},
  eprint = {1406.2661},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1406.2661}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.
  2. Bengio, Y. (2009). Learning deep architectures for AI. Now Publishers. find
  3. Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. (2013a). Better mixing via deep representations. In ICML’13. find
  4. Bengio, Y., Yao, L., Alain, G., and Vincent, P. (2013b). Generalized denoising auto-encoders as generative models. In NIPS26. Nips Foundation. find
  5. Bengio, Y., Thibodeau-Laufer, E., and Yosinski, J. (2014a). Deep generative stochastic networks trainable by backprop. In ICML’14. find
  6. Bengio, Y., Thibodeau-Laufer, E., Alain, G., and Yosinski, J. (2014b). Deep generative stochastic networks trainable by backprop. In Proceedings of the 30th International Conference on Machine Learning (ICML’14). find
  7. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation. find
  8. Breuleux, O., Bengio, Y., and Vincent, P. (2011). Quickly generating representative samples from an RBM-derived process. Neural Computation, 23(8), 2053–2073. find
  9. Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier neural networks. In AISTATS’2011. find
  10. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013a). Maxout networks. In ICML’2013. find
  11. Goodfellow, I. J., Mirza, M., Courville, A., and Bengio, Y. (2013b). Multi-prediction deep Boltzmann machines. In NIPS’2013. find
  12. Goodfellow, I. J., Warde-Farley, D., Lamblin, P., Dumoulin, V., Mirza, M., Pascanu, R., Bergstra, J., Bastien, F., and Bengio, Y. (2013c). Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214. — [arXiv:1308.4214](https://arxiv.org/abs/1308.4214) find
  13. Gutmann, M. and Hyvarinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS’2010. find
  14. Hinton, G., Deng, L., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. (2012a). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6), 82–97. find
  15. Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995). The wake-sleep algorithm for unsupervised neural networks. Science, 268, 1558–1161. find
  16. Hinton, G. E., Osindero, S., and Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554. find
  17. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012b). Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580. find
  18. Hyvärinen, A. (2005). Estimation of non-normalized statistical models using score matching. J. Machine Learning Res., 6. find
  19. Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2009). What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV’09), pages 2146–2153. IEEE. find
  20. Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR).
  21. Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto. find
  22. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). ImageNet classification with deep convolutional neural networks. In NIPS’2012.
  23. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. find
  24. Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. Technical report, arXiv:1401.4082.
  25. Rifai, S., Bengio, Y., Dauphin, Y., and Vincent, P. (2012). A generative process for sampling contractive auto-encoders. In ICML’12. find
  26. Salakhutdinov, R. and Hinton, G. E. (2009). Deep Boltzmann machines. In AISTATS’2009, pages 448– 455. find
  27. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing, volume 1, chapter 6, pages 194–281. MIT Press, Cambridge. find
  28. Susskind, J., Anderson, A., and Hinton, G. E. (2010). The Toronto face dataset. Technical Report UTML TR 2010-001, U. Toronto. find
  29. Tieleman, T. (2008). Training restricted Boltzmann machines using approximations to the likelihood gradient. In W. W. Cohen, A. McCallum, and S. T. Roweis, editors, ICML 2008, pages 1064–1071. ACM. find
  30. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In ICML 2008. find
  31. Younes, L. (1999). On the convergence of Markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics and Stochastic Reports, 65(3), 177–228. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/