Elamur Research
☰
home / papers / 2015 / Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1511.06434

Summary

Convolutional networks dominate supervised computer vision, but their use for learning from unlabeled images has lagged. GANs were a promising route, yet they were notoriously unstable when built from deep CNNs. The authors searched extensively over architectures and arrived at DCGAN, a family of convolutional GANs with a few design rules that make training reliable:

  • Learned strided convolutions in place of pooling.
  • No fully connected hidden layers.
  • Batch normalization in both networks.
  • ReLU/Tanh activations in the generator and LeakyReLU in the discriminator.

They trained these models on three datasets:

  • LSUN bedrooms, deduplicated with an autoencoder-based hashing step.
  • Imagenet-1k.
  • A new web-scraped Faces collection.

The networks produce convincing 64×64 images and learn hierarchical features covering parts through whole scenes. The discriminator's features transfer to classification tasks with results competitive with other unsupervised methods. Individual filters can be linked to specific objects, and arithmetic in the latent space changes semantic attributes of the generated images. The paper matters because it offered a practical recipe for stable GAN training and showed that adversarial training yields reusable visual representations.

Key points

  • The generator projects a 100-dimensional uniform noise vector into a small multi-channel tensor. Four fractionally-strided convolutions then upsample it to a 64×64 image, with no pooling or fully connected layers.
  • Training setup:
    • Adam with learning rate 0.0002 and β1 = 0.5; the default 0.9 caused oscillation.
    • Mini-batch size 128.
    • Weights initialized from N(0, 0.02).
    • LeakyReLU slope 0.2.
    • Batchnorm is omitted at the generator output and the discriminator input.
  • LSUN bedrooms contains just over 3M images. A 3072-128-3072 denoising autoencoder with binarized codes removed about 275,000 near duplicates.
  • The Faces dataset has 3M images of 10K people whose names came from dbpedia. An OpenCV detector extracted about 350,000 face crops from it.
  • In a nearest-neighbor evaluation on MNIST, DCGAN reached 2.98% test error with 50K generated samples, versus 3.1% for real data and 6.28% for a baseline GAN. With 10M samples it reached 1.48%, beating InfiMNIST (2.6%) and nearly matching AlignMNIST (1.4%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

123 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.06434 · arXiv 1511.06434

@misc{radford2015unsupervised,
  title = {{Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks}},
  author = {Radford, Alec and Metz, Luke and Chintala, Soumith},
  year = {2015},
  doi = {10.48550/arxiv.1511.06434},
  eprint = {1511.06434},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06434}
}

References

38 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bergstra, James and Bengio, Yoshua. Random search for hyper-parameter optimization. JMLR, 2012. find
  2. Coates, Adam and Ng, Andrew. Selecting receptive fields in deep networks. NIPS, 2011. find
  3. Coates, Adam and Ng, Andrew Y. Learning feature representations with k-means. In Neural Networks: Tricks of the Trade, pp. 561–580. Springer, 2012. find
  4. Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. find
  5. CVPR 2009. IEEE Conference on, pp. 248–255. find
  6. Denton, Emily, Chintala, Soumith, Szlam, Arthur, and Fergus, Rob. Deep generative image models using a laplacian pyramid of adversarial networks. arXiv preprint arXiv:1506.05751, 2015. — [arXiv:1506.05751](https://arxiv.org/abs/1506.05751) find
  7. Dosovitskiy, Alexey, Springenberg, Jost Tobias, and Brox, Thomas. Learning to generate chairs with convolutional neural networks. arXiv preprint arXiv:1411.5928, 2014. — [arXiv:1411.5928](https://arxiv.org/abs/1411.5928) find
  8. Dosovitskiy, Alexey, Fischer, Philipp, Springenberg, Jost Tobias, Riedmiller, Martin, and Brox, Thomas. Discriminative unsupervised feature learning with exemplar convolutional neural networks. In Pattern Analysis and Machine Intelligence, IEEE Transactions on, volume 99. IEEE, 2015. find
  9. Efros, Alexei, Leung, Thomas K, et al. Texture synthesis by non-parametric sampling. In Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, volume 2, pp. 1033–1038. find
  10. Freeman, William T, Jones, Thouis R, and Pasztor, Egon C. Example-based super-resolution. Computer Graphics and Applications, IEEE, 22(2):56–65, 2002. find
  11. Goodfellow, Ian J, Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. arXiv preprint arXiv:1302.4389, 2013. — [arXiv:1302.4389](https://arxiv.org/abs/1302.4389) find
  12. Goodfellow, Ian J., Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron C., and Bengio, Yoshua. Generative adversarial nets. NIPS, 2014. find
  13. Gregor, Karol, Danihelka, Ivo, Graves, Alex, and Wierstra, Daan. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015. find
  14. Hardt, Moritz, Recht, Benjamin, and Singer, Yoram. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.01240, 2015. — [arXiv:1509.01240](https://arxiv.org/abs/1509.01240) find
  15. Hauberg, Sren, Freifeld, Oren, Larsen, Anders Boesen Lindbo, Fisher III, John W., and Hansen, Lars Kair. Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation. arXiv preprint arXiv:1510.02795, 2015. — [arXiv:1510.02795](https://arxiv.org/abs/1510.02795) find
  16. Hays, James and Efros, Alexei A. Scene completion using millions of photographs. ACM Transactions on Graphics (TOG), 26(3):4, 2007. find
  17. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. find
  18. Kingma, Diederik P and Ba, Jimmy Lei. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  19. Kingma, Diederik P and Welling, Max. arXiv:1312.6114, 2013. find our card
  20. Lee, Honglak, Grosse, Roger, Ranganath, Rajesh, and Ng, Andrew Y. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 609–616. ACM, 2009. find
  21. Loosli, Gaëlle, Canu, Stéphane, and Bottou, Léon. Training invariant support vector machines using selective sampling. In Bottou, Léon, Chapelle, Olivier, DeCoste, Dennis, and Weston, Jason (eds.), Large Scale Kernel Machines, pp. 301–320. MIT Press, Cambridge, MA., 2007. URL http://leon.bottou.… ↗
  22. Maas, Andrew L, Hannun, Awni Y, and Ng, Andrew Y. Rectifier nonlinearities improve neural network acoustic models. In Proc. ICML, volume 30, 2013. find
  23. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013. find
  24. Mordvintsev, Alexander, Olah, Christopher, and Tyka, Mike. Inceptionism : Going deeper into neural networks. http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html. Accessed: 2015-06-17. ↗
  25. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807– 814, 2010. find
  26. Auto-encoding variational bayes. arXiv preprint Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 5. find
  27. Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and transferring mid-level image representations using convolutional neural networks. In CVPR, 2014. find
  28. Portilla, Javier and Simoncelli, Eero P. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision, 40(1):49–70, 2000. find
  29. Rasmus, Antti, Valpola, Harri, Honkala, Mikko, Berglund, Mathias, and Raiko, Tapani. Semi-supervised learning with ladder network. arXiv preprint arXiv:1507.02672, 2015. — [arXiv:1507.02672](https://arxiv.org/abs/1507.02672) find
  30. Sohl-Dickstein, Jascha, Weiss, Eric A, Maheswaranathan, Niru, and Ganguli, Surya. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.03585, 2015. find our card
  31. Springenberg, Jost Tobias, Dosovitskiy, Alexey, Brox, Thomas, and Riedmiller, Martin. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014. — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) find
  32. Srivastava, Rupesh Kumar, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, Jürgen. Understanding locally competitive networks. arXiv preprint arXiv:1410.1165, 2014. — [arXiv:1410.1165](https://arxiv.org/abs/1410.1165) find
  33. Theis, L., van den Oord, A., and Bethge, M. A note on the evaluation of generative models. arXiv:1511.01844, Nov 2015. URL http://arxiv.org/abs/1511.01844. — [arXiv:1511.01844](https://arxiv.org/abs/1511.01844) ↗
  34. Vincent, Pascal, Larochelle, Hugo, Lajoie, Isabelle, Bengio, Yoshua, and Manzagol, Pierre-Antoine. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 11:3371–3408, 2010. find
  35. Xu, Bing, Wang, Naiyan, Chen, Tianqi, and Li, Mu. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015. — [arXiv:1505.00853](https://arxiv.org/abs/1505.00853) find
  36. Yu, Fisher, Zhang, Yinda, Song, Shuran, Seff, Ari, and Xiao, Jianxiong. Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015. — [arXiv:1506.03365](https://arxiv.org/abs/1506.03365) find
  37. Zeiler, Matthew D and Fergus, Rob. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014, pp. 818–833. find our card
  38. Zhao, Junbo, Mathieu, Michael, Goroshin, Ross, and Lecun, Yann. Stacked what-where autoencoders. arXiv preprint arXiv:1506.02351, 2015. — [arXiv:1506.02351](https://arxiv.org/abs/1506.02351) find