Elamur Research⌕
☰
home / papers / 2015 / DRAW: A Recurrent Neural Network For Image Generation

DRAW: A Recurrent Neural Network For Image Generation

Paper · 2015 · Karol Gregor, Ivo Danihelka, Alex Graves et al.

Summary

Most neural generative models of images produce the whole picture in one step from a single latent code. This makes self-correction impossible and scales poorly to large images. DRAW (Deep Recurrent Attentive Writer), from Google DeepMind, instead builds an image gradually, much as a person sketches and then refines. Its encoder and decoder are both LSTM networks inside a variational auto-encoder framework. At every time-step the decoder adds a modification to a cumulative canvas, and the encoder sees what has been drawn so far. A fully differentiable 2D attention mechanism, inspired by eye foveation, decides which region to read and which to write. Because this attention is differentiable, the system trains end-to-end with ordinary backpropagation on a variational bound of the log-likelihood, with no reinforcement learning needed. The authors evaluate DRAW on binarized MNIST, two-digit MNIST, SVHN and CIFAR-10. It beats the previous state of the art on MNIST likelihood, and its SVHN samples are hard to tell apart from real photos. The attention module also improves image classification.

Key points

  • Architecture: a recurrent VAE with LSTM encoder and decoder, diagonal Gaussian latents trained via the reparameterization trick, and a canvas updated additively over T steps. Without attention, the model blurs and then sharpens the whole image; with attention, it traces strokes, much like drawing with a pen.
  • Classification on 100×100 Cluttered Translated MNIST: a differentiable RAM with 8 glimpses of 12×12 reaches 3.36% test error. This compares with 8.11% for RAM (8 glimpses) and 14.35% for a 2-layer convolutional network.
  • Generation on binarized MNIST: DRAW without attention is comparable to DARN, NADE and DBMs. DRAW with attention clearly improves on the state of the art.
  • Two-digit MNIST: given 60×60 canvases holding two digits, the model usually draws one digit and then the other, showing it can compose scenes from parts.
  • SVHN and CIFAR-10:
    • SVHN training used 231,053 images, cropped as 54×54 patches from 64×64 preprocessed images, with 32 glimpses, 800 LSTM units and 100 latent dimensions. The model still underfit the data.
    • CIFAR-10 samples capture shape and colour but are less realistic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende and Daan Wierstra. 2015. DRAW: A Recurrent Neural Network For Image Generation. arXiv:1502.04623.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1502.04623 · arXiv 1502.04623

@misc{gregor2015draw,
  title = {{DRAW: A Recurrent Neural Network For Image Generation}},
  author = {Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende and Daan Wierstra},
  year = {2015},
  doi = {10.48550/arxiv.1502.04623},
  eprint = {1502.04623},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1502.04623}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ba, Jimmy, Mnih, Volodymyr, and Kavukcuoglu, Koray. Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755, 2014. find
  2. Dayan, Peter, Hinton, Geoffrey E, Neal, Radford M, and Zemel, Richard S. The helmholtz machine. Neural computation, 7(5):889–904, 1995. find
  3. Denil, Misha, Bazzani, Loris, Larochelle, Hugo, and de Freitas, Nando. Learning where to attend with deep architectures for image tracking. Neural computation, 24(8):2151–2184, 2012. find
  4. Gers, Felix A, Schmidhuber, Jürgen, and Cummins, Fred. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471, 2000. find
  5. Goodfellow, Ian J, Bulatov, Yaroslav, Ibarz, Julian, Arnoud, Sacha, and Shet, Vinay. Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv preprint arXiv:1312.6082, 2013. — [arXiv:1312.6082](https://arxiv.org/abs/1312.6082) find
  6. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  7. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. find
  8. Gregor, Karol, Danihelka, Ivo, Mnih, Andriy, Blundell, Charles, and Wierstra, Daan. Deep autoregressive networks. In Proceedings of the 31st International Conference on Machine Learning, 2014. find
  9. Hinton, Geoffrey E and Salakhutdinov, Ruslan R. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006. find
  10. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  11. Adam: A arXiv preprint Kingma, Diederik and Ba, Jimmy. method for stochastic optimization. arXiv:1412.6980, 2014.
  12. Kingma, Diederik P and Welling, Max. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
  13. Krizhevsky, Alex. Learning multiple layers of features from tiny images. 2009. find
  14. Larochelle, Hugo and Hinton, Geoffrey E. Learning to combine foveal glimpses with a third-order boltzmann machine. In Advances in Neural Information Processing Systems, pp. 1243–1251. 2010. find
  15. Larochelle, Hugo and Murray, Iain. The neural autoregressive distribution estimator. Journal of Machine Learning Research, 15:29–37, 2011. find
  16. LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278– 2324, 1998. find
  17. Mnih, Andriy and Gregor, Karol. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on Machine Learning, 2014. find
  18. Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pp. 2204–2212, 2014. find
  19. Murray, Iain and Salakhutdinov, Ruslan. Evaluating probabilities under high-dimensional latent variable models. In Advances in neural information processing systems, pp. 1137–1144, 2009. find
  20. Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. 2011. find
  21. Raiko, Tapani, Li, Yao, Cho, Kyunghyun, and Bengio, Yoshua. Iterative neural autoregressive distribution estimator nade-k. In Advances in Neural Information Processing Systems, pp. 325–333. 2014. find
  22. Ranzato, Marc’Aurelio. On learning where to look. arXiv preprint arXiv:1405.5488, 2014. — [arXiv:1405.5488](https://arxiv.org/abs/1405.5488) find
  23. Rezende, Danilo J, Mohamed, Shakir, and Wierstra, Daan. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, pp. 1278– 1286, 2014.
  24. Salakhutdinov, Ruslan and Hinton, Geoffrey E. Deep boltzmann machines. In International Conference on Artificial Intelligence and Statistics, pp. 448–455, 2009. find
  25. Salakhutdinov, Ruslan and Murray, Iain. On the quantitative analysis of Deep Belief Networks. In Proceedings of the 25th Annual International Conference on Machine Learning, pp. 872–879. Omnipress, 2008. find
  26. Salimans, Tim, Kingma, Diederik P, and Welling, Max. Markov chain monte carlo and variational inference: Bridging the gap. arXiv preprint arXiv:1410.6460, 2014. find
  27. Sermanet, Pierre, Frome, Andrea, and Real, Esteban. Attention for fine-grained categorization. arXiv preprint arXiv:1412.7054, 2014. — [arXiv:1412.7054](https://arxiv.org/abs/1412.7054) find
  28. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014.
  29. Tang, Yichuan, Srivastava, Nitish, and Salakhutdinov, Ruslan. Learning generative models with visual attention. arXiv preprint arXiv:1312.6110, 2013. find
  30. Tieleman, Tijmen. Optimizing Neural Networks that Generate Images. PhD thesis, University of Toronto, 2014. find
  31. Uria, Benigno, Murray, Iain, and Larochelle, Hugo. A deep and tractable density estimator. In Proceedings of the 31st International Conference on Machine Learning, pp. 467–475, 2014. find
  32. Zheng, Yin, Zemel, Richard S, Zhang, Yu-Jin, and Larochelle, Hugo. A neural autoregressive approach to attention-based recognition. International Journal of Computer Vision, pp. 1–13, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gregor-2015-draw-recurrent-neural-network-image/