Elamur Research
☰
home / papers / 2014 / Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1401.4082

Summary

The paper addresses a long-standing obstacle for deep directed generative models: sampling from them is easy, but inferring their latent variables efficiently is hard. The authors define deep latent Gaussian models (DLGMs). In these models, each layer's activation is an MLP transformation of the layer above plus Gaussian noise, and observations are drawn from a likelihood conditioned on the lowest layer. Factor analysis and non-linear Gaussian belief networks are special cases. To make inference tractable, the generator is coupled with a recognition network that functions as a randomised encoder, producing an approximation to the posterior. Both networks are trained jointly by minimising a variational free energy, which combines a KL regulariser with a reconstruction term. The enabling tool is stochastic backpropagation, a set of gradient rules that pass through random variables using Gaussian identities and coordinate transformations. Experiments on real-world data sets, including MNIST, show realistic samples, accurate imputation of missing values and useful visualisation of high-dimensional data. Developed concurrently with Kingma & Welling (2014), the work became a cornerstone of modern amortised variational inference.

Key points

  • Gaussian backpropagation (GBP) relies on the theorems of Bonnet (1964) and Price (1958) to express gradients with respect to the mean and covariance as expectations of the gradient and Hessian of the loss.
  • The exact Hessian-based rule costs O(K³). A reparameterisation y = µ + Rε (with C = RRᵀ) yields an unbiased estimator of quadratic complexity, at the cost of somewhat higher variance.
  • Compared with REINFORCE and other control-variate estimators, whose variance grows as O(K) with latent dimensionality, GBP variance stays bounded at O(1).
  • Unlike the wake-sleep algorithm, the method optimises a single consistent bound on the marginal likelihood. The authors also show that denoising auto-encoders can be read as a form of variational inference.
  • The approach targets models with hundreds to thousands of parameters, beyond the reach of INLA (fewer than 10 hyperparameters). MNIST imputation is demonstrated over 15 iterative steps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

181 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 168 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1401.4082 · arXiv 1401.4082

@misc{rezende2014stochastic,
  title = {{Stochastic Backpropagation and Approximate Inference in Deep Generative Models}},
  author = {Rezende, Danilo Jimenez and Mohamed, Shakir and Wierstra, Daan},
  year = {2014},
  doi = {10.48550/arxiv.1401.4082},
  eprint = {1401.4082},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1401.4082}
}

References

32 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bartholomew, D. J. and Knott, M. Latent variable models and factor analysis, volume 7 of Kendall’s library of statistics. Arnold, 2nd edition, 1999. find
  2. Beal, M. J. Variational Algorithms for approximate Bayesian inference. PhD thesis, University of Cambridge, 2003. find
  3. Bengio, Y. and Thibodeau-Laufer, É. Deep generative stochastic networks trainable by backprop. Technical report, University of Montreal, 2013. find
  4. Bengio, Y., Yao, L., Alain, G., and Vincent, P. Generalized denoising auto-encoders as generative models. In Advances in Neural Information Processing Systems (NIPS), pp. 1–9, 2013. find
  5. Blei, D. M., Jordan, M. I., and Paisley, J. W. Variational Bayesian inference with stochastic search. In Proceedings of the 29th International Conference on Machine Learning (ICML), pp. 1367–1374, 2012. find
  6. Bonnet, G. Transformations des signaux aléatoires a travers les systèmes non linéaires sans mémoire. Annales des Télécommunications, 19(9-10):203–220, 1964. find
  7. Damianou, A. C. and Lawrence, N. D. Deep Gaussian processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2013. find
  8. Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. The Helmholtz machine. Neural computation, 7(5):889– 904, September 1995. find
  9. Dayan, P. Helmholtz machines and wake-sleep learning. Handbook of Brain Theory and Neural Network. MIT Press, Cambridge, MA, 44(0), 2000. find
  10. Frey, B. J. Variational inference for continuous sigmoidal Bayesian networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AIS-TATS), 1996. find
  11. Frey, B. J. and Hinton, G. E. Variational learning in non-linear Gaussian belief networks. Neural Computation, 11 (1):193–213, January 1999. find
  12. Gershman, S. J. and Goodman, N. D. Amortized inference in probabilistic reasoning. In Proceedings of the 36th Annual Conference of the Cognitive Science Society, 2014. find
  13. Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems 24 (NIPS), pp. 2348–2356, 2011. find
  14. Gregor, K., Mnih, A., and Wierstra, D. Deep autoregressive networks. In Proceedings of the International Conference on Machine Learning (ICML), October 2014. find
  15. Hoffman, M., Blei, D. M., Wang, C., and Paisley, J. Stochastic variational inference. Journal of Machine Learning Research, 14:1303–1347, May 2013. find
  16. Honkela, A. and Valpola, H. Unsupervised variational Bayesian learning of nonlinear models. In Advances in Neural Information Processing Systems (NIPS), 2004. find
  17. Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. Proceedings of the International Conference on Learning Representations (ICLR), 2014. find our card
  18. Lappalainen, H. and Honkela, A. Bayesian non-linear independent component analysis by multi-layer perceptrons. In Advances in independent component analysis (ICA), pp. 93–121. Springer, 2000. find
  19. Larochelle, H. and Murray, I. The neural autoregressive distribution estimator. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. find
  20. Lawrence, N. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. The Journal of Machine Learning Research, 6:1783– 1816, 2005. find
  21. Little, R. J. and Rubin, D. B. Statistical analysis with missing data, volume 539. Wiley New York, 1987. find
  22. Magdon-Ismail, M. and Purnell, J. T. Approximating the covariance matrix of GMMs with low-rank perturbations. In Proceedings of the 11th international conference on Intelligent data engineering and automated learning (IDEAL), pp. 300–307, 2010. find
  23. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. find
  24. Opper, M. and Archambeau, C. The variational Gaussian approximation revisited. Neural computation, 21 (3):786–92, March 2009. find
  25. Price, R. A useful theorem for nonlinear devices having Gaussian inputs. IEEE Transactions on Information Theory, 4(2):69–72, 1958. find
  26. Ranganath, R., Gerrish, S., and Blei, D. M. Black box variational inference. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AIS-TATS), October 2014. find
  27. Salimans, T. and Knowles, D. A. On using control variates with stochastic approximation for variational bayes and its connection to stochastic linear regression. ArXiv preprint. arXiv:1401.1022, October 2014. — [arXiv:1401.1022](https://arxiv.org/abs/1401.1022) find
  28. Saul, L. K., Jaakkola, T., and Jordan, M. I. Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research (JAIR), 4:61–76, 1996. find
  29. Uria, B., Murray, I., and Larochelle, H. A deep and tractable density estimator. In Proceedings of the International Conference on Machine Learning (ICML), 2014. find
  30. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Manzagol, P. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 11:3371–3408, 2010. find
  31. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229 – 256, 1992. find
  32. Wilson, J. R. Variance reduction techniques for digital simulation. American Journal of Mathematical and Management Sciences, 4(3):277–312, 1984. find