Elamur Research⌕
☰
home / papers / 2015 / Variational Inference with Normalizing Flows

Variational Inference with Normalizing Flows

Paper · 2015 · Danilo Jimenez Rezende and Shakir Mohamed

Summary

Variational inference usually relies on simple posterior families, such as mean-field or diagonal Gaussians, so the approximation can never match the true posterior. This leads to underestimated variance and biased parameter estimates. Rezende and Mohamed make the approximate posterior far more expressive by starting with a simple base density and passing samples through a chain of invertible maps. The change-of-variables rule keeps the density tractable. They plug these normalizing flows into amortized variational inference, which combines inference networks with reparameterized (stochastic backpropagation) gradients. The resulting ELBO gains only linear-cost log-determinant terms. They also introduce infinitesimal flows, which in the limit can represent the true posterior, and show that several earlier methods for richer posteriors are special cases of flows. On deep latent Gaussian models (DLGM) trained on binarized MNIST and CIFAR-10 patches, longer flows steadily tighten the bound and improve test likelihood. Their learned planar flow beats the volume-preserving NICE flow. The work matters because posterior expressiveness becomes a dial set by flow length rather than a fixed limit, and it laid the groundwork for later flow-based models.

Key points

  • Binarized MNIST setup: DLGM with 40 latent variables, trained for 500,000 parameter updates.
  • MNIST test results (−ln p(x) bound): the diagonal-covariance baseline gives ≤ 89.9. DLGM+NF improves with flow length, from ≤ 87.5 at k=10 to ≤ 85.1 at k=80. DLGM+NICE only reaches ≤ 87.2 at k=80.
  • Reference numbers from other papers: Hamiltonian variational inference (HVI, Salimans et al.) reaches 85.51 with 8 leapfrog steps, and DARN (Gregor et al.) reaches about 84.1–84.7. Model specifications differ, so these are not direct comparisons.
  • CIFAR-10 setup and results: random 3×8×8 patches, 30 latent variables and a logit-normal likelihood. The −ln p(x) values reported for K = 0, 2, 5, 10 are −293.7, −308.6, −317.9 and −320.7, a steady trend with flow length.
  • Planar flow vs NICE: on toy densities both may converge to similar quality as flows lengthen. The planar flow needs far fewer parameters, because all its parameters are learned, whereas NICE mixes dimensions using fixed random orthogonal or permutation matrices.

Interesting ideas

  • I1. Complexity as a tunable knob: the posterior's richness is set by the number of stacked invertible steps. This lets one trade compute for accuracy at run time, which is useful wherever a model needs a budget-adjustable level of expressiveness.
  • I2. Choosing transformations with cheap Jacobians: each step is designed so its log-determinant costs only linear time. Asking "is the Jacobian cheap?" is a reusable design test for any density model built from transformations.
  • I3. Learned mixing beats fixed mixing: NICE uses random permutations to mix dimensions, while the planar flow learns all its parameters and does better with fewer of them. Before adding fixed random components to an architecture, it is worth checking whether a small learned component would do the job.
  • I4. Evaluating both the bound and the gap: besides the likelihood bound, the authors track the KL divergence between the approximate and true posterior as flow length varies. Measuring the approximation gap directly, not just the end metric, helps diagnose whether a model is limited by inference or by the model itself.
  • I5. A unifying lens for related methods: recasting prior rich-posterior techniques as special kinds of flows makes their trade-offs comparable. Framing a family of methods under one formalism is a good way to organize the literature in any subfield.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations🏢Google (United Kingdom)London · United Kingdom11 works · 2,450 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksMathematics · 189 worksStatistics and Probability · 19 works

Cite

Cite as Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational Inference with Normalizing Flows. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1505.05770

@misc{rezende2015variational,
  title = {{Variational Inference with Normalizing Flows}},
  author = {Danilo Jimenez Rezende and Shakir Mohamed},
  year = {2015},
  eprint = {1505.05770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1505.05770}
}

References

37 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahn, S., Korattikara, A., and Welling, M. Bayesian posterior sampling via stochastic gradient Fisher scoring. In ICML, 2012. find
  2. Baird, L., Smalenberger, D., and Ingkiriwang, S. One-step neural network inversion with PDF learning and emulation. In IJCNN, volume 2, pp. 966–971. IEEE, 2005. find
  3. Bishop, C. M. Pattern recognition and machine learning. springer New York, 2006. find
  4. Challis, E. and Barber, D. Affine independent variational inference. In NIPS, 2012. find
  5. Dayan, P. Helmholtz machines and wake-sleep learning. Handbook of Brain Theory and Neural Network. MIT Press, Cambridge, MA, 44(0), 2000. find
  6. Dinh, L., Krueger, D., and Bengio, Y. NICE: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014. — [arXiv:1410.8516](https://arxiv.org/abs/1410.8516) find
  7. Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121–2159, 2010. find
  8. Genz, A. Methods for generating random orthogonal matrices. Monte Carlo and Quasi-Monte Carlo Methods, 1998. find
  9. Gershman, S., Hoffman, M., and Blei, D. Nonparametric variational inference. In ICML, 2012. find
  10. Gershman, S. J. and Goodman, N. D. Amortized inference in probabilistic reasoning. In Annual Conference of the Cognitive Science Society, 2014. find
  11. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. Maxout networks. ICML, 2013. find
  12. Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep autoregressive networks. In ICML, 2014. find
  13. Gregor, Karol, Danihelka, Ivo, Graves, Alex, Jimenez Rezende, Danilo, and Wierstra, Daan. Draw: A recurrent neural network for image generation. In ICML, 2015.
  14. Hoffman, M. D., Blei, D. M, Wang, C., and Paisley, J. Stochastic variational inference. JMLR, 14(1):1303– 1347, 2013. find
  15. Jaakkola, T. S. and Jordan, M. I. Improving the mean field approximation via the use of mixture distributions. In Learning in graphical models, pp. 163–173. 1998. find
  16. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999. find
  17. Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In ICLR, 2014.
  18. Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M. Semi-supervised learning with deep generative models. In NIPS, pp. 3581–3589, 2014. find
  19. Krizhevsky, A. and Hinton, G. Convolutional deep belief networks on CIFAR-10. Unpublished manuscript, 2010. find
  20. LeCun, Y. and Cortes, C. The MNIST database of handwritten digits, 1998. find
  21. Mnih, A. and Gregor, K. Neural variational inference and learning in belief networks. In ICML, 2014. find
  22. Neal, R. M. MCMC using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2011. find
  23. Papaspiliopoulos, O., Roberts, G. O., and Sköld, M. Non-centered parameterisations for hierarchical models and data augmentation. In Bayesian Statistics 7, 2003. find
  24. Ranganath, R., Gerrish, S., and Blei, D. M. Black box variational inference. In AISTATS, 2013. find
  25. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
  26. Rippel, O. and Adams, R. P. High-dimensional probability estimation with deep density models. arXiv:1302.5125, 2013. — [arXiv:1302.5125](https://arxiv.org/abs/1302.5125) find
  27. Salimans, T., Kingma, D. P., and Welling, M. Markov chain Monte Carlo and variational inference: Bridging the gap. In ICML, 2015. find
  28. Suykens, J. A. K., Verrelst, H., and Vandewalle, J. Online learning Fokker-Planck machine. Neural processing letters, 7(2):81–89, 1998. find
  29. Tabak, E. G. and Turner, C. V. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145–164, 2013. find
  30. Tabak, E. G and Vanden-Eijnden, E. Density estimation by dual ascent of the log-likelihood. Communications in Mathematical Sciences, 8(1):217–233, 2010. find
  31. Titsias, M. and Lazaro-Gredilla, M. Doubly stochastic variational Bayes for non-conjugate inference. In ICML, 2014. find
  32. Turner, R. E. and Sahani, M. Two problems with variational expectation maximisation for time-series models. In Barber, D., Cemgil, T., and Chiappa, S. (eds.), Bayesian Time series models, chapter 5, pp. 109–130. Cambridge University Press, 2011. find
  33. Uria, B., Murray, I., and Larochelle, H. A deep and tractable density estimator. In ICML, 2014. find
  34. Wang, B. and Titterington, D. M. Convergence and asymptotic normality of variational Bayesian approximations for exponential family models with missing values. In UAI, 2004. find
  35. Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient Langevin dynamics. In ICML, 2011. find
  36. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992. find
  37. Wingate, D. and Weber, T. Automated variational inference in probabilistic programming. In NIPS Workshop on Probabilistic Programming, 2013. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rezende-2015-variational-inference-normalizing-flows/