Elamur Research
☰
home / papers / 2013 / Auto-Encoding Variational Bayes

Auto-Encoding Variational Bayes

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1312.6114

Summary

The paper tackles learning in directed probabilistic models with continuous latent variables when the posterior over those latents cannot be computed exactly and the dataset is too large for batch methods or per-example sampling such as MCMC. The authors rewrite the variational lower bound so that a latent sample becomes a deterministic, differentiable function of the input and independent noise. This is the "reparameterization trick." It produces a low-variance, unbiased estimator of the bound, called SGVB, which can be maximized with ordinary stochastic gradient methods. For i.i.d. data, they build on this estimator with AEVB. In AEVB, a recognition model (a probabilistic encoder) is trained jointly with the generative decoder. When both are neural networks, the result is the variational auto-encoder. On MNIST and Frey Face, AEVB converged faster than wake-sleep and reached better lower bounds. In marginal-likelihood tests it also compared well against Monte Carlo EM. The work matters because it makes scalable variational learning practical for nearly any model with continuous latents.

Key points

  • Two estimators:
    • The generic estimator (SGVB A) uses only sampling.
    • The second variant (SGVB B) computes the KL term between the approximate posterior and the prior analytically, which typically reduces variance. That KL term acts as a regularizer, and the other term corresponds to reconstruction error.
  • Training setup:
    • Minibatches of M = 100 with L = 1 noise sample per datapoint were sufficient.
    • Optimization used Adagrad, with the global step size chosen from {0.01, 0.02, 0.1}.
    • Weights were initialized from N(0, 0.01).
  • Lower-bound experiments:
    • Encoders and decoders had 500 hidden units on MNIST and 200 on the smaller Frey Face dataset.
    • Latent dimensionality ranged from 2 to 200.
    • Adding superfluous latent variables did not cause overfitting.
  • Marginal-likelihood experiments:
    • These used MNIST with 3 latent variables and 100 hidden units.
    • AEVB was compared with wake-sleep and with Monte Carlo EM using a Hybrid Monte Carlo sampler, at 1,000 and 50,000 training points.
    • MCEM does not scale efficiently to the full dataset.
  • Compute: Training took roughly 20–40 minutes per million training samples on an Intel Xeon CPU running at about 40 GFLOPS. With a 2D latent space, the learned encoder can also be used to visualize data manifolds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

790 papers of our library cite it (of the 140673 whose bibliographies we read); 27 of them have a page here, 763 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1312.6114 · arXiv 1312.6114

@misc{kingma2013auto,
  title = {{Auto-Encoding Variational Bayes}},
  author = {Kingma, Diederik P and Welling, Max},
  year = {2013},
  doi = {10.48550/arxiv.1312.6114},
  eprint = {1312.6114},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1312.6114}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. 2013. find our card
  2. David M Blei, Michael I Jordan, and John W Paisley. Variational Bayesian inference with Stochastic Search. In Proceedings of the 29th International Conference on Machine Learning (ICML-12), pages 1367–1374, 2012. find
  3. Yoshua Bengio and Éric Thibodeau-Laufer. Deep generative stochastic networks trainable by backprop. arXiv preprint arXiv:1306.1091, 2013. — [arXiv:1306.1091](https://arxiv.org/abs/1306.1091) find
  4. Luc Devroye. Sample-based non-uniform random variate generation. In Proceedings of the 18th conference on Winter simulation, pages 260–265. ACM, 1986. find
  5. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121– 2159, 2010. find
  6. Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. Hybrid monte carlo. Physics letters B, 195(2):216–222, 1987. find
  7. Karol Gregor, Andriy Mnih, and Daan Wierstra. Deep autoregressive networks. arXiv preprint arXiv:1310.8499, 2013. — [arXiv:1310.8499](https://arxiv.org/abs/1310.8499) find
  8. Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303–1347, 2013. find
  9. Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The” wake-sleep” algorithm for unsupervised neural networks. SCIENCE, pages 1158–1158, 1995. find
  10. Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. Fast inference in sparse coding algorithms with applications to object recognition. Technical Report CBLL-TR-2008-12-01, Computational and Biological Learning Lab, Courant Institute, NYU, 2008. find
  11. Ralph Linsker. An application of the principle of maximum information preservation to linear systems. Morgan Kaufmann Publishers Inc., 1989. find
  12. Rajesh Ranganath, Sean Gerrish, and David M Blei. Black Box Variational Inference. arXiv preprint arXiv:1401.0118, 2013. — [arXiv:1401.0118](https://arxiv.org/abs/1401.0118) find
  13. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and variational inference in deep latent gaussian models. arXiv preprint arXiv:1401.4082, 2014. find our card
  14. Sam Roweis. EM algorithms for PCA and SPCA. Advances in neural information processing systems, pages 626–632, 1998. find
  15. Tim Salimans and David A Knowles. Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8(4), 2013. find
  16. Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In International Conference on Artificial Intelligence and Statistics, pages 693– 700, 2010. find
  17. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 9999:3371–3408, 2010. find