Elamur Research
☰
home / papers / 2015 / Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1503.03585

Summary

Probabilistic models usually force a trade-off. Simple distributions are easy to fit and evaluate but miss rich structure. Expressive ones need intractable normalization constants and expensive Monte Carlo sampling. The authors draw on non-equilibrium statistical physics and annealed importance sampling to get both properties. A fixed forward Markov chain adds small amounts of Gaussian (or binomial) noise over many steps until the data becomes a simple, tractable distribution. A generative model is then trained to undo this corruption one step at a time. When steps are small, each reversal has the same functional form as the forward kernel, so learning only requires estimating per-step means and covariances (or bit-flip rates), optimized through a lower bound on log likelihood. The resulting model, with thousands of time steps, supports exact sampling, cheap probability evaluation and straightforward posterior computation, for example in denoising and inpainting. Experiments cover a 2-D swiss roll, binary sequences, MNIST, CIFAR-10, bark textures and dead leaves images. The work laid the foundation for later diffusion-based generative models.

Key points

  • Framework. Generation runs a learned reverse diffusion from an identity-covariance Gaussian (or independent binomial) back to the data. The model is defined as the endpoint of this chain, so its probabilities can be evaluated analytically.
  • Dead leaves. The model reaches state-of-the-art performance with a log-likelihood bound of 1.489 bits/pixel, compared with 1.244 bits/pixel for MCGSM (Theis et al., 2012).
  • MNIST. Parzen-window estimates using the code from Goodfellow et al. (2014) give 317 ± 2.7 bits. This is comparable to the Deep GSN (309), adversarial net (325) and DBN (199); a "perfect model" scores 349.
  • Binary heartbeat (length-20 sequences). Learning is nearly perfect: the bound is −2.414 bits/seq versus the true −2.322 bits/seq. On CIFAR-10, the bound improves on the Gaussian baseline by 11.5 ± 0.2 bits/pixel.
  • Implementation. An open-source reference implementation built on Blocks and trained with RMSprop is available at github.com/Sohl-Dickstein/Diffusion-Probabilistic-Models. The paper was presented at ICML 2015.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

256 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 241 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1503.03585 · arXiv 1503.03585

@misc{sohldickstein2015deep,
  title = {{Deep Unsupervised Learning using Nonequilibrium Thermodynamics}},
  author = {Sohl-Dickstein, Jascha and Weiss, Eric A. and Maheswaranathan, Niru and Ganguli, Surya},
  year = {2015},
  doi = {10.48550/arxiv.1503.03585},
  eprint = {1503.03585},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1503.03585}
}

References

57 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Barron, J. T., Biggin, M. D., Arbelaez, P., Knowles, D. W., Keranen, S. V., and Malik, J. Volumetric Semantic Segmentation Using Pyramid Context Features. In 2013 IEEE International Conference on Computer Vision, pp. 3448–3455. find
  2. IEEE, December 2013. ISBN 978-1-4799-2840-8. doi: 10.1109/ICCV.2013.428. — [doi:10.1109/iccv.2013.428](https://doi.org/10.1109/iccv.2013.428) find
  3. Bengio, Y. and Thibodeau-Laufer, E. Deep generative stochastic networks trainable by backprop. arXiv preprint arXiv:1306.1091, 2013. — [arXiv:1306.1091](https://arxiv.org/abs/1306.1091) find
  4. Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. Better Mixing via Deep Representations. arXiv preprint arXiv:1207.4404, July 2012. — [arXiv:1207.4404](https://arxiv.org/abs/1207.4404) find
  5. Bergstra, J. and Breuleux, O. Theano: a CPU and GPU math expression compiler. Proceedings of the Python for Scientific Computing Conference (SciPy), 2010. find
  6. Besag, J. Statistical Analysis of Non-Lattice Data. The Statistician, 24(3), 179-195, 1975. find
  7. Bishop, C., Svensén, M., and Williams, C. GTM: The generative topographic mapping. Neural computation, 1998. find
  8. Bornschein, J. and Bengio, Y. Reweighted Wake-Sleep. International Conference on Learning Representations, June 2015. find
  9. Burda, Y., Grosse, R. B., and Salakhutdinov, R. Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealing. arXiv:1412.8566, December 2014. — [arXiv:1412.8566](https://arxiv.org/abs/1412.8566) find
  10. Dayan, P., Hinton, G. E., Neal, R. M., and Zemel, R. S. The helmholtz machine. Neural computation, 7(5):889–904, 1995. find
  11. Dinh, L., Krueger, D., and Bengio, Y. NICE: Non-linear Independent Components Estimation. arXiv:1410.8516, pp. 11, October 2014. — [arXiv:1410.8516](https://arxiv.org/abs/1410.8516) find
  12. Feller, W. On the theory of stochastic processes, with particular reference to applications. In Proceedings of the [First] Berkeley Symposium on Mathematical Statistics and Probability. The Regents of the University of California, 1949. find
  13. Gershman, S. J. and Blei, D. M. A tutorial on Bayesian nonparametric models. Journal of Mathematical Psychology, 56(1):1–12, 2012. find
  14. Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. find
  15. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative Adversarial Nets. Advances in Neural Information Processing Systems, 2014. find
  16. Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep AutoRegressive Networks. arXiv preprint arXiv:1310.8499, October 2013. — [arXiv:1310.8499](https://arxiv.org/abs/1310.8499) find
  17. Grosse, R. B., Maddison, C. J., and Salakhutdinov, R. Annealing between distributions by averaging moments. In Advances in Neural Information Processing Systems, pp. 2769–2777, 2013. find
  18. Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8): 1771–1800, 2002. find
  19. Hinton, G. E. The wake-sleep algorithm for unsupervised neural networks ). Science, 1995. find
  20. Hyvärinen, A. Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, 6:695–709, 2005. find
  21. Jarzynski, C. Equilibrium free-energy differences from nonequilibrium measurements: A master-equation approach. Physical Review E, January 1997. find
  22. Jarzynski, C. Equalities and inequalities: irreversibility and the second law of thermodynamics at the nanoscale. Annu. Rev. Condens. Matter Phys., 2011. find
  23. Jeulin, D. Dead leaves models: from space tesselation to random functions. Proc. of the Symposium on the Advances in the Theory and Applications of Random Sets, 1997. find
  24. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine learning, 37(2):183–233, 1999. find
  25. Kavukcuoglu, K., Ranzato, M., and LeCun, Y. Fast inference in sparse coding algorithms with applications to object recognition. arXiv preprint arXiv:1010.3467, 2010. — [arXiv:1010.3467](https://arxiv.org/abs/1010.3467) find
  26. Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. International Conference on Learning Representations, December 2013. find our card
  27. Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Computer Science Department University of Toronto Tech. Rep., 2009. find
  28. Langevin, P. Sur la théorie du mouvement brownien. CR Acad. Sci. Paris, 146(530-533), 1908. find
  29. Larochelle, H. and Murray, I. The neural autoregressive distribution estimator. Journal of Machine Learning Research, 2011. find
  30. Lazebnik, S., Schmid, C., and Ponce, J. A sparse texture representation using local affine regions. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27 (8):1265–1278, 2005. find
  31. LeCun, Y. and Cortes, C. The MNIST database of handwritten digits. 1998. find
  32. Lee, A., Mumford, D., and Huang, J. Occlusion models for natural images: A statistical study of a scale-invariant dead leaves model. International Journal of Computer Vision, 2001. find
  33. Lyu, S. Unifying Non-Maximum Likelihood Learning Objectives with Minimum KL Contraction. Advances in Neural Information Processing Systems 24, pp. 64–72, 2011. find
  34. MacKay, D. Bayesian neural networks and density networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 1995. find
  35. Murphy, K. P., Weiss, Y., and Jordan, M. I. Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, pp. 467–475. Morgan Kaufmann Publishers Inc., 1999. find
  36. Neal, R. Annealed importance sampling. Statistics and Computing, January 2001. find
  37. Ozair, S. and Bengio, Y. Deep Directed Generative Autoencoders. arXiv:1410.0630, October 2014. — [arXiv:1410.0630](https://arxiv.org/abs/1410.0630) find
  38. Parry, M., Dawid, A. P., Lauritzen, S., and Others. Proper local scoring rules. The Annals of Statistics, 40(1):561– 592, 2012. find
  39. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the 31st International Conference on Machine Learning (ICML-14), January 2014. find our card
  40. Rippel, O. and Adams, R. P. High-Dimensional Probability Estimation with Deep Density Models. arXiv:1410.8516, pp. 12, February 2013. — [arXiv:1410.8516](https://arxiv.org/abs/1410.8516) find
  41. Schmidhuber, J. Learning factorial codes by predictability minimization. Neural Computation, 1992. find
  42. Sminchisescu, C., Kanaujia, A., and Metaxas, D. Learning joint top-down and bottom-up processes for 3D visual inference. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 2, pp. 1743–1752. find
  43. Sohl-Dickstein, J., Battaglino, P., and DeWeese, M. New Method for Parameter Estimation in Probabilistic Models: Minimum Probability Flow. Physical Review Letters, 107(22):11–14, November 2011a. ISSN 0031-9007. doi: 10.1103/PhysRevLett.107.220601. — [doi:10.1103/physrevlett.107.220601](https://doi.org/10.1103/physrevlett.107.220601) find
  44. Sohl-Dickstein, J., Battaglino, P. B., and DeWeese, M. R. Minimum Probability Flow Learning. International Conference on Machine Learning, 107(22):11– 14, November 2011b. ISSN 0031-9007. doi: 10.1103/PhysRevLett.107.220601. — [doi:10.1103/physrevlett.107.220601](https://doi.org/10.1103/physrevlett.107.220601) find
  45. Sohl-Dickstein, J., Poole, B., and Ganguli, S. Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 604–612, 2014. find
  46. Spinney, R. and Ford, I. Fluctuation Relations : A Pedagogical Overview. arXiv preprint arXiv:1201.6381, pp. 3–56, 2013. — [arXiv:1201.6381](https://arxiv.org/abs/1201.6381) find
  47. Stuhlmüller, A., Taylor, J., and Goodman, N. Learning stochastic inverses. Advances in Neural Information Processing Systems, 2013. find
  48. Suykens, J. and Vandewalle, J. Nonconvex optimization using a Fokker-Planck learning machine. In 12th European Conference on Circuit Theory and Design, 1995. find
  49. T, P. Convergence condition of the TAP equation for the infinite-ranged Ising spin glass model. J. Phys. A: Math. Gen. 15 1971, 1982. find
  50. Tanaka, T. Mean-field theory of Boltzmann machine learning. Physical Review Letters E, January 1998. find
  51. Theis, L., Hosseini, R., and Bethge, M. Mixtures of conditional Gaussian scale mixtures applied to multiscale image representations. PloS one, 7(7):e39857, 2012. find
  52. Theis, L., van den Oord, A., and Bethge, M. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844, 2015. — [arXiv:1511.01844](https://arxiv.org/abs/1511.01844) find
  53. Uria, B., Murray, I., and Larochelle, H. RNADE: The real-valued neural autoregressive density-estimator. Advances in Neural Information Processing Systems, 2013a. find
  54. Uria, B., Murray, I., and Larochelle, H. A Deep and Tractable Density Estimator. arXiv:1310.1757, pp. 9, October 2013b. van Merriënboer, B., Chorowski, J., Serdyuk, D., Bengio, Y., Bogdanov, D., Dumoulin, V., and Warde-Farley, D. Blocks and Fuel. — [arXiv:1310.1757](https://arxiv.org/abs/1310.1757) find
  55. Zenodo, May 2015. doi: 10.5281/zenodo.17721. — [doi:10.5281/zenodo.17721](https://doi.org/10.5281/zenodo.17721) find
  56. Welling, M. and Hinton, G. A new learning algorithm for mean field Boltzmann machines. Lecture Notes in Computer Science, January 2002. find
  57. Yao, L., Ozair, S., Cho, K., and Bengio, Y. On the Equivalence Between Deep NADE and Generative Stochastic Networks. In Machine Learning and Knowledge Discovery in Databases, pp. 322–336. Springer, 2014. find