Elamur Research
☰
home / papers / 2012 / Representation Learning: A Review and New Perspectives

Representation Learning: A Review and New Perspectives

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1206.5538

Summary

This survey asks why the encoding of input data matters so much for machine learning, and how such encodings can be learned automatically instead of being hand-built through feature engineering. The authors argue that a useful representation separates, or disentangles, the hidden explanatory factors that generate observed data. They frame the field around generic priors that suit broad AI tasks: smoothness, multiple underlying factors, hierarchical organization of concepts, and the semi-supervised idea that structure useful for modeling P(X) also helps predict P(Y|X). The paper introduces no new dataset or model. Instead, it reviews unsupervised feature learning and deep learning, covering probabilistic models such as Boltzmann Machines, auto-encoders, manifold learning and deep networks. It gathers empirical evidence from speech, vision, NLP and transfer learning showing large gains over earlier methods. It closes with open questions about training objectives, about inference for computing representations, and about how density estimation, manifold learning and representation learning relate geometrically. The result is a conceptual map of a field that had recently gained its own venue, ICLR.

Key points

  • Speech: Microsoft's deep-learning-based MAVIS system (Seide et al., 2011a) lowered word error rate by roughly 30% on four benchmarks compared with Gaussian-mixture acoustic models trained on the same 309 hours (RT03S: 27.4% → 18.5%). Dahl et al. (2012) reported 16–23% relative gains on the 40-hour Bing mobile business search set.
  • Vision: On MNIST, deep models beat the SVM error of 1.4%. Ciresan et al. (2012) reached 0.27% with a convolutional architecture, and Rifai et al. (2011c) reached 0.81% on the knowledge-free version. On ImageNet, Krizhevsky et al. (2012) cut the error from 26.1% to 15.3%.
  • NLP:
    • Word embeddings stem from neural net language models (Bengio et al., 2003).
    • SENNA (Collobert et al., 2011) shares representations across POS tagging, chunking, NER, semantic role labeling and parsing.
    • The recurrent LM of Mikolov et al. (2011) lowered perplexity from 140 to 102 and reduced Wall Street Journal WER from 17.2% to 14.4%.
    • Word sense disambiguation on a Senseval-3 subset improved from 67.8% to 70.2%.
  • Transfer learning: Representation learning won both 2011 transfer learning challenges. The ICML 2011 workshop challenge was won with unsupervised layer-wise pre-training, and the NIPS 2011 challenge was won by Goodfellow et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

218 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 207 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1206.5538 · arXiv 1206.5538

@misc{bengio2012representation,
  title = {{Representation Learning: A Review and New Perspectives}},
  author = {Bengio, Yoshua and Courville, Aaron and Vincent, Pascal},
  year = {2012},
  doi = {10.48550/arxiv.1206.5538},
  eprint = {1206.5538},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1206.5538}
}

References

225 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alain, G. and Bengio, Y. (2012). What regularized auto-encoders learn from the data generating distribution. Technical Report Arxiv report 1211.4246, Université de Montréal. find
  2. Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251–276. find
  3. Bach, F., Jenatton, R., Mairal, J., and Obozinski, G. (2011). Structured sparsity through convex optimization. CoRR, abs/1109.2397. — [arXiv:1109.2397](https://arxiv.org/abs/1109.2397) find
  4. Bagnell, J. A. and Bradley, D. M. (2009). Differentiable sparse coding. In NIPS’2009, pages 113–120. find
  5. Baird, H. (1990). Document image defect models. In IAPR Workshop, Syntactic & Structural Patt. Rec., pages 38–46. find
  6. Becker, S. and Hinton, G. (1992). A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355, 161–163. find
  7. Belkin, M. and Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6), 1373–1396. find
  8. Bell, A. and Sejnowski, T. J. (1997). The independent components of natural scenes are edge filters. Vision Research, 37, 3327–3338. find
  9. Bengio, Y. (1993). A connectionist approach to speech recognition. International Journal on Pattern Recognition and Artificial Intelligence, 7(4), 647–668. find
  10. Bengio, Y. (2008). Neural net language models. Scholarpedia, 3(1). find
  11. Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1), 1–127. Also published as a book. find
  12. Bengio, Y. (2011). Deep learning of representations for unsupervised and transfer learning. In JMLR W&CP: Proc. Unsupervised and Transfer Learning. find
  13. Bengio, Y. (2013). Practical recommendations for gradient-based training of deep architectures. In K.-R. Müller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade. Springer. find
  14. Bengio, Y. and Delalleau, O. (2009). Justifying and generalizing contrastive divergence. Neural Computation, 21(6), 1601–1621. find
  15. Bengio, Y. and Delalleau, O. (2011). On the expressive power of deep architectures. In ALT’2011. find
  16. Bengio, Y. and LeCun, Y. (2007). Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, editors, Large Scale Kernel Machines. MIT Press. find
  17. Bengio, Y. and Monperrus, M. (2005). Non-local manifold tangent learning. In NIPS’2004, pages 129–136. MIT Press. find
  18. Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. find
  19. Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003). A neural probabilistic language model. JMLR, 3, 1137–1155. find
  20. Bengio, Y., Paiement, J.-F., Vincent, P., Delalleau, O., Le Roux, N., and Ouimet, M. (2004). Out-of-sample extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. In NIPS’2003. find
  21. Bengio, Y., Delalleau, O., and Le Roux, N. (2006a). The curse of highly variable functions for local kernel machines. In NIPS’2005. find
  22. Bengio, Y., Larochelle, H., and Vincent, P. (2006b). Non-local manifold Parzen windows. In NIPS’2005. MIT Press. find
  23. Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H. (2007). Greedy layer-wise training of deep networks. In NIPS’2006. find
  24. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In ICML’09. find our card
  25. Bengio, Y., Delalleau, O., and Simard, C. (2010). Decision trees do not generalize to new variations. Computational Intelligence, 26(4), 449–467. find
  26. Bengio, Y., Alain, G., and Rifai, S. (2012). Implicit density estimation by local moment matching to sample from auto-encoders. Technical report, arXiv:1207.0057. — [arXiv:1207.0057](https://arxiv.org/abs/1207.0057) find
  27. Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. (2013). Better mixing via deep representations. In ICML’2013. find
  28. Bergstra, J. and Bengio, Y. (2009). Slow, decorrelated features for pretraining complex cell-like networks. In NIPS’2009. find
  29. Bergstra, J. and Bengio, Y. (2012). Random search for hyperparameter optimization. J. Machine Learning Res., 13, 281–305. find
  30. Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B. (2011). Algorithms for hyper-parameter optimization. In NIPS’2011. find
  31. Berkes, P. and Wiskott, L. (2005). Slow feature analysis yields a rich repertoire of complex cell properties. Journal of Vision, 5(6), 579–602. find
  32. Besag, J. (1975). Statistical analysis of non-lattice data. The Statistician, 24(3), 179–195. find
  33. Bordes, A., Glorot, X., Weston, J., and Bengio, Y. (2012). Joint learning of words and meaning representations for open-text semantic parsing. AISTATS’2012. find
  34. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. In ICML’2012. find
  35. Boureau, Y., Ponce, J., and LeCun, Y. (2010). A theoretical analysis of feature pooling in vision algorithms. In ICML’10. find
  36. Boureau, Y., Le Roux, N., Bach, F., Ponce, J., and LeCun, Y. (2011). Ask the locals: multi-way local pooling for image recognition. In ICCV’11. find
  37. Bourlard, H. and Kamp, Y. (1988). Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59, 291–294. find
  38. Brand, M. (2003). Charting a manifold. In NIPS’2002, pages 961– 968. MIT Press. find
  39. Breuleux, O., Bengio, Y., and Vincent, P. (2011). Quickly generating representative samples from an RBM-derived process. Neural Computation, 23(8), 2053–2073. find
  40. Bruna, J. and Mallat, S. (2011). Classification with scattering operators. In ICPR’2011. find
  41. Cadieu, C. and Olshausen, B. (2009). Learning transformational invariants from natural movies. In NIPS’2009, pages 209–216. MIT Press. find
  42. Carreira-Perpiñan, M. A. and Hinton, G. E. (2005). On contrastive divergence learning. In AISTATS’2005, pages 33–40. find
  43. Chen, M., Xu, Z., Winberger, K. Q., and Sha, F. (2012). Marginalized denoising autoencoders for domain adaptation. In ICML’2012. find
  44. Cho, K., Raiko, T., and Ilin, A. (2010). Parallel tempering is efficient for learning restricted Boltzmann machines. In IJCNN’2010. find
  45. Cho, K., Raiko, T., and Ilin, A. (2011). Enhanced gradient and adaptive learning rate for training restricted Boltzmann machines. In ICML’2011, pages 105–112. find
  46. Ciresan, D., Meier, U., and Schmidhuber, J. (2012). Multi-column deep neural networks for image classification. Technical report, arXiv:1202.2745. find
  47. Ciresan, D. C., Meier, U., Gambardella, L. M., and Schmidhuber, J. (2010). Deep big simple neural nets for handwritten digit recognition. Neural Computation, 22, 1–14. find
  48. Coates, A. and Ng, A. Y. (2011a). The importance of encoding versus training with sparse coding and vector quantization. In ICML’2011. find
  49. Coates, A. and Ng, A. Y. (2011b). Selecting receptive fields in deep networks. In NIPS’2011. find
  50. Collobert, R. and Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML’2008. find
  51. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12, 2493–2537. find our card
  52. Courville, A., Bergstra, J., and Bengio, Y. (2011a). A spike and slab restricted Boltzmann machine. In AISTATS’2011. find
  53. Courville, A., Bergstra, J., and Bengio, Y. (2011b). Unsupervised models of images by spike-and-slab RBMs. In ICML’2011. find
  54. Dahl, G. E., Ranzato, M., Mohamed, A., and Hinton, G. E. (2010). Phone recognition with the mean-covariance restricted Boltzmann machine. In NIPS’2010. find
  55. Dahl, G. E., Yu, D., Deng, L., and Acero, A. (2012). Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1), 33–42. find
  56. Deng, L., Seltzer, M., Yu, D., Acero, A., Mohamed, A., and Hinton, G. (2010). Binary coding of speech spectrograms using a deep auto-encoder. In Interspeech 2010, Makuhari, Chiba, Japan. find
  57. Desjardins, G. and Bengio, Y. (2008). Empirical evaluation of convolutional RBMs for vision. Technical Report 1327, Dept. IRO, U. Montréal. find
  58. Desjardins, G., Courville, A., Bengio, Y., Vincent, P., and Delalleau, O. (2010). Tempered Markov chain Monte Carlo for training of restricted Boltzmann machine. In AISTATS’2010, volume 9, pages 145–152. find
  59. Desjardins, G., Courville, A., and Bengio, Y. (2011). On tracking the partition function. In NIPS’2011. find
  60. Desjardins, G., Courville, A., and Bengio, Y. (2012). On training deep Boltzmann machines. Technical Report arXiv:1203.4416v1, Université de Montréal. — [arXiv:1203.4416](https://arxiv.org/abs/1203.4416) find
  61. DiCarlo, J., Zoccolan, D., and Rust, N. (2012). How does the brain solve visual object recognition? Neuron. find
  62. Donoho, D. L. and Grimes, C. (2003). Hessian eigenmaps: new locally linear embedding techniques for high-dimensional data. Technical Report 2003-08, Dept. Statistics, Stanford University. find
  63. Eisner, J. (2012). Learning approximate inference policies for fast prediction. Keynote talk at ICML Workshop on Inferning: Interactions Between Search and Learning. find
  64. Erhan, D., Courville, A., and Bengio, Y. (2010a). Understanding representations learned in deep architectures. Technical Report 1355, Université de Montréal/DIRO. find
  65. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., and Bengio, S. (2010b). Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11, 625– 660. find
  66. Freund, Y. and Haussler, D. (1994). Unsupervised learning of distributions on binary vectors using two layer networks. Technical Report UCSC-CRL-94-25, University of California, Santa Cruz. find
  67. Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36, 193–202. find
  68. Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In AISTATS’2010. find
  69. Glorot, X., Bordes, A., and Bengio, Y. (2011a). Deep sparse rectifier neural networks. In AISTATS’2011. find
  70. Glorot, X., Bordes, A., and Bengio, Y. (2011b). Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML’2011. find
  71. Goodfellow, I., Le, Q., Saxe, A., and Ng, A. (2009). Measuring invariances in deep networks. In NIPS’2009, pages 646–654. find
  72. Goodfellow, I., Courville, A., and Bengio, Y. (2011). Spike-and-slab sparse coding for unsupervised feature discovery. In NIPS Workshop on Challenges in Learning Hierarchical Models. find
  73. Goodfellow, I. J., Courville, A., and Bengio, Y. (2012). Spike-and-slab sparse coding for unsupervised feature discovery. arXiv:1201.3382. — [arXiv:1201.3382](https://arxiv.org/abs/1201.3382) find
  74. Gregor, K. and LeCun, Y. (2010a). Emergence of complex-like cells in a temporal product network with local receptive fields. Technical report, arXiv:1006.0448. — [arXiv:1006.0448](https://arxiv.org/abs/1006.0448) find
  75. Gregor, K. and LeCun, Y. (2010b). Learning fast approximations of sparse coding. In ICML’2010. find
  76. Gregor, K., Szlam, A., and LeCun, Y. (2011). Structured sparse coding via lateral inhibition. In NIPS’2011. find
  77. Gribonval, R. (2011). Should penalized least squares regression be interpreted as Maximum A Posteriori estimation? IEEE Transactions on Signal Processing, 59(5), 2405–2410. find
  78. Grosse, R., Raina, R., Kwong, H., and Ng, A. Y. (2007). Shift-invariant sparse coding for audio classification. In UAI’2007. find
  79. Grubb, A. and Bagnell, J. A. D. (2010). Boosted backpropagation learning for training deep modular networks. In ICML’2010. find
  80. Gutmann, M. and Hyvarinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS’2010. find
  81. Hamel, P., Lemieux, S., Bengio, Y., and Eck, D. (2011). Temporal pooling and multiscale learning for automatic annotation and ranking of music audio. In ISMIR. find
  82. Håstad, J. (1986). Almost optimal lower bounds for small depth circuits. In STOC’86, pages 6–20. find
  83. Håstad, J. and Goldmann, M. (1991). On the power of small-depth threshold circuits. Computational Complexity, 1, 113–129. find
  84. Henaff, M., Jarrett, K., Kavukcuoglu, K., and LeCun, Y. (2011). Unsupervised learning of sparse features for scalable audio classification. In ISMIR’11. find
  85. Hinton, G., Krizhevsky, A., and Wang, S. (2011). Transforming autoencoders. In ICANN’2011. find
  86. Hinton, G., Deng, L., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6), 82–97. find
  87. Hinton, G. E. (1986). Learning distributed representations of concepts. In Proc. 8th Conf. Cog. Sc. Society, pages 1–12. find
  88. Hinton, G. E. (1999). Products of experts. In ICANN’1999. find
  89. Hinton, G. E. (2000). Training products of experts by minimizing contrastive divergence. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London. find
  90. Hinton, G. E. (2010). A practical guide to training restricted Boltzmann machines. Technical Report UTML TR 2010-003, Department of Computer Science, University of Toronto. find
  91. Hinton, G. E. and Roweis, S. (2003). Stochastic neighbor embedding. In NIPS’2002. find
  92. Hinton, G. E. and Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507. find
  93. Hinton, G. E. and Zemel, R. S. (1994). Autoencoders, minimum description length, and helmholtz free energy. In NIPS’1993. find
  94. Hinton, G. E., Osindero, S., and Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527– 1554. find
  95. Hubel, D. H. and Wiesel, T. N. (1959). Receptive fields of single neurons in the cat’s striate cortex. Journal of Physiology, 148, 574–591. find
  96. Hurri, J. and Hyvärinen, A. (2003). Temporal coherence, natural image sequences, and the visual cortex. In NIPS’2002. find
  97. Hyvärinen, A. (2005). Estimation of non-normalized statistical models using score matching. J. Machine Learning Res., 6. find
  98. Hyvärinen, A. (2007). Some extensions of score matching. Computational Statistics and Data Analysis, 51, 2499–2512. find
  99. Hyvärinen, A. (2008). Optimal approximation of signal priors. Neural Computation, 20(12), 3087–3110. find
  100. Hyvärinen, A. and Hoyer, P. (2000). Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces. Neural Computation, 12(7). find
  101. Hyvärinen, A., Karhunen, J., and Oja, E. (2001a). Independent Component Analysis. Wiley-Interscience. find
  102. Hyvärinen, A., Hoyer, P. O., and Inki, M. (2001b). Topographic independent component analysis. Neural Computation, 13(7), 1527–1558. find
  103. Hyvärinen, A., Hurri, J., and Hoyer, P. O. (2009). Natural Image Statistics: A probabilistic approach to early computational vision. Springer-Verlag. find
  104. Jaeger, H. (2007). Echo state network. Scholarpedia, 2(9), 2330. find
  105. Jain, V. and Seung, S. H. (2008). Natural image denoising with convolutional networks. In NIPS’2008. find
  106. Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2009). What is the best multi-stage architecture for object recognition? In ICCV’09. find
  107. Jenatton, R., Audibert, J.-Y., and Bach, F. (2009). Structured variable selection with sparsity-inducing norms. Technical report, arXiv:0904.3523. — [arXiv:0904.3523](https://arxiv.org/abs/0904.3523) find
  108. Jutten, C. and Herault, J. (1991). Blind separation of sources, part I: an adaptive algorithm based on neuromimetic architecture. Signal Processing, 24, 1–10. find
  109. Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2008). Fast inference in sparse coding algorithms with applications to object recognition. CBLL-TR-2008-12-01, NYU. find
  110. Kavukcuoglu, K., Ranzato, M.-A., Fergus, R., and LeCun, Y. (2009). Learning invariant features through topographic filter maps. In CVPR’2009. find
  111. Kavukcuoglu, K., Sermanet, P., Boureau, Y.-L., Gregor, K., Mathieu, M., and LeCun, Y. (2010). Learning convolutional feature hierarchies for visual recognition. In NIPS’2010. find
  112. Kingma, D. and LeCun, Y. (2010). Regularized estimation of image statistics by score matching. In NIPS’2010. find
  113. Kivinen, J. J. and Williams, C. K. I. (2012). Multiple texture Boltzmann machines. In AISTATS’2012. find
  114. Körding, K. P., Kayser, C., Einhäuser, W., and König, P. (2004). How are complex cell properties adapted to the statistics of natural stimuli? J. Neurophysiology, 91. find
  115. Krizhevsky, A. (2010). Convolutional deep belief networks on CIFAR-10. Technical report, U. Toronto. find
  116. Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical report, U. Toronto. find
  117. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012). ImageNet classification with deep convolutional neural networks. In NIPS’2012. find
  118. Larochelle, H. and Bengio, Y. (2008). Classification using discriminative restricted Boltzmann machines. In ICML’2008. find
  119. Larochelle, H., Bengio, Y., Louradour, J., and Lamblin, P. (2009). Exploring strategies for training deep neural networks. Journal of Machine Learning Research, 10, 1–40. find
  120. Lazebnik, S., Schmid, C., and Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR’2006. find
  121. Le, H.-S., Oparin, I., Allauzen, A., Gauvin, J.-L., and Yvon, F. (2013). Structured output layer neural network language models for speech recognition. IEEE Trans. Audio, Speech & Language Processing. find
  122. Le, Q., Ngiam, J., Chen, Z., hao Chia, D. J., Koh, P. W., and Ng, A. (2010). Tiled convolutional neural networks. In NIPS’2010. find
  123. Le, Q., Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., and Ng, A. (2011a). On optimization methods for deep learning. In ICML’2011. find
  124. Le, Q. V., Karpenko, A., Ngiam, J., and Ng, A. Y. (2011b). ICA with reconstruction cost for efficient overcomplete feature learning. In NIPS’2011. find
  125. Le, Q. V., Zou, W. Y., Yeung, S. Y., and Ng, A. Y. (2011c). Learning hierarchical spatio-temporal features for action recognition with independent subspace analysis. In CVPR’2011. find
  126. Le Roux, N., Bengio, Y., Lamblin, P., Joliveau, M., and Kegl, B. (2008a). Learning the 2-D topology of images. In NIPS’07. find
  127. Le Roux, N., Manzagol, P.-A., and Bengio, Y. (2008b). Topmoumoute online natural gradient algorithm. In NIPS’07. find
  128. LeCun, Y. (1986). Learning processes in an asymmetric threshold network. In Disordered Systems and Biological Organization, pages 233–240. Springer-Verlag. find
  129. LeCun, Y. (1987). Modèles connexionistes de l’apprentissage. Ph.D. thesis, Université de Paris VI. find
  130. LeCun, Y. (1989). Generalization and network design strategies. In Connectionism in Perspective. Elsevier Publishers. find
  131. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation. find
  132. LeCun, Y., Bottou, L., Orr, G. B., and Müller, K. (1998a). Efficient backprop. In Neural Networks, Tricks of the Trade. find
  133. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998b). Gradient based learning applied to document recognition. Proc. IEEE. find
  134. Lee, H., Ekanadham, C., and Ng, A. (2008). Sparse deep belief net model for visual area V2. In NIPS’07. find
  135. Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. (2009a). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In ICML’2009. find
  136. Lee, H., Pham, P., Largman, Y., and Ng, A. (2009b). Unsupervised feature learning for audio classification using convolutional deep belief networks. In NIPS’2009. find
  137. Lin, Y., Tong, Z., Zhu, S., and Yu, K. (2010). Deep coding network. In NIPS’2010. find
  138. Lowe, D. (1999). Object recognition from local scale invariant features. In ICCV’99. find
  139. Mallat, S. (2012). Group invariant scattering. Communications on Pure and Applied Mathematics. find
  140. Marlin, B. and de Freitas, N. (2011). Asymptotic efficiency of deterministic estimators for discrete energy-based models: Ratio matching and pseudolikelihood. In UAI’2011. find
  141. Marlin, B., Swersky, K., Chen, B., and de Freitas, N. (2010). Inductive principles for restricted Boltzmann machine learning. In AISTATS’2010, pages 509–516. find
  142. Martens, J. (2010). Deep learning via Hessian-free optimization. In ICML’2010, pages 735–742. find
  143. Martens, J. and Sutskever, I. (2011). Learning recurrent neural networks with Hessian-free optimization. In ICML’2011. find
  144. Memisevic, R. and Hinton, G. E. (2010). Learning to represent spatial transformations with factored higher-order Boltzmann machines. Neural Comp., 22(6). find
  145. Mesnil, G., Dauphin, Y., Glorot, X., Rifai, S., Bengio, Y., Goodfellow, I., Lavoie, E., Muller, X., Desjardins, G., Warde-Farley, D., Vincent, P., Courville, A., and Bergstra, J. (2011). Unsupervised and transfer learning challenge: a deep learning approach. In JMLR W&CP: Proc. Unsupervised and Tran… find
  146. Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Cernocky, J. (2011). Empirical evaluation and combination of advanced language modeling techniques. In INTERSPEECH’2011. find
  147. Mobahi, H., Collobert, R., and Weston, J. (2009). Deep learning from temporal coherence in video. In ICML’2009. find
  148. Mohamed, A., Dahl, G., and Hinton, G. (2012). Acoustic modeling using deep belief networks. IEEE Trans. on Audio, Speech and Language Processing, 20(1), 14–22. find
  149. Montufar, G. F. and Morton, J. (2012). When does a mixture of products contain a product of mixtures? Technical report, arXiv:1206.0387. — [arXiv:1206.0387](https://arxiv.org/abs/1206.0387) find
  150. Murray, I. and Salakhutdinov, R. (2009). Evaluating probabilities under high-dimensional latent variable models. In NIPS’2008, pages 1137–1144. find
  151. Nair, V. and Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In ICML’10. find
  152. Neal, R. M. (1992). Connectionist learning of belief networks. Artificial Intelligence, 56, 71–113. find
  153. Neal, R. M. (1993). Probabilistic inference using Markov chain Monte-Carlo methods. Technical Report CRG-TR-93-1, Dept. of Computer Science, University of Toronto. find
  154. Ngiam, J., Chen, Z., Koh, P., and Ng, A. (2011). Learning deep energy models. In Proc. ICML’2011. ACM. find
  155. Olshausen, B. A. and Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607–609. find
  156. Orr, G. and Muller, K.-R., editors (1998). Neural networks: tricks of the trade. Lect. Notes Comp. Sc. Springer-Verlag. find
  157. Pascanu, R. and Bengio, Y. (2013). Natural gradient revisited. Technical report, arXiv:1301.3584. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) find
  158. Raiko, T., Valpola, H., and LeCun, Y. (2012). Deep learning made easier by linear transformations in perceptrons. In AISTATS’2012. find
  159. Raina, R., Battle, A., Lee, H., Packer, B., and Ng, A. Y. (2007). Self-taught learning: transfer learning from unlabeled data. In ICML’2007. find
  160. Ranzato, M. and Hinton, G. H. (2010). Modeling pixel means and covariances using factorized third-order Boltzmann machines. In CVPR’2010, pages 2551–2558. find
  161. Ranzato, M., Poultney, C., Chopra, S., and LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. In NIPS’2006. find
  162. Ranzato, M., Boureau, Y., and LeCun, Y. (2008). Sparse feature learning for deep belief networks. In NIPS’2007. find
  163. Ranzato, M., Krizhevsky, A., and Hinton, G. (2010a). Factored 3-way restricted Boltzmann machines for modeling natural images. In AISTATS’2010, pages 621–628. find
  164. Ranzato, M., Mnih, V., and Hinton, G. (2010b). Generating more realistic images using gated MRF’s. In NIPS’2010. find
  165. Ranzato, M., Susskind, J., Mnih, V., and Hinton, G. (2011). On deep generative models with applications to recognition. In CVPR’2011. find
  166. Riesenhuber, M. and Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nature Neuroscience. find
  167. Rifai, S., Vincent, P., Muller, X., Glorot, X., and Bengio, Y. (2011a). Contractive auto-encoders: Explicit invariance during feature extraction. In ICML’2011. find
  168. Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., and Glorot, X. (2011b). Higher order contractive auto-encoder. In ECML PKDD. find
  169. Rifai, S., Dauphin, Y., Vincent, P., Bengio, Y., and Muller, X. (2011c). The manifold tangent classifier. In NIPS’2011. find
  170. Rifai, S., Bengio, Y., Dauphin, Y., and Vincent, P. (2012). A generative process for sampling contractive auto-encoders. In ICML’2012. find
  171. Roweis, S. (1997). EM algorithms for PCA and sensible PCA. CNS Technical Report CNS-TR-97-02, Caltech. find
  172. Roweis, S. and Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500). find
  173. Salakhutdinov, R. (2010a). Learning deep Boltzmann machines using adaptive MCMC. In ICML’2010. find
  174. Salakhutdinov, R. (2010b). Learning in Markov random fields using tempered transitions. In NIPS’2010. find
  175. Salakhutdinov, R. and Hinton, G. E. (2007). Semantic hashing. In SIGIR’2007. find
  176. Salakhutdinov, R. and Hinton, G. E. (2009). Deep Boltzmann machines. In AISTATS’2009, pages 448–455. find
  177. Salakhutdinov, R. and Larochelle, H. (2010). Efficient learning of deep Boltzmann machines. In AISTATS’2010. find
  178. Salakhutdinov, R., Mnih, A., and Hinton, G. E. (2007). Restricted Boltzmann machines for collaborative filtering. In ICML 2007. find
  179. Savard, F. (2011). Réseaux de neurones à relaxation entraı̂nés par critère d’autoencodeur débruitant. Master’s thesis, U. Montréal. find
  180. Schmah, T., Hinton, G. E., Zemel, R., Small, S. L., and Strother, S. (2009). Generative versus discriminative training of RBMs for classification of fMRI images. In NIPS’2008, pages 1409–1416. find
  181. Schölkopf, B., Smola, A., and Müller, K.-R. (1998). Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10, 1299–1319. find
  182. Schwenk, H., Rousseau, A., and Attik, M. (2012). Large, pruned or continuous space language models on a gpu for statistical machine translation. In Workshop on the future of language modeling for HLT. find
  183. Seide, F., Li, G., and Yu, D. (2011a). Conversational speech transcription using context-dependent deep neural networks. In Interspeech 2011, pages 437–440. find
  184. Seide, F., Li, G., and Yu, D. (2011b). Feature engineering in context-dependent deep neural networks for conversational speech transcription. In ASRU’2011. find
  185. Serre, T., Wolf, L., Bileschi, S., and Riesenhuber, M. (2007). Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell., 29(3), 411–426. find
  186. Seung, S. H. (1998). Learning continuous attractors in recurrent networks. In NIPS’1997. find
  187. Simard, D., Steinkraus, P. Y., and Platt, J. C. (2003). Best practices for convolutional neural networks. In ICDAR’2003. find
  188. Simard, P., Victorri, B., LeCun, Y., and Denker, J. (1992). Tangent prop - A formalism for specifying selected invariances in an adaptive network. In NIPS’1991. find
  189. Simard, P. Y., LeCun, Y., and Denker, J. (1993). Efficient pattern recognition using a new transformation distance. In NIPS’92. find
  190. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing, volume 1, chapter 6, pages 194–281. MIT Press, Cambridge. find
  191. Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In NIPS’2012. find
  192. Socher, R., Huang, E. H., Pennington, J., Ng, A. Y., and Manning, C. D. (2011a). Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS’2011. find
  193. Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., and Manning, C. D. (2011b). Semi-supervised recursive autoencoders for predicting sentiment distributions. In EMNLP’2011. find our card
  194. Srivastava, N. and Salakhutdinov, R. (2012). Multimodal learning with deep boltzmann machines. In NIPS’2012. find
  195. Stoyanov, V., Ropson, A., and Eisner, J. (2011). Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In AISTATS’2011. find
  196. Sutskever, I. (2012). Training Recurrent Neural Networks. Ph.D. thesis, Departement of computer science, University of Toronto. find
  197. Sutskever, I. and Tieleman, T. (2010). On the Convergence Properties of Contrastive Divergence. In AISTATS’2010. find
  198. Sutskever, I., Hinton, G., and Taylor, G. (2009). The recurrent temporal restricted Boltzmann machine. In NIPS’2008. find
  199. Swersky, K. (2010). Inductive Principles for Learning Restricted Boltzmann Machines. Master’s thesis, University of British Columbia. find
  200. Swersky, K., Ranzato, M., Buchman, D., Marlin, B., and de Freitas, N. (2011). On score matching for energy based models: Generalizing autoencoders and simplifying deep learning. In Proc. ICML’2011. ACM. find
  201. Taylor, G. and Hinton, G. (2009). Factored conditional restricted Boltzmann machines for modeling motion style. In ICML’2009. find
  202. Taylor, G., Fergus, R., LeCun, Y., and Bregler, C. (2010). Convolutional learning of spatio-temporal features. In ECCV’10. find
  203. Tenenbaum, J., de Silva, V., and Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500), 2319–2323. find
  204. Tieleman, T. (2008). Training restricted Boltzmann machines using approximations to the likelihood gradient. In ICML’2008, pages 1064–1071. find
  205. Tieleman, T. and Hinton, G. (2009). Using fast weights to improve persistent contrastive divergence. In ICML’2009. find
  206. Tipping, M. E. and Bishop, C. M. (1999). Probabilistic principal components analysis. J. Roy. Stat. Soc. B, (3). find
  207. Turaga, S. C., Murray, J. F., Jain, V., Roth, F., Helmstaedter, M., Briggman, K., Denk, W., and Seung, H. S. (2010). Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, 22, 511–538. van der Maaten, L. (2009). Learning a parametric embedding by pre… find
  208. Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7). find
  209. Vincent, P. and Bengio, Y. (2003). Manifold Parzen windows. In NIPS’2002. MIT Press. find
  210. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In ICML 2008. find
  211. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Manzagol, P.-A. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Machine Learning Res., 11. find
  212. Weinberger, K. Q. and Saul, L. K. (2004). Unsupervised learning of image manifolds by semidefinite programming. In CVPR’2004, pages 988–995. find
  213. Welling, M. (2009). Herding dynamic weights for partially observed random field models. In UAI’2009. find
  214. Welling, M., Hinton, G. E., and Osindero, S. (2003). Learning sparse topographic representations with products of Student-t distributions. In NIPS’2002. find
  215. Weston, J., Ratle, F., and Collobert, R. (2008). Deep learning via semi-supervised embedding. In ICML 2008. find
  216. Weston, J., Bengio, S., and Usunier, N. (2010). Large scale image annotation: learning to rank with joint word-image embeddings. Machine Learning, 81(1), 21–35. find
  217. Wiskott, L. and Sejnowski, T. (2002). Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4), 715–770. find
  218. Younes, L. (1999). On the convergence of Markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics and Stochastic Reports, 65(3), 177–228. find
  219. Yu, D., Wang, S., and Deng, L. (2010). Sequential labeling using deep-structured conditional random fields. IEEE Journal of Selected Topics in Signal Processing. find
  220. Yu, K. and Zhang, T. (2010). Improved local coordinate coding using local tangents. In ICML’2010. find
  221. Yu, K., Zhang, T., and Gong, Y. (2009). Nonlinear learning using local coordinate coding. In NIPS’2009. find
  222. Yu, K., Lin, Y., and Lafferty, J. (2011). Learning image representations from the pixel level via hierarchical sparse coding. In CVPR. find
  223. Yuille, A. L. (2005). The convergence of contrastive divergences. In NIPS’2004, pages 1593–1600. find
  224. Zeiler, M., Krishnan, D., Taylor, G., and Fergus, R. (2010). Deconvolutional networks. In CVPR’2010. find
  225. Zou, W. Y., Ng, A. Y., and Yu, K. (2011). Unsupervised learning of visual invariance with temporal coherence. In NIPS 2011 Workshop on Deep Learning and Unsupervised Feature Learning. find