Elamur Research⌕
☰
home / papers / 2012 / Theano: new features and speed improvements

Theano: new features and speed improvements

Paper · 2012 · Frédéric Bastien, Pascal Lamblin, Razvan Pascanu et al.

Summary

This 2012 report describes how Theano, a Python-based compiler that turns symbolic linear-algebra expressions into optimized CPU or GPU code, was extended and made faster. Its authors are the Université de Montréal team. On the feature side, the main additions are the Scan Op, which represents a whole loop as a single graph node that can be differentiated, and an R-operator that provides forward-mode differentiation for Hessian-Free optimization. On the speed side, the authors target time spent outside actual computation. They introduce a VM runtime that supports lazy evaluation, plus a C version (CVM), now the default, that avoids returning to the Python interpreter between operations. More ops were also given C implementations. They then benchmark Theano against Torch7 on feed-forward networks and against RNNLM on recurrent networks. Theano matches or beats Torch7 once mini-batches are used. RNNLM leads on small recurrent models, but Theano catches up as models grow. The work showed that a flexible symbolic framework can compete with hand-tuned libraries.

Key points

  • Scan Op:
    • Gradients use backpropagation through time.
    • Iteration counts can be symbolic, including "do while" loops.
    • Loops can run on the GPU.
    • Compatible loops are merged, and loop-invariant work is moved out of the body.
  • R-operator: Combining TT.Lop with TT.Rop lets Theano compute a Gauss-Newton Hessian-vector product for an RNN in two passes instead of a naive three.
  • Overhead reductions:
    • The cvm_nogc linker skips freeing intermediate results, which matters most on GPU.
    • Setting trust_input skips input type checks and conversions.
    • Calling f.fn(n_calls=N) runs several iterations without a Python loop.
  • Torch7 comparison: The benchmark is 10-class classification of 784-dimensional inputs with three models: logistic regression, one hidden layer of 500 units, and three hidden layers of 1000 units. Torch7 wins only on logistic regression without mini-batches and on GPU with batch size 1. With batches of 10 or 60, Theano is faster or equal.
  • RNNLM comparison: The benchmark uses batch size 1 and hidden layers of 50 to 1500 units. RNNLM is faster for small networks, while Theano closes the gap for larger ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksHardware and Architecture · 167 works

Cite

Cite as Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley and Yoshua Bengio. 2012. Theano: new features and speed improvements. arXiv:1211.5590.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1211.5590 · arXiv 1211.5590

@misc{bastien2012theano,
  title = {{Theano: new features and speed improvements}},
  author = {Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley and Yoshua Bengio},
  year = {2012},
  doi = {10.48550/arxiv.1211.5590},
  eprint = {1211.5590},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1211.5590}
}

References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation. find
  2. Bergstra, J., Bastien, F., Breuleux, O., Lamblin, P., Pascanu, R., Delalleau, O., Desjardins, G., Warde-Farley, D., Goodfellow, I., Bergeron, A., and Bengio, Y. (2011). Theano: Deep learning on gpus with python. In Big Learn workshop, NIPS’11. 500 hidden units 1500 hidden units 10000 8000 6000 4000… find
  3. Collobert, R., Kavukcuoglu, K., and Farabet, C. (2011). Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop. find
  4. Hunter, J. D. (2007). Matplotlib: A 2d graphics environment. Computing in Science and Engineering, 9(3), 90–95. find
  5. Jones, E., Oliphant, T., Peterson, P., et al. (2001). SciPy: Open source scientific tools for Python. find
  6. Martens, J. and Sutskever, I. (2011). Learning recurrent neural networks with hessian-free optimization. In L. Getoor and T. Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML ’11, pages 1033–1040, New York, NY, USA. ACM. find
  7. Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Cernocky, J. (2011). Empirical evaluation and combination of advanced language modeling techniques. In Proc. 12th annual conference of the international speech communication association (INTERSPEECH 2011). find
  8. Oliphant, T. E. (2007). Python for scientific computing. Computing in Science and Engineering, 9, 10–20. find
  9. Pearlmutter, B. A. (1994). Fast exact multiplication by the hessian. Neural Computation, 6, 147–160. find
  10. Pérez, F. and Granger, B. E. (2007). IPython: A system for interactive scientific computing. Computing in Science and Engineering, 9(3), 21–29. find
  11. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning internal representations by error propagation. volume 1, chapter 8, pages 318–362. MIT Press, Cambridge. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bastien-2012-theano-new-features-speed-improvements/