Elamur Research
☰
home / papers / 2016 / Theano: A Python framework for fast computation of mathematical expressions

Theano: A Python framework for fast computation of mathematical expressions

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1605.02688

Summary

Python is convenient for prototyping numerical code, but its interpreter is slow and uses memory inefficiently for heavy array computation. This paper describes Theano, a library developed since 2008 that addresses this problem. Users write symbolic expressions over multi-dimensional arrays through an interface modelled on NumPy. Theano stores these expressions as a typed, bipartite, acyclic graph of Variable and Apply nodes. Before execution it rewrites the graph: it removes unused parts, reuses shared subexpressions, simplifies the mathematics, performs in-place operations, and applies rewrites that improve numerical stability. It then compiles fast code for CPU, or for GPU via CUDA, and a single configuration flag switches between them. It also derives gradients symbolically. Users can extend it with custom operations written in Python, C++ or CUDA. The article covers Theano's design, its community, recently added features and current limitations. It also benchmarks Theano against Torch7 and TensorFlow on several machine learning models. Theano matters because many higher-level deep learning and probabilistic programming tools were built on it.

Key points

  • Expressions are directed acyclic bipartite graphs, similar to SSA form, with strongly typed variables:
    • TensorType for arrays in main memory.
    • CudaNdarrayType for the legacy GPU back-end.
    • GpuArrayType for the new GPU back-end, which has limited, experimental OpenCL support.
    • Sparse for SciPy CSC/CSR matrices.
  • A type records the number of dimensions, the dtype and a broadcastable pattern, but not the array shape or strides.
  • Theano is open source under the 3-clause BSD license. The paper reports these community figures:
    • About 250 code contributors and 1280 GitHub forks since the 2011 move to GitHub.
    • 675 theano-dev subscribers and over 3000 theano-users members.
    • More than 1000 StackOverflow questions and about 38k PyPI downloads per month.
  • Frameworks built on Theano include Pylearn2, Blocks, Lasagne, Keras and the probabilistic programming package PyMC3.
  • Performance is compared with Torch7 and TensorFlow; the specific benchmark figures fall in a part of the text not provided here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

161 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 152 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1605.02688 · arXiv 1605.02688

@misc{team2016theano,
  title = {{Theano: A Python framework for fast computation of mathematical expressions}},
  author = {Team, The Theano Development and Al-Rfou, Rami and Alain, Guillaume and Almahairi, Amjad and Angermueller, Christof and Bahdanau, Dzmitry and Ballas, Nicolas and Bastien, Frédéric and Bayer, Justin and Belikov, Anatoly and Belopolsky, Alexander and Bengio, Yoshua and Bergeron, Arnaud and Bergstra, James and Bisson, Valentin and Snyder, Josh Bleecher and Bouchard, Nicolas and Boulanger-Lewandowski, Nicolas and Bouthillier, Xavier and Brébisson, Alexandre de and Breuleux, Olivier and Carrier, Pierre-Luc and Cho, Kyunghyun and Chorowski, Jan and Christiano, Paul and Cooijmans, Tim and Côté, Marc-Alexandre and Côté, Myriam and Courville, Aaron and Dauphin, Yann N. and Delalleau, Olivier and Demouth, Julien and Desjardins, Guillaume and Dieleman, Sander and Dinh, Laurent and Ducoffe, Mélanie and Dumoulin, Vincent and Kahou, Samira Ebrahimi and Erhan, Dumitru and Fan, Ziye and Firat, Orhan and Germain, Mathieu and Glorot, Xavier and Goodfellow, Ian and Graham, Matt and Gulcehre, Caglar and Hamel, Philippe and Harlouchet, Iban and Heng, Jean-Philippe and Hidasi, Balázs and Honari, Sina and Jain, Arjun and Jean, Sébastien and Jia, Kai and Korobov, Mikhail and Kulkarni, Vivek and Lamb, Alex and Lamblin, Pascal and Larsen, Eric and Laurent, César and Lee, Sean and Lefrancois, Simon and Lemieux, Simon and Léonard, Nicholas and Lin, Zhouhan and Livezey, Jesse A. and Lorenz, Cory and Lowin, Jeremiah and Ma, Qianli and Manzagol, Pierre-Antoine and Mastropietro, Olivier and McGibbon, Robert T. and Memisevic, Roland and Merriënboer, Bart van and Michalski, Vincent and Mirza, Mehdi and Orlandi, Alberto and Pal, Christopher and Pascanu, Razvan and Pezeshki, Mohammad and Raffel, Colin and Renshaw, Daniel and Rocklin, Matthew and Romero, Adriana and Roth, Markus and Sadowski, Peter and Salvatier, John and Savard, François and Schlüter, Jan and Schulman, John and Schwartz, Gabriel and Serban, Iulian Vlad and Serdyuk, Dmitriy and Shabanian, Samira and Simon, Étienne and Spieckermann, Sigurd and Subramanyam, S. Ramana and Sygnowski, Jakub and Tanguay, Jérémie and Tulder, Gijs van and Turian, Joseph and Urban, Sebastian and Vincent, Pascal and Visin, Francesco and Vries, Harm de and Warde-Farley, David and Webb, Dustin J. and Willson, Matthew and Xu, Kelvin and Xue, Lijun and Yao, Li and Zhang, Saizheng and Zhang, Ying},
  year = {2016},
  doi = {10.48550/arxiv.1605.02688},
  eprint = {1605.02688},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1605.02688}
}