Back

Theano: A Python framework for fast computation of mathematical expressions

Rami Al-RfouGuillaume AlainAmjad AlmahairiChristof AngermüllerDzmitry BahdanauNicolas BallasFrédéric BastienJustin BayerAnatoly BelikovAlexander Belopolsky

article2016arXiv2,381 citations

Presents the architecture, symbolic graph compilation, and automatic differentiation mechanisms of Theano, demonstrating how to achieve efficient GPU and CPU mathematical computation for deep learning models.

Listen

Modern machine learning research relies heavily on mathematical compilers to train complex models across central processing units (CPUs) and graphics processing units (GPUs). While the Python language offers rapid prototyping and broad data tools, its native runtime execution is notoriously slow and memory-intensive for large-scale mathematical computations. The article evaluates Theano, an established open-source Python framework designed to bridge this gap by defining, optimizing, and compiling symbolic mathematical expressions into high-performance C++ and GPU code.

The authors conducted a comprehensive technical review and benchmarking study using an enterprise-grade NVIDIA system equipped with four Titan X GPUs. They benchmarked Theano against major competing frameworks—specifically Torch7 and TensorFlow—across three standard machine learning architectures: convolutional neural networks for computer vision, recurrent long short-term memory (LSTM) networks for language processing, and sequence-to-sequence models for video captioning. The evaluation measured raw execution speed, multi-GPU scaling efficiency via the Platoon data-parallel framework, and compilation overhead.

The benchmarks demonstrated that Theano delivers computational throughput highly competitive with, and in several cases exceeding, alternative modern frameworks. On convolutional networks, Theano performed on par with Torch and TensorFlow, while its fast-compilation mode incurred only a modest 10% to 25% execution speed penalty. On medium and large recurrent models, Theano surpassed both TensorFlow and Torch in processing words per second. In sequence-to-sequence tasks, it achieved comparable total runtimes to TensorFlow, performing slightly faster on small batches. Furthermore, multi-GPU scaling through Platoon yielded near-linear speedups—achieving a factor of 2.0 on 2 GPUs and 3.9 to 4.0 on 4 GPUs when synchronizing every 100 batches—though synchronization on every batch reduced scaling efficiency to 1.6 to 1.7 on 2 GPUs and 3.2 on 4 GPUs.

These findings indicate that symbolic graph compilation provides substantial operational performance and flexibility, making Python viable for production-grade machine learning pipelines without sacrificing execution speed. The results validate that Theano remains a robust, cost-effective engine for deep learning research, supported by new features such as memory-pool integration, advanced diagnostic tools, and a modernized GPU back-end supporting half-precision floating-point formats.

To build on these strengths, the authors recommend addressing structural bottlenecks in future releases. Key priorities include restructuring graph optimization algorithms to prevent nonlinear increases in compile time for large graphs, implementing dynamic runtime configurations to curb excessive code recompilation, and refining memory management strategies such as recomputation and memory offloading to maximize limited GPU memory.

The findings are supported by standardized, reproducible hardware benchmarks on single-node systems. However, users should exercise caution regarding specific limitations identified in the article, including compiler scaling bottlenecks on exceptionally large graphs, Python Global Interpreter Lock overhead during multi-threading, and the requirement for external orchestration frameworks when scaling across multiple physical nodes.

AI ImpactsAustrian Research Institute for Artificial IntelligenceBirla Institute of Technology and ScienceCarnegie Mellon UniversityDrexel UniversityENS Paris-SaclayErasmus MCPolitecnico di MilanoSwiftKeyTechnical University of MunichUniversity of UtahUniversity of Warsawcompilercpufast computationcomputation graphssymbolic computationmathematical expressions
arXiv: 1605.02688Theano/Theano
  • Paper: TensorFlow: A system for large-scale machine learning, Martín Abadi et al. (2016). Reading this foundational paper on TensorFlow first provides essential context on dataflow graph compilers and distributed execution models that directly influenced the architecture of Theano.
  • Paper: Automatic differentiation in machine learning: a survey, Atilim Gunes Baydin et al. (2018). Reviewing this comprehensive survey on automatic differentiation beforehand clarifies the core mathematical gradient-computation techniques that frameworks like Theano automate.
Cover for Theano: A Python framework for fast computation of mathematical expressions

Abstract

Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements. Theano is being actively and continuously developed since 2008, multiple frameworks have been built on top of it and it has been used to produce many state-of-the-art machine learning models.

The present article is structured as follows. Section I provides an overview of the Theano software and its community. Section II presents the principal features of Theano and how to use them, and compares them with other similar projects. Section III focuses on recently-introduced functionalities and improvements. Section IV compares the performance of Theano against Torch7 and TensorFlow on several machine learning models. Section V discusses current limitations of Theano and potential ways of improving it.

Topics

compilercpufast computationlibrariesmachine learningmathematicspythonsoftwareefficient computation

Keywords

computation graphssymbolic computationmathematical expressionsdeep learningrecurrent neural networksgradient-based computationdata parallelismCUDA codeconvolution operationscode compilation

Table of Contents

  • I Overview
  • I.1 Vision
  • I.2 Community
  • I.3 Software based on Theano
  • II Main features
  • II.1 Mathematical expressions
  • II.1.1 Graph structure
  • II.1.2 Building a graph
  • II.1.3 Symbolic differentiation
  • II.1.4 Scan: Symbolic loops
  • II.2 The compilation phase
  • II.2.1 Graph optimizations
  • II.2.2 Shared variables
  • II.2.3 C code compilation and caching
  • II.3 Function execution
  • II.4 Extending Theano
  • II.5 Related software
  • III New features
  • III.1 Increased performance
  • III.1.1 Abstract Ops and 2D convolutions
  • III.1.2 Using cuDNN
  • III.1.3 CNMeM integration
  • III.1.4 Improvements in Scan
  • III.1.5 New gpuarray-based back-end
  • III.1.6 Data parallelism with Platoon
  • III.2 Faster compilation of graphs
  • III.2.1 Faster, simpler optimizer
  • III.2.2 Swapping updates without recompiling
  • III.2.3 Save and reload optimized graphs
  • III.3 Visualization, debugging, and diagnostic tools
  • III.3.1 Interactive visualization with d3viz
  • III.3.2 Test values
  • III.3.3 NanGuardMode
  • III.3.4 The PdbBreakPoint Op
  • III.3.5 Keeping the creation stack trace
  • IV Benchmarks
  • IV.1 Setup
  • IV.2 Convolutional networks
  • IV.3 Recurrent neural networks: LSTM on Penn Treebank
  • IV.4 Sequence-to-sequence: Caption generation from video
  • IV.5 Data parallelism for LSTM
  • V Limitations and challenges
  • V.1 Limitations from Python
  • V.2 Graph optimization time
  • V.3 Code compilation time
  • V.4 Loops and control-flow structures
  • V.5 Multi-node parallelism
  • V.6 Improving memory usage
  • V.7 The future of gradient-based computation frameworks
  • VI Conclusion
  • References

Knowls

  1. Knowl 1 — Symbolic Computation Graph Representation

    model/method

    In Theano, mathematical computations are modeled as bipartite directed acyclic graphs (DAGs) in single static assignment (SSA) form composed of two distinct classes of nodes:

    1. Variable nodes: Represent data entities, primarily multi-dimensional tensors. A Variable can serve as an input to multiple operations but is the output of at most one operation (or represents an external graph input). Variables are strongly typed by data type (such as float32 or int64), tensor dimensionality, and a broadcastable pattern (a boolean flag per dimension indicating whether the dimension is statically guaranteed to have length 1). Dynamic array shapes and memory strides are not part of the type signature.

    2. Apply nodes: Represent the application of a mathematical operation (Op) to an ordered list of input Variable nodes, producing an ordered list of output Variable nodes.

    Primary variable types include TensorType (host CPU multi-dimensional arrays backed by NumPy ndarray), CudaNdarrayType (legacy GPU arrays), GpuArrayType (modern GPU arrays backed by libgpuarray), and Sparse (SciPy compressed sparse column and row matrices).

  2. Knowl 2 — Symbolic Differentiation via Vector-Jacobian and Jacobian-Vector Products

    model/method

    Theano performs automatic differentiation directly on symbolic computation graphs through two operational modes:

    • Reverse-mode differentiation (theano.grad): Evaluates the gradient of a scalar cost C:RN→RC: \mathbb{R}^N \to \mathbb{R} with respect to input or intermediate variables xx by traversing the graph in reverse topological order from output to inputs. For an operation g:RN→RMg: \mathbb{R}^N \to \mathbb{R}^M, the Op's grad method computes the vector-Jacobian product (VJP):

    ∇xg(v)=v⋅∂g∂x\nabla_x g(v) = v \cdot \frac{\partial g}{\partial x}

    where v∈RMv \in \mathbb{R}^M is the incoming cotangent vector from downstream operations and ∂g∂x∈RM×N\frac{\partial g}{\partial x} \in \mathbb{R}^{M \times N} is the Jacobian matrix. Because the computed derivatives are themselves symbolic graph expressions, higher-order derivatives and learning rules can be constructed by traversing the resulting graph again.

    • Forward-mode differentiation (theano.Rop): Computes the Jacobian-vector product (JVP) corresponding to the action of the Jacobian on a perturbation vector v∈RNv \in \mathbb{R}^N:

    Rxg(v)=∂g∂x⋅vR_x g(v) = \frac{\partial g}{\partial x} \cdot v

    where Rxg(v)∈RMR_x g(v) \in \mathbb{R}^M. theano.Rop evaluates the R_op method of each Apply node while traversing the graph in forward topological order from inputs to outputs.

  3. Knowl 3 — Graph Optimization and Compilation Pipeline

    model/method

    Compiling a Theano function transforms a symbolic subgraph into an executable callable object through four sequential phases:

    1. Graph Extraction and Cloning: The minimal subgraph connecting specified input Variables to output Variables is extracted and cloned to isolate it from external modifications.

    2. Graph Optimization: Global and local graph rewrites are applied across multiple specialized passes:

    • Canonicalization: Rewrites expressions into canonical forms (e.g., x⋅x⇒x2x \cdot x \Rightarrow x^2), removes duplicate or redundant subtrees (e.g., x⋅y/y⇒xx \cdot y / y \Rightarrow x), and folds constants (e.g., 2+2⇒42 + 2 \Rightarrow 4).
    • Stabilization: Substitutes numerically unstable operations with stable equivalents (e.g., log⁡(1+x)⇒log1p(x)\log(1 + x) \Rightarrow \text{log1p}(x)).
    • Specialization: Substitutes generic operations with hardware-optimized kernels and fuses consecutive element-wise operations to eliminate redundant memory passes.
    • GPU Migration: Converts host CPU operations and variables into GPU counterparts and inserts host-device data transfer nodes.
    • In-Place Execution: Converts operations to write directly into input memory buffers when inputs are no longer read downstream, detecting and preventing dependency cycles.
    • Scan Optimizations: Hoists loop-invariant computations out of recurrent loops and eliminates unnecessary historical state storage.

    1. Code Generation and Compilation: C++ or CUDA source code is emitted for each optimized Op, compiled into Python C-extension modules, and dynamically loaded. An on-disk persistent cache avoids recompilation across identical operations.

    2. Runtime Packaging: A callable wrapper managing input-output data bindings is returned to the user.

  4. Knowl 4 — CVM Runtime Execution Engine and Memory Management

    model/method

    Theano executes compiled graphs using a C-based Virtual Machine (CVM):

    • Execution Loop: The CVM implements the operational scheduling loop in native C, executing compiled C++ or CUDA Op functions directly through function pointers. This eliminates Python interpreter function-call overhead, substantially speeding up graphs containing many operations on small operands.

    • Lazy Evaluation: The runtime schedules operations dynamically, bypassing conditional or unneeded execution branches that do not contribute to the requested output values.

    • Intermediate Buffer Allocation: By default, intermediate tensor memory is reclaimed by garbage collection after consumption. When configured with allow_gc=False, the runtime preserves allocated intermediate ndarray memory buffers across consecutive function calls, eliminating dynamic memory allocation and deallocation overhead during training loops at the cost of higher static memory usage.

  5. Knowl 5 — Scan Symbolic Recurrence and Loop Abstraction

    model/method

    To represent loops, recurrent neural networks, and iterations over variable-length sequences without unrolling the computation graph, Theano provides the Scan operator:

    • Inner Graph Encapsulation: Scan abstracts arbitrary loops into a single Apply node that encapsulates an isolated inner computation graph. The operator handles inputs, sequence indexing, tap access (referencing states from previous steps), recurrent state propagation, and communication between outer and inner graph scopes.

    • Differentiation: The symbolic gradient of a Scan node is constructed as a separate Scan node that iterates over sequences in reverse order, realizing symbolic back-propagation through time (BPTT). The forward-mode RR-operator is likewise generated as a forward-iterating Scan.

    • Buffer Optimization and Strict Mode: The compiler optimizes Scan by hoisting loop invariants to the outer graph and writing iteration outputs directly into pre-allocated destination sequence buffers. Setting strict=True disables implicit capture of outer shared variables, forcing explicit declarations of non-sequence dependencies to prevent redundant calculations from being pulled into the loop body.

  6. Knowl 6 — Abstract Convolution Ops and cuDNN Integration

    model/method

    Theano decouples convolution interfaces from concrete execution kernels using abstract placeholder operations:

    • Abstract Placeholders: Convolution operations are inserted into the initial graph as abstract Ops: AbstractConv2d (forward pass), AbstractConv2d_gradInputs (gradient with respect to inputs), and AbstractConv2d_gradWeights (gradient with respect to filter weights).

    • Backend Replacement: Optimization passes replace abstract nodes with concrete implementations (such as cuDNN, CPU GEMM, GPU GEMM, or FFT-based routines) based on platform availability and configuration settings.

    • cuDNN Algorithm Selection Modes: When using NVIDIA cuDNN, convolution algorithms can be selected statically or dynamically via heuristic profiling policies:

    • guess_once: cuDNN heuristics choose the fastest convolution algorithm once at initialization based on tensor shapes.

    • guess_on_shape_change: Re-evaluates cuDNN selection heuristics whenever input tensor shapes change.

    • time_once: Times all memory-feasible cuDNN convolution algorithms on the initial batch and selects the fastest measured algorithm.

    • time_on_shape_change: Re-benchmarks all feasible cuDNN algorithms whenever input shapes change.

  7. Knowl 7 — libgpuarray GPU Backend Architecture

    model/method

    Theano's GPU backend is implemented via libgpuarray, providing multi-device tensor capabilities:

    • Data Type Support: Supports arbitrary numerical data types on GPU devices, including half-precision floating point (float16), single-precision (float32), double-precision (float64), integers, and booleans.

    • Indexing and Memory Layout: Implements an nn-dimensional array object in device memory with arbitrary stride manipulation, zero-copy slicing views, and 64-bit array indexing to support tensors exceeding 2322^{32} elements.

    • Asynchronous Execution and Stream Concurrency: GPU operations execute asynchronously relative to host CPU code. Memory transfers between host and device are dispatched on dedicated CUDA streams, allowing data movement to overlap concurrently with kernel execution while maintaining dependency tracking.

    • Multi-Device Partitioning: Supports mapping different Apply nodes within the same computation graph to distinct GPU devices on the same system, enabling model parallelism.

  8. Knowl 8 — Platoon Multi-GPU Data Parallelism Framework

    model/method

    Platoon is a multi-process data parallelism framework designed for Theano to circumvent the single-threaded constraints of the Python Global Interpreter Lock (GIL):

    • Multi-Process Architecture: A centralized controller process coordinates multiple worker processes. Each worker manages a dedicated Theano instance running on a distinct CPU or GPU device.

    • Shared Memory Parameter Storage: Global model parameters reside in operating system shared memory, allowing all worker processes on a single host machine to read and update parameters without inter-process communication serialization overhead.

    • Asynchronous Signaling: Worker processes communicate state transitions and coordination signals asynchronously to the controller without blocking compute pipelines.

    • Synchronization Rules: Implements Asynchronous Stochastic Gradient Descent (ASGD) where workers independently apply mini-batch updates to the central shared parameters, as well as Elastic Averaging SGD (EASGD) where local worker parameters periodically synchronize with shared center parameters via an elastic penalty.

  9. Knowl 9 — Penn Treebank LSTM Recurrent Neural Network Benchmarks

    empirical result

    Training throughput on LSTM language models over the Penn Treebank dataset was evaluated on a single NVIDIA Titan X GPU (float32 precision, batch size 20, non-recurrent dropout applied during training) across three model configurations:

    • Small: Single layer, 200 hidden units, sequence length 20.
    • Medium: Single layer, 600 hidden units, sequence length 40.
    • Large: Two layers, 650 hidden units each, sequence length 50.

    Performance was measured in thousands of words processed per second (higher is better):

    • Small Model: TensorFlow achieved ~16,500 words/s, Theano achieved ~13,500 words/s, Torch achieved ~12,000 words/s, and Theano (fast_compile) achieved ~11,000 words/s.
    • Medium Model: Theano achieved the highest throughput at ~12,000 words/s, followed by TensorFlow at ~11,000 words/s, Theano (fast_compile) at ~9,500 words/s, and Torch at ~7,500 words/s.
    • Large Model: Theano achieved the highest throughput at ~9,500 words/s, followed by TensorFlow at ~7,800 words/s, Theano (fast_compile) at ~7,200 words/s, and Torch at ~5,500 words/s.

    On medium and large LSTM configurations, Theano achieved higher processing speed than both TensorFlow and Torch, with Theano's fast_compile mode outperforming Torch on the two larger models.

  10. Knowl 10 — Convolutional Neural Network Execution Benchmarks

    empirical result

    Minibatch forward and backward pass execution times were evaluated across four convolutional network architectures on an NVIDIA Digits DevBox (Core i7-5930K CPU, NVIDIA Titan X GPU, CUDA 7.5, cuDNN v4, float32 precision):

    • AlexNet (one-column, batch size 128): Total processing time was ~80 ms for Theano, ~100 ms for Theano (fast_compile), ~40 ms for Torch, and ~80 ms for TensorFlow.
    • OverFeat (fast variant, batch size 128): Total processing time was ~280 ms for Theano, ~320 ms for Theano (fast_compile), ~260 ms for Torch, and ~270 ms for TensorFlow.
    • VGG Model A (batch size 64): Total processing time was ~620 ms for Theano, ~750 ms for Theano (fast_compile), ~520 ms for Torch, and ~540 ms for TensorFlow.
    • GoogLeNet V1 (batch size 128): Total processing time was ~540 ms for Theano, ~670 ms for Theano (fast_compile), ~460 ms for Torch, and ~440 ms for TensorFlow.

    Theano achieved execution performance comparable to Torch and TensorFlow across all architectures, and using the fast_compile optimizer resulted in a 10% to 25% execution slowdown while substantially reducing graph compilation time.

  11. Knowl 11 — Multi-GPU Data Parallelism Scaling with Platoon

    empirical result

    Training throughput was measured using Platoon with Asynchronous SGD (ASGD) on Small, Medium, and Large LSTM models across 1, 2, and 4 NVIDIA Titan X GPUs on a single node:

    • Synchronizing After Every Batch: Inter-process communication and parameter contention yielded sub-linear scaling:

    • 2 GPUs achieved a 1.6×1.6\times to 1.7×1.7\times speedup over 1 GPU across all three models.

    • 4 GPUs achieved an approximate 3.2×3.2\times speedup over 1 GPU (reaching ~44,000 words/s on Small, ~39,000 words/s on Medium, and ~31,000 words/s on Large).

    • Synchronizing Every 100 Batches: Amortizing synchronization overhead produced near-optimal linear scaling:

    • 2 GPUs achieved a 2.0×2.0\times speedup over 1 GPU.

    • 4 GPUs achieved a 3.9×3.9\times to 4.0×4.0\times speedup over 1 GPU (reaching ~54,000 words/s on Small, ~46,000 words/s on Medium, and ~38,000 words/s on Large).

  12. Knowl 12 — Sequence-to-Sequence Video Captioning Benchmark

    empirical result

    Computational throughput for a sequence-to-sequence video captioning model (an LSTM conditioned on 1024-dimensional GoogLeNet frame features) was evaluated in Theano and TensorFlow on an NVIDIA Titan X GPU across three minibatch sizes:

    • Batch size 32: Total minibatch processing time was ~290 ms for Theano (~60 ms forward, ~230 ms backward) versus ~310 ms for TensorFlow (~100 ms forward, ~210 ms backward).
    • Batch size 64: Total minibatch processing time was ~510 ms for Theano (~100 ms forward, ~410 ms backward) versus ~510 ms for TensorFlow (~140 ms forward, ~370 ms backward).
    • Batch size 128: Total minibatch processing time was ~920 ms for Theano (~190 ms forward, ~730 ms backward) versus ~860 ms for TensorFlow (~230 ms forward, ~630 ms backward).

    Theano achieved faster forward pass computation across all batch sizes, while TensorFlow exhibited faster backward pass computation, resulting in comparable overall execution time. Theano configured with fast_compile could not complete this benchmark due to excessive memory consumption.

  13. Knowl 13 — Interactive Visualization and Diagnostic Tooling

    model/method

    Theano incorporates dedicated diagnostic and introspection tools for debugging symbolic graphs:

    • d3viz Interactive Visualizer: Renders computation graphs into interactive D3.js and Graphviz HTML representations. Features include execution time profiling heatmaps (color-coding nodes by duration), memory view indicators (blue edges denote zero-copy memory views, red edges denote destructive in-place modifications), and interactive expansion/collapse of nested subgraphs (such as OpFromGraph nodes).

    • Test Values (test_value): Binds concrete numeric arrays to symbolic inputs at graph creation time, eagerly evaluating shapes and values of intermediate nodes during graph construction to detect dimensionality and shape mismatches before compilation.

    • NanGuardMode: An execution mode that monitors all Apply node inputs and outputs at runtime, immediately throwing an exception upon encountering NaN, infinity (±∞\pm\infty), or abnormal numerical values.

    • PdbBreakPoint Op: A conditional operation embedded in the symbolic graph that monitors a symbolic boolean expression; when the condition evaluates to true, execution halts and drops into the interactive Python debugger (pdb) with access to specified monitored tensor values.

  14. Knowl 14 — Architectural Limitations and Scaling Bottlenecks in Theano

    limitation

    Several architectural limitations constrain Theano's scalability:

    • Python Global Interpreter Lock (GIL): Reliance on Python C API wrappers for tensor manipulation prevents multithreaded intra-process CPU execution, requiring multi-process architectures to utilize multiple CPU cores or GPUs.

    • Supra-Linear Optimization Time: The graph optimizer applies local rewrites iteratively until convergence. On large computation graphs, the optimization phase scales supra-linearly with node count, resulting in long compilation times.

    • Redundant Code Compilation: Ops generate separate C++/CUDA modules for variations in compile-time properties (e.g., specific data types, inplace settings, device IDs) rather than passing configurations dynamically at runtime, causing compilation delays and disk load.

    • Control Flow via Subgraphs: Recurrence and branching rely on isolated subgraphs (Scan, ifelse) rather than native cyclic dataflow primitives (e.g., switch and merge nodes), limiting dynamic execution flexibility.

Coverage note — No substantial contributed material was omitted; general summaries of third-party libraries built on top of Theano (e.g., Keras, Blocks, PyMC3) and standard background comparisons were excluded.

References

  1. 1.James Bergstra, Olivier Breuleux, Fred́ eric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David ́ Warde-Farley, and Yoshua Bengio, “Theano: A CPU and GPU math expression compiler,” in Proceedings of the Python for Scientific Computing Conference (SciPy) (2010).
  2. 2.James Bergstra, Fred́ eric Bastien, Olivier Breuleux, Pascal Lamblin, Razvan Pascanu, Olivier Delalleau, Guillaume Desjardins, David ́ Warde-Farley, Ian J. Goodfellow, Arnaud Bergeron, and Yoshua Bengio, “Theano: Deep learning on GPUs with Python,” in Big Learning Workshop, NIPS (2011).
  3. 3.Fred́ eric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua ́ Bengio, “Theano: New features and speed improvements,” Deep Learning and Unsupervised Feature Learning Workshop, NIPS (2012).
  4. 4.Ronan Collobert, Koray Kavukcuoglu, and Clement Farabet, “Torch7: A matlab-like environment for machine learning,” in ́ Big Learning Workshop, NIPS (2011).
  5. 5.Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, ́ Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, ́ Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” (2015), software available from tensorflow.org.
  6. 6.Stefan van der Walt, S. Chris Colbert, and Gael Varoquaux, “The NumPy array: A structure for efficient numerical computation,” Computing in Science and Eng. 13, 22–30 (2011).
  7. 7.Eric Jones, Travis Oliphant, Pearu Peterson, et al., “SciPy: Open source scientific tools for Python,” (2001–), [Online; accessed 2016- 04-19].
  8. 8.Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu, James Bergstra, Fred́ eric ́ Bastien, and Yoshua Bengio, “Pylearn2: A machine learning research library,” arXiv e-prints abs/1308.4214 (2013).
  9. 9.Bart van Merrienboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde-Farley, Jan Chorowski, and Yoshua ¨ Bengio, “Blocks and Fuel: Frameworks for deep learning,” arXiv e-prints abs/1506.00619 (2015).
  10. 10.Sander Dieleman, Jan Schluter, Colin Raffel, Eben Olson, Søren Kaae Sønderby, Daniel Nouri, Daniel Maturana, Martin Thoma, Eric ¨ Battenberg, Jack Kelly, Jeffrey De Fauw, Michael Heilman, diogo149, Brian McFee, Hendrik Weideman, takacsg84, peterderivaz, Jon, instagibbs, Dr. Kashif Rasul, CongLiu, Britefury, and Jonas Degrave, “Lasagne: First release.” (2015).
  11. 11.François Chollet, “Keras,” https://github.com/fchollet/keras (2015).
  12. 12.John Salvatier, Thomas V. Wiecki, and Christopher Fonnesbeck, “Probabilistic programming in Python using PyMC3,” PeerJ Computer Science 2, e55 (2016).
  13. 13.Arvind and David E. Culler, “Dataflow architectures,” Annual Review of Computer Science 1, 225–253 (1986).
  14. 14.Barak A. Pearlmutter, “Fast exact multiplication by the Hessian,” Neural Computation 6, 147–160 (1994).
  15. 15.Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang, “MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems,” arXiv e-prints abs/1512.01274 (2015).
  16. 16.Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell, “Caffe: Convolutional architecture for fast feature embedding,” arXiv e-prints abs/1408.5093 (2014).
  17. 17.Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton, “Chainer: a next-generation open source framework for deep learning,” in Workshop on Machine Learning Systems (LearningSys), NIPS (2015).
  18. 18.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems (2012) pp. 1097–1105.
  19. 19.V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” arXiv e-prints abs/1603.07285 (2016).
  20. 20.Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer, “cuDNN: Efficient primitives for deep learning,” arXiv e-prints abs/1410.0759 (2014).
  21. 21.Fred́ eric Bastien, Arnaud Bergeron, Andreas Kl ́ ockner, Pascal Vincent, and Yoshua Bengio, “A common GPU n-dimensional array for ¨ Python and C,” in Big Learning Workshop, NIPS (2011).
  22. 22.Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc V. Le, and Andrew Y. Ng, “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems (2012) pp. 1223–1231.
  23. 23.Sixin Zhang, Anna E Choromanska, and Yann LeCun, “Deep learning with elastic averaging SGD,” in Advances in Neural Information Processing Systems (2015) pp. 685–693.
  24. 24.Alex Krizhevsky, “One weird trick for parallelizing convolutional neural networks,” arXiv e-prints abs/1404.5997 (2014).
  25. 25.Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, and Yann LeCun, “OverFeat: Integrated recognition, ¨ localization and detection using convolutional networks,” arXiv e-prints abs/1312.6229 (2013).
  26. 26.Karen Simonyan and Andrew Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv e-prints abs/1409.1556 (2014).
  27. 27.Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, “Going deeper with convolutions,” in Computer Vision and Pattern Recognition (CVPR) (2015).
  28. 28.Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals, “Recurrent neural network regularization,” arXiv e-prints abs/1409.2329 (2014).
  29. 29.Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville, “Describing videos by exploiting temporal structure,” in Computer Vision (ICCV), 2015 IEEE International Conference on (IEEE, 2015).
  30. 30.Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W. Keckler, “Virtualizing Deep Neural Networks for Memory-Efficient Neural Network Design,” arXiv e-prints abs/1602.08124 (2016).
  31. 31.Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin, “训Training deep nets with sublinear memory cost,” arXiv e-prints abs/1604.06174 (2016).
  32. 32.SymPy Development Team, SymPy: Python library for symbolic mathematics (2016).

Citation

MLA
Team, T. T. D., et al. “Theano: A Python Framework for Fast Computation of Mathematical Expressions”. arXiv, 2016, http://arxiv.org/abs/1605.02688v1.
APA
Team, T. T. D., Al-Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., Belopolsky, A., Bengio, Y., Bergeron, A., Bergstra, J., Bisson, V., Snyder, J. B., Bouchard, N., Boulanger-Lewandowski, N., Bouthillier, X., … Zhang, Y. (2016). Theano: A Python framework for fast computation of mathematical expressions. arXiv. http://arxiv.org/abs/1605.02688v1
Chicago
Team, T. T. D., R. Al-Rfou, G. Alain, et al. 2016. “Theano: A Python Framework for Fast Computation of Mathematical Expressions”. arXiv. http://arxiv.org/abs/1605.02688v1.
Harvard
Team, T.T.D. et al. (2016) “Theano: A Python framework for fast computation of mathematical expressions”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1605.02688v1.
Vancouver
1. Team TTD, Al-Rfou R, Alain G, et al (2016) Theano: A Python framework for fast computation of mathematical expressions. arXiv

BibTeX

@article{team2016theano,
  title = {Theano: A Python framework for fast computation of mathematical expressions},
  author = {Team, The Theano Development and Al-Rfou, Rami and Alain, Guillaume and Almahairi, Amjad and Angermueller, Christof and Bahdanau, Dzmitry and Ballas, Nicolas and Bastien, Frédéric and Bayer, Justin and Belikov, Anatoly and Belopolsky, Alexander and Bengio, Yoshua and Bergeron, Arnaud and Bergstra, James and Bisson, Valentin and Snyder, Josh Bleecher and Bouchard, Nicolas and Boulanger-Lewandowski, Nicolas and Bouthillier, Xavier and Brébisson, Alexandre de and Breuleux, Olivier and Carrier, Pierre-Luc and Cho, Kyunghyun and Chorowski, Jan and Christiano, Paul and Cooijmans, Tim and Côté, Marc-Alexandre and Côté, Myriam and Courville, Aaron and Dauphin, Yann N. and Delalleau, Olivier and Demouth, Julien and Desjardins, Guillaume and Dieleman, Sander and Dinh, Laurent and Ducoffe, Mélanie and Dumoulin, Vincent and Kahou, Samira Ebrahimi and Erhan, Dumitru and Fan, Ziye and Firat, Orhan and Germain, Mathieu and Glorot, Xavier and Goodfellow, Ian and Graham, Matt and Gulcehre, Caglar and Hamel, Philippe and Harlouchet, Iban and Heng, Jean-Philippe and Hidasi, Balázs and Honari, Sina and Jain, Arjun and Jean, Sébastien and Jia, Kai and Korobov, Mikhail and Kulkarni, Vivek and Lamb, Alex and Lamblin, Pascal and Larsen, Eric and Laurent, César and Lee, Sean and Lefrancois, Simon and Lemieux, Simon and Léonard, Nicholas and Lin, Zhouhan and Livezey, Jesse A. and Lorenz, Cory and Lowin, Jeremiah and Ma, Qianli and Manzagol, Pierre-Antoine and Mastropietro, Olivier and McGibbon, Robert T. and Memisevic, Roland and Merriënboer, Bart van and Michalski, Vincent and Mirza, Mehdi and Orlandi, Alberto and Pal, Christopher and Pascanu, Razvan and Pezeshki, Mohammad and Raffel, Colin and Renshaw, Daniel and Rocklin, Matthew and Romero, Adriana and Roth, Markus and Sadowski, Peter and Salvatier, John and Savard, François and Schlüter, Jan and Schulman, John and Schwartz, Gabriel and Serban, Iulian Vlad and Serdyuk, Dmitriy and Shabanian, Samira and Simon, Étienne and Spieckermann, Sigurd and Subramanyam, S. Ramana and Sygnowski, Jakub and Tanguay, Jérémie and Tulder, Gijs van and Turian, Joseph and Urban, Sebastian and Vincent, Pascal and Visin, Francesco and Vries, Harm de and Warde-Farley, David and Webb, Dustin J. and Willson, Matthew and Xu, Kelvin and Xue, Lijun and Yao, Li and Zhang, Saizheng and Zhang, Ying},
  year = {2016},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1605.02688v1},
  eprint = {1605.02688}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission