Back

Learning to learn by gradient descent by gradient descent

Marcin AndrychowiczMisha DenilSergio Gomez ColmenarejoMatthew W. HoffmanDavid PfauTom SchaulNando de Freitas

article2016NeurIPS2,241 citations

Presents a meta-learning framework that uses recurrent neural networks to automatically learn optimization algorithms via gradient descent, showing that learned update rules can outperform traditional hand-designed optimizers across various machine learning tasks.

Listen

Modern machine learning relies heavily on optimization algorithms to train complex models. While machine learning has successfully transitioned from manual feature engineering to automated, data-driven feature learning, the optimization algorithms used to train these systems are still designed by hand. These general-purpose, hand-crafted optimizers often fail to exploit the specialized structures inherent in specific problem domains. The article addresses this limitation by investigating whether the design of optimization algorithms can itself be cast as an automated learning problem.

The primary objective of the article is to demonstrate that an optimization algorithm parameterized by a recurrent neural network can be trained via gradient descent to optimize other mathematical functions and machine learning models, effectively learning domain-specific update strategies that surpass standard hand-designed optimizers.

To achieve this, the authors implemented an optimizer using a coordinatewise recurrent network—specifically a Long Short-Term Memory (LSTM) architecture. In this setup, a small neural network learns to propose parameter updates based on incoming gradient information while maintaining an internal state across optimization steps. This coordinate-level design enables the optimizer to scale to tens of thousands of parameters without an explosion in model size. The approach was evaluated across synthetic convex quadratic functions, multilayer perceptrons on image classification tasks (MNIST), convolutional neural networks on more complex image classification (CIFAR-10), and neural style transfer problems.

The findings demonstrate that the learned optimizers consistently converge faster and reach lower loss values than established, hand-crafted baselines such as standard gradient descent, momentum-based methods, and adaptive step-size algorithms like ADAM and RMSprop. First, on synthetic quadratic tasks, the learned optimizer converged substantially faster than all baseline methods. Second, when applied to neural networks, the learned optimizer generalized well to variations in network size, layer count, and extended training iterations, though it struggled when tested on activation functions fundamentally different from its training regime, such as switching from sigmoid to rectified linear units. Third, in convolutional network tasks, splitting the optimizer into separate modules for convolutional and fully connected layers allowed rapid convergence, even when transferring to datasets with unseen class labels. Finally, in neural art styling tasks, an optimizer trained on low-resolution images with a single artistic style successfully generalized to double the resolution and entirely new artistic styles, handling nearly 50,000 parameters effectively.

These results imply that automating optimizer design can significantly improve optimization efficiency and reduce the computational time required to train specialized models. By treating algorithm design as a transfer learning problem, organizations can develop customized optimizers that exploit repeated problem structures across workflows, rather than relying on generic rules. However, the inability of the learned optimizer to generalize across fundamentally different non-linear activation functions highlights clear operational boundaries.

Organizations seeking to apply these techniques should consider learned optimizers for recurring, structurally similar optimization workloads, such as continuous neural rendering or specialized domain retraining. Prior to deployment, teams should conduct pilot tests to verify performance and establish safeguards, as the learned optimizers should not be applied to architectures or mathematical properties that diverge substantially from the training distribution. Further research is warranted to expand cross-architecture transferability and explore richer coordinate-communication mechanisms.

CIFARGoogleUniversity of Oxfordneural networksoptimizationoptimization problemsStepOptimizerLearning
arXiv: 1606.04474
  • Paper: Optimization as a Model for Few-Shot Learning, S. Ravi et al. (2017). This paper applies the learned optimization concepts introduced in the source directly to the challenge of few-shot image classification.
  • Paper: On First-Order Meta-Learning Algorithms, Alex Nichol et al. (2018). This work continues the exploration of meta-learning initiated by the source by investigating first-order approximations to simplify gradient-based adaptation.
Cover for Learning to learn by gradient descent by gradient descent

Abstract

The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.

Topics

neural networksoptimizationoptimization problemsadaptive algorithms

Keywords

StepOptimizerLearningLSTMGradientGradient DescentLSTM OptimizerNeuralDifferentLearn

Table of Contents

  • 1 Introduction
  • 1.1 Transfer learning and generalization
  • 1.2 A brief history and related work
  • 2 Learning to learn with recurrent neural networks
  • 2.1 Coordinatewise LSTM optimizer
  • 3 Experiments
  • 3.1 Quadratic functions
  • 3.2 Training a small neural network on MNIST
  • 3.3 Training a convolutional network on CIFAR-10
  • 3.4 Neural Art
  • 4 Conclusion
  • References
  • A Gradient preprocessing
  • B Visualizations
  • C Neural Art
  • D Information sharing between coordinates

Knowls

  1. Knowl 1 — Meta-Learning Formulation for Learned Gradient Optimizers

    model/method

    The task of designing an optimization algorithm is framed as a meta-learning problem where the update rule is parameterized by a recurrent neural network (RNN) mm with parameters ϕ\phi and internal hidden state hth_t. Given an optimizee objective function f(θ)f(\theta) with parameters θ∈Θ\theta \in \Theta, the parameter update at step tt is generated according to:

    θt+1=θt+gt,[gtht+1]=m(∇t,ht,ϕ)\theta_{t+1} = \theta_t + g_t, \quad \begin{bmatrix} g_t \\ h_{t+1} \end{bmatrix} = m(\nabla_t, h_t, \phi)

    where ∇t=∇θf(θt)\nabla_t = \nabla_\theta f(\theta_t).

    To train the optimizer across a distribution of functions ff, the meta-objective minimizes the expected weighted cumulative loss over a finite optimization trajectory horizon TT:

    L(ϕ)=Ef[∑t=1Twtf(θt)]\mathcal{L}(\phi) = \mathbb{E}_f \left[ \sum_{t=1}^T w_t f(\theta_t) \right]

    where wt∈R≥0w_t \in \mathbb{R}_{\ge 0} are non-negative weights associated with each step (set to wt=1w_t = 1 for all tt to enable efficient training on intermediate trajectories via truncated backpropagation through time).

  2. Knowl 2 — Coordinatewise Architecture for Scalable Recurrent Optimizers

    model/method

    To scale a learned optimizer to optimizees with tens of thousands of parameters without requiring an intractable number of optimizer parameters or state dimensions, the recurrent optimizer operates coordinatewise on the optimizee parameter vector θ\theta.

    A single small recurrent network (such as a two-layer Long Short-Term Memory network with 20 hidden units per layer) processes one parameter coordinate k∈{1,…,dim⁡(θ)}k \in \{1, \dots, \dim(\theta)\} at a time. The network receives the scalar gradient component ∇tk=∂f∂θk(θt)\nabla_t^k = \frac{\partial f}{\partial \theta^k}(\theta_t) and its coordinate-specific hidden state htkh_t^k, producing the scalar parameter step gtkg_t^k and updated state ht+1kh_{t+1}^k.

    The optimizer parameters ϕ\phi are shared across all coordinates, while each coordinate maintains its own independent hidden state. This makes the optimizer invariant to coordinate order permutations and allows the optimizer model size to remain independent of the optimizee's parameter count.

  3. Knowl 3 — Zero-Second-Derivative Approximation for Optimizer Meta-Gradients

    assumption

    When optimizing the meta-loss L(ϕ)\mathcal{L}(\phi) with respect to optimizer parameters ϕ\phi using backpropagation through time on the unrolled computational graph, gradients flowing through the optimizee gradient evaluation ∇t=∇θf(θt)\nabla_t = \nabla_\theta f(\theta_t) to ϕ\phi are dropped by assuming:

    ∂∇t∂ϕ=0\frac{\partial \nabla_t}{\partial \phi} = 0

    This assumption treats the evaluated gradient ∇t\nabla_t as an external input at each recurrent time step rather than a function of the historical parameters θt(ϕ)\theta_t(\phi). This avoids computing second derivatives (Hessian-vector products) of the optimizee objective ff during meta-gradient computation.

  4. Knowl 4 — Logarithmic Gradient Preprocessing Transformation

    equation

    To accommodate input gradient magnitudes that vary across multiple orders of magnitude without saturating neural network activations, individual gradient components ∇\nabla are mapped to a two-dimensional vector (∇(1),∇(2))(\nabla^{(1)}, \nabla^{(2)}) using the transformation:

    ∇↦{(log⁡(∣∇∣)p, sgn⁡(∇))if ∣∇∣≥e−p(−1, ep∇)otherwise\nabla \mapsto \begin{cases} \left( \frac{\log(|\nabla|)}{p}, \, \operatorname{sgn}(\nabla) \right) & \text{if } |\nabla| \ge e^{-p} \\ \left( -1, \, e^p \nabla \right) & \text{otherwise} \end{cases}

    where sgn⁡(⋅)\operatorname{sgn}(\cdot) denotes the sign function, and p>0p > 0 is a scale threshold parameter controlling the cutoff for small gradients (set to p=10p = 10 across experiments). The first component encodes a normalized log-magnitude for large values, and the second component carries sign or scaled linear magnitude information.

  5. Knowl 5 — Global Averaging Cells and NTM-BFGS for Inter-Coordinate Communication

    model/method

    To capture cross-coordinate correlations and emulate second-order optimization methods (such as BFGS), coordinatewise LSTMs are augmented with communication mechanisms:

    1. Global Averaging Cells (GACs): A subset of LSTM hidden units in each layer whose outgoing activations are averaged across all coordinates at each time step, allowing the optimizer to compute global statistics such as L2L_2 gradient norms for adaptive step clipping.
    2. NTM-BFGS Architecture: The optimizer controller (an LSTM+GAC) is augmented with an external dynamic memory matrix Mt∈Rn×nM_t \in \mathbb{R}^{n \times n} representing an inverse Hessian approximation, accessed via Neural Turing Machine (NTM) operations. Coordinatewise read heads produce read vectors rtr_t yielding input it=Mtrti_t = M_t r_t, and coordinatewise write heads produce vectors at,bta_t, b_t to execute low-rank outer-product memory updates:

    Mt+1=Mt+atbtTM_{t+1} = M_t + a_t b_t^T

    1. NTM-L-BFGS: In memory-constrained settings where storing MtM_t is infeasible, the optimizer maintains a sliding temporal history of write vectors {(at,bt)}\{(a_t, b_t)\} to execute low-rank matrix-vector products efficiently without materializing the full matrix.

  6. Knowl 6 — Layer-Specific Recurrent Optimizers for Heterogeneous Network Architectures

    model/method

    For neural network architectures containing structurally distinct parameter classes—such as convolutional filters and fully connected weight matrices—a single shared coordinatewise LSTM optimizer can suffer from disparate parameter dynamics.

    To address this, the optimizer is partitioned into multiple distinct LSTMs with separate parameters ϕconv\phi_{\text{conv}} and ϕfc\phi_{\text{fc}}. Each parameter set operates coordinatewise with separate hidden states, but weights are shared exclusively among optimizee parameters of the same functional layer type (all convolutional coordinates share ϕconv\phi_{\text{conv}}, and all fully connected coordinates share ϕfc\phi_{\text{fc}}).

  7. Knowl 7 — MNIST Generalization Across Horizons, Architectures, and Activation Functions

    empirical result

    A two-layer coordinatewise LSTM optimizer (20 hidden units per layer) trained for 100 steps on an MLP with 1 hidden layer of 20 sigmoid units on MNIST achieved the following generalization properties when tested against tuned baselines (SGD, NAG, ADAM, RMSprop):

    • Time Horizon: When evaluated for 200 steps (twice its training horizon), the LSTM optimizer continued to decrease the loss monotonically and maintained its performance margin over standard optimizers.
    • Architecture Dimensions: When transferred to an MLP with 40 hidden units or an MLP with 2 hidden layers (using sigmoid activations), the learned optimizer outperformed hand-designed optimizers.
    • Activation Shift: When transferred to an MLP using ReLU activations instead of sigmoid activations, the learned optimizer failed to optimize effectively due to the significant shift in gradient dynamics.

  8. Knowl 8 — Resolution and Style Generalization in Neural Art Transfer

    empirical result

    An LSTM optimizer trained on the neural artistic style transfer loss (matching content, style, and total variation regularization) using 64×6464 \times 64 content images from ImageNet and 1 fixed style (12,288 optimizee parameters, unrolled for 32 steps over 128 total steps) was evaluated on out-of-distribution styling tasks:

    • Same-Domain Performance: On unseen 64×6464 \times 64 content images with the training style, the LSTM optimizer converged faster and achieved lower objective loss over 128 steps than tuned hand-designed baselines (ADAM, RMSprop, NAG, SGD).
    • Zero-Shot Transfer: When tested on unseen content images styling with a novel artistic style at 128×128128 \times 128 resolution (49,152 parameters, a fourfold parameter increase), the LSTM optimizer generalized successfully and continued to outperform the baseline optimizers.

  9. Knowl 9 — Disjoint-Class Transfer on CIFAR-10 Convolutional Networks

    empirical result

    A multi-LSTM optimizer (using separate LSTMs for convolutional and fully connected layers) trained to minimize classification loss on CIFAR-10 using a network with 3 convolutional layers (batch normalization, ReLU) and 1 dense layer (32 units) outperformed tuned ADAM, RMSprop, SGD, and NAG in convergence speed.

    Furthermore, when the optimizer was meta-trained exclusively on a subset of CIFAR-10 classes (LSTM-sub) and tested on disjoint label subsets (CIFAR-2 and CIFAR-5), it transferred with no noticeable loss in optimization efficiency, substantially outperforming standard hand-designed optimizers on the new datasets.

  10. Knowl 10 — Learned Optimizer Convergence on Quadratic Functions

    empirical result

    On synthetic 10-dimensional quadratic objective functions of the form f(θ)=∥Wθ−y∥22f(\theta) = \|W\theta - y\|_2^2, where elements of W∈R10×10W \in \mathbb{R}^{10 \times 10} and y∈R10y \in \mathbb{R}^{10} were sampled IID from a Gaussian distribution, a two-layer coordinatewise LSTM optimizer (20 units per layer, trained for 100 steps with 20-step unrolling) converged significantly faster and attained a lower final loss within 100 steps compared to tuned baseline algorithms including ADAM, RMSprop, SGD, and Nesterov accelerated gradient (NAG).

Coverage note — Qualitative trajectory visualizations of optimizer step directions as a function of gradient values (Appendix B) and baseline hyperparameter random search grid configurations were omitted as secondary supporting analyses.

References

  1. 1.F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning, 4(1):1–106, 2012.
  2. 2.S. Bengio, Y. Bengio, and J. Cloutier. On the search for new learning rules for ANNs. Neural Processing Letters, 2(4):26–30, 1995.
  3. 3.Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. Université de Montréal, Département d’informatique et de recherche opérationnelle, 1990.
  4. 4.Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu. Advances in optimizing recurrent networks. In International Conference on Acoustics, Speech and Signal Processing, pages 8624–8628. IEEE, 2013.
  5. 5.F. Bobolas. brain-neurons, 2009. URL https://www.flickr.com/photos/fbobolas/3822222947. Creative Commons Attribution-ShareAlike 2.0 Generic.
  6. 6.N. E. Cotter and P. R. Conwell. Fixed-weight networks can learn. In International Joint Conference on Neural Networks, pages 553–559, 1990.
  7. 7.C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In Association for the Advancement of Artificial Intelligence, 2016.
  8. 8.J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009.
  9. 9.D. L. Donoho. Compressed sensing. Transactions on Information Theory, 52(4):1289–1306, 2006.
  10. 10.J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, 2011.
  11. 11.L. A. Feldkamp and G. V. Puskorius. A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification. Proceedings of the IEEE, 86(11): 2259–2277, 1998.
  12. 12.L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015.
  13. 13.A. Graves, G. Wayne, and I. Danihkela. Neural Turing machines. arXiv Report 1410.5401, 2014.
  14. 14.S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  15. 15.S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87–94. Springer, 2001.
  16. 16.D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  17. 17.A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
  18. 18.B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like people. arXiv Report 1604.00289, 2016.
  19. 19.T. Maley. neuron, 2011. URL https://www.flickr.com/photos/taylortotz101/6280077898. Creative Commons Attribution 2.0 Generic.
  20. 20.J. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning, pages 2408–2417, 2015.
  21. 21.G. L. Nemhauser and L. A. Wolsey. Integer and combinatorial optimization. John Wiley & Sons, 1988.
  22. 22.Y. Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady, volume 27, pages 372–376, 1983.
  23. 23.J. Nocedal and S. Wright. Numerical optimization. Springer Science & Business Media, 2006.
  24. 24.M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In International Conference on Neural Networks, pages 586–591, 1993.
  25. 25.T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks, pages 59–63. IEEE, 2000.
  26. 26.A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning, 2016.
  27. 27.J. Schmidhuber. Evolutionary principles in self-referential learning; On learning how to learn: The meta-meta-... hook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
  28. 28.J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992.
  29. 29.J. Schmidhuber. A neural network that embeds its own meta-levels. In International Conference on Neural Networks, pages 407–412. IEEE, 1993.
  30. 30.J. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28(1):105–130, 1997.
  31. 31.N. N. Schraudolph. Local gain adaptation in stochastic gradient descent. In International Conference on Artificial Neural Networks, volume 2, pages 569–574, 1999.
  32. 32.R. S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Association for the Advancement of Artificial Intelligence, pages 171–176, 1992.
  33. 33.S. Thrun and L. Pratt. Learning to learn. Springer Science & Business Media, 1998.
  34. 34.T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4:2, 2012.
  35. 35.P. Tseng. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. Journal on Optimization, 8(2):506–531, 1998.
  36. 36.D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. Transactions on Evolutionary Computation, 1(1):67–82, 1997.
  37. 37.A. S. Younger, P. R. Conwell, and N. E. Cotter. Fixed-weight on-line learning. Transactions on Neural Networks, 10(2):272–283, 1999.
  38. 38.A. S. Younger, S. Hochreiter, and P. R. Conwell. Meta-learning with backpropagation. In International Joint Conference on Neural Networks, 2001.

Citation

MLA
Andrychowicz, M., et al. “Learning to Learn by Gradient Descent by Gradient Descent”. arXiv, 2016, http://arxiv.org/abs/1606.04474v2.
APA
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., & Freitas, N. de . (2016). Learning to learn by gradient descent by gradient descent. arXiv. http://arxiv.org/abs/1606.04474v2
Chicago
Andrychowicz, M., M. Denil, S. Gomez, et al. 2016. “Learning to Learn by Gradient Descent by Gradient Descent”. arXiv. http://arxiv.org/abs/1606.04474v2.
Harvard
Andrychowicz, M. et al. (2016) “Learning to learn by gradient descent by gradient descent”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1606.04474v2.
Vancouver
1. Andrychowicz M, Denil M, Gomez S, Hoffman MW, Pfau D, Schaul T, Shillingford B, Freitas N de (2016) Learning to learn by gradient descent by gradient descent. arXiv

BibTeX

@article{andrychowicz2016learning,
  title = {Learning to learn by gradient descent by gradient descent},
  author = {Andrychowicz, Marcin and Denil, Misha and Gomez, Sergio and Hoffman, Matthew W. and Pfau, David and Schaul, Tom and Shillingford, Brendan and Freitas, Nando de},
  year = {2016},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1606.04474v2},
  eprint = {1606.04474}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission