Optimization as a Model for Few-Shot Learning

S. RaviH. Larochelle

article2017ICLR3,726 citations

Proposes an LSTM-based meta-learner that learns both parameter initializations and gradient-based update rules, enabling neural networks to rapidly adapt to new classification tasks from only a few training examples.

Listen

The paper addresses the challenge of few-shot learning, in which a model must classify new examples after seeing only one or five labeled instances per class. Standard deep networks trained with gradient descent perform poorly in this regime because typical optimizers require many iterations and examples to converge, and each new task normally begins from a random initialization that slows progress further.

The work sets out to learn an optimization procedure itself, rather than hand-design one, so that a learner network can reach good performance after a fixed, small number of updates on each new task.

The authors frame the problem as meta-learning over many small training sets drawn from Mini-ImageNet. They replace the usual gradient-descent update with an LSTM whose cell state stores the learner’s parameters. The LSTM is trained end-to-end to produce both a useful starting point for the learner and a sequence of parameter updates that minimize loss on held-out test examples from the same small task. Experiments compare the resulting meta-learner against nearest-neighbor and fine-tuning baselines as well as Matching Networks on 5-class, 1-shot and 5-shot classification.

On the 5-shot task the meta-learner reaches 60 percent accuracy, materially above Matching Networks (52 percent) and the baselines (49–52 percent). On the harder 1-shot task it achieves 44 percent, statistically indistinguishable from the strongest Matching-Network variant. Visualizations of the learned input and forget gates show that the meta-learner adopts a consistent weight-decay strategy while still varying its updates across tasks, indicating it has not collapsed to a fixed learning-rate schedule.

These results demonstrate that an explicitly learned optimizer can capture both a task-common initialization and an adaptive update rule that together overcome the data and iteration limits of conventional training. Consequently, models can be deployed in domains where collecting millions of labels is impractical, and the same meta-learner can be reused across many related but distinct classification problems.

The authors note that the current implementation is tuned only for the narrow few-shot, few-class setting. Extending the approach to a wider range of example counts and class cardinalities is identified as the next step required before broader deployment. The main limitations are the use of a custom Mini-ImageNet split and a gradient-independence assumption that simplifies training but may affect scaling behavior on larger models.

Princeton University Twitter few-shot learning high capacity long short term memory neural networks Meta‑Learner LSTM Few‑Shot Learning Gradient Descent

twitter-research/meta-learning-lstm Ravi et al (2017).pdf

Paper: On the importance of initialization and momentum in deep learning, Ilya Sutskever et al. (2013). Understanding how properly tuned momentum and sparse initialization enable first-order training of deep networks provides necessary foundation for the source's meta-learned optimization.
Paper: An overview of gradient descent optimization algorithms, Sebastian Ruder (2016). Reviewing standard gradient descent variants and adaptive optimization algorithms helps clarify the mechanics of gradient-based parameter updates that the source's meta-learner seeks to generate.
Book: Introduction to stochastic gradient methods, Simon Weissmann (2026). Familiarity with the mathematical foundations and convergence properties of stochastic gradient methods establishes the theoretical baseline for analyzing gradient-based learning.

Paper: Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Chelsea Finn et al. (2017). Building directly upon the source's idea of optimizing initialization for fast gradient-based adaptation, this work introduces Model-Agnostic Meta-Learning (MAML) as a streamlined and broadly applicable optimization-based meta-learner.

Cover for Optimization as a Model for Few-Shot Learning

Abstract

Though deep neural networks have shown great success in the large data domain, they generally perform poorly on few-shot learning tasks, where a model has to quickly generalize after seeing very few examples from each class. The general belief is that gradient-based optimization in high capacity models requires many iterative steps over many examples to perform well. Here, we propose an LSTM-based meta-learner model to learn the exact optimization algorithm used to train another learner neural network in the few-shot regime. The parametrization of our model allows it to learn appropriate parameter updates specifically for the scenario where a set amount of updates will be made, while also learning a general initialization of the learner network that allows for quick convergence of training. We demonstrate that this meta-learning model is competitive with deep metric-learning techniques for few-shot learning.

Topics

few-shot learning high capacity long short term memory neural networks meta-learning optimization optimization problems parameterization network architecture

Keywords

Meta‑Learner LSTM Few‑Shot Learning Gradient Descent Initialize Batch Normalization Matching Network Mini‑ImageNet Adam Parameter Sharing Transfer Learning

1 INTRODUCTION
2 TASK DESCRIPTION
3 MODEL
3.1 MODEL DESCRIPTION
3.2 PARAMETER SHARING & PREPROCESSING
3.3 TRAINING
3.3.1 GRADIENT INDEPENDENCE ASSUMPTION
3.3.2 INITIALIZATION OF META-LEARNER LSTM
3.4 BATCH NORMALIZATION
4 RELATED WORK
4.1 META-LEARNING
4.2 FEW-SHOT LEARNING
5 EVALUATION
5.1 EXPERIMENT RESULTS
5.2 VISUALIZATION OF META-LEARNER
6 CONCLUSION
ACKNOWLEDGMENTS
REFERENCES

Knowls

Knowl 1 — LSTM Meta-Learner Parameter Update Formulation
model/method
The meta-learner frames the optimization of a base neural network (the "learner") as a recurrent update analogous to an LSTM cell state update. In standard gradient descent, learner parameters $\theta \in \mathbb{R}^d$ are updated as:

$\theta_t = \theta_{t-1} - \alpha_t \nabla_{\theta_{t-1}} \mathcal{L}_t$

where $\alpha_t$ is the learning rate, $\mathcal{L}_t$ is the training loss at step $t$ , and $\nabla_{\theta_{t-1}} \mathcal{L}_t$ is the gradient of that loss. In an LSTM, the cell state updates according to:

$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$

By mapping the LSTM cell state to the learner parameters ( $c_t = \theta_t$ ) and the candidate cell state to the negative gradient ( $ilde{c}_t = -\nabla_{\theta_{t-1}} \mathcal{L}_t$ or $\nabla_{\theta_{t-1}} \mathcal{L}_t$ modulated by sign), the LSTM gating mechanisms control parameter updates:

$i_t = \sigma(W_I [\nabla_{\theta_{t-1}} \mathcal{L}_t, \mathcal{L}_t, \theta_{t-1}, i_{t-1}] + b_I)$

$f_t = \sigma(W_F [\nabla_{\theta_{t-1}} \mathcal{L}_t, \mathcal{L}_t, \theta_{t-1}, f_{t-1}] + b_F)$

where $W_I, W_F$ and $b_I, b_F$ are learnable weight matrices and bias vectors of the meta-learner, and $\sigma(\cdot)$ is the element-wise sigmoid function. The input gate $i_t$ acts as a coordinate-wise adaptive step size conditioned on the current gradient, loss, parameter value, and prior step size. The forget gate $f_t$ allows shrinking existing parameters (acting as adaptive weight decay) to facilitate escaping poor local optima when loss is high but gradients are small.
Knowl 2 — Episode-Based Meta-Learner Training Algorithm
algorithm
The meta-learner LSTM is trained across a meta-dataset $\mathcal{D}_{\text{meta-train}}$ composed of episodic classification tasks. For each episode, a task dataset $D = (D_{\text{train}}, D_{\text{test}})$ is sampled, where $D_{\text{train}}$ contains $k$ examples per class for $N$ classes ( $k$ -shot, $N$ -class task) and $D_{\text{test}}$ contains validation examples for the same $N$ classes. The base learner model $M$ starts from learned initial parameters $\theta_0 = c_0$ and is updated for $T$ steps using the meta-learner $R(\cdot; \Theta)$ . The meta-learner parameters $\Theta$ (including the LSTM weights and base learner initial parameters $c_0$ ) are updated via gradient descent on the learner's test loss $\mathcal{L}_{\text{test}}$ evaluated on $D_{\text{test}}$ .
Input: Meta-training set $\mathcal{D}_{\text{meta-train}}$ , Learner $M$ with parameters $\theta$ , Meta-Learner $R$ with parameters $\Theta$ , number of meta-training iterations $n$ , number of inner update steps $T$
Output: Trained meta-learner parameters $\Theta$
$\Theta_0 \leftarrow \text{random initialization}$
for $d = 1$ to $n$ do
     $D_{\text{train}}, D_{\text{test}} \leftarrow \text{sample random dataset from } \mathcal{D}_{\text{meta-train}}$
     $\theta_0 \leftarrow c_0$
    for $t = 1$ to $T$ do
         $X_t, Y_t \leftarrow \text{sample batch from } D_{\text{train}}$
         $\mathcal{L}_t \leftarrow \mathcal{L}(M(X_t; \theta_{t-1}), Y_t)$
         $c_t \leftarrow R((\nabla_{\theta_{t-1}} \mathcal{L}_t, \mathcal{L}_t); \Theta_{d-1})$
         $\theta_t \leftarrow c_t$
    end for
     $X, Y \leftarrow D_{\text{test}}$
     $\mathcal{L}_{\text{test}} \leftarrow \mathcal{L}(M(X; \theta_T), Y)$
    Update $\Theta_d$ using $\nabla_{\Theta_{d-1}} \mathcal{L}_{\text{test}}$
end for
return $\Theta_n$
Knowl 3 — Coordinate-wise Parameter Sharing and Gradient Preprocessing
model/method
To prevent an explosion of meta-learner parameters when training deep neural networks with tens of thousands of weights, the LSTM meta-learner shares its parameters across all individual parameter coordinates of the learner network. Each coordinate maintains its own hidden state and cell state, but uses the same transition matrices $W_I, W_F$ and biases $b_I, b_F$ .

To accommodate wide variations in gradient and loss scales across coordinates and iterations, a coordinate-wise preprocessing function is applied to both the gradient coordinates $\nabla_{\theta_{t-1}} \mathcal{L}_t$ and the loss $\mathcal{L}_t$ :

$x \mapsto \begin{cases} \left(\frac{\log(|x|)}{p}, \operatorname{sgn}(x)\right) & \text{if } |x| \ge e^{-p} \\ \left(-1, e^p x\right) & \text{otherwise} \end{cases}$

where $p = 10$ . This transformation separates magnitude on a logarithmic scale from the sign of each scalar input, ensuring stable inputs to the LSTM meta-learner.
Knowl 4 — Meta-Learned Base Model Initialization
model/method
Rather than initializing the base learner network randomly for each new few-shot task, the initial cell state $c_0$ of the LSTM meta-learner is treated as a learnable meta-parameter. Because $c_0 = \theta_0$ , optimizing $c_0$ alongside the meta-learner weights during meta-training yields a task-common initialization that captures inductive bias shared across all training tasks. This initialization provides an optimal starting point from which the base network can adapt to a new task in very few optimization updates (e.g., 5 to 12 gradient steps).
Knowl 5 — First-Order Gradient Independence Assumption in Meta-Optimization
assumption
In computing the gradient of the meta-objective $\mathcal{L}_{\text{test}}$ with respect to the meta-learner parameters $\Theta$ , the intermediate base learner losses $\mathcal{L}_t$ and gradients $\nabla_{\theta_{t-1}} \mathcal{L}_t$ formally depend on $\Theta$ . Computing the exact meta-gradient would require backpropagating through the gradient operator $\nabla_{\theta_{t-1}} \mathcal{L}_t$ , introducing computationally expensive second derivatives (Hessian-vector products). The meta-learner assumes that the contributions of $\nabla_{\Theta} \nabla_{\theta_{t-1}} \mathcal{L}_t$ are negligible and ignores them during backpropagation. This first-order approximation avoids second derivatives while preserving effective meta-training.
Knowl 6 — Task-Isolated Batch Normalization for Episodic Meta-Learning
model/method
Standard Batch Normalization accumulates global running averages of batch mean and variance during training and freezes them during evaluation. In few-shot meta-learning, accumulating statistics across test tasks leads to information leakage between tasks, while reusing static running averages from meta-training creates a train-test mismatch. To preserve identical conditions between meta-training and meta-testing without cross-task leakage, batch normalization statistics (mean and standard deviation) are collected locally per task dataset $D \in \mathcal{D}$ across meta-training, meta-validation, and meta-testing, and are reset when moving to the next dataset.
Knowl 7 — Meta-Learner LSTM Gate Bias Initialization Strategy
model/method
To ensure numerical stability at the start of meta-training, the LSTM meta-learner is initialized so that its initial update behavior approximates standard gradient descent with a small learning rate:
- The forget gate bias $b_F$ is set to a large positive value so that $f_t \approx 1$ , preserving prior learner parameters and facilitating gradient flow through time.
- The input gate bias $b_I$ is set to a large negative value so that $i_t \approx 0$ , ensuring the initial effective learning rate starts small to prevent initial divergence.
- The remaining LSTM weights are initialized with small random values.
Knowl 8 — Mini-ImageNet Few-Shot Classification Benchmark Protocol
experimental setup
The Mini-ImageNet benchmark consists of 100 randomly selected ImageNet classes with 600 images per class ( $84 \times 84$ resolution), split into 64 training classes, 16 validation classes, and 20 test classes. Tasks are framed as 5-class, $k$ -shot classification ( $k \in \{1, 5\}$ ) with 15 test examples per class in each episode ( $D_{\text{test}}$ has 75 examples).

The learner architecture is a 4-layer convolutional network (each layer having 32 $3 \times 3$ filters, Batch Normalization, ReLU, and $2 \times 2$ max pooling) followed by a linear classification layer with softmax. The meta-learner is a 2-layer LSTM where the first layer is standard and the second implements the coordinate-wise cell update. The meta-learner is optimized using ADAM with learning rate 0.001 and gradient clipping threshold 0.25, running 12 inner update steps for 1-shot tasks and 5 inner update steps for 5-shot tasks.

Knowl 9 — Few-Shot Classification Performance on Mini-ImageNet

data/table

Classification accuracies on 5-class Mini-ImageNet few-shot tasks comparing the LSTM meta-learner against transfer baselines and metric-learning approaches:

Model	5-class 1-shot	5-class 5-shot
Baseline-nearest-neighbor	38.9%	48.1%
Baseline-finetune	37.3%	51.9%
Matching Network	42.4%	50.9%
Matching Network FCE	45.1%	52.2%
Meta-Learner LSTM (Ours)	44.3%	60.0%

The LSTM meta-learner achieves 60.0% on 5-shot classification, outperforming both baseline models and Matching Networks by a substantial margin (+7.8% absolute gain over Matching Network with Fully-Conditional Embeddings). On 1-shot classification, the meta-learner achieves 44.3%, competitive with Matching Network FCE (45.1%, with overlapping confidence intervals) and superior to basic Matching Networks (42.4%).

Knowl 10 — Qualitative Dynamics of Meta-Learned Gates
empirical result
Visualization of the LSTM meta-learner's internal gate activations throughout inner optimization steps demonstrates distinct mechanisms:
- Forget gate $f_t$ : Learns values consistently slightly below 1 across layers and datasets, acting as a stable, meta-learned weight decay mechanism.
- Input gate $i_t$ : Shows high variability across different datasets and between 1-shot and 5-shot settings, confirming that the meta-learner learns a dynamic, instance-dependent step size schedule rather than a fixed decay rule.

Coverage note — None omitted; all core contributions—the LSTM meta-learner update formulation, training algorithm, parameter sharing and preprocessing, learned initialization, gradient independence assumption, task-isolated batch normalization, gate initialization, Mini-ImageNet benchmark, empirical evaluation, and gate behavior analysis—are represented in the knowls.

References

1.Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. CoRR, abs/1606.04474, 2016. URL http://arxiv.org/abs/1606.04474.
2.Yoshua Bengio et al. Deep learning of representations for unsupervised and transfer learning. ICML Unsupervised and Transfer Learning, 27:17–36, 2012.
3.Tom Bosc. Learning to learn neural networks.
4.Rich Caruana. Learning many related tasks at the same time with backpropagation. Advances in neural information processing systems, pp. 657–664, 1995.
5.Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. CoRR, abs/1310.1531, 2013. URL http://arxiv.org/abs/1310.1531.
6.John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121–2159, July 2011. ISSN 1532-4435. URL http://dl.acm.org/citation.cfm?id=1953048.2021068.
7.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385.
8.Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In IN LECTURE NOTES ON COMP. SCI. 2130, PROC. INTL. CONF. ON ARTI NEURAL NETWORKS (ICANN-2001, pp. 87–94. Springer, 2001.
9.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL http://arxiv.org/abs/1502.03167.
10.Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL http://arxiv.org/abs/1412.6980.
11.Gregory Koch. Siamese neural networks for one-shot image recognition. PhD thesis, University of Toronto, 2015.
12.Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. CoRR, abs/1604.00289, 2016. URL http://arxiv.org/abs/1604.00289.
13.Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Gradient-based hyperparameter optimization through reversible learning. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
14.Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). 1983.
15.Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016.
16.Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. Oneshot learning with memory-augmented neural networks. CoRR, abs/1605.06065, 2016. URL http://arxiv.org/abs/1605.06065.
17.Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28(1): 105–130, 1997.
18.Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pp. 181–209. Springer, 1998.
19.Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. CoRR, abs/1606.04080, 2016. URL http://arxiv.org/abs/1606.04080.
20.Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
21.Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? CoRR, abs/1411.1792, 2014. URL http://arxiv.org/abs/1411.1792.
22.Wojciech Zaremba. An empirical exploration of recurrent network architectures. 2015.
23.Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012. URL http://arxiv.org/abs/1212.5701.

Citation

MLA

Ravi, S., and H. Larochelle. “Optimization as a Model for Few-Shot Learning”. International Conference on Learning Representations, 2017, https://openreview.net/pdf?id=rJY0-Kcll.

APA

Ravi, S., & Larochelle, H. (2017). Optimization as a Model for Few-Shot Learning. International Conference on Learning Representations. https://openreview.net/pdf?id=rJY0-Kcll

Chicago

Ravi, S., and H. Larochelle. 2017. “Optimization as a Model for Few-Shot Learning”. International Conference on Learning Representations. https://openreview.net/pdf?id=rJY0-Kcll.

Harvard

Ravi, S. and Larochelle, H. (2017) “Optimization as a Model for Few-Shot Learning”, International Conference on Learning Representations [Preprint]. Available at: https://openreview.net/pdf?id=rJY0-Kcll.

Vancouver

1. Ravi S, Larochelle H (2017) Optimization as a Model for Few-Shot Learning. International Conference on Learning Representations

BibTeX

@article{ravi2017optimization,
  title = {Optimization as a Model for Few-Shot Learning},
  author = {Ravi, Sachin and Larochelle, Hugo},
  year = {2017},
  journal = {International Conference on Learning Representations},
  url = {https://openreview.net/pdf?id=rJY0-Kcll}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Optimization as a Model for Few-Shot Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — LSTM Meta-Learner Parameter Update Formulation

Knowl 2 — Episode-Based Meta-Learner Training Algorithm

Knowl 3 — Coordinate-wise Parameter Sharing and Gradient Preprocessing

Knowl 4 — Meta-Learned Base Model Initialization

Knowl 5 — First-Order Gradient Independence Assumption in Meta-Optimization

Knowl 6 — Task-Isolated Batch Normalization for Episodic Meta-Learning

Knowl 7 — Meta-Learner LSTM Gate Bias Initialization Strategy