Back

Continual Learning with Deep Generative Replay

Hanul ShinJung Kwon LeeJaehong KimJiwon Kim

article2017NeurIPS2,573 citations

Proposes Deep Generative Replay, a dual-model framework that prevents catastrophic forgetting in continual learning by training a generative network to synthesize past data rather than storing old datasets.

Listen

The article addresses catastrophic forgetting, a core barrier to building artificial intelligence systems that learn multiple tasks sequentially without large memory stores or access to prior data. This issue arises because training neural networks on new objectives overwrites previously learned representations, limiting practical deployment in real-world settings where data privacy or storage constraints apply.

The work evaluates whether a cooperative dual-model architecture called the scholar—pairing a deep generative model with a task-solving model—can enable continual learning by replaying synthetic data that mimics past inputs and targets. Experiments involved sequential training on image classification tasks using MNIST permutations, transitions between MNIST and SVHN domains, and disjoint class subsets, with performance compared against exact replay, noise replay, and no-replay baselines.

The analysis shows that generative replay preserved accuracy near the level of joint training on all accumulated data, often within 1-2 percentage points on cumulative test sets, while naive sequential training caused near-total loss on earlier tasks. It also matched or exceeded regularization-based alternatives like elastic weight consolidation and learning-without-forgetting when augmented with generated samples, and transferred knowledge across scholar models without requiring identical network structures.

These outcomes indicate that high-quality generative replay can substitute for stored data, lowering memory demands and supporting privacy-sensitive applications while maintaining balanced performance across old and new tasks. The approach scales to moderately complex image domains but reveals trade-offs when generator fidelity declines.

Next steps include extending the framework to reinforcement learning, continuously evolving networks that reference their own prior copies, and hybrid combinations with weight-protection methods. The primary limitation is reliance on generator quality, which produced measurable degradation on SVHN within the tested settings; results should be interpreted cautiously outside image-classification domains until further validation on broader data types.

Massachusetts Institute of TechnologySK T-Brainclassificationgenerative modelrandom access memoryDeep Generative ReplayCatastrophic ForgettingScholar Model
arXiv: 1705.08690
  • Paper: Generative Adversarial Networks, Ian J. Goodfellow et al. (2014). Introduces Generative Adversarial Networks (GANs), the foundational deep generative modeling framework directly adopted by the Scholar architecture to generate synthetic replay data.
  • Paper: Learning without Forgetting, Zhizhong Li et al. (2016). Presents Learning without Forgetting (LwF), establishing distillation-based continual learning that serves as a primary baseline and conceptual contrast to generative replay.
  • Paper: Overcoming catastrophic forgetting in neural networks, James Kirkpatrick et al. (2017). Introduces Elastic Weight Consolidation (EWC), providing the cornerstone parameter-regularization baseline against which the Scholar framework is benchmarked.
  • Paper: iCaRL: Incremental Classifier and Representation Learning, Sylvestre-Alvise Rebuffi et al. (2016). Demonstrates exemplar-based replay and distillation for incremental learning, framing the data storage constraints that motivate synthetic generative replay.
  • Paper: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Alec Radford et al. (2016). Establishes Deep Convolutional GANs (DCGANs), providing the standard architectural principles for stable image generation used in deep generative replay.
  • Paper: Adversarial Autoencoders, Alireza Makhzani et al. (2015). Details Adversarial Autoencoders, an alternative generative model class evaluated within the Scholar framework alongside GANs for data distribution matching.
  • Paper: Improved Techniques for Training GANs, Tim Salimans et al. (2016). Develops training and architectural stability techniques for generative adversarial networks necessary to synthesize high-fidelity replay images.
Cover for Continual Learning with Deep Generative Replay

Abstract

Attempts to train a comprehensive artificial intelligence capable of solving multiple tasks have been impeded by a chronic problem called catastrophic forgetting. Although simply replaying all previous data alleviates the problem, it requires large memory and even worse, often infeasible in real world applications where the access to past data is limited. Inspired by the generative nature of hippocampus as a short-term memory system in primate brain, we propose the Deep Generative Replay, a novel framework with a cooperative dual model architecture consisting of a deep generative model ("generator") and a task solving model ("solver"). With only these two models, training data for previous tasks can easily be sampled and interleaved with those for a new task. We test our methods in several sequential learning settings involving image classification tasks.

Topics

classificationgenerative modelrandom access memorymemory systemsinterleaversarchitecture designssupervised classification

Keywords

Deep Generative ReplayCatastrophic ForgettingScholar ModelGenerative Adversarial NetworksElastic Weight ConsolidationLearning Without ForgettingMNISTSVHNComplementary Learning SystemPseudo‑Rehearsal

Table of Contents

  • 1 Introduction
  • 2 Related Works
  • 2.1 Comparable methods
  • 2.2 Complementary Learning System(CLS) theory
  • 2.3 Deep Generative Models
  • 3 Generative Replay
  • 3.1 Proposed Method
  • 3.2 Preliminary Experiment
  • 4 Experiments
  • 4.1 Learning independent tasks
  • 4.2 Learning new domains
  • 4.3 Learning new classes
  • 5 Discussion
  • References

Knowls

  1. Knowl 1 — Deep Generative Replay Framework (Scholar Architecture)

    model/method

    Deep Generative Replay (DGR) is a continual learning framework designed to mitigate catastrophic forgetting without storing past raw data. The framework introduces a dual-network module termed a scholar H=⟨G,S⟩H = \langle G, S \rangle, composed of:

    1. A Generator GG: A deep generative model (such as a Generative Adversarial Network or Wasserstein GAN with Gradient Penalty) that learns and reproduces the cumulative input data distribution across all previously encountered tasks.
    2. A Solver SS: A task-solving model parameterized by weights θ\theta that performs the target task (e.g., classification) on inputs from any seen task.

    When presented with a new task TiT_i with data distribution DiD_i, the previous scholar Hi−1=⟨Gi−1,Si−1⟩H_{i-1} = \langle G_{i-1}, S_{i-1} \rangle acts as an episodic memory surrogate. Pseudo-inputs x′x' are sampled from Gi−1G_{i-1} and labeled with pseudo-targets y′=Si−1(x′)y' = S_{i-1}(x'). The new generator GiG_i is trained to reconstruct the combined distribution of real inputs x∼Dix \sim D_i and generated inputs x′∼Gi−1x' \sim G_{i-1}. Simultaneously, the new solver SiS_i is trained on an interleaved stream of real pairs (x,y)∼Di(x, y) \sim D_i and replayed pairs (x′,y′)(x', y'), retaining historical capabilities while mastering new tasks.

  2. Knowl 2 — Solver Objective Function for Generative Replay

    equation

    When training the ii-th solver S(⋅;θi)S(\cdot; \theta_i) on task TiT_i with current data distribution DiD_i using pseudo-data generated from the previous scholar Hi−1=⟨Gi−1,Si−1⟩H_{i-1} = \langle G_{i-1}, S_{i-1} \rangle, the training loss function Ltrain(θi)\mathcal{L}_{\text{train}}(\theta_i) is:

    Ltrain(θi)=rE(x,y)∼Di[L(S(x;θi),y)]+(1−r)Ex′∼Gi−1[L(S(x′;θi),S(x′;θi−1))]\mathcal{L}_{\text{train}}(\theta_i) = r \mathbb{E}_{(x,y) \sim D_i} \left[ L(S(x; \theta_i), y) \right] + (1 - r) \mathbb{E}_{x' \sim G_{i-1}} \left[ L(S(x'; \theta_i), S(x'; \theta_{i-1})) \right]

    where:

    • θi\theta_i is the parameter vector of the current solver SiS_i,
    • θi−1\theta_{i-1} is the parameter vector of the previous solver Si−1S_{i-1}, kept fixed during the update of SiS_i,
    • DiD_i is the real data distribution for current task TiT_i,
    • Gi−1G_{i-1} is the generator network trained up through task Ti−1T_{i-1},
    • L(⋅,⋅)L(\cdot, \cdot) is the task-specific loss function (e.g., cross-entropy),
    • r∈[0,1]r \in [0, 1] is a mixing coefficient determining the proportion of current task samples relative to replayed past samples.

    For the initial task (i=1i = 1), the replayed loss term is omitted and the solver is optimized solely on real training examples from D1D_1.

  3. Knowl 3 — Sequential Scholar Training Procedure

    algorithm

    The training procedure for sequentially learning NN tasks (T1,T2,…,TN)(T_1, T_2, \dots, T_N) with Deep Generative Replay iteratively updates the generator and solver at each task step using mixed real and generated batches:

    Input: Sequence of tasks (T1,…,TN)(T_1, \dots, T_N) with data distributions (D1,…,DN)(D_1, \dots, D_N), mixing ratio r∈[0,1]r \in [0, 1]
    Output: Final scholar model HN=⟨GN,SN⟩H_N = \langle G_N, S_N \rangle
    Initialize G1G_1 and S1S_1
    Train G1G_1 on real inputs x∼D1x \sim D_1
    Train S1S_1 on real pairs (x,y)∼D1(x, y) \sim D_1 to minimize E(x,y)∼D1[L(S(x;θ1),y)]\mathbb{E}_{(x,y) \sim D_1}[L(S(x; \theta_1), y)]
    for i=2i = 2 to NN do
        Initialize new scholar Hi=⟨Gi,Si⟩H_i = \langle G_i, S_i \rangle
        repeat
            Sample current real inputs x∼Dix \sim D_i with proportion rr
            Sample replayed inputs x′∼Gi−1x' \sim G_{i-1} with proportion 1−r1 - r
            Update GiG_i on the combined input batch (x,x′)(x, x')
        until GiG_i converges
        repeat
            Sample current real pairs (x,y)∼Di(x, y) \sim D_i with proportion rr
            Sample replayed inputs x′∼Gi−1x' \sim G_{i-1} with proportion 1−r1 - r
            Compute replayed targets y′=Si−1(x′)y' = S_{i-1}(x')
            Update parameters θi\theta_i to minimize rL(S(x;θi),y)+(1−r)L(S(x′;θi),y′)r L(S(x; \theta_i), y) + (1 - r) L(S(x'; \theta_i), y')
        until SiS_i converges
    end for
    return HN=⟨GN,SN⟩H_N = \langle G_N, S_N \rangle

  4. Knowl 4 — Knowledge Transfer Across Sequentially Trained Scholars on MNIST

    data/table

    To evaluate if a scholar model alone can transfer learned knowledge to subsequent networks without access to the original training data, a sequence of five scholar models was trained on the full MNIST digit classification dataset. Solver1\text{Solver}_1 was trained directly on real MNIST data, whereas each subsequent solver (Solver2\text{Solver}_2 through Solver5\text{Solver}_5) was trained from scratch using exclusively generated input-target pairs produced by the preceding scholar.

    Model Solver1\text{Solver}_1 Solver2\text{Solver}_2 Solver3\text{Solver}_3 Solver4\text{Solver}_4 Solver5\text{Solver}_5
    Test Accuracy (%) 98.81% 98.64% 98.58% 98.53% 98.56%

    The accuracy on the full MNIST test set dropped by less than 0.3 percentage points across five successive generations of knowledge transfer, demonstrating that the scholar network can accurately preserve and transfer classification knowledge across iterations without catastrophic information loss.

  5. Knowl 5 — Continual Learning on Permuted MNIST with Generative Replay

    empirical result

    In the permuted MNIST benchmark, each task is constructed by applying a unique fixed random permutation to the image pixels of the MNIST dataset, making input distributions statistically independent across tasks. Evaluating sequential learning on permuted MNIST reveals:

    • Naive training (None): Sequentially training a classifier without any memory mechanism causes rapid catastrophic forgetting; test accuracy on earlier tasks drops sharply toward near-zero upon training on subsequent tasks.
    • Noise pseudo-rehearsal (Noise): Generating random Gaussian noise paired with the previous solver's predicted labels fails to prevent catastrophic forgetting, confirming that mimicking the true input distribution is necessary.
    • Deep Generative Replay (GR): Using a generator trained with WGAN-GP to reproduce permuted image distributions enables the solver to maintain high classification accuracy across all past tasks throughout the training sequence, matching the performance of exact replay (ER) where original historical data is replayed with past solver predictions.

  6. Knowl 6 — Cross-Domain Continual Learning on MNIST and SVHN

    empirical result

    When sequentially training across two distinct image domains—MNIST and Street View House Numbers (SVHN)—evaluating performance demonstrates that:

    • A naive solver trained without replay suffers severe performance degradation on the first domain after training on the second domain (e.g., MNIST classification accuracy drops markedly below 60% when learning SVHN, and vice versa).
    • Deep Generative Replay (GR) maintains the performance on the initial domain while achieving high accuracy on the new domain. Over training iterations on the second task, the generator learns a bimodal distribution that simultaneously synthesizes both MNIST-like and SVHN-like digit images.
    • The retention of the first domain under generative replay matches the upper bound achieved by exact replay (ER) of real previous domain images paired with old solver predictions.

  7. Knowl 7 — Augmenting Learning without Forgetting with Generative Replay (LwF-GR)

    model/method

    Learning without Forgetting (LwF) prevents forgetting by using the network's outputs on current task data prior to fine-tuning as pseudo-labels for earlier task-specific heads. However, standard LwF relies only on new task inputs to evoke past knowledge, which leads to performance loss when shared parameters are fine-tuned across disparate domains.

    Generative Replay can augment LwF (forming LwF-GR) by providing task-specific generated samples representing past domains. In a domain transition setting (training on SVHN followed by MNIST):

    • Standard LwF experiences a sharp decline in test accuracy on SVHN as soon as fine-tuning of the shared layers begins.
    • LwF-GR feeds generated SVHN samples to preserve the SVHN task representations during fine-tuning, mitigating the performance drop on SVHN while achieving high accuracy on MNIST.

  8. Knowl 8 — Class-Incremental Continual Learning on Disjoint MNIST Subsets

    empirical result

    In a class-incremental continual learning benchmark where the MNIST dataset is partitioned into 5 disjoint tasks of 2 classes each (Task 1: digits 0 and 1; Task 2: digits 2 and 3; ...; Task 5: digits 8 and 9), the model is evaluated on all 10 digit classes after sequentially learning the tasks:

    • Naive training (None): The solver forgets previously seen classes after each task, retaining proficiency solely on the current 2 classes, resulting in an overall 10-class accuracy of approximately 20%.
    • Noise baseline (Noise): Replaying random noise paired with recorded solver outputs fails to preserve class decision boundaries, also yielding ~20% accuracy.
    • Deep Generative Replay (GR): The generator iteratively learns to reproduce samples across all accumulated classes seen up to the current task. When trained on the interleaved real and generated samples, the solver retains discrimination across all 10 classes, achieving full 10-class test performance close to the exact replay (ER) upper bound.

  9. Knowl 9 — Generative Model Quality Constraint in Deep Generative Replay

    limitation

    The primary limitation of the Deep Generative Replay framework is that the solver's task retention is strictly bounded by the fidelity and capacity of the generative model. If the generator fails to faithfully recover the underlying data distribution of past tasks, the replayed inputs x′x' will not adequately cover the required feature space, leading to memory degradation in the solver. While generative replay performs near the exact replay upper bound on datasets with simpler geometries like MNIST, noticeable performance loss occurs on more complex visual domains (such as class-incremental learning on SVHN) due to difficulties in generating high-quality complex images.

Coverage note — No substantial contributed material was omitted; all primary concepts, equations, training procedures, experimental benchmarks (Permuted MNIST, MNIST-SVHN domain transfer, LwF-GR integration, class-incremental MNIST), and framework limitations were included.

References

  1. 1.W. C. Abraham and A. Robins. Memory retention–the synaptic stability versus plasticity dilemma. Trends in neurosciences, 28(2):73–78, 2005.
  2. 2.B. Ans and S. Rousset. Avoiding catastrophic forgetting by coupling two reverberating neural networks. Comptes Rendus de l’Académie des Sciences-Series III-Sciences de la Vie, 320(12):989–997, 1997.
  3. 3.D. A. Baldwin, E. M. Markman, and R. L. Melartin. Infants’ ability to draw inferences about nonobvious object properties: Evidence from exploratory play. Child development, 64(3):711–728, 1993.
  4. 4.M. H. Bornstein and M. E. Arterberry. The development of object categorization in young children: Hierarchical inclusiveness, age, perceptual attribute, and group versus individual analyses. Developmental psychology, 46(2):350, 2010.
  5. 5.T. G. Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1–15. Springer, 2000.
  6. 6.J. Fagot and R. G. Cook. Evidence for large long-term memory capacities in baboons and pigeons and its implications for learning and the evolution of cognition. Proceedings of the National Academy of Sciences, 103(46):17564–17567, 2006.
  7. 7.R. M. French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999.
  8. 8.H. Gelbard-Sagiv, R. Mukamel, M. Harel, R. Malach, and I. Fried. Internally generated reactivation of single neurons in human hippocampus during free recall. Science, 322(5898):96–101, 2008.
  9. 9.R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  10. 10.I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014.
  11. 11.I. J. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. CoRR, abs/1701.00160, 2017.
  12. 12.I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
  13. 13.J. Grutzendler, N. Kasthuri, and W.-B. Gan. Long-term dendritic spine stability in the adult cortex. Nature, 420(6917):812–816, 2002.
  14. 14.I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of wasserstein gans. arXiv preprint arXiv:1704.00028, 2017.
  15. 15.M. Hattori. A biologically inspired dual-network memory model for reduction of catastrophic forgetting. Neurocomputing, 134:262–268, 2014.
  16. 16.G. E. Hinton and D. C. Plaut. Using fast weights to deblur old memories. In Proceedings of the ninth annual conference of the Cognitive Science Society, pages 177–186, 1987.
  17. 17.D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  18. 18.J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  19. 19.Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  20. 20.S.-W. Lee, J.-H. Kim, J.-W. Ha, and B.-T. Zhang. Overcoming catastrophic forgetting by incremental moment matching. arXiv preprint arXiv:1703.08475, 2017.
  21. 21.Z. Li and D. Hoiem. Learning without forgetting. In European Conference on Computer Vision, pages 614–629. Springer, 2016.
  22. 22.M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 24:109–165, 1989.
  23. 23.V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  24. 24.D. C. Mocanu, M. T. Vega, E. Eaton, P. Stone, and A. Liotta. Online contrastive divergence with generative replay: Experience replay without storing data. CoRR, abs/1610.05555, 2016.
  25. 25.Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 5, 2011.
  26. 26.R. C. O’Reilly and K. A. Norman. Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework. Trends in cognitive sciences, 6(12):505–510, 2002.
  27. 27.J. O’Neill, B. Pleydell-Bouverie, D. Dupret, and J. Csicsvari. Play it again: reactivation of waking experience and memory. Trends in neurosciences, 33(5):220–229, 2010.
  28. 28.S. Ramirez, X. Liu, P.-A. Lin, J. Suh, M. Pignatelli, R. L. Redondo, T. J. Ryan, and S. Tonegawa. Creating a false memory in the hippocampus. Science, 341(6144):387–391, 2013.
  29. 29.R. Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological review, 97(2):285–308, 1990.
  30. 30.A. Robins. Catastrophic forgetting in neural networks: the role of rehearsal mechanisms. In Artificial Neural Networks and Expert Systems, 1993. Proceedings., First New Zealand International Two-Stream Conference on, pages 65–68. IEEE, 1993.
  31. 31.A. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123–146, 1995.
  32. 32.R. Salakhutdinov and G. Hinton. Deep boltzmann machines. In Artificial Intelligence and Statistics, pages 448–455, 2009.
  33. 33.N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  34. 34.R. K. Srivastava, J. Masci, S. Kazerounian, F. Gomez, and J. Schmidhuber. Compete to compute. In Advances in neural information processing systems, pages 2310–2318, 2013.
  35. 35.R. Stickgold and M. P. Walker. Sleep-dependent memory consolidation and reconsolidation. Sleep medicine, 8(4):331–343, 2007.

Citation

MLA
Shin, H., et al. “Continual Learning with Deep Generative Replay”. arXiv, 2017, https://doi.org/10.48550/arxiv.1705.08690.
APA
Shin, H., Lee, J. K., Kim, J., & Kim, J. (2017). Continual Learning with Deep Generative Replay. arXiv. https://doi.org/10.48550/arxiv.1705.08690
Chicago
Shin, H., J. K. Lee, J. Kim, and J. Kim. 2017. “Continual Learning with Deep Generative Replay”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1705.08690.
Harvard
Shin, H. et al. (2017) “Continual Learning with Deep Generative Replay”. arXiv. Available at: https://doi.org/10.48550/arxiv.1705.08690.
Vancouver
1. Shin H, Lee JK, Kim J, Kim J (2017) Continual Learning with Deep Generative Replay. https://doi.org/10.48550/arxiv.1705.08690

BibTeX

@misc{https://doi.org/10.48550/arxiv.1705.08690,
  doi = {10.48550/ARXIV.1705.08690},
  url = {https://arxiv.org/abs/1705.08690},
  author = {Shin, Hanul and Lee, Jung Kwon and Kim, Jaehong and Kim, Jiwon},
  keywords = {Artificial Intelligence (cs.AI), Computer Vision and Pattern Recognition (cs.CV), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Continual Learning with Deep Generative Replay},
  publisher = {arXiv},
  year = {2017},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors