Continual Learning Through Synaptic Intelligence

Friedemann ZenkeBen PooleSurya Ganguli

article2017ICML3,479 citations

Proposes Synaptic Intelligence, a biologically inspired continual learning method that tracks parameter importance online to prevent catastrophic forgetting across sequential tasks with minimal computational overhead.

Listen

The paper addresses catastrophic forgetting in artificial neural networks, where sequential training on new tasks overwrites parameters critical to prior tasks and erases earlier performance. This limitation becomes especially relevant as networks move from static, curated datasets toward real-world streams of shifting data distributions, in contrast to the lifelong adaptation seen in biological systems.

The work sets out to show that endowing each synapse with a simple local importance measure—accumulated along the full training trajectory—allows networks to protect parameters that mattered most for past tasks while still learning new ones efficiently.

The authors derive an online importance score for every parameter by integrating the product of its gradient and its update step across each task’s learning path, then use this score to construct a quadratic penalty that anchors important weights near their previous values. They test the resulting algorithm on split and permuted MNIST as well as split CIFAR-10/100, comparing against plain stochastic gradient descent, dropout, and the related elastic-weight-consolidation method, with multiple random seeds and both single- and multi-head output configurations.

Networks equipped with the consolidation mechanism retain near-original accuracy on earlier tasks even after training on up to ten subsequent tasks, whereas standard fine-tuning drops to chance levels on the first tasks; the same networks also generalize better on new tasks with limited data than either fine-tuning or training from scratch. The per-synapse importance values remain largely uncorrelated across tasks when consolidation is active, confirming that distinct subsets of weights are recruited for each new problem. Performance matches or slightly exceeds elastic weight consolidation while requiring only quantities already available during ordinary back-propagation.

These results indicate that modest additional state per synapse can eliminate the need to store old data, replay examples, or duplicate entire networks, thereby lowering both memory cost and training time for continual-learning applications. The approach therefore offers a practical route to more robust sequential training in production systems that must accommodate distribution shifts without periodic full retraining.

Because the importance estimate is noisy under stochastic gradients, a tunable strength parameter must be chosen below its theoretical optimum, and the method has so far been validated only on image-classification benchmarks of modest size. Larger-scale experiments on language, reinforcement-learning, or streaming sensor data would be required before confident deployment at industrial scale.

Stanford University classification classification methods computational efficiency Synaptic Intelligence Catastrophic Forgetting Importance Measure ω

arXiv: 1703.04200 ganguli-lab/pathint

Paper: Overcoming catastrophic forgetting in neural networks, James Kirkpatrick et al. (2017). Reading this paper on Elastic Weight Consolidation is essential because Synaptic Intelligence directly builds upon and compares against its parameter-importance regularization strategy for continual learning.
Paper: Why There Are Complementary Learning Systems in the Hippocampus and Neocortex, James L. McClelland et al. (1995). Understanding the biological foundations of complementary learning systems provides vital context for how artificial models balance stability and plasticity to prevent catastrophic forgetting.

Paper: Gradient Episodic Memory for Continual Learning, David Lopez-Paz et al. (2017). This paper extends continual learning research beyond regularization by introducing Gradient Episodic Memory, offering a natural architectural progression from weight-based protection methods.

Cover for Continual Learning Through Synaptic Intelligence

Abstract

While deep learning has led to remarkable advances across diverse applications, it struggles in domains where the data distribution changes over the course of learning. In stark contrast, biological neural networks continually adapt to changing domains, possibly by leveraging complex molecular machinery to solve many tasks simultaneously. In this study, we introduce intelligent synapses that bring some of this biological complexity into artificial neural networks. Each synapse accumulates task relevant information over time, and exploits this information to rapidly store new memories without forgetting old ones. We evaluate our approach on continual learning of classification tasks, and show that it dramatically reduces forgetting while maintaining computational efficiency.

Topics

classification classification methods computational efficiency deep learning random access memory online learning network architecture recurrent neural networks

Keywords

Synaptic Intelligence Catastrophic Forgetting Importance Measure ω Path Integral Elastic Weight Consolidation Split MNIST Permuted MNIST Consolidation Strength c Online Computation Hessian

1. Introduction
2. Prior work
3. Synaptic framework
4. Theoretical analysis of special cases
5. Experiments
5.1. Split MNIST
5.2. Permuted MNIST benchmark
5.3. Split CIFAR-10/CIFAR-100 benchmark
6. Discussion
Acknowledgements
References
A. Split CIFAR-10/100 CNN architecture
B. Additional split CIFAR-10 experiments
C. Comparison of path integral approach to other metrics

Knowls

Knowl 1 — Path-Integral Synaptic Importance Measure
model/method
During training of a neural network on task $\mu$ over the time interval $[t^{\mu-1}, t^\mu]$ , the parameter vector traces a continuous trajectory $\theta(t)$ in parameter space. The change in the task loss $L_\mu(\theta)$ along this trajectory is given by the path integral of the gradient field $g(\theta(t)) = \nabla_\theta L_\mu(\theta(t))$ :

$\int_{t^{\mu-1}}^{t^\mu} g(\theta(t)) \cdot \theta'(t) dt = \sum_k \int_{t^{\mu-1}}^{t^\mu} g_k(\theta(t)) \theta'_k(t) dt \equiv - \sum_k \omega_k^\mu$

where $k$ indexes individual parameters (synaptic weights and biases), $\theta'_k(t) = \frac{d\theta_k(t)}{dt}$ is the rate of parameter change, and $\omega_k^\mu$ is defined as the parameter-specific importance measure:

$\omega_k^\mu = - \int_{t^{\mu-1}}^{t^\mu} g_k(\theta(t)) \theta'_k(t) dt$

In discrete-time optimization using gradient descent or stochastic gradient descent (SGD), $\omega_k^\mu$ is computed online as a running sum over optimization steps $t$ of the product between the parameter gradient $g_k(t) = \frac{\partial L_\mu}{\partial \theta_k}(t)$ and the parameter displacement $\Delta \theta_k(t) = \theta_k(t) - \theta_k(t-1)$ :

$\omega_k^\mu \approx - \sum_t g_k(t) \Delta \theta_k(t)$

The quantity $\omega_k^\mu$ measures the local contribution of parameter $\theta_k$ toward the cumulative reduction of the task loss $L_\mu$ .
Knowl 2 — Quadratic Surrogate Loss with Cumulative Synaptic Consolidation
equation
To train on a new task $\mu$ without catastrophically forgetting previously learned tasks $\nu < \mu$ , the optimization minimizes a surrogate regularized loss function $\tilde{L}_\mu(\theta)$ :

$\tilde{L}_\mu(\theta) = L_\mu(\theta) + c \sum_k \Omega_k^\mu (\theta_k - \tilde{\theta}_k)^2$

where $L_\mu(\theta)$ is the loss function of the current task $\mu$ , $c > 0$ is a dimensionless strength parameter that balances performance on new tasks against preservation of past memories, and $\tilde{\theta}_k = \theta_k(t^{\mu-1})$ is the reference parameter value at the completion of training on task $\mu-1$ .

The per-parameter cumulative regularization weight $\Omega_k^\mu$ aggregates normalized importance across all prior tasks $\nu < \mu$ :

$\Omega_k^\mu = \sum_{\nu < \mu} \frac{\omega_k^\nu}{(\Delta_k^\nu)^2 + \xi}$

where $\omega_k^\nu$ is the path-integral task importance of parameter $\theta_k$ accumulated during task $\nu$ , $\Delta_k^\nu = \theta_k(t^\nu) - \theta_k(t^{\nu-1})$ is the total displacement of parameter $\theta_k$ during task $\nu$ , and $\xi > 0$ is a small positive damping parameter introduced to bound $\Omega_k^\mu$ when $\Delta_k^\nu \to 0$ . The squared distance term $(\Delta_k^\nu)^2$ ensures that the regularization penalty carries the same physical units as the loss function $L_\mu$ .
Knowl 3 — Synaptic Intelligence Algorithm for Continual Learning
algorithm
The Synaptic Intelligence (SI) continual learning algorithm tracks local parameter importances $\omega_k$ online during training and consolidates important parameters via cumulative penalties $\Omega_k$ when switching tasks.
Input: Sequence of tasks $\mu \in \{1, \dots, M\}$ , regularization strength $c$ , damping factor $\xi$ , learning rate $\eta$
Initialize parameters $\theta$ , cumulative importance $\Omega_k \leftarrow 0$ for all parameters $k$
for task $\mu = 1$ to $M$ do
     $\tilde{\theta}_k \leftarrow \theta_k$ for all parameters $k$
     $\omega_k \leftarrow 0$ for all parameters $k$
    while training on task $\mu$ do
        Sample minibatch data for task $\mu$
        Compute task loss $L_\mu(\theta)$
        Compute surrogate loss $\tilde{L}_\mu(\theta) = L_\mu(\theta) + c \sum_k \Omega_k (\theta_k - \tilde{\theta}_k)^2$
        Compute regularized gradients $g_k = \frac{\partial \tilde{L}_\mu}{\partial \theta_k}$ for all $k$
        Compute parameter updates $\Delta \theta_k = -\eta g_k$ (or via an adaptive optimizer)
        for all parameters $k$ do
             $\omega_k \leftarrow \omega_k - \frac{\partial L_\mu}{\partial \theta_k} \cdot \Delta \theta_k$
             $\theta_k \leftarrow \theta_k + \Delta \theta_k$
        end for
    end while
    for all parameters $k$ do
         $\Delta_k^\mu \leftarrow \theta_k - \tilde{\theta}_k$
         $\Omega_k \leftarrow \Omega_k + \frac{\omega_k}{(\Delta_k^\mu)^2 + \xi}$
    end for
end for
The importance vector $\omega_k$ is updated locally at each parameter update step without requiring additional gradient passes. After each task completes, $\Omega_k$ and $\tilde{\theta}_k$ are updated and $\omega_k$ is reset to zero for the subsequent task.
Knowl 4 — Correspondence Between Path-Integral Importance and the Hessian in Quadratic Error Functions
theoretical result
Consider a quadratic loss function $E(\theta) = \frac{1}{2}(\theta - \theta^*)^T H (\theta - \theta^*)$ with Hessian matrix $H$ and minimum at $\theta^*$ . Continuous-time gradient descent dynamics $\tau \frac{d\theta}{dt} = -\frac{\partial E}{\partial \theta} = -H(\theta - \theta^*)$ from initial condition $\theta(0)$ produces the trajectory $\theta(t) = \theta^* + e^{-Ht/\tau}(\theta(0) - \theta^*)$ .

The integrated trajectory outer product matrix $Q = \tau \int_0^\infty \frac{d\theta}{dt} \left(\frac{d\theta}{dt}\right)^T dt$ has matrix elements:

$Q_{ij} = \sum_{\alpha, \beta} u_i^\alpha d^\alpha \frac{\lambda_\alpha \lambda_\beta}{\lambda_\alpha + \lambda_\beta} d^\beta u_j^\beta$

where $\lambda_\alpha$ and $u^\alpha$ denote the eigenvalues and eigenvectors of $H$ , and $d^\alpha = u^\alpha \cdot (\theta(0) - \theta^*)$ is the projection of the initial displacement onto the $\alpha$ -th eigenvector.

Under this formulation:
1. Random Initial Conditions: Averaging over zero-mean i.i.d. initial discrepancies with variance $\sigma^2$ (such that $\langle d^\alpha d^\beta \rangle = \sigma^2 \delta_{\alpha\beta}$ ) yields $\langle Q_{ij} \rangle = \frac{1}{2}\sigma^2 H_{ij}$ . Dividing by displacement variance $\sigma^2$ recovers the exact Hessian matrix.
2. Diagonal Hessian: When $H$ is diagonal, $Q_{ij} = \delta_{ij} (d^i)^2 H_{ii}$ . Dividing by $(d^i)^2$ in the normalized metric $\Omega_i$ recovers the diagonal Hessian elements $H_{ii}$ .
3. Rank-1 Hessian: When $H$ has a single non-zero eigenvalue $\lambda_1$ , $Q_{ij} = \frac{1}{2}(d^1)^2 H_{ij}$ , directly recovering the low-rank Hessian structure up to a scale factor.
Knowl 5 — Comparison Between Synaptic Intelligence and Fisher Information Metrics
model/method
Synaptic Intelligence (SI) differs fundamentally from Fisher information-based regularization methods such as Elastic Weight Consolidation (EWC) along three dimensions:
1. Trajectory vs. Endpoint Evaluation: EWC relies on a point estimate of the diagonal of the Fisher Information Matrix $F = \mathbb{E}_{x, y \sim p_\theta} [(\nabla_\theta \log p_\theta)(\nabla_\theta \log p_\theta)^T]$ computed offline at the final parameters $\theta^*$ after a task is finished. In contrast, SI accumulates importance $\omega_k$ online by integrating the path product $g_k(t) \theta'_k(t)$ continuously over the entire learning trajectory $\theta(t)$ .
2. Behavior at a Minimum: For a quadratic error surface, the gradient at the minimum is zero, causing the empirical Fisher metric $\bar{F} = \mathbb{E}_{x, y \sim \mathcal{D}} [g(\theta) g(\theta)^T]$ to vanish at $\theta^*$ . In contrast, the path integral $\omega_k$ evaluated along the trajectory leading to the minimum remains strictly non-zero and proportional to the diagonal of the Hessian.
3. Computational Overhead: Computing the exact Fisher matrix requires summing or sampling over the entire label space, making it linear in the number of classes or requiring an extra backpropagation pass. SI updates parameter importances using quantities already computed during the standard forward and backward training passes.
Knowl 6 — Permuted MNIST Continual Learning Performance
empirical result
On the Permuted MNIST benchmark (10 sequential tasks, each with a distinct random permutation applied to all 784 input pixels), an MLP with two hidden layers of 2,000 ReLU units each was trained sequentially for 20 epochs per task using Adam (batch size 256, learning rate $10^{-3}$ , damping parameter $\xi = 0.1$ , regularizer weight $c = 0.1$ ).

Results demonstrated:
- Baseline Failure: Standard SGD without consolidation ( $c=0$ ) and SGD with dropout ( $p=0.5$ on hidden layers) suffer severe catastrophic forgetting, with average classification accuracy across all learned tasks falling toward chance levels (fraction correct $\approx 0.50$ ) as the task count reaches 10.
- Consolidation Retention: Synaptic Intelligence with $c=0.1$ maintains an average classification accuracy of approximately $0.97$ across all 10 tasks, matching the performance of Elastic Weight Consolidation (EWC) and performing only slightly below the upper bound of joint simultaneous training on all tasks.
Knowl 7 — Split MNIST Continual Learning Performance
empirical result
On the Split MNIST benchmark, the standard MNIST dataset is split into 5 sequential binary classification tasks corresponding to consecutive digit pairs (0/1, 2/3, 4/5, 6/7, 8/9). An MLP with two hidden layers of 256 ReLU units each was trained for 10 epochs per task using Adam (learning rate $10^{-3}$ , batch size 64) with a multi-head readout evaluated only on current-task classes.

Comparing consolidation ( $c = 1, \xi = 10^{-3}$ ) to unregularized fine-tuning ( $c = 0$ ):
- Without consolidation ( $c = 0$ ), accuracy on earlier tasks drops to chance level ( $0.5$ ) after training on all 5 tasks, despite the network achieving near $1.0$ accuracy on each task immediately upon training.
- With consolidation ( $c = 1$ ), accuracy on earlier tasks undergoes only minor degradation, maintaining average accuracy across all learned tasks close to $1.0$ .
Knowl 8 — Decorrelation of Synaptic Importance Across Tasks Under Consolidation
empirical result
Measuring the pairwise Pearson correlation of parameter importance vectors $\omega_k^\mu$ across sequential tasks $\mu \in \{1, \dots, 10\}$ on the Permuted MNIST benchmark reveals how synaptic consolidation modifies network representation:
- First Layer Weights: Input-to-hidden layer weight importances are mutually uncorrelated across tasks for both unregularized training ( $c = 0$ ) and consolidation ( $c = 0.1$ ), due to the independent random pixel permutations applied to the input at each task.
- Second Layer Weights Without Consolidation ( $c = 0$ ): Second hidden layer weight importances become progressively correlated across tasks as more tasks are learned. This causes subsequent tasks to continually overwrite parameters that were critical for earlier tasks, leading directly to catastrophic forgetting.
- Second Layer Weights With Consolidation ( $c = 0.1$ ): Synaptic consolidation suppresses cross-task correlations in $\omega_k^\mu$ , maintaining near-zero correlation between task importance profiles and forcing the optimizer to utilize previously unused parameter subspaces to learn new tasks.
Knowl 9 — Split CIFAR-10 and CIFAR-100 Continual Learning Performance
empirical result
To evaluate continual learning in deep convolutional networks, a CNN consisting of 4 convolutional layers (two 32-filter $3\times3$ layers, $2\times2$ max-pool with dropout 0.25; two 64-filter $3\times3$ layers, $2\times2$ max-pool with dropout 0.25) and a 512-unit dense hidden layer (dropout 0.5) was trained sequentially across 6 tasks: Task 1 consisted of the full 10-class CIFAR-10 dataset (60 epochs), followed by 5 sequential tasks each containing 10 disjoint classes from CIFAR-100 (60 epochs each), using multi-head readout and Adam (learning rate $10^{-3}$ , batch size 256).

Key empirical findings across $n=5$ runs (chance level $= 0.1$ ):
- Fine-Tuning Failure: Without consolidation ( $c=0$ ), validation accuracy exhibits an age-dependent decline, with performance on older tasks steadily deteriorating (e.g., Task 1 validation accuracy drops to $\approx 0.40$ ).
- Memory Retention: Synaptic consolidation ( $c=0.1$ ) maintains validation accuracy above $0.70 - 0.75$ across all tasks, outperforming unregularized fine-tuning on every past task.
- Generalization Improvement: On Tasks 2 through 6, networks trained with consolidation achieve validation accuracy equal to or higher than networks trained from scratch on those individual tasks alone, while exhibiting lower training accuracy, demonstrating that consolidation acts as a regularizer that mitigates overfitting on limited-data tasks.
Knowl 10 — Memory Retention and Forward Transfer on Split CIFAR-10
empirical result
In a split CIFAR-10 experiment consisting of Task A (first 5 classes) and Task B (remaining 5 classes), a 4-layer CNN with 2 dense layers was trained sequentially for 60 epochs per task with an optimizer state reset between tasks (chance level $= 0.2$ , $n=10$ runs):
- Task A (Backward Retention): Unregularized fine-tuning ( $c=0$ ) suffers severe forgetting, dropping Task A validation accuracy from near $1.0$ down to $\approx 0.60 \pm 0.05$ . Synaptic consolidation ( $c=0.1$ ) prevents forgetting, maintaining Task A validation accuracy at $\approx 0.92 \pm 0.02$ .
- Task B (Forward Transfer / Generalization): Networks trained with consolidation achieve a validation accuracy of $\approx 0.86 \pm 0.02$ on Task B, compared to $\approx 0.82 \pm 0.02$ for fine-tuning without consolidation. This represents a statistically significant $\approx 4.5\%$ absolute accuracy improvement on the new task, an effect observed consistently across regularizer values $c \in [0.1, 10]$ .

Coverage note — General neurobiological background on molecular synaptic complexity and standard default Adam hyperparameter derivations were omitted as non-contributed background.

References

1.Benna, Marcus K. and Fusi, Stefano. Computational principles of synaptic memory consolidation. Nat Neurosci, advance online publication, October 2016. ISSN 1097-6256. doi: 10.1038/nn.4401.
2.Choy, Min Chee, Srinivasan, Dipti, and Cheu, Ruey Long. Neural networks for continuous online learning and control. IEEE Trans Neural Netw, 17(6):1511–1531, November 2006. ISSN 1045-9227. doi: 10.1109/TNN.2006.881710.
3.Donahue, Jeff, Jia, Yangqing, Vinyals, Oriol, Hoffman, Judy, Zhang, Ning, Tzeng, Eric, and Darrell, Trevor. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference in Machine Learning (ICML), 2014.
4.Fukushima, Kunihiko and Miyake, Sei. Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition. In Competition and Cooperation in Neural Nets, pp. 267–285. Springer, Berlin, Heidelberg, 1982. DOI: 10.1007/978-3-642-46466-9_18.
5.Fusi, Stefano, Drew, Patrick J., and Abbott, Larry F. Cascade models of synaptically stored memories. Neuron, 45(4):599–611, February 2005. ISSN 0896-6273. doi: 10.1016/j.neuron.2005.02.001.
6.Goodfellow, Ian J., Mirza, Mehdi, Xiao, Da, Courville, Aaron, and Bengio, Yoshua. An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks. arXiv:1312.6211 [cs, stat], December 2013. arXiv: 1312.6211.
7.Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. NIPS Deep Learning and Representation Learning Workshop, 2014.
8.Jung, Heechul, Ju, Jeongwoo, Jung, Minju, and Kim, Junmo. Less-forgetting Learning in Deep Neural Networks. arXiv:1607.00122 [cs], July 2016. arXiv: 1607.00122.
9.Kingma, Diederik and Ba, Jimmy. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], December 2014. arXiv: 1412.6980.
10.Kirkpatrick, James, Pascanu, Razvan, Rabinowitz, Neil, Veness, Joel, Desjardins, Guillaume, Rusu, Andrei A., Milan, Kieran, Quan, John, Ramalho, Tiago, Grabska-Barwinska, Agnieszka, Hassabis, Demis, Clopath, Claudia, Kumaran, Dharshan, and Hadsell, Raia. Overcoming catastrophic forgetting in neural networks. PNAS, pp. 201611835, March 2017. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1611835114.
11.Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. 2009.
12.Lahiri, Subhaneil and Ganguli, Surya. A memory frontier for complex synapses. In Advances in Neural Information Processing Systems, volume 26, pp. 1034–1042, Tahoe, USA, 2013. Curran Associates, Inc.
13.LeCun, Yann, Cortes, Corinna, and Burges, Christopher JC. The MNIST database of handwritten digits. 1998.
14.LeCun, Yann, Bengio, Yoshua, and Hinton, Geoffrey. Deep learning. Nature, 521(7553):436–444, May 2015. ISSN 0028-0836. doi: 10.1038/nature14539.
15.Li, Zhizhong and Hoiem, Derek. Learning without forgetting. In European Conference on Computer Vision, pp. 614–629. Springer, 2016.
16.Martens, James. Second-order optimization for neural networks. PhD thesis, University of Toronto, 2016.
17.Martens, James, Sutskever, Ilya, and Swersky, Kevin. Estimating the hessian by back-propagating curvature. arXiv preprint arXiv:1206.6464, 2012.
18.Montgomery, Johanna M. and Madison, Daniel V. State-Dependent Heterogeneity in Synaptic Depression between Pyramidal Cell Pairs. Neuron, 33(5):765–777, February 2002. ISSN 0896-6273. doi: 10.1016/S0896-6273(02)00606-2.
19.Pascanu, Razvan and Bengio, Yoshua. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013.
20.Razavian, Ali Sharif, Azizpour, Hossein, Sullivan, Josephine, and Carlsson, Stefan. Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 806–813, 2014.
21.Redondo, Roger L. and Morris, Richard G. M. Making memories last: the synaptic tagging and capture hypothesis. Nat Rev Neurosci, 12(1):17–30, January 2011. ISSN 1471-003X. doi: 10.1038/nrn2963.
22.Rosenblatt, Frank. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
23.Rusu, Andrei A., Rabinowitz, Neil C., Desjardins, Guillaume, Soyer, Hubert, Kirkpatrick, James, Kavukcuoglu, Koray, Pascanu, Razvan, and Hadsell, Raia. Progressive Neural Networks. arXiv:1606.04671 [cs], June 2016. arXiv: 1606.04671.
24.Srivastava, Rupesh K, Masci, Jonathan, Kazerounian, Sohrob, Gomez, Faustino, and Schmidhuber, Juergen. Compete to Compute. In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 26, pp. 2310–2318. Curran Associates, Inc., 2013.
25.Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson, Hod. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320–3328, 2014.
26.Zenke, Friedemann, Agnes, Everton J., and Gerstner, Wulfram. Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nat Commun, 6, April 2015. doi: doi: 10.1038/ncomms7922.
27.Zhou, Qiang, Tao, Huizhong W., and Poo, Mu-Ming. Reversal and Stabilization of Synaptic Modifications in a Developing Visual System. Science, 300(5627):1953–1957, June 2003. doi: 10.1126/science.1082212.
28.Ziegler, Lorric, Zenke, Friedemann, Kastner, David B., and Gerstner, Wulfram. Synaptic Consolidation: From Synapses to Behavioral Modeling. J Neurosci, 35(3):1319–1334, January 2015. ISSN 0270-6474, 1529-2401. doi: 10.1523/JNEUROSCI.3989-14.2015.

Citation

MLA

Zenke, F., et al. “Continual Learning Through Synaptic Intelligence”. arXiv, 2017, http://arxiv.org/abs/1703.04200v3.

APA

Zenke, F., Poole, B., & Ganguli, S. (2017). Continual Learning Through Synaptic Intelligence. arXiv. http://arxiv.org/abs/1703.04200v3

Chicago

Zenke, F., B. Poole, and S. Ganguli. 2017. “Continual Learning Through Synaptic Intelligence”. arXiv. http://arxiv.org/abs/1703.04200v3.

Harvard

Zenke, F., Poole, B. and Ganguli, S. (2017) “Continual Learning Through Synaptic Intelligence”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1703.04200v3.

Vancouver

1. Zenke F, Poole B, Ganguli S (2017) Continual Learning Through Synaptic Intelligence. arXiv

BibTeX

@article{zenke2017continual,
  title = {Continual Learning Through Synaptic Intelligence},
  author = {Zenke, Friedemann and Poole, Ben and Ganguli, Surya},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1703.04200v3},
  eprint = {1703.04200}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Continual Learning Through Synaptic Intelligence

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Path-Integral Synaptic Importance Measure

Knowl 2 — Quadratic Surrogate Loss with Cumulative Synaptic Consolidation

Knowl 3 — Synaptic Intelligence Algorithm for Continual Learning

Knowl 4 — Correspondence Between Path-Integral Importance and the Hessian in Quadratic Error Functions

Knowl 5 — Comparison Between Synaptic Intelligence and Fisher Information Metrics

Knowl 6 — Permuted MNIST Continual Learning Performance

Knowl 7 — Split MNIST Continual Learning Performance

Knowl 8 — Decorrelation of Synaptic Importance Across Tasks Under Consolidation

Knowl 9 — Split CIFAR-10 and CIFAR-100 Continual Learning Performance

Knowl 10 — Memory Retention and Forward Transfer on Split CIFAR-10

References

Citation

BibTeX

Source Code

Access the Paper