Forget-free Continual Learning with Winning Subnetworks

Haeyong KangRusty John Lloyd MinaSultan Rizky Hikmawan MadjidJaehong YoonMark Hasegawa-JohnsonSung Ju HwangChang D. Yoo

article2022ICML187 citations

Proposes a continual learning framework that completely prevents catastrophic forgetting by isolating and reusing task-specific sparse subnetworks within a single model, compressing the resulting binary masks with Huffman coding to achieve sub-linear memory growth across sequential tasks.

Listen

Continual learning requires artificial intelligence systems to learn a sequence of new tasks without degrading performance on previously learned tasks. In conventional deep neural networks, updating network parameters on new data typically causes catastrophic forgetting, where earlier knowledge is overwritten and lost. Existing remedies often expand network architectures as tasks accumulate or store historical data in memory buffers to replay past experiences. However, both approaches lead to substantial memory and computation overhead, making them impractical for resource-constrained or long-term operational environments.

The article aims to introduce and validate a continual learning framework called Winning SubNetworks. This method seeks to prevent catastrophic forgetting completely while maintaining high task accuracy and enforcing sub-linear growth in network memory capacity.

The proposed framework builds on the concept that dense neural networks contain compact, highly performant subnetworks. The approach jointly learns the model weights alongside separate task-adaptive binary masks, which identify and activate the most critical subnetwork connections for each incoming task. Once a subnetwork is selected for a task, its weights are frozen to prevent future tasks from altering them, while new tasks selectively reuse these frozen connections alongside a small set of unallocated weights. To store subnetwork configurations efficiently across long task sequences without ballooning storage, the article applies Huffman coding to compress the accumulated binary masks. The authors validated the method through multi-task classification experiments across six standard benchmark datasets—ranging up to 100 sequential tasks—using diverse network architectures.

The experimental evaluations yielded four principal findings. First, the method achieves complete immunity to catastrophic forgetting, maintaining zero performance loss on prior tasks across all tested datasets. Second, the framework achieved higher overall accuracy than competing continual learning techniques, including top average accuracies of 87.28% on Omniglot Rotation and 71.96% on TinyImageNet. Third, the selective reuse of weights and Huffman coding reduced capacity requirements significantly; for example, on TinyImageNet, the model operated within 48.65% capacity compared to baselines exceeding 100% to 200%. Finally, selective knowledge reuse substantially improved computational efficiency, leading to faster training convergence across all evaluated benchmarks.

These findings demonstrate that artificial intelligence systems can continually adapt to new operational requirements without requiring linear infrastructure expansion or expensive retraining cycles. For organizations deploying machine learning at scale, this method provides a way to reduce compute costs, cut storage footprints, and guarantee that existing capabilities remain stable. The strategy shifts continual learning from resource-heavy memory buffers toward structural subnetwork isolation.

Organizations developing edge computing devices or sequential machine learning pipelines should consider evaluating modular subnetwork masking architectures as a viable strategy to manage capacity constraints. In systems where past tasks share structural commonalities, tuning target capacity ratios—such as reserving roughly 10% capacity per task—provides a balanced trade-off between task accuracy and memory growth. Before wide production deployment, teams should conduct pilot evaluations to assess layer-specific sensitivity and verify performance on complex domain shifts, as the current experiments rely on supervised classification setups where task identities are explicitly known during inference.

AITRICS Korea Advanced Institute of Science and Technology University of Illinois Urbana-Champaign huffman wireless sensor networks sensor networks forget-free continual learning winning subnetworks Selective Reuse Expansion

ihaeyong/WSN Kang et al (2022).pdf

Paper: PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning, Arun Mallya et al. (2017). PackNet’s iterative pruning and task-specific parameter masks provide the closest earlier blueprint for WSN’s reuse of sparse subnetworks to prevent forgetting.
Paper: Overcoming catastrophic forgetting in neural networks, James Kirkpatrick et al. (2017). EWC establishes the continual-learning problem and contrasts weight-importance regularization with WSN’s strategy of isolating task-specific weights.
Paper: Overcoming catastrophic forgetting with hard attention to the task, Joan Serrà et al. (2018). HAT shows how learned task-specific masks protect neural pathways, making its gating approach a useful precursor to WSN’s binary subnetwork masks.
Paper: Progressive Neural Networks, Andrei A. Rusu et al. (2016). Progressive Neural Networks introduce isolated task-specific network components and selective reuse of prior features, clarifying the architectural trade-offs WSN addresses.
Paper: A Continual Learning Survey: Defying Forgetting in Classification Tasks, Matthias De Lange et al. (2019). This survey organizes continual-learning methods, including parameter-isolation approaches, and supplies the taxonomy needed to situate WSN’s winning subnetworks.

Paper: Task-Specific Skill Localization in Fine-tuned Language Models, Abhishek Panigrahi et al. (2023). Task-Specific Skill Localization carries binary-mask isolation into fine-tuned language models, extending WSN’s subnetwork idea to pinpointing compact task-specific capabilities.
Paper: Localizing Task Information for Improved Model Merging and Compression, Ke Wang et al. (2024). This work uses task-specific binary masks to localize information in merged models, extending the mask-based separation central to WSN toward model merging and compression.
Paper: Dense Network Expansion for Class Incremental Learning, Zhiyuan Hu et al. (2023). Dense Network Expansion continues the search for scalable task-specific capacity by reusing features across sequential tasks instead of allowing model growth to run unchecked.
Paper: Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning, Kai Zhu et al. (2022). This approach extends architectural adaptation for continual learning with temporary branches that absorb new classes and are then consolidated without increasing model size.

Cover for Forget-free Continual Learning with Winning Subnetworks

Abstract

Inspired by Lottery Ticket Hypothesis that competitive subnetworks exist within a dense network, we propose a continual learning method referred to as Winning SubNetworks (WSN) which sequentially learns and selects an optimal subnetwork for each task. Specifically, WSN jointly learns the model weights and task-adaptive binary masks pertaining to subnetworks associated with each task whilst attempting to select a small set of weights to be activated (winning ticket) by reusing weights of the prior subnetworks. The proposed method is inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other subnetworks. Binary masks spawned per winning ticket are encoded into one N-bit binary digit mask, then compressed using Huffman coding for a sub-linear increase in network capacity with respect to the number of tasks. Code is available at https://github.com/ihaeyong/WSN.

Topics

huffman wireless sensor networks sensor networks huffman code

Keywords

forget-free continual learning winning subnetworks Selective Reuse Expansion catastrophic forgetting layer-wise forgetting iterative pruning accuracy-capacity tradeoff prior subnetworks

1. Introduction
2. Related Works
3. Forget-Free Continual Learning with Winning SubNetworks
3.1. Winning SubNetworks
3.2. Optimization Procedure for Winning SubNetworks
3.3. Binary Mask Encoding
4. Experiments
5. Results and Discussion
5.1. Comparisons with the Baseline
5.2. Comparisons with the SOTA
5.3. Forget-Free Performance and Model Capacity
5.4. Catastrophic Forgetting From WSN's Viewpoint
5.5. Re-initialized WSN v.s. Initialized WSN
5.6. Sparse Binary Maps
6. Conclusion
References
A. Experimental Details
A.1. Datasets for Baseline and SOTA
A.2. Architecture Details
A.3. Training Details
A.4. List of Hyperparameters
B. Additional Results
B.1. Additional Analysis on Baseline Comparisons.
B.2. Forget-Free Performances and Capacities.
B.3. Comparisons with the SOTA
B.4. Comparisons of SupSup and WSN
B.5. Capacities of Re-used Weights.
B.6. Sparse Binary Maps
B.7. Additional Analysis

Knowls

Knowl 1 — Winning SubNetworks (WSN) Continual Learning Framework
model/method
Winning SubNetworks (WSN) is a pruning-based continual learning method that jointly optimizes model parameters and task-specific binary masks without requiring explicit iterative pruning, retraining, or parameter rewinding. For a sequence of $T$ supervised tasks where task $t \in \{1, \dots, T\}$ has dataset $\mathcal{D}_t = \{(x_{i,t}, y_{i,t})\}_{i=1}^{n_t}$ , WSN defines a shared dense neural network parameterized by weights $\theta \in \mathbb{R}^{|\theta|}$ and a corresponding set of continuous learnable weight scores $s \in \mathbb{R}^{|\theta|}$ .

For task $t$ , a binary mask $m_t \in \{0, 1\}^{|\theta|}$ is generated by selecting the weights corresponding to the top- $c\%$ highest values of $s$ in each layer, where $c \in (0, 100]$ represents the target layer-wise capacity ratio:

$m_t = \mathbf{1}_c(s)$

where $\mathbf{1}_c(\cdot)$ evaluates to $1$ for weights with scores within the top- $c\%$ percentile of their respective layer and $0$ otherwise. The task-specific subnetwork $\hat{\theta}_t$ is computed via the Hadamard element-wise product:

$\hat{\theta}_t = \theta \odot m_t$

WSN searches for an optimal subnetwork mask $m_t^*$ by optimizing the objective:

$m_t^* = \arg\min_{m_t \in \{0,1\}^{|\theta|}} \frac{1}{n_t} \sum_{i=1}^{n_t} \mathcal{L}\left(f(x_{i,t}; \theta \odot m_t), y_{i,t}\right) \quad \text{subject to } |m_t^*| \le c$

where $\mathcal{L}(\cdot, \cdot)$ is a standard classification loss (such as cross-entropy) and $f(x; \theta \odot m_t)$ denotes the network output parameterized by the masked weights.
Knowl 2 — Optimization and Selective Weight Update Algorithm for WSN
algorithm
To train sequentially without catastrophic forgetting, WSN freezes all weights that were selected for any previous task, while allowing the current task to selectively reuse those frozen weights and update only previously unselected weights. Because the thresholding operation $\mathbf{1}_c(s)$ is non-differentiable with a zero derivative almost everywhere, weight scores $s$ are updated using the Straight-Through Estimator (STE), passing gradients directly to $s$ .

Let $M_{t-1} = \bigvee_{i=1}^{t-1} m_i$ be the cumulative binary mask across all tasks prior to task $t$ , with $M_0 = \mathbf{0}$ . For an objective loss $\mathcal{L}$ and learning rate $\eta$ , model weights $\theta$ and weight scores $s$ are updated via:

$\theta \leftarrow \theta - \eta \left( \frac{\partial \mathcal{L}}{\partial \theta} \odot (1 - M_{t-1}) \right)$

$s \leftarrow s - \eta \frac{\partial \mathcal{L}}{\partial s}$
Input: Dataset sequence { $D_t$ } $_{t=1}^T$ , initial weights $\theta$ , initial score weights $s$ , layer-wise capacity $c$ , learning rate $\eta$
Output: Trained weights $\theta$ and task masks { $m_t$ } $_{t=1}^T$
$M_0 \leftarrow \mathbf{0}^{|\theta|}$
Randomly initialize $\theta$ and $s$
for $t = 1, \dots, T$ do
    for batch $b_t \sim D_t$ do
        Obtain binary mask $m_t$ from top- $c$ \% scores $s$ at each layer
        Compute loss $\mathcal{L}(\theta \odot m_t; b_t)$
         $\theta \leftarrow \theta - \eta \left( \frac{\partial \mathcal{L}}{\partial \theta} \odot (1 - M_{t-1}) \right)$
         $s \leftarrow s - \eta \frac{\partial \mathcal{L}}{\partial s}$
    end for
     $M_t \leftarrow M_{t-1} \lor m_t$
end for
Knowl 3 — Lossless Task Binary Mask Compression via Huffman Coding
model/method
Storing individual binary masks $m_t \in \{0, 1\}^{|\theta|}$ for each task $t \in \{1, \dots, T\}$ would linearly scale memory consumption as tasks accumulate. WSN avoids this overhead by compressing the sequence of masks using Huffman coding.

Across $T$ tasks, each parameter position across layers is associated with a $T$ -bit binary vector indicating its activation state across tasks. This $T$ -bit binary sequence is converted into an accumulated integer/decimal value, and each integer is mapped to a unique ASCII symbol. The resulting stream of symbols is compressed losslessly using Huffman coding.

Because WSN repeatedly reuses subsets of previously trained weights for subsequent tasks, the resulting symbol distribution has high redundancy (frequently reused weight patterns yield high-probability symbols, which receive shorter Huffman codewords). Huffman encoding of 7-bit binary maps achieves an empirical compression rate of approximately $\alpha \approx 0.78$ without bit loss, causing overall subnetwork mask capacity to grow sub-linearly with the number of tasks.
Knowl 4 — Continual Learning Evaluation Metrics: ACC, CAP, and BWT
definition
Continual learning models in WSN are evaluated across three primary metrics over a sequence of $T$ tasks:
1. Average Accuracy (ACC): Measures the mean test classification accuracy across all $T$ tasks after training on the final task $T$ : $\text{ACC} = \frac{1}{T} \sum_{i=1}^T A_{T,i}$ where $A_{T,i}$ is the test accuracy on task $i$ after training on task $T$ .
2. Model Capacity (CAP): Measures the percentage of memory used relative to a single dense 32-bit float backbone network, accounting for non-zero weights and compressed binary masks: $\text{CAP} = (1 - S) + \frac{(1 - \alpha)T}{32}$ where $S$ is the parameter sparsity of the cumulative mask $M_T$ , $T$ is the total number of tasks, $\alpha$ is the average Huffman compression rate (empirically $\alpha \approx 0.78$ for 7-bit encoding), and weights are assumed to be 32-bit precision.
3. Backward Transfer (BWT): Measures catastrophic forgetting on previous tasks: $\text{BWT} = \frac{1}{T-1} \sum_{i=1}^{T-1} (A_{T,i} - A_{i,i})$ where $A_{i,i}$ is the test accuracy on task $i$ immediately after training on task $i$ . A negative BWT indicates forgetting, whereas $\text{BWT} = 0.0$ indicates zero catastrophic forgetting (forget-free learning).

Knowl 5 — Empirical Performance on Permuted MNIST, 5-Datasets, and Omniglot Rotation

data/table

WSN was evaluated against regularization-based, rehearsal-based, and pruning-based baselines across Permuted MNIST (10 tasks, 2-layer MLP with 100 hidden units), 5-Datasets (5 vision datasets, reduced ResNet-18), and Omniglot Rotation (100 tasks, modified LeNet). Results report mean (standard deviation) across 5 runs.

Method	Permuted MNIST			5 Datasets			Omniglot Rotation
	ACC (%)	CAP (%)	BWT	ACC (%)	CAP (%)	BWT	ACC (%)	CAP (%)	BWT
STL	97.37 ( 0.01)	1000.0	-	93.44 ( 0.12)	500.0	-	82.13 ( 0.08)	10000.0	-
FINETUNE	78.22 ( 0.84)	100.0	-0.21 ( 0.01)	80.06 ( 0.74)	100.0	-0.17 ( 0.01)	44.48 ( 1.68)	100.0	-0.45 ( 0.02)
EWC	92.01 ( 0.56)	100.0	-0.03 ( 0.00)	88.64 ( 0.26)	100.0	-0.04 ( 0.01)	68.66 ( 1.92)	100.0	-
HAT	-	-	-	91.32 ( 0.18)	100.0	-0.03 ( 0.00)	-	-	-
GPM	94.96 ( 0.07)	100.0	-0.02 ( 0.01)	91.22 ( 0.20)	100.0	-0.01 ( 0.00)	85.24 ( 0.37)	100.0	-0.01 ( 0.00)
PackNet	96.37 ( 0.04)	96.38	0.0	92.81 ( 0.12)	82.86	0.0	30.70 ( 1.50)	399.2	0.0
SupSup	96.31 ( 0.09)	122.89 ( 0.07)	0.0	93.28 ( 0.21)	104.27 ( 0.21)	0.0	58.14 ( 2.42)	407.12 ( 0.17)	0.0
WSN, $c = 0.03$	94.84 ( 0.11)	19.87 ( 0.16)	0.0	90.57 ( 0.65)	12.11 ( 0.06)	0.0	80.68 ( 2.60)	75.87 ( 1.24)	0.0
WSN, $c = 0.05$	95.65 ( 0.03)	26.49 ( 0.16)	0.0	91.61 ( 0.21)	17.26 ( 0.25)	0.0	87.28 ( 0.72)	79.85 ( 1.19)	0.0
WSN, $c = 0.1$	96.14 ( 0.03)	40.41 ( 0.54)	0.0	92.67 ( 0.12)	28.01 ( 0.28)	0.0	83.10 ( 1.56)	83.08 ( 1.61)	0.0
WSN, $c = 0.3$	96.41 ( 0.07)	77.73 ( 0.36)	0.0	93.22 ( 0.32)	62.30 ( 0.69)	0.0	81.89 ( 1.15)	102.2 ( 0.89)	0.0
WSN, $c = 0.5$	96.24 ( 0.11)	98.10 ( 0.25)	0.0	93.41 ( 0.13)	86.10 ( 0.57)	0.0	79.80 ( 2.16)	121.2 ( 0.50)	0.0
MTL	96.70 ( 0.02)	100.0	-	91.54 ( 0.28)	100.0	-	81.23 ( 0.52)	100.0	-

WSN achieves strictly zero forgetting ( $\text{BWT} = 0.0$ ) across all settings. On the 100-task Omniglot Rotation setup, non-reusing pruning methods like PackNet collapse ( $30.70\%$ accuracy, $399.2\%$ capacity) due to parameter exhaustion, whereas WSN ( $c = 0.05$ ) attains $87.28\%$ accuracy with only $79.85\%$ capacity usage.

Knowl 6 — Empirical Performance on Complex Visual Benchmarks

data/table

WSN was evaluated on CIFAR-100 Split (10 tasks of 10 classes, modified AlexNet), CIFAR-100 Superclass (20 tasks of 5 classes, modified LeNet), and TinyImageNet (40 tasks of 5 classes, 4 Conv + 3 FC network). Results report mean (standard deviation) over 5 random seeds.

Method	CIFAR-100 Split			CIFAR-100 Superclass			TinyImageNet
	ACC (%)	CAP (%)	BWT (%)	ACC (%)	CAP (%)	BWT (%)	ACC (%)	CAP (%)	BWT (%)
La-MaML	71.37 ( 0.67)	100.0	-5.39 ( 0.53)	54.44 ( 1.36)	100.0	-6.65 ( 0.85)	66.90 ( 1.65)	100.0	-9.13 ( 0.90)
GPM	73.18 ( 0.52)	100.0	-1.17 ( 0.27)	57.33 ( 0.37)	100.0	-0.37 ( 0.12)	67.39 ( 0.47)	100.0	1.45 ( 0.22)
FS-DGPM	74.33 ( 0.31)	100.0	-2.71 ( 0.17)	58.81 ( 0.34)	100.0	-2.97 ( 0.35)	70.41 ( 1.30)	100.0	-2.11 ( 0.84)
PackNet	72.39 ( 0.37)	96.38 ( 0.00)	0.0	58.78 ( 0.52)	126.65 ( 0.00)	0.0	55.46 ( 1.22)	188.67 ( 0.00)	0.0
SupSup	75.47 ( 0.30)	129.00 ( 0.03)	0.0	61.70 ( 0.31)	162.49 ( 0.00)	0.0	59.60 ( 1.05)	214.52 ( 0.89)	0.0
WSN, $c = 0.03$	70.65 ( 0.36)	18.56 ( 0.25)	0.0	54.99 ( 0.71)	22.30 ( 0.22)	0.0	68.72 ( 1.63)	37.19 ( 0.21)	0.0
WSN, $c = 0.05$	72.44 ( 0.27)	25.09 ( 0.42)	0.0	57.99 ( 1.34)	27.37 ( 0.33)	0.0	71.22 ( 0.94)	41.98 ( 0.52)	0.0
WSN, $c = 0.1$	74.55 ( 0.47)	39.87 ( 0.62)	0.0	60.45 ( 0.37)	38.55 ( 0.20)	0.0	71.96 ( 1.41)	48.65 ( 3.03)	0.0
WSN, $c = 0.3$	75.98 ( 0.68)	80.26 ( 1.53)	0.0	61.47 ( 0.30)	63.47 ( 1.33)	0.0	70.92 ( 1.37)	73.44 ( 2.35)	0.0
WSN, $c = 0.5$	76.38 ( 0.34)	99.13 ( 0.48)	0.0	61.79 ( 0.23)	80.93 ( 1.58)	0.0	69.06 ( 0.82)	92.03 ( 1.80)	0.0
Multitask	79.75 ( 0.38)	100.0	-	61.00 ( 0.20)	100.0	-	77.10 ( 1.06)	100.0	-

WSN outperforms existing state-of-the-art continual learning methods, reaching peak average accuracies of $76.38\%$ , $61.79\%$ , and $71.96\%$ on CIFAR-100 Split, CIFAR-100 Superclass, and TinyImageNet respectively, while preserving strictly zero backward forgetting and maintaining total capacity under $100\%$ of the base dense model.

Knowl 7 — Training Time and Computational Efficiency Comparison

data/table

Because WSN directly transfers and reuses existing weights from previous tasks without requiring iterative pruning loops or rewinding, it converges faster than competing subnetwork and pruning baselines (PackNet and SupSup) across all benchmark datasets. Training times measured in hours on an NVIDIA V100 GPU are summarized below:

Method	PMNIST	5-Datasets	Omniglot Rot.	CIFAR-100 Split	CIFAR-100 SC	TinyImageNet
PackNet	0.49 ( 0.03)	3.38 ( 0.11)	7.30 ( 0.01)	1.04 ( 0.19)	0.46 ( 0.01)	1.39 ( 0.03)
SupSup	0.48 ( 0.06)	3.20 ( 0.01)	6.92 ( 0.03)	0.79 ( 0.14)	0.37 ( 0.00)	0.92 ( 0.00)
WSN (best)	0.35 ( 0.02)	3.02 ( 0.03)	6.33 ( 0.04)	0.71 ( 0.09)	0.36 ( 0.00)	0.89 ( 0.00)

WSN consistently achieves the lowest wall-clock training time across all datasets while yielding higher accuracy and lower memory capacity.

Knowl 8 — Weight Score Persistence vs. Per-Task Re-initialization
empirical result
Maintaining continuous weight scores $s$ across tasks is essential to WSN's parameter efficiency and forward transfer capability. If weight scores $s$ are re-initialized at the beginning of each new task rather than updated continuously from previous tasks:
1. Premature Network Saturation: Re-initializing $s$ causes the learner to randomly explore subnetworks across the entire parameter space. Within a few initial tasks, all weights in the network are allocated, driving the newly learned parameters to $0\%$ while the proportion of weights shared across all tasks drops to $0\%$ .
2. Performance Degradation and Capacity Inflation: At $c = 0.5$ , re-initialized WSN achieves lower accuracy and requires higher capacity than continuously initialized WSN on CIFAR-100 Split ( $75.68\%$ ACC at $104.47\%$ CAP vs. $76.38\%$ ACC at $99.13\%$ CAP), CIFAR-100 Superclass ( $61.72\%$ ACC at $113.74\%$ CAP vs. $61.79\%$ ACC at $80.93\%$ CAP), and TinyImageNet ( $61.13\%$ ACC at $127.49\%$ CAP vs. $69.06\%$ ACC at $92.03\%$ CAP).
By retaining weight scores across tasks, previously learned weights carry an implicit "premium" in their scores, allowing subsequent tasks to selectively reuse appropriate prior weights and finetune only a small fraction of unallocated weights.
Knowl 9 — Layer-wise Weight Reuse Dynamics and Sensitivity Analysis
empirical result
Ablation and layer-wise analysis on TinyImageNet and CIFAR-100 Split demonstrate how weight reuse governs continual learning performance:
1. Reused Weights Significance: Removing reused weights (setting active reused mask entries to 0 during inference) causes severe performance collapse. For instance, on CIFAR-100 Split and TinyImageNet, inference using only newly added task weights results in catastrophic degradation, demonstrating that the subnetwork relies heavily on representations shared from earlier tasks.
2. Capacity Saturation: In early tasks, the proportion of newly activated weights is high. As the task index increases, the proportion of new weights per task diminishes, and total progressive capacity per task saturates around the configured layer-wise capacity ratio $c$ .
3. Layer Sensitivity: Network layers exhibit distinct sensitivities to weight reuse. In convolutional networks, removing reused weights at the first convolutional layer (Conv1) induces the largest accuracy drop, indicating that low-level visual feature representations are the most critical and universally reused parameters across sequential tasks.

Coverage note — None. All major contributed methods, algorithms, compression mechanisms, experimental setups, quantitative benchmark results, and empirical analyses have been captured as self-contained knowls.

References

1.Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Caccia, M., Lin, M., and Page-Caccia, L. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
2.Bengio, Y., Léonard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 2013.
3.Bulatov, Y. notmnist dataset. 2011.
4.Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem. In Proceedings of the International Conference on Learning Representations (ICLR), 2019a.
5.Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. Continual learning with tiny episodic memories. arXiv preprint arXiv:1902.10486, 2019b.
6.Chaudhry, A., Khan, N., Dokania, P. K., and Torr, P. H. Continual learning in low-rank orthogonal subspaces. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
7.Chen, T., Zhang, Z., Liu, S., Chang, S., and Wang, Z. Long live the lottery: The existence of winning tickets in lifelong learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
8.Deng, D., Chen, G., Hao, J., Wang, Q., and Heng, P.-A. Flattening sharpness for dynamic gradient projection memory benefits continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
9.Denil, M., Shakibi, B., Dinh, L., Ranzato, M. A., and de Freitas, N. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
10.Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
11.Golkar, S., Kagan, M., and Cho, K. Continual learning via neural pruning. arXiv preprint arXiv:1903.04476, 2019.
12.Gupta, G., Yadav, K., and Paull, L. La-maml: Look-ahead meta learning for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
13.Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
14.Hinton, G. Neural networks for machine learning, 2012.
15.Huffman, D. A. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098–1101, 1952.
16.Jung, S., Ahn, H., Cha, S., and Moon, T. Continual learning with node-importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
17.Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
18.Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
19.Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097–1105, 2012.
20.Kumar, A. and Daume III, H. Learning task grouping and overlap in multi-task learning. In Proceedings of the International Conference on Machine Learning (ICML), 2012.
21.LeCun, Y. The mnist database of handwritten digits. 1998.
22.Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
23.Li, X., Zhou, Y., Wu, T., Socher, R., and Xiong, C. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
24.Li, Z. and Hoiem, D. Learning without forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
25.Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
26.Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
27.Mallya, A., Davis, D., and Lazebnik, S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
28.McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109–165. Elsevier, 1989.
29.Mirzadeh, S. I., Farajtabar, M., Gorur, D., Pascanu, R., and Ghasemzadeh, H. Linear mode connectivity in multitask and continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
30.Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
31.Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., and Rastegari, M. What’s hidden in a randomly weighted neural network? In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
32.Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001–2010, 2017.
33.Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
34.Saha, G., Garg, I., and Roy, K. Gradient projection memory for continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
35.Serrà, J., Suris, D., Miron, M., and Karatzoglou, A. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
36.Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
37.Stanford. Available online at http://cs231n.stanford.edu/tiny-imagenet-200.zip. CS 231N, 2021.
38.Thrun, S. A Lifelong Learning Perspective for Mobile Robot Control. Elsevier, 1995.
39.Titsias, M. K., Schwarz, J., Matthews, A. G. d. G., Pascanu, R., and Teh, Y. W. Functional regularisation for continual learning with gaussian processes. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
40.Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., and Farhadi, A. Supermasks in superposition. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
41.Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv, 2017.
42.Xu, J. and Zhu, Z. Reinforced continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
43.Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
44.Yoon, J., Kim, S., Yang, E., and Hwang, S. J. Scalable and order-robust continual learning with additive parameter decomposition. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.

Citation

MLA

Kang, H., et al. “Forget-free Continual Learning with Winning Subnetworks”. International Conference on Machine Learning, vol. 162, 2022, pp. 10734–50, https://proceedings.mlr.press/v162/kang22b.html.

APA

Kang, H., Mina, R. J. L., Madjid, S. R. H., Yoon, J., Hasegawa-Johnson, M., Hwang, S. J., & Yoo, C. D. (2022). Forget-free Continual Learning with Winning Subnetworks. International Conference on Machine Learning, 162, 10734–10750. https://proceedings.mlr.press/v162/kang22b.html

Chicago

Kang, H., R. J. L. Mina, S. R. H. Madjid, et al. 2022. “Forget-free Continual Learning with Winning Subnetworks”. International Conference on Machine Learning 162: 10734–50. https://proceedings.mlr.press/v162/kang22b.html.

Harvard

Kang, H. et al. (2022) “Forget-free Continual Learning with Winning Subnetworks”, International Conference on Machine Learning. PMLR, pp. 10734–10750. Available at: https://proceedings.mlr.press/v162/kang22b.html.

Vancouver

1. Kang H, Mina RJL, Madjid SRH, Yoon J, Hasegawa-Johnson M, Hwang SJ, Yoo CD (2022) Forget-free Continual Learning with Winning Subnetworks. In: International Conference on Machine Learning. PMLR, pp 10734–10750

BibTeX

@InProceedings{pmlr-v162-kang22b,
  title = 	 {Forget-free Continual Learning with Winning Subnetworks},
  author =       {Kang, Haeyong and Mina, Rusty John Lloyd and Madjid, Sultan Rizky Hikmawan and Yoon, Jaehong and Hasegawa-Johnson, Mark and Hwang, Sung Ju and Yoo, Chang D.},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {10734--10750},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/kang22b/kang22b.pdf},
  url = 	 {https://proceedings.mlr.press/v162/kang22b.html},
  abstract = 	 {Inspired by Lottery Ticket Hypothesis that competitive subnetworks exist within a dense network, we propose a continual learning method referred to as Winning SubNetworks (WSN), which sequentially learns and selects an optimal subnetwork for each task. Specifically, WSN jointly learns the model weights and task-adaptive binary masks pertaining to subnetworks associated with each task whilst attempting to select a small set of weights to be activated (winning ticket) by reusing weights of the prior subnetworks. The proposed method is inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other subnetworks. Binary masks spawned per winning ticket are encoded into one N-bit binary digit mask, then compressed using Huffman coding for a sub-linear increase in network capacity with respect to the number of tasks.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Forget-free Continual Learning with Winning Subnetworks

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Winning SubNetworks (WSN) Continual Learning Framework

Knowl 2 — Optimization and Selective Weight Update Algorithm for WSN

Knowl 3 — Lossless Task Binary Mask Compression via Huffman Coding

Knowl 4 — Continual Learning Evaluation Metrics: ACC, CAP, and BWT

Knowl 5 — Empirical Performance on Permuted MNIST, 5-Datasets, and Omniglot Rotation

Knowl 6 — Empirical Performance on Complex Visual Benchmarks

Knowl 7 — Training Time and Computational Efficiency Comparison

Knowl 8 — Weight Score Persistence vs. Per-Task Re-initialization

Knowl 9 — Layer-wise Weight Reuse Dynamics and Sensitivity Analysis

References

Citation

BibTeX

Source Code

Access the Paper