SNIP: Single-shot Network Pruning based on Connection Sensitivity

Namhoon LeeThalaiyasingam AjanthanPhilip H. S. Torr

article2018ICLR1,530 citations

Proposes a connection sensitivity metric to prune neural networks in a single step at initialization, eliminating costly pretraining and iterative pruning schedules while preserving accuracy across diverse architectures.

Listen

Modern deep learning models achieve impressive accuracy but are heavily overparameterized, demanding significant computational power and memory. Compressing these models through network pruning makes them viable for resource-constrained environments, yet standard techniques require expensive cycles of pretraining, pruning, and fine-tuning governed by complex schedules and tuning parameters.

The article introduces and evaluates Single-Shot Network Pruning (SNIP), a method that prunes redundant connections in a neural network in a single step at random initialization prior to training. The objective is to demonstrate that structurally vital connections can be identified and isolated before any standard learning occurs, removing the computational overhead of iterative retraining.

The approach calculates a connection sensitivity score for each parameter using a single forward-backward pass over a small batch of training data at variance scaling initialization. This score quantifies how much a connection influences the loss function. Connections with the lowest sensitivity are removed immediately, and the remaining sparse network is trained using standard optimization techniques. The researchers evaluated this method across several benchmark image classification datasets—MNIST, CIFAR-10, and Tiny-ImageNet—using fully connected, convolutional, residual, and recurrent network architectures.

The findings show that SNIP removes 90% to 99% of parameters across architectures while maintaining classification error rates within roughly 1% of dense baseline models. On LeNet architectures, removing up to 99% of parameters caused only negligible performance loss (0.3% to 0.7%), matching or exceeding the accuracy of methods requiring complex iterative retraining. Across modern convolutional, residual, and recurrent networks, pruning 90% to 95% of connections yielded comparable accuracy. Layer visualizations confirmed that retained connections directly align with discriminative input features, and models pruned with SNIP were unable to fit randomized labels, proving that the algorithm preserves task-relevant structural capacity rather than arbitrary connections.

These results demonstrate that extensive pretraining is unnecessary for effective network compression. Implementing single-shot pruning upfront significantly reduces the computational expense, training duration, and engineering complexity associated with model compression pipelines. Furthermore, because it functions as an architecture-agnostic preprocessor, it minimizes the risk and implementation friction of deploying compact deep learning models to embedded or edge devices.

Engineering and research teams should consider adopting single-shot pruning as a lightweight preprocessing step before training deep neural networks. Practitioners should ensure proper variance scaling initialization, as experiments indicate it is essential for model stability, particularly in recurrent networks. Before large-scale deployment, teams should run pilot evaluations on their specific datasets and target architectures to confirm appropriate sparsity thresholds.

The conclusions are supported with high confidence across standard computer vision benchmarks and common network architectures. However, decision-makers should note that evaluations were limited to classification tasks on benchmark datasets rather than production-scale workloads such as full ImageNet, large language models, or real-time control systems. Performance tradeoffs on highly customized architectures or non-classification tasks should be validated independently.

University of Oxford classification network architecture optimization single-shot network pruning connection sensitivity SNIP

arXiv: 1810.02340 namhoonlee/snip-public

Paper: Optimal Brain Damage, Yann LeCun et al. (1989). Introduces foundational saliency-based network pruning using loss Taylor expansions, establishing the theoretical lineage for SNIP's gradient-based connection sensitivity criterion.
Paper: Second Order Derivatives for Network Pruning: Optimal Brain Surgeon, Babak Hassibi et al. (1992). Develops analytical second-order weight importance metrics for pruning, providing fundamental concepts of parameter sensitivity that SNIP simplifies into a single-shot first-order formulation at initialization.
Paper: Learning both Weights and Connections for Efficient Neural Networks, Song Han et al. (2015). Establishes the standard iterative three-stage pipeline of training, pruning, and fine-tuning that SNIP directly seeks to replace with single-shot pruning before training.
Paper: Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding, Song Han et al. (2015). Details the classic multi-step compression paradigm relying on iterative post-training pruning, representing the baseline methodologies SNIP aims to overcome.
Paper: Pruning Convolutional Neural Networks for Resource Efficient Inference, Pavlo Molchanov et al. (2016). Formulates a first-order Taylor expansion criterion to measure parameter importance on network loss, directly inspiring SNIP's gradient-weight product sensitivity metric.
Paper: Pruning Filters for Efficient ConvNets, Hao Li et al. (2016). Demonstrates heuristic sensitivity-driven pruning across network layers, motivating SNIP's unified, hyperparameter-free sensitivity score across architectures.
Paper: Learning Efficient Convolutional Networks through Network Slimming, Zhuang Liu et al. (2017). Presents structured channel pruning using scaling factors during training, highlighting the need for methods like SNIP that eliminate the overhead of pre-training.

Paper: The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks., Jonathan Frankle et al. (2019). Discovers the Lottery Ticket Hypothesis, providing the broader theoretical foundation for why sparse subnetworks identified at initialization (such as by SNIP) can be trained in isolation.
Paper: Pruning neural networks without any data by iteratively conserving synaptic flow, Hidenori Tanaka et al. (2020). Diagnoses the layer-collapse failure mode in single-shot pruning methods like SNIP and introduces SynFlow to preserve synaptic flow across layers without data.
Paper: Rethinking the Value of Network Pruning, Zhuang Liu et al. (2019). Critically evaluates whether inherited weights or the underlying sparse architectures are responsible for pruned model performance, directly contextualizing SNIP's training-from-scratch paradigm.
Paper: Rigging the Lottery: Making All Tickets Winners, Utku Evci et al. (2020). Extends pruning at initialization by allowing dynamic sparse topology updates during training using gradient signals, overcoming static connectivity limits in single-shot pruning.
Paper: Comparing Rewinding and Fine-tuning in Neural Network Pruning, Alex Renda et al. (2020). Investigates weight and learning rate rewinding protocols, contrasting initialization-based subnetwork discovery with post-training pruning and fine-tuning dynamics.
Paper: DepGraph: Towards Any Structural Pruning, Gongfan Fang et al. (2023). Generalizes structural dependency tracking across arbitrary modern architectures, solving inter-layer coupling challenges that single-shot connection-level pruning cannot address.

Cover for SNIP: Single-shot Network Pruning based on Connection Sensitivity

Abstract

Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

Topics

classification network architecture optimization recurrent networks sparse approximations architecture designs optimization problems regularization parameters classification models network components

Keywords

single-shot network pruning connection sensitivity SNIP network compression weight pruning connection saliency network initialization extreme sparsity

1 Introduction
2 Related work
3 Neural network pruning
4 Single-shot network pruning based on connection sensitivity
4.1 Connection sensitivity: architectural perspective
4.2 Single-shot pruning at initialization
5 Experiments
5.1 Pruning LeNets with varying levels of sparsity
5.2 Comparisons to existing approaches
5.3 Various modern architectures
5.4 Understanding which connections are being pruned
5.5 Effects of data and weight initialization
5.6 Fitting random labels
6 Discussion and future work
References
A Visualizing pruned parameters on (inverted) (fashion-)mnist
B Fitting random labels: varying sparsity levels
C Tiny-imagenet
D Architecture details

Knowls

Knowl 1 — Masked optimization formulation for single-shot pruning
model/method
SNIP separates whether a connection exists from the numerical value of its weight. For a training set $D=\{(x_i,y_i)\}_{i=1}^{n}$ , a neural network with $m$ parameters, weights $w\in\mathbb{R}^{m}$ , and target number of retained parameters $\kappa$ , let $c\in\{0,1\}^{m}$ be a binary connectivity mask and let $\odot$ denote elementwise multiplication. Pruning is formulated as
$\min_{c,w} L(c\odot w;D)=\frac{1}{n}\sum_{i=1}^{n}\ell(c\odot w;(x_i,y_i)) \quad\text{subject to}\quad \|c\|_0\leq\kappa,$
where $\ell$ is the per-example training loss, $L$ is the empirical loss, and $\|c\|_0$ counts active connections. The mask $c$ is selected before training; the surviving weights are then learned by ordinary optimization with the mask fixed.
Knowl 2 — Connection sensitivity as a data-dependent saliency criterion
equation
For a network with weights $w\in\mathbb{R}^{m}$ , binary mask $c\in\{0,1\}^{m}$ , loss $L(c\odot w;D)$ , and dataset $D$ , the exact loss change from removing connection $j$ is
$\Delta L_j(w;D)=L(\mathbf{1}\odot w;D)-L((\mathbf{1}-e_j)\odot w;D),$
where $\mathbf{1}\in\mathbb{R}^{m}$ is the all-ones vector and $e_j\in\mathbb{R}^{m}$ is one at index $j$ and zero elsewhere. SNIP approximates this discrete removal influence by the derivative with respect to a multiplicative connectivity variable:
$g_j(w;D)=\left.\frac{\partial L(c\odot w;D)}{\partial c_j}\right|_{c=\mathbf{1}}, \qquad \Delta L_j(w;D)\approx g_j(w;D).$
The connection-sensitivity score is the normalized absolute influence
$s_j=\frac{|g_j(w;D)|}{\sum_{k=1}^{m}|g_k(w;D)|}, \qquad j\in\{1,\ldots,m\}.$
The mask retains the $\kappa$ connections with the largest $s_j$ values, breaking ties arbitrarily. Taking the absolute value preserves connections with large influence regardless of whether an infinitesimal reduction in connectivity would increase or decrease the loss. Unlike $\partial L/\partial w_j$ , $g_j$ measures sensitivity to a multiplicative removal of a connection rather than to an additive change in its weight.
Knowl 3 — SNIP single-shot pruning procedure
algorithm
SNIP takes a differentiable loss function $L$ , training dataset $D$ , desired retained-connection count $\kappa$ , and a network with $m$ parameters. It returns a fixed binary mask and a trained sparse network. The saliency computation uses one mini-batch and one automatic-differentiation pass.
Input: Loss function $L$ , training dataset $D$ , retained-connection count $\kappa$
Output: Binary mask $c$ and trained sparse weights $w^*$
Initialize weights $w$ with variance scaling.
Sample one pruning mini-batch $D_b$ from $D$ .
Set all connectivity variables to one: $c \leftarrow \mathbf{1}$ .
Compute $g_j \leftarrow \partial L(c \odot w;D_b)/\partial c_j$ for every connection $j$ in one forward-backward pass.
Compute $s_j \leftarrow |g_j|/\sum_k |g_k|$ for every connection $j$ .
Find the $\kappa$ -th largest score $\tilde{s}_{\kappa}$ .
Set $c_j \leftarrow 1$ if $s_j \geq \tilde{s}_{\kappa}$ and zero otherwise; break ties to retain exactly $\kappa$ connections.
Train $w$ by ordinary network training on the full training data while applying the fixed mask $c$ .
Return $w^* \leftarrow c \odot w$ .
The reported MNIST and CIFAR-10 training used SGD with momentum $0.9$ , batch sizes $100$ and $128$ , weight decay $0.0005$ , initial learning rate $0.1$ , and learning-rate reductions by $0.1$ every $25{,}000$ or $30{,}000$ iterations, respectively. These are ordinary training settings rather than pruning schedules; SNIP itself adds no pretraining, iterative prune–retrain cycles, or pruning-specific hyperparameters.
Knowl 4 — Variance-scaled initialization makes the criterion architecture-robust
assumption
SNIP computes sensitivity before learning, so the initialization must produce informative, non-saturated activations and gradients. The paper therefore recommends variance-scaling initialization, which keeps signal variance approximately stable across layers. A fixed-variance initialization can make the scores depend strongly on layer width and depth, while overly large weights can saturate nonlinearities and yield uninformative derivatives.

The scores are task-dependent because they use the loss function and examples from the task. A practitioner may compute them from the full training set, one mini-batch, several accumulated batches, or an exponential moving average. The paper's experiments show that a single reasonable training mini-batch is often sufficient, enabling pruning before any pretraining. This design is intended to make the same pruning procedure applicable without architectural modifications to fully connected, convolutional, residual, and recurrent networks.
Knowl 5 — Extreme sparsity on MNIST LeNets
empirical result
SNIP was evaluated on MNIST using LeNet-300-100, containing approximately $267$ k parameters, and LeNet-5-Caffe, containing approximately $431$ k parameters. Each sparsity setting was repeated $20$ times with different dataset and initialization seeds. Across sparsity levels, LeNet-300-100 had test performance close to its dense reference through $90\%$ sparsity, while the degradation for LeNet-5-Caffe was nearly invisible.

At more extreme sparsity, the dense reference errors were $1.7\%$ for LeNet-300-100 and $0.9\%$ for LeNet-5-Caffe. SNIP retained only $2\%$ of LeNet-300-100's parameters ( $98\%$ sparsity) with error $2.4\%$ , and only $1\%$ of LeNet-5-Caffe's parameters ( $99\%$ sparsity) with error $1.1\%$ . At less extreme settings, $95\%$ sparsity on LeNet-300-100 produced $1.6\%$ error and $98\%$ sparsity on LeNet-5-Caffe produced $0.8\%$ error, both numerically below the corresponding dense-reference errors. These results show that connections selected before training can provide nearly all of the task-relevant capacity of the dense networks.
Knowl 6 — Comparison with iterative and Bayesian pruning methods
data/table
On MNIST LeNets, SNIP was compared with magnitude-based LWC, DNS, and LC; Bayesian SWS and SVD; and Hessian-based OBD and L-OBS. The dense reference errors were $1.7\%$ for LeNet-300-100 and $0.9\%$ for LeNet-5-Caffe. The comparison reported the following representative operating points:
- LWC: $91.7\%$ sparsity and $1.6\%$ error on LeNet-300-100; $91.7\%$ and $0.8\%$ on LeNet-5-Caffe.
- DNS: $98.2\%$ and $2.0\%$ ; $99.1\%$ and $0.9\%$ .
- LC: $99.0\%$ and $3.2\%$ ; $99.0\%$ and $1.1\%$ .
- SWS: $95.6\%$ and $1.9\%$ ; $99.5\%$ and $1.0\%$ .
- SVD: $98.5\%$ and $1.9\%$ ; $99.6\%$ and $0.8\%$ .
- OBD: $92.0\%$ and $2.0\%$ ; $92.0\%$ and $2.7\%$ .
- L-OBS: $98.5\%$ and $2.0\%$ ; $99.0\%$ and $2.1\%$ .
- SNIP: $95.0\%$ sparsity and $1.6\%$ error, or $98.0\%$ and $2.4\%$ , on LeNet-300-100; and $98.0\%$ and $0.8\%$ , or $99.0\%$ and $1.1\%$ , on LeNet-5-Caffe.
Unlike the comparison methods, SNIP used no pretraining, one pruning operation, no additional pruning hyperparameters, no augmented training objective, and no architecture-dependent pruning constraints. The comparison therefore demonstrates not only competitive accuracy at extreme sparsity but also the substantially simpler training procedure claimed by the paper.
Knowl 7 — Applicability to convolutional, residual, and recurrent networks
data/table
The paper applied the unchanged SNIP procedure to CIFAR-10 convolutional and residual models and to sequential-MNIST recurrent models. Each entry below gives sparsity, parameter count before and after pruning, and classification error before and after pruning; $\Delta$ is the after-minus-before error in percentage points.
- AlexNet-s: $90.0\%$ sparsity; $5.1$ m $\rightarrow$ $507$ k parameters; error $14.12\%\rightarrow14.99\%$ ; $\Delta=+0.87$ .
- AlexNet-b: $90.0\%$ ; $8.5$ m $\rightarrow$ $849$ k; $13.92\%\rightarrow14.50\%$ ; $\Delta=+0.58$ .
- VGG-C: $95.0\%$ ; $10.5$ m $\rightarrow$ $526$ k; $6.82\%\rightarrow7.27\%$ ; $\Delta=+0.45$ .
- VGG-D: $95.0\%$ ; $15.2$ m $\rightarrow$ $762$ k; $6.76\%\rightarrow7.09\%$ ; $\Delta=+0.33$ .
- VGG-like: $97.0\%$ ; $15.0$ m $\rightarrow$ $449$ k; $8.26\%\rightarrow8.00\%$ ; $\Delta=-0.26$ .
- WRN-16-8: $95.0\%$ ; $10.0$ m $\rightarrow$ $548$ k; $6.21\%\rightarrow6.63\%$ ; $\Delta=+0.42$ .
- WRN-16-10: $95.0\%$ ; $17.1$ m $\rightarrow$ $856$ k; $5.91\%\rightarrow6.43\%$ ; $\Delta=+0.52$ .
- WRN-22-8: $95.0\%$ ; $17.2$ m $\rightarrow$ $858$ k; $6.14\%\rightarrow5.85\%$ ; $\Delta=-0.29$ .
- LSTM-s: $95.0\%$ ; $137$ k $\rightarrow$ $6.8$ k; $1.88\%\rightarrow1.57\%$ ; $\Delta=-0.31$ .
- LSTM-b: $95.0\%$ ; $535$ k $\rightarrow$ $26.8$ k; $1.15\%\rightarrow1.35\%$ ; $\Delta=+0.20$ .
- GRU-s: $95.0\%$ ; $104$ k $\rightarrow$ $5.2$ k; $1.87\%\rightarrow2.41\%$ ; $\Delta=+0.54$ .
- GRU-b: $95.0\%$ ; $404$ k $\rightarrow$ $20.2$ k; $1.71\%\rightarrow1.52\%$ ; $\Delta=-0.19$ .
The convolutional models were evaluated on CIFAR-10 and the recurrent models on sequential MNIST, where each recurrent cell processed one image row at a time. Every model retained at least $90\%$ sparsity, and every reported error change was below $1$ percentage point, supporting the claim that the score is not tied to a particular architecture type.
Knowl 8 — Performance on the more difficult Tiny-ImageNet task
empirical result
SNIP was also tested on Tiny-ImageNet, which has $200$ classes, $500$ training images and $50$ validation images per class, with $64\times64$ input resolution. The reported parameter counts and errors were:
- AlexNet-s at $90.0\%$ sparsity: $5.1$ m $\rightarrow$ $507$ k parameters and error $62.52\%\rightarrow65.27\%$ ( $\Delta=+2.75$ percentage points).
- AlexNet-b at $90.0\%$ : $8.5$ m $\rightarrow$ $849$ k and $62.76\%\rightarrow65.54\%$ ( $\Delta=+2.78$ ).
- VGG-C at $95.0\%$ : $10.5$ m $\rightarrow$ $526$ k and $56.49\%\rightarrow57.48\%$ ( $\Delta=+0.99$ ).
- VGG-D at $95.0\%$ : $15.2$ m $\rightarrow$ $762$ k and $56.85\%\rightarrow57.00\%$ ( $\Delta=+0.15$ ).
- VGG-like at $95.0\%$ : $15.0$ m $\rightarrow$ $749$ k and $54.86\%\rightarrow55.73\%$ ( $\Delta=+0.87$ ).
Thus, even on this more complex classification problem, the method removed $90$ – $95\%$ of parameters with small error increases for the VGG models. The paper notes that the larger AlexNet degradation may be related to its first convolution using stride $4$ on the higher-resolution images, which can lose information when connections are pruned.
Knowl 9 — Retained connections expose task-relevant image structure
empirical result
To inspect what SNIP preserves, the paper used the first fully connected layer of LeNet-300-100, whose weights connect $784$ input pixels to $300$ hidden units. For a class-specific mini-batch, the binary mask was averaged across the $300$ input-to-hidden columns and reshaped from $784$ entries into a $28\times28$ image. The visualizations on page 9 show that, as sparsity increases from $10\%$ to $90\%$ , retained connections concentrate on discriminative regions: digit foregrounds for MNIST and clothing silhouettes for Fashion-MNIST, while much of the background is removed.

This behavior is class-dependent: a batch of digit-0 examples preserves connections covering the foreground of the zero, and the Fashion-MNIST masks reveal the identity and shape of the clothing class. The appendix visualization on page 13 shows that inverting image intensities produces essentially the same SNIP masks, whereas directly pruning by the weight gradient $\partial L/\partial w$ gives different and inconsistent patterns across sparsity levels and between original and inverted data. These observations provide the paper's interpretability evidence that SNIP selects connections relevant to the task rather than merely pruning blindly.
Knowl 10 — Effects of pruning-batch size and weight initialization
empirical result
The paper measured how the pruning data and initialization affect SNIP on LeNet-300-100 at $90\%$ sparsity. For pruning batches of sizes $|D_b|=1,10,100,1000,10000$ , the resulting classification errors were respectively $1.94\%$ , $1.72\%$ , $1.64\%$ , $1.56\%$ , and $1.40\%$ . The $|D_b|=1$ mask closely reflected the individual sampled example, while larger batches produced masks increasingly similar to the average pattern over the training set and lower errors. The training batch size remained $100$ in this analysis.

Initialization was evaluated over $20$ runs at $90\%$ sparsity using random normal (RN), truncated random normal (TN), Glorot variance scaling (VS-X), and He variance scaling (VS-H). The reported mean classification errors with standard deviations were:
- LeNet-300-100: RN $1.90\pm0.09\%$ , TN $1.96\pm0.11\%$ , VS-X $1.91\pm0.10\%$ , VS-H $1.88\pm0.10\%$ .
- LeNet-5-Caffe: RN $0.89\pm0.04\%$ , TN $0.87\pm0.05\%$ , VS-X $0.88\pm0.07\%$ , VS-H $0.85\pm0.05\%$ .
- LSTM-s: RN $2.93\pm0.20\%$ , TN $3.03\pm0.17\%$ , VS-X $1.48\pm0.09\%$ , VS-H $1.47\pm0.08\%$ .
- GRU-s: RN $47.61\pm20.49\%$ , TN $46.48\pm22.25\%$ , VS-X $1.80\pm0.10\%$ , VS-H $1.80\pm0.14\%$ .
Variance scaling made little difference for the LeNets but was essential for the recurrent models, especially the GRU, where non-variance-scaled initializations often produced unusable sparse networks.
Knowl 11 — SNIP-selected networks resist memorizing random labels
empirical result
The paper tested whether SNIP's sparse subnetworks merely retain enough capacity to memorize arbitrary labels. LeNet-5-Caffe was trained on MNIST with true labels and with randomly shuffled labels; the sparse model used $99\%$ sparsity, and its SNIP mask was always computed using the true labels. With true labels, both the dense reference and the SNIP-pruned model rapidly reached nearly zero training loss. With random labels, the dense reference also reached very low training loss, even when an explicit $L_2$ regularizer was added, whereas the SNIP-pruned model failed to fit the random labels and retained high training error.

The appendix's varying-sparsity experiment showed that less sparse SNIP models, which retain more parameters, achieve lower training loss on random labels, while all tested sparse models still learn the genuine MNIST classification task without substantial accuracy loss. The paper interprets this contrast as evidence that SNIP preserves task-relevant capacity while removing capacity useful for memorizing arbitrary labels, although it also notes that insufficient capacity is a possible explanation.

Coverage note — Low-level architecture specifications, the full related-work discussion, and supplementary visualization details were omitted because they support the experiments without adding separate load-bearing method or result claims.

References

1.Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. ICML, 2018.
2.Leo Breiman. Better subset regression using the nonnegative garrote. Technometrics, 1995.
3.Miguel A. Carreira-Perpiñan and Yerlan Idelbayev. “Learning-compression” algorithms for neural net pruning. CVPR, 2018.
4.Yves Chauvin. A back-propagation algorithm with optimal use of hidden units. NIPS, 1989.
5.Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP, 2014.
6.Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. NIPS, 2017.
7.Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010.
8.Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
9.Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT press Cambridge, 2016.
10.Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. NIPS, 2016.
11.Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. ICML, 2015.
12.Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. NIPS, 2015.
13.Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. Neural Networks, 1993.
14.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. ICCV, 2015.
15.Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. NIPS, 2016.
16.Masumi Ishikawa. Structural learning with forgetting. Neural Networks, 1996.
17.Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. BMVC, 2014.
18.Ehud D Karnin. A simple procedure for pruning back-propagation trained neural networks. Neural Networks, 1990.
19.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
20.Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. ICML, 2017.
21.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. 2012.
22.Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. CoRR, 2015.
23.Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. NIPS, 1990.
24.Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. Neural Networks: Tricks of the Trade, 1998.
25.Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017.
26.Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l0 regularization. ICLR, 2018.
27.Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 2018.
28.Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. ICML, 2017a.
29.Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. ICLR, 2017b.
30.Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. NIPS, 1989.
31.Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. ICLR, 2017.
32.Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. NIPS, 2015.
33.Steven J Nowlan and Geoffrey E Hinton. Simplifying neural networks by soft weight-sharing. Neural Computation, 1992.
34.Ameya Prabhu, Girish Varma, and Anoop Namboodiri. Deep expander networks: Efficient deep networks from graph theory. ECCV, 2018.
35.Russell Reed. Pruning algorithms-a survey. Neural Networks, 1993.
36.Abigail See, Minh-Thang Luong, and Christopher D Manning. Compression of neural machine translation models via pruning. CoNLL, 2016.
37.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
38.Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. ICLR, 2017.
39.Andreas S Weigend, David E Rumelhart, and Bernardo A Huberman. Generalization by weight-elimination with application to forecasting. NIPS, 1991.
40.Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. NIPS, 2016.
41.Sergey Zagoruyko. 92.45% on cifar-10 in torch. Torch Blog, 2015.
42.Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016.
43.Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.
44.Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.

Citation

MLA

Lee, N., et al. “SNIP: Single-shot Network Pruning Based on Connection Sensitivity”. arXiv, 2018, http://arxiv.org/abs/1810.02340v2.

APA

Lee, N., Ajanthan, T., & Torr, P. H. S. (2018). SNIP: Single-shot Network Pruning based on Connection Sensitivity. arXiv. http://arxiv.org/abs/1810.02340v2

Chicago

Lee, N., T. Ajanthan, and P. H. S. Torr. 2018. “SNIP: Single-shot Network Pruning Based on Connection Sensitivity”. arXiv. http://arxiv.org/abs/1810.02340v2.

Harvard

Lee, N., Ajanthan, T. and Torr, P.H.S. (2018) “SNIP: Single-shot Network Pruning based on Connection Sensitivity”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1810.02340v2.

Vancouver

1. Lee N, Ajanthan T, Torr PHS (2018) SNIP: Single-shot Network Pruning based on Connection Sensitivity. arXiv

BibTeX

@article{lee2018snip,
  title = {SNIP: Single-shot Network Pruning based on Connection Sensitivity},
  author = {Lee, Namhoon and Ajanthan, Thalaiyasingam and Torr, Philip H. S.},
  year = {2018},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1810.02340v2},
  eprint = {1810.02340}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

SNIP: Single-shot Network Pruning based on Connection Sensitivity

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Masked optimization formulation for single-shot pruning

Knowl 2 — Connection sensitivity as a data-dependent saliency criterion

Knowl 3 — SNIP single-shot pruning procedure

Knowl 4 — Variance-scaled initialization makes the criterion architecture-robust

Knowl 5 — Extreme sparsity on MNIST LeNets

Knowl 6 — Comparison with iterative and Bayesian pruning methods

Knowl 7 — Applicability to convolutional, residual, and recurrent networks

Knowl 8 — Performance on the more difficult Tiny-ImageNet task

Knowl 9 — Retained connections expose task-relevant image structure

Knowl 10 — Effects of pruning-batch size and weight initialization

Knowl 11 — SNIP-selected networks resist memorizing random labels

References

Citation

BibTeX

Source Code

Access the Paper