Learning Structured Sparsity in Deep Neural Networks

Wei WenChunpeng WuYandan WangYiran ChenHai Li

article2016NeurIPS2,526 citations

Introduces Structured Sparsity Learning, a regularization framework that prunes filters, channels, and entire layers to generate hardware-friendly deep neural networks that accelerate CPU and GPU inference while maintaining or improving accuracy.

Listen

The article addresses the challenge of deploying large deep neural networks on resource-limited devices, where high computation and memory demands create barriers to practical use. Non-structured sparsity methods often fail to deliver real speed gains because of irregular memory access patterns, while low-rank approximations require repeated decomposition steps and fix layer structures in ways that limit flexibility.

The article set out to develop and test a Structured Sparsity Learning method that applies group Lasso regularization to enforce compact structures across filters, channels, filter shapes, and network depth during training.

The approach involved training and evaluating the method on standard models including LeNet, ConvNet, ResNet, and AlexNet using the MNIST, CIFAR-10, and ImageNet datasets. Structured sparsity was measured through changes in floating-point operations, layer dimensions, and runtime on both CPU and GPU platforms with common libraries, while accuracy was tracked before and after regularization and fine-tuning.

The experiments showed that SSL produced average speedups of 5.1 times on CPU and 3.1 times on GPU for AlexNet convolutional layers, roughly double the gains from non-structured sparsity at similar accuracy levels. On CIFAR-10, depth regularization reduced a 20-layer ResNet to 18 layers while raising accuracy from 91.25 percent to 92.60 percent, exceeding the original 32-layer ResNet. For AlexNet on ImageNet, the method lowered top-1 error by about one percent or maintained accuracy with substantially fewer parameters. Structured sparsity also enabled direct reductions in weight-matrix dimensions for efficient matrix-multiplication routines.

These results indicate that enforcing structured compactness during training yields models that run faster on existing hardware without custom optimizations, while the regularization effect can sometimes improve classification performance. The approach therefore supports faster inference on constrained devices and may reduce both energy use and latency in deployed systems.

The findings support combining SSL with complementary techniques such as low-rank approximation for further gains. Additional testing on newer architectures and broader hardware platforms would strengthen confidence before large-scale adoption.

The work relies on specific published models and datasets from 2016, and depth regularization assumes the presence of shortcut connections; results may vary under different training conditions or without fine-tuning after structure removal.

University of Pittsburgh classification cpu libraries structured sparsity sparsity regularization depth-wise sparsity

arXiv: 1608.03665 tree/scnn

Paper: Learning both Weights and Connections for Efficient Neural Networks, Song Han et al. (2015). Introduces foundational non-structured connection pruning and fine-tuning, which directly motivates the source paper's shift toward regularized structured sparsity.
Paper: Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding, Song Han et al. (2015). Establishes the three-stage deep compression pipeline that the source contrasts against to highlight why unstructured sparsity struggles to achieve direct hardware speedups without specialized accelerators.
Paper: Deep Residual Learning for Image Recognition, Kaiming He et al. (2016). Presents deep residual learning and skip connections, which the source explicitly builds upon and adapts for depth-regularization structured sparsity.
Paper: Optimal Brain Damage, Yann LeCun et al. (1989). Provides the seminal framework of weight saliency and network pruning to simplify architectures and improve generalization.

Paper: Learning Efficient Convolutional Networks through Network Slimming, Zhuang Liu et al. (2017). Extends structured channel-level sparsity by applying regularized scaling factors in batch normalization layers rather than directly regularizing weight tensors.
Paper: Channel Pruning for Accelerating Very Deep Neural Networks, Yihui He et al. (2017). Applies LASSO-based channel selection to reconstruct layer outputs iteratively for deep network acceleration, building on structured filter reduction principles.
Paper: Rethinking the Value of Network Pruning, Zhuang Liu et al. (2019). Critically re-evaluates structured pruning algorithms like the source paper's by investigating whether inherited weights or discovered compact architectures drive performance.
Paper: DepGraph: Towards Any Structural Pruning, Gongfan Fang et al. (2023). Generalizes structured pruning across arbitrary neural network architectures by building explicit dependency graphs to resolve complex inter-layer coupling.
Paper: Efficient Processing of Deep Neural Networks: A Tutorial and Survey, Vivienne Sze et al. (2017). Synthesizes structured sparsity, low-rank approximations, and hardware-software co-design into a comprehensive survey on efficient deep learning execution.

Cover for Learning Structured Sparsity in Deep Neural Networks

Abstract

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in this https URL

1 Introduction
2 Related works
3 Structured Sparsity Learning Method for DNNs
3.1 Proposed structured sparsity learning for generic structures
3.2 Structured sparsity learning for structures of filters, channels, filter shapes and depth
3.3 Structured sparsity learning for computationally efficient structures
4 Experiments
4.1 LeNet and multilayer perceptron on MNIST
4.2 ConvNet and ResNet on CIFAR-10
4.3 AlexNet on ImageNet
5 Conclusion
References

Knowls

Knowl 1 — Generic Objective Function for Structured Sparsity Learning
model/method
Structured Sparsity Learning (SSL) regularizes deep neural network structures during training by grouping weights and applying Group Lasso penalties to zero out entire structural components.

Let the weights of $L$ convolutional layers in a deep neural network be represented by a sequence of 4-D tensors $W^{(l)} \in \mathbb{R}^{N_l \times C_l \times M_l \times K_l}$ for $l \in \{1, \dots, L\}$ , where $N_l$ is the number of filters, $C_l$ is the number of input channels, $M_l$ is the spatial filter height, and $K_l$ is the spatial filter width. Let $W$ represent all trainable parameters in the network. The general optimization objective is:

$E(W) = E_D(W) + \lambda \cdot R(W) + \lambda_g \sum_{l=1}^L R_g\left(W^{(l)}\right)$

where $E_D(W)$ is the empirical data loss, $R(W)$ is a non-structured penalty (such as an $\ell_2$ -norm weight decay) with hyperparameter $\lambda$ , and $R_g\left(W^{(l)}\right)$ is the structured group Lasso regularizer with hyperparameter $\lambda_g$ .

For a weight subset $w$ partitioned into $G$ (possibly overlapping) groups $w^{(g)}$ , the Group Lasso penalty is defined as:

$R_g(w) = \sum_{g=1}^G \|w^{(g)}\|_g = \sum_{g=1}^G \sqrt{\sum_{i=1}^{|w^{(g)}|} \left(w_i^{(g)}\right)^2}$

where $|w^{(g)}|$ denotes the number of parameters in group $w^{(g)}$ and $w_i^{(g)}$ is the $i$ -th parameter of that group.
Knowl 2 — Simultaneous Filter-Wise and Channel-Wise Structured Sparsification
model/method
To prune unimportant 3D filters and channels simultaneously from convolutional layers, Structured Sparsity Learning groups the weights of each layer tensor $W^{(l)} \in \mathbb{R}^{N_l \times C_l \times M_l \times K_l}$ along the filter dimension and channel dimension.

Let $W^{(l)}_{n_l, :, :, :}$ denote the $n_l$ -th 3D filter in the $l$ -th layer (where $1 \le n_l \le N_l$ ), and let $W^{(l)}_{:, c_l, :, :}$ denote the $c_l$ -th channel across all filters in the $l$ -th layer (where $1 \le c_l \le C_l$ ). The optimization objective is formulated as:

$E(W) = E_D(W) + \lambda_n \sum_{l=1}^L \sum_{n_l=1}^{N_l} \|W^{(l)}_{n_l, :, :, :}\|_g + \lambda_c \sum_{l=1}^L \sum_{c_l=1}^{C_l} \|W^{(l)}_{:, c_l, :, :}\|_g$

where $E_D(W)$ is the data loss, $\lambda_n$ is the regularization coefficient for filter-wise sparsity, and $\lambda_c$ is the regularization coefficient for channel-wise sparsity.

Zeroing out an entire 3D filter in layer $l$ produces an all-zero output feature map, rendering the corresponding input channel in layer $l+1$ useless. Co-regularizing filters and channels enables removing both redundant output maps and inactive input slices.
Knowl 3 — Shape-Wise Structured Sparsity via Shape Fibers
model/method
Traditional convolutional neural networks constrain filters to fixed cuboid geometries (e.g., $M_l \times K_l$ ). Arbitrary, non-cubic filter geometries can be learned dynamically by defining and sparsifying shape fibers.

A shape fiber, denoted by $W^{(l)}_{:, c_l, m_l, k_l} \in \mathbb{R}^{N_l}$ , is the 1D vector of weights across all $N_l$ filters in layer $l$ located at the specific spatial coordinate $(m_l, k_l)$ within channel $c_l$ , where $1 \le c_l \le C_l$ , $1 \le m_l \le M_l$ , and $1 \le k_l \le K_l$ .

The shape-wise structured sparsity optimization objective is:

$E(W) = E_D(W) + \lambda_s \sum_{l=1}^L \sum_{c_l=1}^{C_l} \sum_{m_l=1}^{M_l} \sum_{k_l=1}^{K_l} \|W^{(l)}_{:, c_l, m_l, k_l}\|_g$

where $E_D(W)$ is the loss on data and $\lambda_s$ is the shape-wise regularization coefficient. Zeroing out specific shape fibers eliminates computations at corresponding spatial locations across all filters in a given channel, yielding homogeneous non-cubic spatial filter footprints.
Knowl 4 — Depth-Wise Structured Sparsification in Residual Networks
model/method
Network depth can be regularized by applying Group Lasso over entire weight tensors of convolutional layers: $R_g(W^{(l)}) = \|W^{(l)}\|_g$ . Because completely zeroing out all filters in a standard feedforward layer terminates signal propagation, depth-wise structured sparsity is combined with shortcut connections (such as identity or $1 \times 1$ convolutional shortcuts in Deep Residual Networks).

During training, Group Lasso is applied to the convolutional layer weights between each pair of shortcut endpoints, excluding the initial convolutional layer and convolutional shortcuts. When all weights in an intermediate layer reach zero, the entire layer is removed, and feature maps bypass the eliminated layer directly through the shortcut connection without disrupting network evaluation. The pruned architecture is subsequently fine-tuned using a reduced base learning rate.
Knowl 5 — 2D-Filter-Wise Sparsity Formulation
model/method
A 3D convolution in a convolutional layer consists of a collection of 2D convolutions. In 2D-filter-wise structured sparsity, Group Lasso is applied separately to every individual 2D filter $W^{(l)}_{n_l, c_l, :, :} \in \mathbb{R}^{M_l \times K_l}$ for each filter index $n_l \in \{1, \dots, N_l\}$ and channel index $c_l \in \{1, \dots, C_l\}$ .

Because the parameter group size of an individual 2D slice ( $M_l \cdot K_l$ ) is substantially smaller than that of a full 3D filter ( $C_l \cdot M_l \cdot K_l$ ), weight updating gradients are sharper. This enables Group Lasso to rapidly drive a large fraction of 2D filter groups to zero in deep networks, reducing 2D convolution FLOPs in direct proportion to the percentage of zeroed 2D filters.
Knowl 6 — GEMM Matrix Dimension Reduction via Row-Wise and Column-Wise Sparsity
model/method
Convolutional layers implemented via General Matrix Multiplication (GEMM) lower 4D weight tensors into 2D matrices where each 3D filter $W^{(l)}_{n_l, :, :, :}$ corresponds to a row and each shape fiber $W^{(l)}_{:, c_l, m_l, k_l}$ corresponds to a column.

Applying filter-wise structured sparsity corresponds to row-wise matrix sparsity, while shape-wise structured sparsity corresponds to column-wise matrix sparsity. Combining filter-wise and shape-wise regularization zeroes out entire rows and columns of the lowered weight matrix. The remaining nonzero rows and columns are concatenated into dense submatrices in contiguous memory. This allows standard dense GEMM routines (such as BLAS or cuBLAS) to execute directly on the smaller dense matrix without the indexing overhead, irregular memory access, or Compressed Sparse Row (CSR) storage costs associated with non-structured element-wise sparsity.
Knowl 7 — Fully-Connected Layer Neuron Pruning via Group Lasso
model/method
Structured Sparsity Learning extends to fully-connected layers by grouping all incoming or outgoing connection weights of each individual neuron under Group Lasso regularization.

For a fully-connected layer, if all incoming weights to a neuron are zeroed out, the neuron degenerates into a constant bias term for the subsequent layer. If all outgoing weights from a neuron are zeroed out, the neuron contributes nothing to subsequent layers and is pruned as a removable dummy neuron. Applying this regularization to the input layer zeroes out uninformative input pixels (e.g., boundary pixels in centered digit images).

Knowl 8 — Speedup and Sparsity Comparison Between SSL and Non-Structured Pruning on AlexNet

data/table

The performance of Structured Sparsity Learning (SSL) was compared against non-structured $\ell_1$ -norm regularization and connection pruning on AlexNet using ImageNet ILSVRC 2012. Dense GEMM routines computed SSL pruned models after concatenating nonzero rows and columns, whereas non-structured models were stored in Compressed Sparse Row (CSR) format and evaluated via sparse-dense matrix multiplication libraries (cuSPARSE on an NVIDIA GTX TITAN Black GPU and Intel MKL on an Intel Xeon E5-2630 CPU).

#	Method	Top-1 Err.	Statistics	conv1	conv2	conv3	conv4	conv5
1	$\ell_1$	44.67%	Sparsity	67.6%	92.4%	97.2%	96.6%	94.3%
			CPU $\times$	0.80	2.91	4.84	3.83	2.76
			GPU $\times$	0.25	0.52	1.38	1.04	1.36
2	SSL	44.66%	Column Sparsity	0.0%	63.2%	76.9%	84.7%	80.7%
			Row Sparsity	9.4%	12.9%	40.6%	46.9%	0.0%
			CPU $\times$	1.05	3.37	6.27	9.73	4.93
			GPU $\times$	1.00	2.37	4.94	4.03	3.05
3	Pruning	42.80%	Sparsity	16.0%	62.0%	65.0%	63.0%	63.0%
4	$\ell_1$	42.51%	Sparsity	14.7%	76.2%	85.3%	81.5%	76.3%
			CPU $\times$	0.34	0.99	1.30	1.10	0.93
			GPU $\times$	0.08	0.17	0.42	0.30	0.32
5	SSL	42.53%	Column Sparsity	0.00%	20.9%	39.7%	39.7%	24.6%
			CPU $\times$	1.00	1.27	1.64	1.68	1.32
			GPU $\times$	1.00	1.25	1.63	1.72	1.36

Non-structured sparsity requires extremely high zero ratios (>90%) to yield any speedup, often experiencing slowdowns (<1.0x) on GPUs and low-sparsity layers due to scattered memory access. In contrast, SSL achieves consistent speedups across layers, averaging 5.1x on CPU and 3.1x on GPU for model 2, and 1.4x on both CPU and GPU without accuracy loss for model 5.

Knowl 9 — Depth Regularization and Accuracy Trade-Offs on ResNet-20
empirical result
When depth-wise Structured Sparsity Learning was evaluated on a 20-layer ResNet (ResNet-20, baseline top-1 error 8.82%) on CIFAR-10, removing complete residual layers between shortcut endpoints produced shallower architectures with improved classification performance:
- SSL-ResNet-14 reduced the network to 14 layers while achieving a lower error than the original 20-layer baseline.
- SSL-ResNet-18 reduced the network to 18 layers and achieved a top-1 error of 7.40% (accuracy 92.60%), outperforming both the original ResNet-20 (error 8.82% / accuracy 91.25%) and the original 32-layer ResNet-32 (error 7.51%).
This shows that depth-wise SSL acts as an effective structural regularizer that mitigates degradation and over-parameterization.

Knowl 10 — Filter and Channel Pruning on LeNet for MNIST

data/table

Filter-wise and channel-wise structured sparsity were applied to LeNet on the MNIST dataset to penalize redundant filters and channels across convolutional layers conv1 and conv2.

LeNet #	Error	Filter # (conv1–conv2)	Channel # (conv1–conv2)	FLOP (conv1–conv2)	Speedup (conv1–conv2)
1 (baseline)	0.9%	20–50	1–20	100%–100%	1.00 $\times$ –1.00 $\times$
2	0.8%	5–19	1–4	25%–7.6%	1.64 $\times$ –5.23 $\times$
3	1.0%	3–12	1–3	15%–3.6%	1.99 $\times$ –7.44 $\times$

Applying SSL allowed conv1 filters to be reduced from 20 to 5 (LeNet 2) and 3 (LeNet 3), and conv2 channels from 20 to 4 and 3, reducing conv2 FLOPs down to 7.6% and 3.6% of the baseline while preserving classification error within $\pm 0.1\%$ , resulting in single-thread CPU speedups up to 7.44x on conv2.

Knowl 11 — Low-Rank Approximability Induced by Structured Sparsity Learning
empirical result
Applying Structured Sparsity Learning imposes a smoothness and structural regularization that restricts parameter search to lower-dimensional subspaces. Evaluating the normalized reconstruction error of weight matrices via Principal Component Analysis (PCA) across varying percentages of rank basis vectors demonstrates that weight tensors trained with SSL exhibit substantially lower reconstruction errors at low rank percentages than unregularized baseline models across LeNet, ConvNet, and AlexNet layers. As a result, filters regularized by SSL naturally possess lower intrinsic rank, enabling SSL to serve as a pre-regularization step before Low-Rank Approximation (LRA) matrix decomposition.

Coverage note — All primary contributed techniques (SSL formulation for filters, channels, shapes, depth, 2D filters, GEMM, and MLPs) and their corresponding empirical results across MNIST, CIFAR-10, and ImageNet are included.

References

1.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105. 2012.
2.Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
3.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
4.Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2015.
5.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
6.Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
7.Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135–1143. 2015.
8.Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
9.Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, pages 2148–2156. 2013.
10.Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, pages 1269–1277. 2014.
11.Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.
12.Yani Ioannou, Duncan P. Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. arXiv preprint arXiv:1511.06744, 2015.
13.Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E. Convolutional neural networks with low-rank regularization. arXiv preprint arXiv:1511.06067, 2015.
14.Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 68(1):49–67, 2006.
15.Seyoung Kim and Eric P Xing. Tree-guided group lasso for multi-task regression with structured sparsity. In Proceedings of the 27th International Conference on Machine Learning, 2010.
16.Jiashi Feng and Trevor Darrell. Learning the structure of deep convolutional networks. In The IEEE International Conference on Computer Vision (ICCV), 2015.
17.Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.
18.Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.
19.Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
20.Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
21.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.

Citation

MLA

Wen, W., et al. “Learning Structured Sparsity in Deep Neural Networks”. arXiv, 2016, https://doi.org/10.48550/arxiv.1608.03665.

APA

Wen, W., Wu, C., Wang, Y., Chen, Y., & Li, H. (2016). Learning Structured Sparsity in Deep Neural Networks. arXiv. https://doi.org/10.48550/arxiv.1608.03665

Chicago

Wen, W., C. Wu, Y. Wang, Y. Chen, and H. Li. 2016. “Learning Structured Sparsity in Deep Neural Networks”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1608.03665.

Harvard

Wen, W. et al. (2016) “Learning Structured Sparsity in Deep Neural Networks”. arXiv. Available at: https://doi.org/10.48550/arxiv.1608.03665.

Vancouver

1. Wen W, Wu C, Wang Y, Chen Y, Li H (2016) Learning Structured Sparsity in Deep Neural Networks. https://doi.org/10.48550/arxiv.1608.03665

BibTeX

@misc{https://doi.org/10.48550/arxiv.1608.03665,
  doi = {10.48550/ARXIV.1608.03665},
  url = {https://arxiv.org/abs/1608.03665},
  author = {Wen, Wei and Wu, Chunpeng and Wang, Yandan and Chen, Yiran and Li, Hai},
  keywords = {Neural and Evolutionary Computing (cs.NE), Machine Learning (cs.LG), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2.6; I.5.1},
  title = {Learning Structured Sparsity in Deep Neural Networks},
  publisher = {arXiv},
  year = {2016},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Learning Structured Sparsity in Deep Neural Networks

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Generic Objective Function for Structured Sparsity Learning

Knowl 2 — Simultaneous Filter-Wise and Channel-Wise Structured Sparsification

Knowl 3 — Shape-Wise Structured Sparsity via Shape Fibers

Knowl 4 — Depth-Wise Structured Sparsification in Residual Networks

Knowl 5 — 2D-Filter-Wise Sparsity Formulation

Knowl 6 — GEMM Matrix Dimension Reduction via Row-Wise and Column-Wise Sparsity

Knowl 7 — Fully-Connected Layer Neuron Pruning via Group Lasso