Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Emily L. DentonWojciech ZarembaJoan BrunaYann LeCunR. Fergus

article2014NeurIPS1,859 citations

Develops low-rank tensor approximation methods for convolutional filters that cut deep neural network inference latency in half across CPUs and GPUs while preserving original model accuracy within one percent.

Listen

Large convolutional neural networks deliver state-of-the-art accuracy in image recognition, but their evaluation requires millions of floating-point operations and vast memory storage. This high computational burden makes deployment difficult on resource-constrained mobile devices and expensive on large-scale cloud servers that process hundreds of millions of images daily. The article evaluates linear compression techniques designed to speed up test-time model execution and reduce memory overhead by exploiting structural redundancies within the network parameters, all while preserving prediction accuracy.

To achieve this, the article introduces specialized low-rank tensor approximation methods combined with brief fine-tuning of network layers. For the initial convolutional layer, it applies a monochromatic approximation that projects redundant color channels onto a lower-dimensional subspace. For subsequent convolutional layers, it groups filters using balanced biclustering and applies low-rank decompositions, using data-driven distance metrics to preserve critical features. In fully connected layers, standard low-rank matrix decomposition is applied. The approach was validated on a deep 15-layer architecture trained on the ImageNet dataset across standard central processing unit (CPU) and graphics processing unit (GPU) hardware.

Key findings demonstrate that the first two convolutional layers account for the majority of execution time, making them prime targets for optimization. Applying the proposed linear approximations to these layers achieved practical 2x to 2.5x speedups on both CPU and GPU hardware, with less than a 1% drop in classification accuracy. When cascading approximations across multiple layers alongside a single pass of fine-tuning, accuracy remained within 1% of the original uncompressed model. Furthermore, the techniques reduced parameter counts in the convolutional layers by a factor of 3x to 5x and compressed fully connected layers by a factor of 5x to 13x with negligible impact on error rates.

These results demonstrate that organizations can significantly decrease hardware latency, lower cloud power and cooling costs, and deploy sophisticated computer vision models to edge devices without meaningful sacrifices in accuracy. The compression methods are complementary to other acceleration techniques, such as integer quantization and Fourier-domain processing, allowing them to be combined for further performance gains. Decision-makers evaluating computer vision deployments should adopt these linear compression pipelines for performance-critical systems and consider brief retraining passes to restore baseline accuracy. However, stakeholders should note that empirical speedups depend heavily on specific hardware and software implementations, and practical deployment should be tested on target devices before full-scale operational rollout.

Meta New York University convolutional neural networks cpu floating points convolutional networks efficient evaluation linear structure

arXiv: 1404.0736

Paper: ImageNet Classification with Deep Convolutional Neural Networks, Alex Krizhevsky et al. (2012). It introduces the foundational deep convolutional neural network architecture whose computationally expensive convolutional layers motivate the need for linear filter approximations and acceleration.
Paper: Network In Network, Min Lin et al. (2014). It explores structure within convolutional feature spaces and 1x1 convolutions, establishing concepts of inter-channel linear combinations relevant to low-rank filter decomposition.
Paper: Learning Fast Approximations of Sparse Coding, Karol Gregor et al. (2010). It provides foundational principles for learning fast feed-forward approximations to computationally intensive linear and sparse visual representations.

Paper: Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding, Song Han et al. (2015). It extends the paradigm of accelerating convolutional networks by combining structured pruning with trained quantization and entropy coding.
Paper: Channel Pruning for Accelerating Very Deep Neural Networks, Yihui He et al. (2017). It builds directly on layer-wise linear reconstruction techniques to speed up convolutional inference via channel selection and least-squares feature reconstruction.
Paper: Pruning Filters for Efficient ConvNets, Hao Li et al. (2016). It advances convolutional acceleration by pruning whole filters to reduce FLOPs on dense hardware without requiring low-rank tensor decomposition.
Paper: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, Andrew G. Howard et al. (2017). It translates the principle of factorizing standard convolutions into linear and spatial stages into an efficient, lightweight architecture using depthwise separable convolutions.
Paper: Learning Structured Sparsity in Deep Neural Networks, Wei Wen et al. (2016). It provides an alternative regularization approach to discover structured, hardware-friendly linear sparsity directly during network training.
Paper: GhostNet: More Features From Cheap Operations, Kai Han et al. (2019). It exploits redundancy across convolutional feature maps by generating intrinsic features with standard convolutions and deriving remaining features through cheap linear operations.
Paper: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size, Forrest N. Iandola et al. (2016). It redesigns convolutional blocks using 1x1 squeeze layers to reduce parameter and computational redundancy while maintaining AlexNet-level accuracy.
Paper: Learning Efficient Convolutional Networks through Network Slimming, Zhuang Liu et al. (2017). It proposes an end-to-end channel slimming method via batch normalization scaling factors to automatically prune redundant convolutional channels.

Cover for Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Abstract

We present techniques for speeding up the test-time evaluation of large convolutional networks, designed for object recognition tasks. These models deliver impressive accuracy but each image evaluation requires millions of floating point operations, making their deployment on smartphones and Internet-scale clusters problematic. The computation is dominated by the convolution operations in the lower layers of the model. We exploit the linear structure present within the convolutional filters to derive approximations that significantly reduce the required computation. Using large state-of-the-art models, we demonstrate we demonstrate speedups of convolutional layers on both CPU and GPU by a factor of 2x, while keeping the accuracy within 1% of the original model.

Topics

convolutional neural networks cpu floating points internet object recognition smart phones network architecture communication networks mesh network

Keywords

convolutional networks efficient evaluation linear structure low-rank approximations tensor approximations outer product decomposition approximation monochromatic convolution convolution operations

1 Introduction
2 Related Work
3 Convolutional Tensor Compression
3.1 Approximation Metric
3.2 Low-rank Tensor Approximations
3.2.1 Matrix Decomposition
3.2.2 Higher Order Tensor Approximations
3.3 Monochromatic Convolution Approximation
3.4 Biclustering Approximations
3.5 Fine-tuning
4 Experiments
4.1 Speedup
4.1.1 First Layer
4.1.2 Second Layer
4.2 Combining approximations
4.3 Reduction in memory overhead
5 Discussion
References
A Forward propagation time breakdown
B Theoretical speedups
C Combined results

Knowls

Knowl 1 — Monochromatic Convolution Approximation for the First Convolutional Layer
model/method
In standard convolutional neural networks, the first layer processes an input tensor $I \in \mathbb{R}^{C \times N \times M}$ with $C$ color channels (typically $C=3$ for RGB) and spatial dimensions $N \times M$ using a weight tensor $W \in \mathbb{R}^{C \times X \times Y \times F}$ , where $X, Y$ are filter spatial dimensions and $F$ is the number of output feature maps.

The monochromatic approximation exploits the fact that the color profiles of learned first-layer filters lie predominantly in a low-dimensional subspace. The approximation is constructed as follows:
1. For each output feature $f \in \{1, \dots, F\}$ , combine the spatial filter dimensions to form a matrix $W_f \in \mathbb{R}^{C \times (XY)}$ .
2. Compute the Singular Value Decomposition (SVD) of $W_f = U_f S_f V_f^T$ , and truncate it to a rank-1 approximation $\tilde{W}_f = \tilde{U}_f \tilde{S}_f \tilde{V}_f^T$ , where $\tilde{U}_f \in \mathbb{R}^{C \times 1}$ , $\tilde{S}_f \in \mathbb{R}$ , and $\tilde{V}_f \in \mathbb{R}^{1 \times XY}$ .
3. Cluster the $F$ left singular vectors $\{\tilde{U}_f\}_{f=1}^F$ into $C' < F$ equal-sized clusters with cluster centroids $U_{c_f} \in \mathbb{R}^{C \times 1}$ using balanced $k$ -means.
4. Approximate each filter as $\tilde{W}_f = U_{c_f} \tilde{S}_f \tilde{V}_f^T$ .
At inference time, the computation decouples into two stages:
1. Project the input image $I$ onto $C'$ intermediate color channels via a point-wise linear projection using the cluster centers $U_{c_f}$ requiring $C' C N M$ operations.
2. Perform $C'$ independent 2D spatial convolutions using the spatial filters $\tilde{S}_f \tilde{V}_f^T$ requiring $X Y F N M \Delta^{-2}$ operations, where $\Delta$ is the convolutional stride.
This reduces total floating point operations from $X Y C F N M \Delta^{-2}$ to $C' C N M + X Y F N M \Delta^{-2}$ , and reduces parameters from $C X Y F$ to $C C' + X Y F$ .
Knowl 2 — Biclustering Framework for Higher Convolutional Layers
model/method
Higher convolutional layers contain 4D weight tensors $W \in \mathbb{R}^{C \times X \times Y \times F}$ with large numbers of input channels $C$ and output channels $F$ . To improve low-rank tensor factorizations across diverse filters, the biclustering framework partitions the tensor into homogeneous sub-tensors before decomposition:
1. Reshape $W$ into an input-channel unfolding matrix $W_C \in \mathbb{R}^{C \times (XYF)}$ and cluster its rows into $G$ equal-sized groups $C_1, \dots, C_G$ .
2. Reshape $W$ into an output-channel unfolding matrix $W_F \in \mathbb{R}^{(CXY) \times F}$ and cluster its columns into $H$ equal-sized groups $F_1, \dots, F_H$ .
3. Perform the clustering via a balanced $k$ -means algorithm implemented using the Floyd algorithm with a subspace projection distance metric to ensure equal cluster partition sizes for parallel execution on multi-core CPU and GPU architectures.
4. This yields $G \times H$ distinct sub-tensors $\{W_S^{g,h}\}_{g=1,\dots,G;\, h=1,\dots,H}$ , where each sub-tensor is a 3-tensor $W_S^{g,h} \in \mathbb{R}^{(C/G) \times (XY) \times (F/H)}$ combining the spatial kernel dimensions $X$ and $Y$ .
Each sub-tensor contains structurally similar filters and is subsequently approximated with a low-rank tensor decomposition (such as rank- $K$ outer product decomposition or multi-stage SVD).
Knowl 3 — Low-Rank Approximations for Convolutional Sub-Tensors
model/method
Given a 3-tensor sub-tensor $W_S \in \mathbb{R}^{m \times n \times k}$ from a biclustered convolutional weight tensor (where $m = C/G$ is the input cluster size, $n = XY$ combines kernel spatial dimensions, and $k = F/H$ is the output cluster size), two low-rank factorizations are used:
1. Outer Product Decomposition (Rank- $K$ Canonical Polyadic Decomposition):
$\tilde{W}_S = \sum_{k=1}^K \alpha_k \otimes \beta_k \otimes \gamma_k$

where $\alpha_k \in \mathbb{R}^m$ , $\beta_k \in \mathbb{R}^n$ , $\gamma_k \in \mathbb{R}^k$ , and $\otimes$ denotes the tensor outer product. The decomposition is obtained greedily by solving rank-1 subproblems $\min_{\alpha, \beta, \gamma} \|W_S^{(k)} - \alpha \otimes \beta \otimes \gamma\|_F$ via alternating least squares and updating residuals $W_S^{(k+1)} = W_S^{(k)} - \alpha_k \otimes \beta_k \otimes \gamma_k$ .

Evaluating the full biclustered outer product decomposition on an input feature map of size $N \times M$ with stride $\Delta$ requires $G H K \left( N M \frac{C}{G} + X Y N M \Delta^{-2} + \frac{F}{H} N M \Delta^{-2} \right)$ operations and $G H K \left( \frac{C}{G} + X Y + \frac{F}{H} \right)$ parameters.
1. SVD Decomposition:
$W_S$ is flattened along the first dimension to $W_m \in \mathbb{R}^{m \times (nk)}$ and decomposed via rank- $K_1$ SVD as $W_m \approx \tilde{U} \tilde{S} \tilde{V}^T$ , where $\tilde{U} \in \mathbb{R}^{m \times K_1}$ and $\tilde{V} \in \mathbb{R}^{(nk) \times K_1}$ . The factor $\tilde{V}^T$ is then reshaped and compressed via a second SVD with rank $K_2$ .

Evaluating this decomposition across all $G \times H$ sub-tensors requires $G H N M \left( \frac{C}{G} K_1 + K_1 X Y K_2 \Delta^{-2} + K_2 \frac{F}{H} \right)$ operations and $G H \left( \frac{C}{G} K_1 + K_1 X Y K_2 + K_2 \frac{F}{H} \right)$ parameters.
Knowl 4 — Error-Aware and Data Covariance Distance Metrics for Filter Approximation
model/method
Standard tensor approximations minimize the unweighted Frobenius reconstruction norm $\|\tilde{W} - W\|_F$ , which assumes isotropic sensitivity across weight parameters. To preserve network classification performance during low-rank fitting, two alternative metrics are used:
1. Diagonal Mahalanobis Distance Metric:
Let $\Theta = \{W_1, \dots, W_S\}$ be the network parameters, $U(I_n; \Theta)$ the softmax output for image $I_n$ , and $\{\beta_n\}$ the indices of the $h$ highest predicted incorrect classes for input $n$ . Gradients representing mistake sensitivity for layer $s$ are defined as:

$d_{n,l,s} = \nabla_{W_s} \left( U(I_n, \Theta) - \delta(i - l) \right), \quad l \in \{\beta_n\}$

where $\delta$ is the Dirac delta. The approximate diagonal Mahalanobis metric is defined as:

$\|W\|_{\hat{\text{maha}}} := \sum_p \alpha_p W(p), \quad \text{with } \alpha_p = \left( \sum_{n,l} d_{n,l,s}(p)^2 \right)^{1/2}$

where $p$ indexes tensor elements. Fitting is performed by setting $W' = \alpha \odot W$ (element-wise product), computing the low-rank approximation $\tilde{W}'$ using the standard Frobenius norm, and recovering $\tilde{W} = \alpha^{-1} \odot \tilde{W}'$ .
1. Data Covariance Distance Metric:
Weights are weighted by the empirical covariance $\hat{\Sigma} \in \mathbb{R}^{CXY \times CXY}$ of the layer's input activations:

$\|W\|_{\text{data}} = \|\hat{\Sigma}^{1/2} W_F\|_F$

where $W_F \in \mathbb{R}^{(CXY) \times F}$ is obtained by unfolding the first three dimensions of $W$ into matrix form.
Knowl 5 — Sequential Layer-by-Layer Network Compression and Fine-Tuning
algorithm
To compress deep convolutional neural networks without compounding error across layers, compression is performed sequentially from lower to upper layers, fine-tuning the remaining layers after each stage:
Input: Pretrained $S$ -layer network with parameters $\Theta = \{W_1, \dots, W_S\}$ , training set $\mathcal{D}$
Output: Compressed network parameters $\{\tilde{W}_1, \dots, \tilde{W}_S\}$
for layer $s = 1$ to $S$ :
    if layer $s$ is the first convolutional layer:
        Compute monochromatic approximation $\tilde{W}_s$
    else if layer $s$ is a higher convolutional layer:
        Compute biclustering with outer product or SVD decomposition $\tilde{W}_s$
    else if layer $s$ is a fully connected layer:
        Compute truncated matrix SVD $\tilde{W}_s = \tilde{U}_s \tilde{S}_s \tilde{V}_s^T$ with rank $K$
    Replace layer weights $W_s \leftarrow \tilde{W}_s$
    Freeze weights of layers $1, \dots, s$
    Fine-tune unfrozen upper layers $s+1, \dots, S$ using SGD on $\mathcal{D}$ for $< 2$ passes
return $\{\tilde{W}_1, \dots, \tilde{W}_S\}$
Fine-tuning restores accuracy even when aggressive low-rank compression is applied to individual layers.

Knowl 6 — Computational Operations and Parameter Counts Across Approximation Methods

data/table

Analytical formulas for the number of arithmetic operations and number of parameters required by uncompressed and compressed convolutional layers:

Approximation Technique	Number of Operations	Number of Parameters
No approximation (Standard)	$X Y C F N M \Delta^{-2}$	$C X Y F$
Monochromatic	$C' C N M + X Y F N M \Delta^{-2}$	$C C' + X Y F$
Biclustering + Outer Product	$G H K \left(N M \frac{C}{G} + X Y N M \Delta^{-2} + \frac{F}{H} N M \Delta^{-2}\right)$	$G H K \left(\frac{C}{G} + X Y + \frac{F}{H}\right)$
Biclustering + SVD	$G H N M \left(\frac{C}{G} K_1 + K_1 X Y K_2 \Delta^{-2} + K_2 \frac{F}{H}\right)$	$G H \left(\frac{C}{G} K_1 + K_1 X Y K_2 + K_2 \frac{F}{H}\right)$

Variable definitions:

$C$ : Number of input channels
$F$ : Number of output feature maps / filters
$X, Y$ : Spatial kernel dimensions
$N, M$ : Spatial dimensions of input feature map
$\Delta$ : Convolutional stride
$C'$ : Number of intermediate color channels in monochromatic approximation ( $C' < F$ )
$G$ : Number of input channel clusters
$H$ : Number of output feature clusters
$K$ : Decomposition rank for outer product tensor decomposition
$K_1, K_2$ : Ranks for the first and second matrix SVD decompositions

Knowl 7 — Forward Propagation Latency Breakdown Across CNN Layers

data/table

Evaluation time per batch (batch size 128, averaged over 8 runs) for a 15-layer ImageNet CNN on CPU (Eigen3 with Intel MKL, multi-threaded) and GPU (Nvidia Titan with CUDA):

Layer	CPU Time (s)	CPU Fraction	GPU Time (s)	GPU Fraction
Conv1	$2.8317 \pm 0.1030$	21.97%	$0.0604 \pm 0.0112$	5.14%
MaxPool1	$0.1059 \pm 0.0154$	0.82%	$0.0072 \pm 0.0040$	0.61%
LRNormal1	$0.1918 \pm 0.0162$	1.49%	$0.0041 \pm 0.0043$	0.35%
Conv2	$4.2626 \pm 0.0740$	33.07%	$0.4663 \pm 0.0072$	39.68%
MaxPool2	$0.0705 \pm 0.0029$	0.55%	$0.0032 \pm 0.0000$	0.27%
LRNormal2	$0.0772 \pm 0.0027$	0.60%	$0.0015 \pm 0.0003$	0.13%
Conv3	$1.8689 \pm 0.0577$	14.50%	$0.2219 \pm 0.0014$	18.88%
MaxPool3	$0.0532 \pm 0.0018$	0.41%	$0.0016 \pm 0.0000$	0.14%
Conv4	$1.5261 \pm 0.0386$	11.84%	$0.1991 \pm 0.0001$	16.94%
Conv5	$1.4222 \pm 0.0416$	11.03%	$0.1958 \pm 0.0002$	16.66%
MaxPool5	$0.0102 \pm 0.0006$	0.08%	$0.0005 \pm 0.0001$	0.04%
FC1	$0.3777 \pm 0.0233$	2.93%	$0.0077 \pm 0.0013$	0.66%
FC2	$0.0709 \pm 0.0038$	0.55%	$0.0017 \pm 0.0001$	0.14%
FC3	$0.0168 \pm 0.0018$	0.13%	$0.0007 \pm 0.0002$	0.06%
Softmax	$0.0028 \pm 0.0015$	0.02%	$0.0038 \pm 0.0098$	0.32%
Total	12.8885	100.00%	1.1752	100.00%

Convolution layers account for over 92% of total forward-propagation latency on CPU and over 97% on GPU. The first two convolutional layers (Conv1 and Conv2) account for 55.04% of total execution time on CPU and 44.82% on GPU.

Knowl 8 — Empirical Speedup and Accuracy for First-Layer Monochromatic Approximation
empirical result
On the first convolutional layer ( $C=3$ input channels, $F=96$ filters of size $7 \times 7$ ) of an ImageNet classification CNN evaluated on 20,000 ImageNet 2012 validation images:
- Monochromatic approximation with $C'=6$ intermediate color channels yields a $3\times$ reduction in layer weights and a $2.95\times$ theoretical floating-point speedup.
- Prediction error increase before fine-tuning is $16.1\%$ with the standard Frobenius metric, $2.4\%$ using the data covariance distance metric ( $\|W\|_{\text{data}}$ ), and drops to $0.4\%$ after fine-tuning (under 2 passes of training data).
- Setting $C'=12$ yields a $2.91\times$ theoretical speedup and achieves $0\%$ validation error increase after fine-tuning ( $3.5\%$ original, $0.7\%$ with $\|W\|_{\text{data}}$ ).
- Empirical speedups on both CPU and GPU reach $2.0\times - 2.5\times$ relative to optimized baselines while keeping classification accuracy degradation under $1.0\%$ .
Knowl 9 — Empirical Speedup and Accuracy for Second-Layer Biclustering Approximations
empirical result
On the second convolutional layer ( $C=96$ input channels, $F=256$ output channels, $5 \times 5$ spatial kernels) of an ImageNet classification CNN:
- Biclustering with Outer Product Decomposition ( $G=48$ input clusters, $H=2$ output clusters, rank $K=6$ ): achieves a $5.3\times$ reduction in parameters and an empirical GPU speedup of $2.0\times - 2.5\times$ with a $0.68\%$ increase in classification error after fine-tuning.
- Biclustering with SVD Decomposition ( $G=2$ input clusters, $H=2$ output clusters, ranks $K_1=19, K_2=24$ ): achieves a $3.9\times$ reduction in parameters and an empirical CPU speedup of $2.0\times - 2.5\times$ with a $0.90\%$ increase in classification error after fine-tuning.
- Both Mahalanobis ( $\text{maha}$ ) and data covariance ( $\text{data}$ ) approximation metrics consistently produce lower initial classification error prior to fine-tuning across all rank settings ( $K=5$ to $16$ ) compared to standard unweighted Frobenius minimization.

Knowl 10 — Parameter Reduction and Accuracy in Fully Connected Layers via Matrix SVD

data/table

Truncated SVD applied to fully connected layer weight matrices $W \in \mathbb{R}^{N \times M} \approx U \tilde{S} V^T$ with rank $K$ reduces parameters from $NM$ to $K(N+M)$ :

Layer	Rank $K$	Uncompressed Parameters	Weight Reduction	Increase in Error
FC Layer 1	$K = 250$	$NM$	$13.4\times$	$0.8394\%$
FC Layer 1	$K = 950$	$NM$	$3.5\times$	$0.09\%$
FC Layer 2	$K = 350$	$NM$	$5.8\times$	$0.19\%$
FC Layer 2	$K = 650$	$NM$	$3.14\times$	$0.06\%$
FC Layer 3	$K = 250$	$NM$	$8.1\times$	$0.67\%$
FC Layer 3	$K = 850$	$NM$	$2.4\times$	$0.02\%$

Fully connected layers contain the vast majority of parameters in the CNN architecture. Applying truncated SVD achieves a $5\times$ to $13.4\times$ parameter compression ratio per layer with less than $0.85\%$ loss in classification performance, substantially reducing the model memory footprint for embedded deployment.

Knowl 11 — Cumulative Accuracy and Speedup in Cascaded Multi-Layer Network Compression
empirical result
When low-rank approximations are applied sequentially across multiple layers and fine-tuned for a single pass through the ImageNet 2012 training dataset:
- Compressing Conv1 using monochromatic approximation with $C'=6$ colors, followed by compressing Conv2 using biclustering with outer product decomposition ( $G=48, H=2, K=8$ ), results in a total increase in validation classification error of $1.0\%$ .
- Combining Conv1 ( $C'=6$ ) with Conv2 biclustering + outer product decomposition ( $G=48, H=2, K=6$ ) yields a cumulative error increase of $1.5\%$ .
- Combining Conv1 ( $C'=6$ ) with Conv2 biclustering + SVD ( $G=2, H=2, K_1=19, K_2=64$ ) yields a cumulative error increase of $1.2\%$ .
- Combining Conv1 ( $C'=6$ ) with Conv2 biclustering + SVD ( $G=2, H=2, K_1=19, K_2=51$ ) yields a cumulative error increase of $1.4\%$ .
This confirms that sequential layer-by-layer compression with intermediate fine-tuning successfully prevents compound error accumulation across cascaded convolutional layers.

Coverage note — None was omitted; all contributed low-rank approximation methods, error metrics, fine-tuning algorithms, complexity formulas, latency profiles, layer-wise speedups, fully connected compression data, and cascaded compression results are fully covered.

References

1.Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 (2013)
2.Denil, M., Shakibi, B., Dinh, L., Ranzato, M., de Freitas, N.: Predicting parameters in deep learning. arXiv preprint arXiv:1306.0543 (2013)
3.Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
4.Vanhoucke, V., Senior, A., Mao, M.Z.: Improving the speed of neural networks on cpus. In: Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop. (2011)
5.Mathieu, M., Henaff, M., LeCun, Y.: Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851 (2013)
6.Jaderberg, M., Vedaldi, Andrea, Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 (2014)
7.Zhang, T., Golub, G.H.: Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl. 23(2) (February 2001) 534–550
8.Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional neural networks. arXiv preprint arXiv:1311.2901 (2013)
9.Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: CVPR09. (2009)
10.Guennebaud, G., Jacob, B., et al.: Eigen v3. http://eigen.tuxfamily.org (2010)
11.Zeiler, M.D., Taylor, G.W., Fergus, R.: Adaptive deconvolutional networks for mid and high level feature learning. In: Computer Vision (ICCV), 2011 IEEE International Conference on, IEEE (2011) 2018–2025
12.Le, Q.V., Ngiam, J., Chen, Z., Chia, D., Koh, P.W., Ng, A.Y.: Tiled convolutional neural networks. In: Advances in Neural Information Processing Systems. (2010)
13.Le, Q.V., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G.S., Dean, J., Ng, A.Y.: Building high-level features using large scale unsupervised learning. arXiv preprint arXiv:1112.6209 (2011)
14.Lowe, D.G.: Object recognition from local scale-invariant features. In: Computer vision, 1999. The proceedings of the seventh IEEE international conference on. Volume 2., Ieee (1999) 1150–1157
15.Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25. (2012) 1106–1114

Citation

MLA

Denton, R., et al. “Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation”. arXiv, 2014, http://arxiv.org/abs/1404.0736v2.

APA

Denton, R., Zaremba, W., Bruna, J., LeCun, Y., & Fergus, R. (2014). Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. arXiv. http://arxiv.org/abs/1404.0736v2

Chicago

Denton, R., W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. 2014. “Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation”. arXiv. http://arxiv.org/abs/1404.0736v2.

Harvard

Denton, R. et al. (2014) “Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1404.0736v2.

Vancouver

1. Denton R, Zaremba W, Bruna J, LeCun Y, Fergus R (2014) Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. arXiv

BibTeX

@article{denton2014exploiting,
  title = {Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation},
  author = {Denton, Remi and Zaremba, Wojciech and Bruna, Joan and LeCun, Yann and Fergus, Rob},
  year = {2014},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1404.0736v2},
  eprint = {1404.0736}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Monochromatic Convolution Approximation for the First Convolutional Layer

Knowl 2 — Biclustering Framework for Higher Convolutional Layers

Knowl 3 — Low-Rank Approximations for Convolutional Sub-Tensors

Knowl 4 — Error-Aware and Data Covariance Distance Metrics for Filter Approximation

Knowl 5 — Sequential Layer-by-Layer Network Compression and Fine-Tuning

Knowl 6 — Computational Operations and Parameter Counts Across Approximation Methods

Knowl 7 — Forward Propagation Latency Breakdown Across CNN Layers

Knowl 8 — Empirical Speedup and Accuracy for First-Layer Monochromatic Approximation

Knowl 9 — Empirical Speedup and Accuracy for Second-Layer Biclustering Approximations

Knowl 10 — Parameter Reduction and Accuracy in Fully Connected Layers via Matrix SVD

Knowl 11 — Cumulative Accuracy and Speedup in Cascaded Multi-Layer Network Compression

References

Citation

BibTeX

Access the Paper