Learning Fast Approximations of Sparse Coding

article2010ICML2,105 citations

Introduces Learned ISTA (LISTA), a deep learning approach that unrolls iterative optimization algorithms into trainable, fixed-depth neural networks to compute sparse codes an order of magnitude faster than standard iterative solvers while maintaining differentiability for end-to-end vision pipelines.

Listen

Extracting meaningful visual features using sparse coding—representing data using a small combination of basic building blocks—is essential for modern computer vision tasks such as object recognition and image restoration. However, conventional inference algorithms solve a costly optimization problem for every image patch, creating a severe computational bottleneck that prevents sparse coding from running in real-time environments. The article demonstrates that a trainable, feed-forward neural predictor with fixed depth can produce accurate approximations of sparse codes in a fraction of the time required by standard iterative methods.

The authors designed two learning-based approximator architectures: Learned Iterative Shrinkage-Thresholding Algorithm (LISTA) and Learned Coordinate Descent (LCoD). Instead of using fixed mathematical operators derived from the dictionary, these architectures learn input filters, mutual inhibition matrices, and shrinkage thresholds directly from training data using stochastic gradient descent. The models were evaluated on natural image patches from the Berkeley image database across complete and overcomplete dictionary settings, as well as on digit recognition tasks using the standard MNIST handwritten digit dataset.

The findings show that learned approximators deliver dramatic speed improvements with negligible loss in accuracy. In sparse code prediction, LISTA required only 1 iteration to match the accuracy that standard iterative shrinkage methods achieved after 18 to 35 iterations, representing roughly a twenty-fold speedup. Similarly, LCoD matched the approximation accuracy of 100 conventional coordinate descent iterations within just 5 iterations. Furthermore, pruning up to 80% of the internal interaction matrix connections in LISTA yielded significant computational savings with only a minor rise in prediction error. When applied to downstream digit recognition on MNIST, LCoD with 10 iterations achieved a classification error of 1.42%, performing comparably to exact inference algorithms (1.33%) while running substantially faster.

These results demonstrate that sparse feature extraction is viable for real-time applications and low-latency computer vision pipelines without sacrificing accuracy. Because the learned encoders are continuous and differentiable, they can be directly integrated and fine-tuned end-to-end within larger pattern recognition architectures. Organizations operating visual perception systems can drastically cut processing overhead by switching to learned feed-forward approximators or by using them to quickly initialize exact solvers when higher precision is mandatory.

For practical implementation, teams should adopt LCoD for maximum inference efficiency in latency-critical pipelines, or prune LISTA matrices when memory and matrix operations must be minimized. While the study demonstrates high confidence and clear empirical gains on image patches and digit classification, future work should evaluate performance across broader benchmarks, including full-scale object recognition and real-world image restoration tasks.

New York University basis vector coordinate descent inference approximate sparse codes sparse representation iterative shrinkage algorithms

arXiv: 1006.0448

Paper: Emergence of simple-cell receptive field properties by learning a sparse code for natural images, Bruno A. Olshausen et al. (1996). Reading the foundational sparse coding formulation first provides the core statistical objective and generative image model that the source source approximates.
Paper: Efficient sparse coding algorithms, Honglak Lee et al. (2006). Understanding efficient coordinate-descent solvers for sparse coding is essential because the source directly builds upon and accelerates these iterative optimization steps.

Paper: Learning a Deep Convolutional Network for Image Super-Resolution, Chao Dong et al. (2014). This work extends the shift from slow patch-based sparse coding toward efficient, end-to-end feedforward mappings introduced in the source.

Cover for Learning Fast Approximations of Sparse Coding

Abstract

In Sparse Coding (SC), input vectors are reconstructed using a sparse linear combination of basis vectors. SC has become a popular method for extracting features from data. For a given input, SC minimizes a quadratic reconstruction error with an L1 penalty term on the code. The process is often too slow for applications such as real-time pattern recognition. We proposed two versions of a very fast algorithm that produces approximate estimates of the sparse code that can be used to compute good visual features, or to initialize exact iterative algorithms. The main idea is to train a non-linear, feed-forward predictor with a specific architecture and a fixed depth to produce the best possible approximation of the sparse code. A version of the method, which can be seen as a trainable version of Li and Osher's coordinate descent method, is shown to produce approximate solutions with 10 times less computation than Li and Osher's for the same approximation error. Unlike previous proposals for sparse code predictors, the system allows a kind of approximate "explaining away" to take place during inference. The resulting predictor is differentiable and can be included into globally-trained recognition systems.

Topics

basis vector coordinate descent inference reconstruction sparse approximations visual feature exact solution sparse recovery

Keywords

approximate sparse codes sparse representation iterative shrinkage algorithms coordinate descent stochastic gradient descent trainable encoder encoder architecture dictionary matrix sparsity penalty shrinkage function

1. Introduction
1.1. Sparse Coding
2. Iterative Shrinkage Algorithms
2.1. ISTA and Fast ISTA
2.2. Coordinate Descent (CoD)
3. Trainable Sparse Code Predictors
3.1. Baseline Encoder Architecture
3.2. Mutual Inhibition and Explaining Away
3.3. Learned Iterative Shrinkage-Thresholding Algorithm (LISTA)
3.4. Learned Coordinate Descent (LCoD)
4. Results
5. Conclusions
References

Knowls

Knowl 1 — Learned Iterative Shrinkage-Thresholding Algorithm (LISTA) Architecture
model/method
The Learned Iterative Shrinkage-Thresholding Algorithm (LISTA) is a parameterized feed-forward neural network architecture of fixed depth $T$ designed to produce fast approximations to optimal sparse codes. Standard sparse coding seeks a code vector $Z^* \in \mathbb{R}^m$ for an input $X \in \mathbb{R}^n$ by minimizing:

$E_{W_d}(X, Z) = \frac{1}{2}\|X - W_d Z\|_2^2 + \alpha \|Z\|_1$

where $W_d \in \mathbb{R}^{n \times m}$ is a dictionary matrix with normalized columns and $\alpha > 0$ controls sparsity. Classical Iterative Shrinkage and Thresholding (ISTA) minimizes this objective by iterating $Z(t) = h_{\alpha/L}(W_e X + S Z(t-1))$ , where $W_e = \frac{1}{L} W_d^T$ , $S = I - \frac{1}{L} W_d^T W_d$ , $L$ is an upper bound on the largest eigenvalue of $W_d^T W_d$ , and $h_\theta$ is a coordinate-wise soft-shrinkage function defined by $[h_\theta(V)]_i = \text{sign}(V_i)(|V_i| - \theta_i)_+$ with $(u)_+ = \max(u, 0)$ .

Instead of pre-computing $W_e$ , $S$ , and thresholds from $W_d$ , LISTA unrolls $T$ steps of the recurrence as a time-unfolded feed-forward network with weight sharing across layers and treats the filter matrix $W_e \in \mathbb{R}^{m \times n}$ , the mutual inhibition matrix $S \in \mathbb{R}^{m \times m}$ , and the threshold vector $\theta \in \mathbb{R}^m$ as trainable parameters learned directly from data. The forward computation is:

$B = W_e X$

$Z(0) = h_\theta(B)$

$Z(t) = h_\theta(B + S Z(t-1)) \quad \text{for } t = 1, \dots, T$

The final output is $Z = Z(T)$ .
Knowl 2 — LISTA Forward and Backward Propagation Algorithms
algorithm
LISTA parameters $W = (W_e, S, \theta)$ are trained by supervised back-propagation through time (BPTT) using the squared error loss $\frac{1}{2}\|Z^* - Z(T)\|_2^2$ , where $Z^* \in \mathbb{R}^m$ is the ground-truth optimal sparse code computed by an exact solver for input $X \in \mathbb{R}^n$ . In the algorithm below, $h'_\theta(V)$ denotes the Jacobian of the shrinkage function with respect to its input (a diagonal matrix with $[h'_\theta(V)]_{ii} = 1$ if $|V_i| > \theta_i$ and $0$ otherwise), and $\text{sign}(V)$ is the component-wise sign.
procedure LISTA_Forward(X, We, S, theta, T)
    Input: Input vector $X \in \mathbb{R}^n$ , weights $W_e \in \mathbb{R}^{m \times n}$ , $S \in \mathbb{R}^{m \times m}$ , thresholds $\theta \in \mathbb{R}^m$ , depth $T$
    Output: Predicted sparse code $Z(T) \in \mathbb{R}^m$
     $B = W_e X$
     $Z(0) = h_\theta(B)$
    for $t = 1$ to $T$ do
         $C(t) = B + S Z(t - 1)$
         $Z(t) = h_\theta(C(t))$
    end for
    return $Z(T)$
end procedure
procedure LISTA_Backward(Z_star, X, We, S, theta, T, saved_variables)
    Input: Ground truth code $Z^* \in \mathbb{R}^m$ , input $X \in \mathbb{R}^n$ , parameters and forward-pass variables $B, C(t), Z(t)$
    Output: Gradients $\delta W_e, \delta S, \delta \theta, \delta X$
     $\delta B = 0$
     $\delta S = 0$
     $\delta \theta = 0$
     $\delta Z(T) = Z(T) - Z^*$
    for $t = T$ down to $1$ do
         $\delta C(t) = h'_\theta(C(t)) \cdot \delta Z(t)$
         $\delta \theta = \delta \theta - \text{sign}(C(t)) \cdot \delta C(t)$
         $\delta B = \delta B + \delta C(t)$
         $\delta S = \delta S + \delta C(t) (Z(t - 1))^T$
         $\delta Z(t - 1) = S^T \delta C(t)$
    end for
     $\delta B = \delta B + h'_\theta(B) \cdot \delta Z(0)$
     $\delta \theta = \delta \theta - \text{sign}(B) \cdot h'_\theta(B) \cdot \delta Z(0)$
     $\delta W_e = \delta B X^T$
     $\delta X = W_e^T \delta B$
    return $\delta W_e, \delta S, \delta \theta, \delta X$
end procedure
Knowl 3 — Learned Coordinate Descent (LCoD) Architecture
model/method
Learned Coordinate Descent (LCoD) is a trainable sparse code predictor inspired by the Coordinate Descent (CoD) algorithm of Li and Osher. In standard CoD, only one coordinate of the code $Z \in \mathbb{R}^m$ is updated at each step by selecting the component that yields the largest magnitude change, requiring $O(m)$ operations per step instead of the $O(m^2)$ or $O(mn)$ required by parallel full-vector updates.

LCoD unrolls this greedy coordinate selection and update mechanism into a feed-forward architecture with $T$ iterations while making the filter matrix $W_e \in \mathbb{R}^{m \times n}$ , the mutual inhibition matrix $S \in \mathbb{R}^{m \times m}$ , and the threshold vector $\theta \in \mathbb{R}^m$ trainable parameters. Given an input $X \in \mathbb{R}^n$ , the system computes initial activations $B = W_e X$ in $O(nm)$ operations, initializes $Z = 0$ , and at each step $t = 1, \dots, T-1$ updates:

$\bar{Z} = h_\theta(B)$

$k = \arg\max_{j \in \{1, \dots, m\}} |Z_j - \bar{Z}_j|$

$e(t) = \bar{Z}_k - Z_k$

$B_j \leftarrow B_j + S_{jk} e(t) \quad \forall j \in \{1, \dots, m\}$

$Z_k \leftarrow \bar{Z}_k$

After $T-1$ coordinate descent iterations, a final soft-thresholding step is performed: $Z = h_\theta(B)$ . Each internal coordinate update step requires only $O(m)$ operations.
Knowl 4 — LCoD Forward and Backward Propagation Algorithms
algorithm
The parameters $W = (W_e, S, \theta)$ of the Learned Coordinate Descent (LCoD) network are optimized via stochastic sub-gradient descent. The index selection $\arg\max$ introduces kinks of measure zero in the loss function, but gradients can still be back-propagated effectively through the continuous pieces. Each backward iteration step requires $O(m)$ operations and $O(m)$ memory.
procedure LCoD_Forward(X, We, S, theta, T)
    Input: Input vector $X \in \mathbb{R}^n$ , parameters $W_e \in \mathbb{R}^{m \times n}$ , $S \in \mathbb{R}^{m \times m}$ , $\theta \in \mathbb{R}^m$ , number of steps $T$
    Output: Predicted sparse code $Z \in \mathbb{R}^m$
     $B = W_e X$
     $Z = 0$
    for $t = 1$ to $T - 1$ do
         $\bar{Z} = h_\theta(B)$
         $k = \arg\max_{j \in \{1, \dots, m\}} |Z_j - \bar{Z}_j|$
         $k(t) = k$
         $b(t) = B_k$
         $e(t) = \bar{Z}_k - Z_k$
        for $j = 1$ to $m$ do
             $B_j = B_j + S_{jk} e(t)$
        end for
         $Z_k = \bar{Z}_k$
    end for
     $B(T) = B$
     $Z = h_\theta(B)$
    return $Z$
end procedure
procedure LCoD_Backward(Z_star, X, We, S, theta, T, saved_variables)
    Input: Ground truth optimal code $Z^* \in \mathbb{R}^m$ , input $X \in \mathbb{R}^n$ , parameters and forward-pass variables $B(T), b(t), k(t), e(t)$
    Output: Gradients $\delta W_e, \delta S, \delta \theta, \delta X$
     $\delta S = 0$
     $\delta Z = 0$
     $B = B(T)$
     $\delta B = h'_\theta(B) \cdot (Z - Z^*)$
    for $t = T - 1$ down to $1$ do
         $k = k(t)$
         $\delta e = \sum_{j=1}^m \delta B_j S_{jk}$
        for $j = 1$ to $m$ do
             $\delta S_{jk} = \delta S_{jk} + \delta B_j e(t)$
        end for
         $\delta B_k = \delta B_k + h'_{\theta_k}(b(t)) (\delta Z_k + \delta e)$
         $\delta \theta_k = \delta \theta_k - \text{sign}(b(t)) h'_{\theta_k}(b(t)) (\delta Z_k + \delta e)$
         $\delta Z_k = -\delta e$
    end for
     $\delta W_e = \delta B X^T$
     $\delta X = W_e^T \delta B$
    return $\delta W_e, \delta S, \delta \theta, \delta X$
end procedure
Knowl 5 — Supervised Objective for Training Sparse Code Predictors
equation
Let $\{X^p\}_{p=0}^{P-1}$ be a training set of input vectors with $X^p \in \mathbb{R}^n$ , and let $Z^{*p} \in \mathbb{R}^m$ be the corresponding optimal sparse code obtained by an exact optimization solver (such as Coordinate Descent) minimizing the sparse coding energy:

$Z^{*p} = \arg\min_{Z \in \mathbb{R}^m} \left( \frac{1}{2} \|X^p - W_d Z\|_2^2 + \alpha \|Z\|_1 \right)$

where $W_d \in \mathbb{R}^{n \times m}$ is a fixed dictionary matrix and $\alpha > 0$ is the sparsity penalty coefficient. A trainable encoder $f_e(W, X)$ with parameters $W$ is optimized by minimizing the mean squared error loss over the dataset:

$\mathcal{L}(W) = \frac{1}{P} \sum_{p=0}^{P-1} \mathcal{L}(W, X^p) = \frac{1}{P} \sum_{p=0}^{P-1} \frac{1}{2} \|Z^{*p} - f_e(W, X^p)\|_2^2$

Training is performed via stochastic gradient descent:

$W^{(j+1)} = W^{(j)} - \eta(j) \frac{\partial \mathcal{L}(W, X^{(j \bmod P)})}{\partial W}$

where $\eta(j) \propto 1/j$ is a decaying learning rate schedule ensuring convergence.
Knowl 6 — Explaining Away and Non-Linearity Selection in Feed-Forward Encoders
model/method
Single-layer feed-forward encoders of the form $Z = g(W_e X)$ suffer from two major limitations when predicting sparse codes for overcomplete dictionaries ( $m > n$ ):
1. Inability to Explain Away: In overcomplete representations, multiple basis vectors can partially reconstruct the same input feature (e.g., two Gabor filters with similar orientations). In exact sparse coding, activating one unit suppresses the other ("explaining away"). In a single-layer feed-forward encoder, code activations are conditionally independent given the input $X$ , causing both units to activate simultaneously without mutual inhibition. Multilayer architectures with lateral interaction matrices $S$ (such as LISTA and LCoD) introduce iterative mutual inhibition to resolve this competition.
2. Thresholding and Differentiability: Standard non-linearities such as $g(x) = D \tanh(x)$ (where $D$ is a diagonal gain matrix) have a high derivative at $x = 0$ , making it difficult to produce exact zero coefficients. The coordinate-wise shrinkage function $h_\theta(x) = \text{sign}(x)(|x| - \theta)_+$ provides exact zero regions. Alternatively, a smooth, everywhere-differentiable "double tanh" non-linearity can be used:
$g(X) = D \left(\tanh(X + U) + \tanh(X - U)\right)$

where $U$ is a trainable threshold vector and $D$ is a trainable diagonal matrix. Both $h_\theta$ and the double tanh substantially outperform $D \tanh(x)$ in predicting sparse codes.
Knowl 7 — Sparsification vs. Low-Rank Factorization of the Mutual Inhibition Matrix
empirical result
In LISTA, multiplying by the mutual inhibition matrix $S \in \mathbb{R}^{m \times m}$ requires $O(m^2)$ operations per iteration, which can be computationally expensive compared to $O(mn)$ in dictionary-factored ISTA. Two strategies to reduce this computational burden were evaluated on $10 \times 10$ natural image patches ( $n = 100$ ) with complete ( $m = 100$ ) and 4-times overcomplete ( $m = 400$ ) dictionaries:
1. Low-Rank Factorization: Representing $S$ as $S = U_1^T U_2$ , where $U_1, U_2 \in \mathbb{R}^{q \times m}$ and $U_1$ has normalized rows, reducing the operations by a factor $c_f = 2q/m$ .
2. Element Pruning (Sparsification): Constraining a fraction of elements in $S$ to remain zero throughout training, selecting which connections to drop by pruning the elements with the smallest absolute values in the theoretical ISTA matrix $I - W_d^T W_d$ .
Pruning small elements is substantially more efficient than low-rank factorization for a given computational budget $c_f$ . Pruning 80% of the connections in $S$ ( $c_f = 0.2$ ) leads to only a minor increase in 1-iteration LISTA prediction error (from approximately 1.6 to 2.0 for $m = 400$ ), while preserving fast sparse matrix multiplication over active units.

Knowl 8 — Sparse Code Prediction Error on Natural Image Patches

data/table

The prediction error (mean squared error between the predicted code and the optimal ground-truth code $Z^*$ ) was evaluated on $10 \times 10$ mean-subtracted, variance-normalized natural image patches from the Berkeley image database with sparsity parameter $\alpha = 0.5$ . Two dictionary sizes were tested: $m = 100$ (complete) and $m = 400$ (4 times overcomplete).

Non-linearity / Architecture	100 units	400 units
$D \tanh(x)$	8.60	10.70
$D(\tanh(x + u) + \tanh(x - u))$	3.33	4.62
$h_\alpha(x)$	3.29	4.82
LISTA 1 iteration	1.50	2.45
LISTA 3 iterations	0.98	2.12
LISTA 7 iterations	0.52	1.62
FISTA 1 iteration	21.0	22.0

A single iteration of LISTA reduces prediction error by nearly a factor of 2 compared to the baseline single-layer shrinkage encoder and is vastly superior to 1 iteration of FISTA. To match the prediction accuracy of 1 iteration of LISTA, standard FISTA requires 18 iterations for $m = 100$ and 35 iterations for $m = 400$ , representing an approximate $20\times$ speedup for approximate sparse coding. Similarly, 5 iterations of LCoD achieve the approximation accuracy of 100 iterations of standard Coordinate Descent (CoD).

Knowl 9 — MNIST Digit Classification Using Truncated Sparse Predictors

data/table

The utility of fast approximate sparse codes for classification was evaluated on the MNIST dataset using linear logistic regression. Two feature configurations were evaluated:

Global 784-D codes trained directly on the $28 \times 28$ digit images.
Local 256-D codes extracted from $16 \times 16$ patches with a 3-pixel stride (25 patches concatenated into a 6400-D feature vector).

Comparisons were made between exact Coordinate Descent (CoD conv), fixed-iteration Coordinate Descent (CoD), Coordinate Descent trained and evaluated with fixed iterations (CoD'), and Learned Coordinate Descent (LCoD).

		Pred. Error (784-D)		Classification Error (%, 784-D)
iter	cpu	CoD	LCoD	CoD	CoD'	LCoD
0	1.00	2143	1.07	6.74	-	2.65
1	1.02	370	0.99	5.53	-	2.55
5	1.05	31.7	0.78	6.71	5.04	2.33
10	1.10	12.0	0.66	5.24	4.82	2.32
20	1.21	5.81	0.55	3.77	4.17	2.39
50	1.50	2.14	0.51	2.57	3.54	2.29
conv	4.56	0	-	2.15	2.15	-

		Pred. Error (256-D)		Classification Error (%, 256-D)
iter	cpu	CoD	LCoD	CoD	CoD'	LCoD
0	1.00	273	0.70	2.24	-	1.66
1	1.15	80.5	0.58	2.30	-	1.60
5	1.55	5.58	0.34	1.82	2.18	1.47
10	2.10	2.75	0.22	1.58	1.99	1.42
20	2.95	1.44	0.14	1.55	1.55	1.42
50	5.45	0.44	0.07	1.46	1.48	1.39
conv	37.25	0	-	1.33	1.33	-

For both configurations, LCoD with only 10 iterations reaches a classification error rate very close to that of fully converged exact inference (2.32% vs 2.15% on 784-D codes; 1.42% vs 1.33% on 256-D patch codes, where differences $< 0.1\%$ are statistically insignificant on MNIST), while standard CoD requires 50+ iterations or exact convergence to approach competitive error rates.

Knowl 10 — Distribution-Specific Approximation and Lack of Asymptotic Convergence
limitation
Unlike classical convex optimization algorithms for sparse coding (such as ISTA, FISTA, or Coordinate Descent), which possess mathematical guarantees of asymptotic convergence to the unique global minimum of the LASSO objective for any arbitrary input vector $X$ , learned predictors such as LISTA and LCoD exhibit two fundamental theoretical and practical limitations:
1. Lack of Asymptotic Convergence Beyond Trained Depth: LISTA and LCoD parameters are optimized specifically to minimize prediction error within a fixed truncation depth $T$ . If the unfolded network is run for more steps than the number of iterations it was trained for, it is not guaranteed to converge to the true optimal sparse code $Z^*$ .
2. Distributional Specificity: The learned parameters $(W_e, S, \theta)$ are tuned to minimize expected prediction error specifically for input vectors drawn from the training distribution $P(X)$ . They do not represent general-purpose solvers for arbitrary out-of-distribution inputs.
3. Random Initialization Degradation at High Iteration Counts: When trained with large numbers of iterations from random initializations, LCoD performance degrades relative to standard CoD. Pre-initializing the encoder matrices with their analytically derived CoD values ( $W_e = W_d^T$ , $S = I - W_d^T W_d$ , $\theta = \alpha$ ) improves performance in the high-iteration regime, but can degrade performance in the low-iteration regime.

Coverage note — All primary contributions—the LISTA architecture and backpropagation algorithm, the LCoD architecture and sub-gradient backpropagation algorithm, the supervised MSE learning framework, the explaining-away analysis, mutual inhibition matrix pruning/factorization, natural image patch approximation experiments, MNIST classification benchmarks, and fundamental convergence/distributional limitations—are covered. Standard background algorithms (unmodified classical ISTA, FISTA, and basic dictionary learning) were omitted as standalone knowls.

References

1.Beck, A. and Teboulle, M. A fast iterative shrinkage-thresholding algorithm with application to wavelet-based image deblurring. ICASSP'09, pp. 693–696, 2009.
2.Chen, S.S., Donoho, D.L., and Saunders, M.A. Atomic decomposition by basis pursuit. SIAM review, 43(1): 129–159, 2001.
3.Daubechies, I, Defrise, M., and De Mol, C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. on Pure and Applied Mathematics, 57:1413–1457, 2004.
4.Donoho, D.L. and Elad, M. Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ 1 minimization. PNAS, 100(5):2197–2202, 2003.
5.Elad, M. and Aharon, M. Image denoising via learned dictionaries and sparse representation. In CVPR'06, 2006.
6.Hale, E.T., Yin, W., and Zhang, Y. Fixed-point continuation for l1-minimization: Methodology and convergence. SIAM J. on Optimization, 19:1107, 2008.
7.Hoyer, P. O. Non-negative matrix factorization with sparseness constraints. JMLR, 5:1457–1469, 2004.
8.Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. What is the best multi-stage architecture for object recognition? In ICCV'09. IEEE, 2009.
9.Kavukcuoglu, Koray, Ranzato, Marc'Aurelio, and LeCun, Yann. Fast inference in sparse coding algorithms with applications to object recognition. Technical Report CBLL-TR-2008-12-01, Computational and Biological Learning Lab, Courant Institute, NYU, 2008.
10.Lee, H., Battle, A., Raina, R., and Ng, A.Y. Efficient sparse coding algorithms. In NIPS'06, 2006.
11.Lee, H., Chaitanya, E., and Ng, A. Y. Sparse deep belief net model for visual area v2. In Advances in Neural Information Processing Systems, 2007.
12.Lee, H., Grosse, R., Ranganath, R., and Ng, A.Y. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In International Conference on Machine Learning. ACM New York, 2009.
13.Li, Y. and Osher, S. Coordinate descent optimization for l1 minimization with application to compressed sensing; a greedy algorithm. Inverse Problems and Imaging, 3 (3):487–503, 2009.
14.Mairal, J., Elad, M., and Sapiro, G. Sparse representation for color image restoration. IEEE T. Image Processing, 17(1):53–69, January 2008.
15.Mairal, J., Bach, F., Ponce, J., and Sapiro, G. Online dictionary learning for sparse coding. In ICML'09, 2009.
16.Olshausen, B.A. and Field, D. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996.
17.Ranzato, M., Huang, F.-J., Boureau, Y.-L., and LeCun, Y. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In CVPR'07. IEEE, 2007a.
18.Ranzato, M.-A., Boureau, Y.-L., Chopra, S., and LeCun, Y. A unified energy-based framework for unsupervised learning. In AI-Stats'07, 2007b.
19.Rozell, C.J., Johnson, D.H, Baraniuk, R.G., and Olshausen, B.A. Sparse coding via thresholding and local competition in neural circuits. Neural Computation, 20: 2526–2563, 2008.
20.Vonesch, C. and Unser, M. A fast iterative thresholding algorithm for wavelet-regularized deconvolution. In IEEE ISBI, 2007.
21.Wu, T.T. and Lange, K. Coordinate descent algorithms for lasso penalized regression. Ann. Appl. Stat, 2(1): 224–244, 2008.
22.Yang, Jianchao, Yu, Kai, Gong, Yihong, and Huang, Thomas. Linear spatial pyramid matching using sparse coding for image classification. In CVPR'09, 2009.
23.Yu, Kai, Zhang, Tong, and Gong, Yihong. Nonlinear learning using local coordinate coding. In NIPS'09, 2009.

Citation

MLA

Gregor, K., and Y. LeCun. “Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields”. arXiv, 2010, http://arxiv.org/abs/1006.0448v1.

APA

Gregor, K., & LeCun, Y. (2010). Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields. arXiv. http://arxiv.org/abs/1006.0448v1

Chicago

Gregor, K., and Y. LeCun. 2010. “Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields”. arXiv. http://arxiv.org/abs/1006.0448v1.

Harvard

Gregor, K. and LeCun, Y. (2010) “Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1006.0448v1.

Vancouver

1. Gregor K, LeCun Y (2010) Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields. arXiv

BibTeX

@article{gregor2010emergence,
  title = {Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields},
  author = {Gregor, Karo and LeCun, Yann},
  year = {2010},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1006.0448v1},
  eprint = {1006.0448}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Learning Fast Approximations of Sparse Coding

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Learned Iterative Shrinkage-Thresholding Algorithm (LISTA) Architecture

Knowl 2 — LISTA Forward and Backward Propagation Algorithms

Knowl 3 — Learned Coordinate Descent (LCoD) Architecture

Knowl 4 — LCoD Forward and Backward Propagation Algorithms

Knowl 5 — Supervised Objective for Training Sparse Code Predictors

Knowl 6 — Explaining Away and Non-Linearity Selection in Feed-Forward Encoders

Knowl 7 — Sparsification vs. Low-Rank Factorization of the Mutual Inhibition Matrix