Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

Honglak LeeRoger GrosseRajesh RanganathAndrew Y. Ng

article2009ICML2,704 citationsBest Application Paper Award, Best Student Paper Award

Introduces convolutional deep belief networks and probabilistic max-pooling to scale unsupervised hierarchical feature learning to full-sized images, enabling both bottom-up recognition and top-down reconstruction of occluded parts.

Listen

The article addresses the challenge of scaling unsupervised hierarchical generative models, such as deep belief networks, to realistic high-dimensional images. These models hold promise for learning multi-level visual features from unlabeled data, yet they struggle with computational demands and the need for representations that remain stable under small shifts in object position.

The article set out to develop and test a convolutional deep belief network that incorporates weight sharing and a new probabilistic max-pooling step to achieve translation-invariant inference while remaining tractable on full-sized images.

The authors trained the model layer by layer on natural-scene photographs and evaluated it through classification experiments on the Caltech-101 and MNIST data sets, as well as qualitative inspection of features learned from single object categories.

The model learned oriented edge detectors in the first layer, object-part detectors in the second, and whole-object detectors in the third. When these features were used for object recognition, accuracy reached 57.7 percent on Caltech-101 with 15 training images per class and 65.4 percent with 30 images; on MNIST the test error fell to 0.8 percent with the full training set. The network also combined bottom-up and top-down information to infer occluded object parts more accurately than a purely feed-forward pass.

These results indicate that large unlabeled image collections can yield general-purpose hierarchical features competitive with hand-engineered descriptors, lowering the cost of building visual recognition systems and supporting inference under partial occlusion or ambiguity.

The article recommends extending the approach to larger and more varied data collections and exploring tighter integration with supervised fine-tuning. Additional validation on video and three-dimensional data would strengthen before widespread deployment.

The main limitations are reliance on mean-field approximations for inference and evaluation on only a few benchmark collections; performance on domains with very different statistics remains untested.

Stanford University deep belief networks generative model inference Convolutional Deep Belief Network Probabilistic Max-Pooling Translation Invariance

honglaklee/convDBN Lee et al (2009).pdf

Paper: A Fast Learning Algorithm for Deep Belief Nets, Geoffrey E. Hinton et al. (2006). This seminal paper introduces the greedy layer-wise training of Deep Belief Networks using Restricted Boltzmann Machines, providing the theoretical and algorithmic foundation that the source extends to convolutional architectures.
Paper: Greedy Layer-Wise Training of Deep Networks, Yoshua Bengio et al. (2007). This work establishes how greedy layer-wise unsupervised pre-training effectively initializes deep hierarchical networks and extends Restricted Boltzmann Machines to continuous inputs.
Paper: Extracting and composing robust features with denoising autoencoders, Pascal Vincent et al. (2008). This paper presents the principles of learning robust intermediate representations and stacking unsupervised layer-wise models for deep hierarchical feature extraction.
Paper: What is the best multi-stage architecture for object recognition?, Kevin Jarrett et al. (2009). This contemporary study provides crucial empirical insights into the roles of filter learning, spatial pooling, and rectification stages in multi-stage visual recognition architectures.
Paper: One-shot learning of object categories, Li Fei-Fei et al. (2006). This paper establishes benchmark visual recognition challenges on Caltech-101 that provide the evaluation context for unsupervised hierarchical feature extractors.
Paper: A Bayesian hierarchical model for learning natural scene categories, Li Fei-Fei et al. (2005). This work introduces probabilistic hierarchical modeling of natural scene categories, motivating the transition toward generative models capable of scalable visual feature discovery.

Paper: 3D ShapeNets: A deep representation for volumetric shapes, Zhirong Wu et al. (2014). This work directly extends convolutional deep belief networks from 2D images to 3D volumetric shapes for generative shape completion and object recognition.
Paper: An Analysis of Single-Layer Networks in Unsupervised Feature Learning, Adam Coates et al. (2011). This paper rigorously analyzes single-layer feature extraction and pooling dynamics in unsupervised visual learning pipelines, building upon the foundational feature representations explored in CDBNs.
Paper: Building high-level features using large scale unsupervised learning, Quoc V. Le et al. (2011). This paper scales up unsupervised hierarchical feature learning to massive unlabeled datasets to autonomously discover high-level semantic concepts like faces and bodies.
Paper: Why Does Unsupervised Pre-training Help Deep Learning?, D. Erhan et al. (2010). This work provides an in-depth empirical investigation into the optimization and regularization mechanisms that make unsupervised pre-training effective for deep architectures.
Paper: Representation Learning: A Review and New Perspectives, Yoshua Bengio et al. (2012). This comprehensive review synthesizes the principles of deep unsupervised representation learning, probabilistic models, and hierarchical abstraction established in early deep learning literature.
Paper: Visualizing and Understanding Convolutional Networks, Matthew D. Zeiler et al. (2014). This paper develops deconvolutional visualization techniques to interpret the hierarchical edge, part, and object features formed within deep convolutional networks.
Paper: DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition, Jeff Donahue et al. (2013). This work examines how deep convolutional activations can serve as general-purpose, reusable feature extractors across diverse downstream recognition tasks.

Cover for Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

Abstract

There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.

1. Introduction
2. Preliminaries
2.1. Restricted Boltzmann machines
2.2. Deep belief networks
3. Algorithms
3.1. Notation
3.2. Convolutional RBM
3.3. Probabilistic max-pooling
3.4. Training via sparsity regularization
3.5. Convolutional deep belief network
3.6. Hierarchical probabilistic inference
3.7. Discussion
4. Experimental results
4.1. Learning hierarchical representations from natural images
4.2. Self-taught learning for object recognition
4.3. Handwritten digit classification
4.4. Unsupervised learning of object parts
4.5. Hierarchical probabilistic inference
5. Conclusion
Acknowledgment
References

Knowls

Knowl 1 — Probabilistic Max-Pooling in Convolutional Restricted Boltzmann Machines
model/method
Probabilistic max-pooling is a mechanism that allows higher layers of an undirected generative graphical model to shrink spatial representations while preserving exact probabilistic semantics and supporting both bottom-up and top-down inference.

Consider an input/visible layer $V \in \{0,1\}^{N_V \times N_V}$ , a detection hidden layer $H$ consisting of $K$ groups of size $N_H \times N_H$ , and a pooling layer $P$ consisting of $K$ groups of size $N_P \times N_P$ , where $N_P = N_H / C$ for an integer pooling ratio $C$ (such as $2$ or $3$ ). In each group $k \in \{1, \dots, K\}$ , the detection layer $H^k$ is partitioned into non-overlapping spatial blocks $B_\alpha$ of size $C \times C$ , where each block $\alpha$ is wired to exactly one binary pooling unit $p_\alpha^k$ .

The joint potential over block $B_\alpha$ enforces two hard constraints:
1. At most one detection unit within block $B_\alpha$ can be active ( $\sum_{(i,j) \in B_\alpha} h_{ij}^k \le 1$ ).
2. The pooling unit $p_\alpha^k$ is active if and only if exactly one detection unit in $B_\alpha$ is active ( $p_\alpha^k = \sum_{(i,j) \in B_\alpha} h_{ij}^k$ ).
Thus, the $C^2$ detection units together with the pooling unit act as a single discrete variable with $C^2 + 1$ mutually exclusive states: either one specific detection unit is $1$ (and $p_\alpha^k = 1$ ), or all detection units are $0$ (and $p_\alpha^k = 0$ ).

Given visible input $v$ , the bottom-up input to detection unit $(i,j)$ in group $k$ is: $I(h_{ij}^k) \triangleq b_k + (\tilde{W}^k * v)_{ij}$ where $W^k \in \mathbb{R}^{N_W \times N_W}$ is the shared filter for group $k$ , $\tilde{W}^k$ denotes $W^k$ flipped horizontally and vertically, $*$ denotes convolution, and $b_k$ is the group bias.

Conditioned on the visible layer $v$ , detection and pooling units in block $B_\alpha$ follow the multinomial conditional probabilities: $P(h_{ij}^k = 1 \mid v) = \frac{\exp(I(h_{ij}^k))}{1 + \sum_{(i',j') \in B_\alpha} \exp(I(h_{i'j'}^k))}$ $P(p_\alpha^k = 0 \mid v) = \frac{1}{1 + \sum_{(i',j') \in B_\alpha} \exp(I(h_{i'j'}^k))}$ for each $(i,j) \in B_\alpha$ .
Knowl 2 — Convolutional Restricted Boltzmann Machine Formulation
model/method
The Convolutional Restricted Boltzmann Machine (CRBM) is a two-layer undirected bipartite graphical model that incorporates 2D spatial weight-sharing into the Restricted Boltzmann Machine framework.

The visible layer $V$ consists of an $N_V \times N_V$ array of units $v_{ij}$ , and the hidden layer $H$ consists of $K$ feature groups, where each group $k \in \{1, \dots, K\}$ is an $N_H \times N_H$ array of binary units $h_{ij}^k$ . Each hidden group $k$ is parameterized by a shared filter $W^k \in \mathbb{R}^{N_W \times N_W}$ (with $N_W = N_V - N_H + 1$ ) and a hidden bias scalar $b_k$ . All visible units share a single scalar bias $c$ .

For binary visible units $v \in \{0, 1\}^{N_V \times N_V}$ , the energy function is defined as: $E(v, h) = -\sum_{k=1}^K h^k \bullet (\tilde{W}^k * v) - \sum_{k=1}^K b_k \sum_{i,j} h_{ij}^k - c \sum_{i,j} v_{ij}$ where $\tilde{W}^k$ denotes $W^k$ flipped horizontally and vertically, $*$ denotes 2D convolution, and $\bullet$ denotes the Frobenius inner product (element-wise product summed over all entries, $A \bullet B = \operatorname{tr}(A^T B)$ ).

The conditional distributions for block Gibbs sampling are: $P(h_{ij}^k = 1 \mid v) = \sigma\left((\tilde{W}^k * v)_{ij} + b_k\right)$ $P(v_{ij} = 1 \mid h) = \sigma\left(\left(\sum_{k=1}^K W^k * h^k\right)_{ij} + c\right)$ where $\sigma(z) = \frac{1}{1 + \exp(-z)}$ is the logistic sigmoid function.

For real-valued visible units $v \in \mathbb{R}^{N_V \times N_V}$ , adding a quadratic visible term $\frac{1}{2}\sum_{i,j} v_{ij}^2$ to the energy yields Gaussian conditional distributions with unit variance given the hidden layer: $v_{ij} \mid h \sim \mathcal{N}\left(\left(\sum_{k=1}^K W^k * h^k\right)_{ij} + c, 1\right).$
Knowl 3 — Convolutional Deep Belief Network Architecture and Layer-Wise Training
model/method
The Convolutional Deep Belief Network (CDBN) is a hierarchical generative model formed by stacking multiple Convolutional Restricted Boltzmann Machines with probabilistic max-pooling.

The complete model defines an energy function equal to the sum of the energy functions of its constituent adjacent layer pairs. CDBNs are trained in a greedy, layer-wise unsupervised manner:
1. The first-layer CRBM is trained on raw image inputs using contrastive divergence.
2. Once layer $l$ is trained, its filter weights and biases are frozen, and its expected activations (or pooled unit samples) $P^{(l)}$ serve as the visible inputs for training layer $l+1$ .
Because the representation is overcomplete (the total number of hidden units $K \times N_H^2$ typically exceeds the input image size $N_V^2$ ), models run the risk of learning trivial features such as single-pixel detectors. To prevent this, a sparsity regularization term is added to the log-likelihood objective during training, penalizing the divergence between each hidden unit's mean activation and a small target constant $\rho \ll 1$ .
Knowl 4 — Bidirectional Hierarchical Probabilistic Inference in Multi-Layer CDBNs
algorithm
In a multi-layer CDBN with undirected interlayer connections, hidden states at intermediate layers are inferred by fusing bottom-up evidence from lower layers with top-down contextual signals from higher layers.

For a subnetwork consisting of visible layer $V$ , intermediate detection layer $H$ , pooling layer $P$ , and subsequent higher detection layer $H'$ , let $W^k$ denote the filter connecting $V$ to hidden group $H^k$ , and let $\Gamma^{k,\ell}$ denote the filter connecting pooling group $P^k$ to higher detection group ${H'}^\ell$ . The energy function is: $E(v, h, p, h') = -\sum_k v \bullet (W^k * h^k) - \sum_k b_k \sum_{i,j} h_{ij}^k - \sum_{k,\ell} p^k \bullet (\Gamma^{k,\ell} * {h'}^\ell) - \sum_\ell b'_\ell \sum_{i,j} {h'}_{ij}^\ell$ subject to $\sum_{(i,j) \in B_\alpha} h_{ij}^k = p_\alpha^k \le 1$ for all groups $k$ and blocks $B_\alpha$ .

The inference procedure computes the joint posterior distribution over intermediate layers as follows:
Input: Visible activations $v$ , higher detection layer activations $h'$
Output: Updated intermediate detection activations $h$ and pooling activations $p$
for each group $k \in \{1, \dots, K\}$ do
    Compute bottom-up input: $I(h_{ij}^k) \leftarrow b_k + (\tilde{W}^k * v)_{ij}$ for all $(i,j)$
    Compute top-down input: $I(p_\alpha^k) \leftarrow \sum_\ell (\tilde{\Gamma}^{k,\ell} * {h'}^\ell)_\alpha$ for all blocks $\alpha$
    for each pooling block $B_\alpha$ do
        for each $(i,j) \in B_\alpha$ do
             $P(h_{ij}^k = 1 \mid v, h') \leftarrow \frac{\exp(I(h_{ij}^k) + I(p_\alpha^k))}{1 + \sum_{(i',j') \in B_\alpha} \exp(I(h_{i'j'}^k) + I(p_\alpha^k))}$
        end for
         $P(p_\alpha^k = 0 \mid v, h') \leftarrow \frac{1}{1 + \sum_{(i',j') \in B_\alpha} \exp(I(h_{i'j'}^k) + I(p_\alpha^k))}$
        Sample $(h_{B_\alpha}^k, p_\alpha^k)$ from the above multinomial distribution
    end for
end for
return $h, p$
Full posterior estimation across all network layers is executed via parallel block Gibbs sampling or approximated deterministically using mean-field iterations (where 5 iterations empirically suffice).

Knowl 5 — Caltech-101 Object Recognition Accuracy via Self-Taught Learning

data/table

A two-layer CDBN was trained in an unsupervised manner entirely on unlabeled natural images from the Kyoto dataset (24 first-layer $10 \times 10$ bases, 100 second-layer $10 \times 10$ bases, pooling factor $C=2$ ). The model was evaluated on Caltech-101 classification under the self-taught learning protocol, where unsupervised pre-training uses generic data unrelated to the target task. Activations from the first and second pooling layers at full and half resolution were pooled using spatial pyramid matching and classified with an SVM. Accuracy was normalized over all classes and averaged across 10 random splits:

Method	15 Training Examples	30 Training Examples
CDBN (first layer)	$53.2 \pm 1.2\%$	$60.5 \pm 1.1\%$
CDBN (first + second layers)	$57.7 \pm 1.5\%$	$65.4 \pm 0.5\%$
Raina et al. (2007)	$46.6\%$	–
Ranzato et al. (2007)	–	$54.0\%$
Mutch and Lowe (2006)	$51.0\%$	$56.0\%$
Lazebnik et al. (2006)	$54.0\%$	$64.6\%$
Zhang et al. (2006)	$59.0 \pm 0.56\%$	$66.2 \pm 0.5\%$

Combining second-layer features with first-layer features improves classification accuracy over first-layer features alone by $4.5\%$ (with 15 training images/class) and $4.9\%$ (with 30 training images/class), reaching performance competitive with hand-engineered single visual descriptors (such as SIFT, geometric blur, and shape-context).

Knowl 6 — MNIST Handwritten Digit Classification Test Error

data/table

A two-layer CDBN (40 first-layer $12 \times 12$ bases, 40 second-layer $6 \times 6$ bases, pooling factor $C=2$ for each layer) was trained unsupervised on MNIST digit images. Feature vectors constructed by concatenating first- and second-pooling-layer activations were classified with an SVM across varying sizes of labeled training data. Test errors are reported as percentages (averaged over 10 random training subsets for smaller sample sizes):

Method	1,000	2,000	3,000	5,000	60,000 (Full)
CDBN	$2.62 \pm 0.12\%$	$2.13 \pm 0.10\%$	$1.91 \pm 0.09\%$	$1.59 \pm 0.11\%$	$0.82\%$
Ranzato et al. (2007)	$3.21\%$	$2.53\%$	–	$1.52\%$	$0.64\%$
Hinton and Salakhutdinov (2006)	–	–	–	–	$1.20\%$
Weston et al. (2008)	$2.73\%$	–	$1.83\%$	–	$1.50\%$

On limited labeled training sizes (1,000 to 3,000 examples), the CDBN achieves lower classification error than contemporary deep learning methods, and reaches $0.82\%$ error when trained on the full 60,000-example training set.

Knowl 7 — Hierarchical Unsupervised Part Learning and Category Specificity
empirical result
When trained on unlabeled images of object categories (such as Caltech-101 faces, cars, motorbikes, airplanes, elephants, and chairs) without bounding boxes or part annotations:
- The first layer learns localized, oriented edge filters.
- The second layer learns object parts (such as noses, eyes, wheels, handlebars, and car windows).
- The third layer combines parts into full-object representations or object shapes.
The category specificity of individual features increases monotonically with layer depth:
1. Class-Specific Models (Area Under Precision-Recall Curve, AUC-PR): In three separate three-layer CDBNs trained individually on faces, motorbikes, and cars, the average individual feature AUC-PR for distinguishing the target category from the other two categories increases at higher layers:
  - Faces: Layer 1 = $0.39 \pm 0.17$ , Layer 2 = $0.86 \pm 0.13$ , Layer 3 = $0.95 \pm 0.03$ .
  - Motorbikes: Layer 1 = $0.44 \pm 0.21$ , Layer 2 = $0.69 \pm 0.22$ , Layer 3 = $0.81 \pm 0.13$ .
  - Cars: Layer 1 = $0.43 \pm 0.19$ , Layer 2 = $0.72 \pm 0.23$ , Layer 3 = $0.87 \pm 0.15$ .
2. Multi-Class Mixture Models (Conditional Entropy): In a CDBN trained on an unlabeled mixture of four classes (faces, cars, airplanes, motorbikes), the conditional entropy $H(\text{class} \mid \gamma > 0.95)$ of the top quantile activation $\gamma$ decreases progressively from layer 1 to layer 3, demonstrating that higher-layer units spontaneously develop class selectivity without supervision.
Knowl 8 — Hierarchical Occlusion Filling via Bidirectional Gibbs Sampling
empirical result
CDBNs perform hierarchical Bayesian inference to complete occluded visual parts by combining bottom-up sensory evidence with top-down generative context.

When presented with face images in which the entire left half is occluded (set to zero intensity):
- A single bottom-up feed-forward pass activates only the right side of the second layer; because no low-level evidence exists on the left half, the reconstructed second-layer representation fails to infer the missing facial features.
- Running 20 iterations of bidirectional block Gibbs sampling across all hidden layers enables the third layer (which identifies the global face structure from the unoccluded right half) to transmit top-down contextual signals to the second and first layers, successfully reconstructing the occluded left eye, nose boundary, and mouth.
Knowl 9 — Hierarchical Representation Learning on Natural Images and the Role of Sparsity
empirical result
When a two-layer CDBN is trained on unlabeled natural images from the Kyoto dataset using real-valued Gaussian visible units (layer 1: 24 filters of size $10 \times 10$ ; layer 2: 100 filters of size $10 \times 10$ ; pooling ratio $C=2$ ):
- Layer 1: The first layer learns localized, oriented Gabor-like edge filters. Sparsity regularization during training is strictly necessary for this behavior; removing the sparsity penalty prevents the network from learning oriented edge filters.
- Layer 2: The second-layer filters (which cover roughly twice the spatial area of layer 1 due to pooling) combine first-layer edge responses to selectively detect contours, corners, angles, and surface boundaries.
Knowl 10 — Computational Speedup from Weight Sharing and Probabilistic Max-Pooling
empirical result
In a three-layer network evaluated on $200 \times 200$ pixel images:
- Inference with convolutional weight sharing but without probabilistic max-pooling is approximately 10 times slower than inference with probabilistic max-pooling.
- Inference without convolutional weight sharing (a standard fully connected deep belief network) is more than 100 times slower than the convolutional deep belief network with probabilistic max-pooling.

Coverage note — None was omitted; all key theoretical models, algorithms, experimental setups, and quantitative/qualitative results were converted into knowls.

References

1.Bell, A. J., & Sejnowski, T. J. (1997). The ‘independent components’ of natural scenes are edge filters. Vision Research, 37, 3327–3338.
2.Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2006). Greedy layer-wise training of deep networks. Adv. in Neural Information Processing Systems.
3.Berg, A. C., Berg, T. L., & Malik, J. (2005). Shape matching and object recognition using low distortion correspondence. IEEE Conference on Computer Vision and Pattern Recognition (pp. 26–33).
4.Desjardins, G., & Bengio, Y. (2008). Empirical evaluation of convolutional RBMs for vision (Technical Report).
5.Fei-Fei, L., Fergus, R., & Perona, P. (2004). Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. CVPR Workshop on Gen.-Model Based Vision.
6.Grosse, R., Raina, R., Kwong, H., & Ng, A. (2007). Shift-invariant sparse coding for audio classification. Proceedings of the Conference on Uncertainty in AI.
7.Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14, 1771–1800.
8.Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554.
9.Hinton, G. E., & Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504–507.
10.Ito, M., & Komatsu, H. (2004). Representation of angles embedded within contour stimuli in area V2 of macaque monkeys. J. Neurosci., 24, 3313–3324.
11.Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. IEEE Conference on Computer Vision and Pattern Recognition.
12.LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1, 541–551.
13.Lee, H., Ekanadham, C., & Ng, A. Y. (2008). Sparse deep belief network model for visual area V2. Advances in Neural Information Processing Systems.
14.Lee, T. S., & Mumford, D. (2003). Hierarchical bayesian inference in the visual cortex. Journal of the Optical Society of America A, 20, 1434–1448.
15.Mutch, J., & Lowe, D. G. (2006). Multiclass object recognition with sparse, localized features. IEEE Conf. on Computer Vision and Pattern Recognition.
16.Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607–609.
17.Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007). Self-taught learning: Transfer learning from unlabeled data. International Conference on Machine Learning (pp. 759–766).
18.Raina, R., Madhavan, A., & Ng, A. Y. (2009). Large-scale deep unsupervised learning using graphics processors. International Conf. on Machine Learning.
19.Ranzato, M., Huang, F.-J., Boureau, Y.-L., & LeCun, Y. (2007). Unsupervised learning of invariant feature hierarchies with applications to object recognition. IEEE Conference on Computer Vision and Pattern Recognition.
20.Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2006). Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems (pp. 1137–1144).
21.Taylor, G., Hinton, G. E., & Roweis, S. (2007). Modeling human motion using binary latent variables. Adv. in Neural Information Processing Systems.
22.Varma, M., & Ray, D. (2007). Learning the discriminative power-invariance trade-off. International Conference on Computer Vision.
23.Weston, J., Ratle, F., & Collobert, R. (2008). Deep learning via semi-supervised embedding. International Conference on Machine Learning.
24.Yu, K., Xu, W., & Gong, Y. (2009). Deep learning with kernel regularization for visual recognition. Adv. Neural Information Processing Systems.
25.Zhang, H., Berg, A. C., Maire, M., & Malik, J. (2006). SVM-KNN: Discriminative nearest neighbor classification for visual category recognition. IEEE Conference on Computer Vision and Pattern Recognition.

Citation

MLA

Lee, H., et al. “Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations”. Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 609–16, https://doi.org/10.1145/1553374.1553453.

APA

Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th Annual International Conference on Machine Learning, 609–616. https://doi.org/10.1145/1553374.1553453

Chicago

Lee, H., R. Grosse, R. Ranganath, and A. Y. Ng. 2009. “Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations”. Proceedings of the 26th Annual International Conference on Machine Learning, 609–16. https://doi.org/10.1145/1553374.1553453.

Harvard

Lee, H. et al. (2009) “Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations”, Proceedings of the 26th Annual International Conference on Machine Learning. ACM, pp. 609–616. Available at: https://doi.org/10.1145/1553374.1553453.

Vancouver

1. Lee H, Grosse R, Ranganath R, Ng AY (2009) Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of the 26th Annual International Conference on Machine Learning. ACM, pp 609–616

BibTeX

@inproceedings{Lee_2009, series={ICML ’09}, title={Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations}, url={http://dx.doi.org/10.1145/1553374.1553453}, DOI={10.1145/1553374.1553453}, booktitle={Proceedings of the 26th Annual International Conference on Machine Learning}, publisher={ACM}, author={Lee, Honglak and Grosse, Roger and Ranganath, Rajesh and Ng, Andrew Y.}, year={2009}, month=June, pages={609–616}, collection={ICML ’09} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Probabilistic Max-Pooling in Convolutional Restricted Boltzmann Machines

Knowl 2 — Convolutional Restricted Boltzmann Machine Formulation

Knowl 3 — Convolutional Deep Belief Network Architecture and Layer-Wise Training

Knowl 4 — Bidirectional Hierarchical Probabilistic Inference in Multi-Layer CDBNs

Knowl 5 — Caltech-101 Object Recognition Accuracy via Self-Taught Learning

Knowl 6 — MNIST Handwritten Digit Classification Test Error

Knowl 7 — Hierarchical Unsupervised Part Learning and Category Specificity

Knowl 8 — Hierarchical Occlusion Filling via Bidirectional Gibbs Sampling

Knowl 9 — Hierarchical Representation Learning on Natural Images and the Role of Sparsity

Knowl 10 — Computational Speedup from Weight Sharing and Probabilistic Max-Pooling

References

Citation

BibTeX

Source Code

Access the Paper