Back

Learning Hierarchical Features for Scene Labeling

C. FarabetC. CouprieLaurent NajmanYann LeCun

article2013TPAMI2,784 citations

Proposes a multiscale convolutional network that learns hierarchical features directly from raw pixels across image pyramids, enabling near-real-time scene parsing while eliminating the need for hand-engineered features and complex graphical model post-processing.

Listen

The article addresses the challenge of scene labeling, also known as scene parsing, in which every pixel in an image is assigned the category of the object or region it belongs to. This task combines detection, segmentation, and recognition and supports practical image-understanding applications, yet remains difficult because pixel categories often depend on both local texture and wide contextual information.

The article evaluates a feed-forward system that extracts dense feature vectors from raw pixels using a multiscale convolutional network applied to a Laplacian pyramid of the input image. Three post-processing strategies are compared: simple voting within superpixels, a conditional random field over superpixels, and an optimal-cover procedure that selects the most class-pure segments from a family of over-segmentations. The network is trained end-to-end in a supervised manner on fully labeled images; no hand-engineered features are used.

Experiments on three standard benchmarks show that the multiscale network alone already produces competitive pixel-wise and per-class accuracies. Adding the optimal-cover or CRF post-processing yields record results on the SIFT Flow dataset (33 classes) and the Barcelona dataset (170 classes) and near-record accuracy on the Stanford Background dataset (8 classes). The complete pipeline labels a 320 × 240 image in less than one second on a conventional CPU—roughly an order of magnitude faster than prior methods—while requiring no per-image parameter tuning.

These outcomes indicate that a wide contextual window learned directly from pixels can capture most scene-level relationships, thereby reducing reliance on complex graphical-model inference. The speed and accuracy open the door to real-time applications, yet performance remains modest once the number of categories grows large, and the system still needs fully pixel-labeled training data.

Further progress would benefit from improved evaluation metrics that reward correct detection of rare objects, from hierarchical label sets, and from joint training of the network with the post-processor so that weakly labeled images can be exploited. The main uncertainties concern generalization to new domains and the sensitivity of the learned features to class imbalance during training.

New York UniversityUniversité Paris-Estconvolutional neural networksfeature extractionfeature vectorsMultiscale Convolutional NetworkScene ParsingSuperpixel
Farabet et al (2013).pdf
Cover for Learning Hierarchical Features for Scene Labeling

Abstract

Scene labeling consists in labeling each pixel in an image with the category of the object it belongs to. We propose a method that uses a multiscale convolutional network trained from raw pixels to extract dense feature vectors that encode regions of multiple sizes centered on each pixel. The method alleviates the need for engineered features, and produces a powerful representation that captures texture, shape and contextual information. We report results using multiple post-processing methods to produce the final labeling. Among those, we propose a technique to automatically retrieve, from a pool of segmentation components, an optimal set of components that best explain the scene; these components are arbitrary, e.g. they can be taken from a segmentation tree, or from any family of over-segmentations. The system yields record accuracies on the Sift Flow Dataset (33 classes) and the Barcelona Dataset (170 classes) and near-record accuracy on Stanford Background Dataset (8 classes), while being an order of magnitude faster than competing approaches, producing a 320 × 240 image labeling in less than a second, including feature extraction.

Topics

convolutional neural networksfeature extractionfeature vectorssiftfeature space

Keywords

Multiscale Convolutional NetworkScene ParsingSuperpixelConditional Random FieldOptimal CoverFeature ExtractionLaplacian PyramidReal‑timeEntropy PurityDataset

Table of Contents

  • 1 INTRODUCTION
  • 2 RELATED WORK
  • 3 MULTISCALE FEATURE EXTRACTION FOR SCENE PARSING
  • 3.1 Scale-invariant, scene-level feature extraction
  • 3.2 Learning discriminative scale-invariant features
  • 4 SCENE LABELING STRATEGIES
  • 4.1 Superpixels
  • 4.2 Conditional Random Fields
  • 4.3 Parameter-free multilevel parsing
  • 4.3.1 Optimal purity cover
  • 4.3.2 Producing the confidence costs
  • 4.3.3 Training procedure
  • 5 EXPERIMENTS
  • 5.1 Multiscale feature extraction
  • 5.2 Parsing with superpixels
  • 5.3 Multilevel parsing
  • 5.4 Conditional random field
  • 5.5 Some comments on the learned features
  • 5.6 Some comments on real-world generalization
  • 6 DISCUSSION
  • 7 CONCLUSION AND FUTURE WORK
  • ACKNOWLEDGMENT
  • REFERENCES

Knowls

  1. Knowl 1 — Multiscale Convolutional Network Architecture for Dense Scene Feature Extraction

    model/method

    To extract scale-invariant visual features encompassing both short-range details and wide scene-level context without incurring an unmanageable parameter count, the system processes an input image II using a multiscale convolutional network.

    The input image II is converted to YUV color space, and each channel is locally contrast-normalized such that local 15×1515 \times 15 patches exhibit zero mean and unit variance. A Laplacian pyramid produces N=3N=3 octave scales of the image: X1X_1 (320×240320 \times 240), X2X_2 (160×120160 \times 120), and X3X_3 (80×6080 \times 60).

    A three-stage convolutional neural network fs(Xs;θ0)f_s(X_s; \theta_0) with shared parameters θ0\theta_0 is evaluated across all scales s∈{1,…,N}s \in \{1, \dots, N\}:

    1. Stage 1: A bank of 16 filters of spatial dimension 7×77 \times 7 (10 filters connected to the Y channel, 6 connected to the U and V channels), followed by pointwise tanh⁡\tanh non-linearities and 2×22 \times 2 spatial max-pooling.
    2. Stage 2: A bank of 64 filters of size 7×77 \times 7, where each feature map is generated from a combination of 8 randomly chosen maps from Stage 1, followed by pointwise tanh⁡\tanh and 2×22 \times 2 spatial max-pooling.
    3. Stage 3: A bank of 256 filters of size 7×77 \times 7, where each map combines 32 randomly selected feature maps from Stage 2, with no subsequent pooling.

    The network has an effective receptive field of 46×4646 \times 46 pixels at each scale, corresponding to contextual windows of 46×4646 \times 46 at full resolution, 92×9292 \times 92 at half resolution, and 184×184184 \times 184 at quarter resolution.

    The resulting feature maps from coarser scales s>1s > 1 are spatially upsampled via function u(⋅)u(\cdot) to match the resolution of the finest scale. Concatenating these maps produces a dense per-pixel feature map FF with feature dimensionality 256×3=768256 \times 3 = 768: F=[f1(X1;θ0),u(f2(X2;θ0)),…,u(fN(XN;θ0))]F = [f_1(X_1; \theta_0), u(f_2(X_2; \theta_0)), \dots, u(f_N(X_N; \theta_0))]

  2. Knowl 2 — Shape-Invariant Spatial Grid Feature Aggregation for Arbitrary Segments

    model/method

    To describe an arbitrary image segment or component Ck⊆{1,…,H}×{1,…,W}C_k \subseteq \{1, \dots, H\} \times \{1, \dots, W\} using the dense multiscale feature map F∈RH×W×DF \in \mathbb{R}^{H \times W \times D} (where D=768D=768), a shape-invariant attention and pooling operator a(Ck,F)a(C_k, F) is applied.

    The attention function isolates the feature vectors inside component CkC_k, setting the exterior background to zero (background suppression). An elastic max-pooling operation resamples the bounding box of CkC_k onto a fixed G×GG \times G spatial grid (with G=3G=3). For each grid cell (r,c)∈{1,…,G}×{1,…,G}(r, c) \in \{1, \dots, G\} \times \{1, \dots, G\}, the aggregated feature vector is computed by taking the component-wise maximum across all pixel feature vectors falling within the intersection of CkC_k and cell (r,c)(r,c): Ok,(r,c)=max⁡i∈Ck∩cell(r,c)FiO_{k,(r,c)} = \max_{i \in C_k \cap \text{cell}(r,c)} F_i

    Concatenating the aggregated vectors across all G×G=9G \times G = 9 grid cells yields a fixed-size descriptor Ok∈R9×768=R6912O_k \in \mathbb{R}^{9 \times 768} = \mathbb{R}^{6912}. This representation encodes the spatial arrangements and dominant parts of the segment in a scale- and shape-invariant manner while being robust to elongated or irregular component boundaries.

  3. Knowl 3 — Optimal Purity Cover Algorithm for Multilevel Scene Parsing

    algorithm

    Given a collection of candidate image segments T={Ck}k=1K\mathcal{T} = \{C_k\}_{k=1}^K (organized as a hierarchical segmentation tree or as multi-threshold superpixels) and predicted impurity costs Sk∈RS_k \in \mathbb{R} for each segment, the optimal purity cover identifies the observation level that best explains each pixel ii by minimizing the impurity cost: k∗(i)=arg⁡min⁡k∣i∈CkSkk^*(i) = \arg\min_{k \mid i \in C_k} S_k

    The final label assigned to pixel ii is the predicted dominant category of its optimal segment: li=arg⁡max⁡a∈Cd^k∗(i),al_i = \arg\max_{a \in \mathcal{C}} \hat{d}_{k^*(i), a}, where d^k,a\hat{d}_{k, a} is the predicted probability of class aa in segment CkC_k and C\mathcal{C} is the set of semantic classes.

    Input: Set of image pixels VV, family of candidate segments T={Ck}k=1K\mathcal{T} = \{C_k\}_{k=1}^K, predicted class distributions {d^k}k=1K\{\hat{d}_k\}_{k=1}^K, predicted impurity costs {Sk}k=1K\{S_k\}_{k=1}^K
    Output: Pixel-level semantic labeling l={li}i∈Vl = \{l_i\}_{i \in V}
    if T\mathcal{T} is a hierarchical segmentation tree TT then
        for each leaf node i∈Vi \in V do
            Pathi←Path_i \leftarrow sequence of tree nodes from leaf ii to the root of TT
            k∗(i)←arg⁡min⁡k∈PathiSkk^*(i) \leftarrow \arg\min_{k \in Path_i} S_k
            li←arg⁡max⁡a∈Cd^k∗(i),al_i \leftarrow \arg\max_{a \in \mathcal{C}} \hat{d}_{k^*(i), a}
        end for
    else
        for each pixel i∈Vi \in V do
            CoveringNodesi←{k∈{1,…,K}∣i∈Ck}CoveringNodes_i \leftarrow \{k \in \{1, \dots, K\} \mid i \in C_k\}
            k∗(i)←arg⁡min⁡k∈CoveringNodesiSkk^*(i) \leftarrow \arg\min_{k \in CoveringNodes_i} S_k
            li←arg⁡max⁡a∈Cd^k∗(i),al_i \leftarrow \arg\max_{a \in \mathcal{C}} \hat{d}_{k^*(i), a}
        end for
    end if
    return ll

    For a segmentation tree with MM nodes, finding the minimal-cost node along all leaf-to-root paths can be performed via a single depth-first search in O(M)\mathcal{O}(M) time. For a general family of LL overlapping segmentation layers (such as multi-threshold superpixels), the complexity is O(L⋅∣V∣)\mathcal{O}(L \cdot |V|).

  4. Knowl 4 — Segment Class Distribution and Impurity Cost Prediction

    model/method

    To evaluate the semantic content and quality of arbitrary segmentation components CkC_k, a classifier c(Ok;θc)c(O_k; \theta_c) predicts the class probability histogram present within the segment from its spatial grid descriptor Ok∈R6912O_k \in \mathbb{R}^{6912}.

    The classifier is implemented as a two-layer neural network: yk=W2tanh⁡(W1Ok+b1)y_k = W_2 \tanh(W_1 O_k + b_1) d^k,a=exp⁡(yk,a)∑b∈Cexp⁡(yk,b)\hat{d}_{k,a} = \frac{\exp(y_{k,a})}{\sum_{b \in \mathcal{C}} \exp(y_{k,b})} where W1∈R1024×6912W_1 \in \mathbb{R}^{1024 \times 6912}, b1∈R1024b_1 \in \mathbb{R}^{1024}, W2∈R∣C∣×1024W_2 \in \mathbb{R}^{|\mathcal{C}| \times 1024}, and C\mathcal{C} is the set of semantic categories.

    The network is trained on candidate segment collections computed over training images by minimizing the Kullback-Leibler (KL) divergence between predicted distribution d^k\hat{d}_k and ground truth class histogram dkd_k (the normalized proportions of pixel labels within CkC_k) with L2L_2 weight regularization (coefficient 10−210^{-2}): ldiv=∑a∈Cd^k,aln⁡(d^k,adk,a)l_{\text{div}} = \sum_{a \in \mathcal{C}} \hat{d}_{k,a} \ln\left(\frac{\hat{d}_{k,a}}{d_{k,a}}\right)

    The impurity cost SkS_k, representing the degree to which a segment contains mixed semantic categories, is defined as the cross-entropy of the class distribution: Sk=−∑a∈Cdk,aln⁡(d^k,a)S_k = -\sum_{a \in \mathcal{C}} d_{k,a} \ln(\hat{d}_{k,a}) Segments covering a single pure category achieve minimal entropy score SkS_k.

  5. Knowl 5 — Superpixel-Based Scene Labeling Strategies

    model/method

    Two post-processing schemes enforce spatial boundary alignment on top of dense multiscale feature vectors Fi∈R768F_i \in \mathbb{R}^{768}:

    1. Superpixel Probability Averaging: Pixelwise class probabilities d^i,a\hat{d}_{i,a} are estimated using a two-layer neural network with 768 input units, 1024 hidden tanh units, and ∣C∣|\mathcal{C}| softmax outputs: yi=W2tanh⁡(W1Fi+b1)y_i = W_2 \tanh(W_1 F_i + b_1). Predictions are averaged across each superpixel component kk with surface area s(k)s(k): d^k,a=1s(k)∑i∈kd^i,a\hat{d}_{k,a} = \frac{1}{s(k)} \sum_{i \in k} \hat{d}_{i,a} The entire superpixel kk is assigned the label lk=arg⁡max⁡a∈Cd^k,al_k = \arg\max_{a \in \mathcal{C}} \hat{d}_{k,a}.

    2. Conditional Random Field (CRF) over Superpixels/Pixels: A graph G=(V,E)G = (V, E) is constructed over nodes i∈Vi \in V with pairwise edges eij∈Ee_{ij} \in E, minimizing the energy: E(l)=∑i∈VΦ(d^i,li)+γ∑eij∈EΨ(li,lj)E(l) = \sum_{i \in V} \Phi(\hat{d}_i, l_i) + \gamma \sum_{e_{ij} \in E} \Psi(l_i, l_j) where unary terms penalize discordance with pixelwise predictions: Φ(d^i,a,li)=exp⁡(−αd^i,a)1(li≠a)\Phi(\hat{d}_{i,a}, l_i) = \exp(-\alpha \hat{d}_{i,a}) \mathbf{1}(l_i \neq a) and pairwise terms penalize adjacent label disagreements scaled by image gradient magnitude ∥∇Ii∥2\|\nabla I_i\|_2: Ψ(li,lj)=exp⁡(−β∥∇Ii∥2)1(li≠lj)\Psi(l_i, l_j) = \exp(-\beta \|\nabla I_i\|_2) \mathbf{1}(l_i \neq l_j) The energy is minimized using α\alpha-expansion graph cuts (with hyperparameters α=0.1\alpha = 0.1, β=20\beta = 20, γ=200\gamma = 200).

  6. Knowl 6 — Two-Stage Supervised Training Procedure for Dense Scene Labeling

    algorithm

    The multiscale scene parsing system is trained in two decoupled stages:

    Input: Training images {Im}\{I_m\}, ground truth pixel labelings {cm}\{c_m\}, scale count N=3N=3, learning rate η=10−3\eta = 10^{-3}, weight decay λ=10−5\lambda = 10^{-5}
    Output: Trained multiscale feature parameters θ0\theta_0, trained segment classifier parameters θc\theta_c
    // Stage 1: Multiscale Feature Learning
    Initialize shared ConvNet parameters θ0\theta_0 and temporary linear classifier weights ww
    while not converged (10 to 50 million patches sampled with jitter augmentation) do
        Sample image ImI_m and apply random horizontal flip, rotation ∈[−8∘,8∘]\in [-8^\circ, 8^\circ], and scale factor ∈[0.9,1.1]\in [0.9, 1.1]
        Compute normalized Laplacian pyramid X1,X2,X3X_1, X_2, X_3
        Compute feature maps fs(Xs;θ0)f_s(X_s; \theta_0) and concatenate upsampled maps to form FF
        Compute per-pixel class predictions c^i,a=exp⁡(waTFi)∑bexp⁡(wbTFi)\hat{c}_{i,a} = \frac{\exp(w_a^T F_i)}{\sum_b \exp(w_b^T F_i)}
        Compute cross-entropy loss Lcat=−∑i∑aci,aln⁡(c^i,a)L_{\text{cat}} = -\sum_{i} \sum_{a} c_{i,a} \ln(\hat{c}_{i,a})
        Update θ0\theta_0 and ww via SGD with batch size 1, learning rate η\eta, and L2L_2 decay λ\lambda
    end while
    Discard linear classifier weights ww
    // Stage 2: Segment Classifier Training
    Extract feature maps FF for all training images using frozen parameters θ0\theta_0
    Generate segmentation families T={Ck}\mathcal{T} = \{C_k\} for all training images
    For each component CkC_k, compute descriptor Ok∈R6912O_k \in \mathbb{R}^{6912} via spatial grid pooling a(Ck,F)a(C_k, F)
    Compute true class distribution histograms dkd_k for each component CkC_k
    Train 2-layer classifier c(Ok;θc)c(O_k; \theta_c) to minimize KL-divergence ldiv=∑ad^k,aln⁡(d^k,a/dk,a)l_{\text{div}} = \sum_a \hat{d}_{k,a} \ln(\hat{d}_{k,a} / d_{k,a}) with L2L_2 regularization coefficient 10−210^{-2}
    return θ0,θc\theta_0, \theta_c

  7. Knowl 7 — Stanford Background Dataset Parsing Performance

    data/table

    The multiscale convolutional network and its post-processing variants were evaluated on the Stanford Background dataset (715 outdoor scene images of size ≈320×240\approx 320 \times 240 pixels with 8 classes, evaluated via 5-fold cross-validation with 572 train and 143 test images per fold).

    Method Pixel Acc. Class Acc. Compute Time (sec.)
    Gould et al. 2009 76.4% - 10 to 600s
    Munoz et al. 2010 76.9% 66.2% 12s
    Tighe et al. 2010 77.5% - 10 to 300s
    Socher et al. 2011 78.1% - ?
    Kumar et al. 2010 79.4% - < 600s
    Lempitzky et al. 2011 81.9% 72.4% > 60s
    singlescale convnet 66.0% 56.5% 0.35s
    multiscale convnet 78.8% 72.4% 0.6s
    multiscale net + superpixels 80.4% 74.56% 0.7s
    multiscale net + gPb + cover 80.4% 75.24% 61s
    multiscale net + CRF on gPb 81.4% 76.0% 60.5s

    The multiscale ConvNet without any segmentation post-processing outperforms the single-scale network by 12.812.8 percentage points in pixel accuracy and 15.915.9 percentage points in per-class accuracy, demonstrating the critical importance of large context (184×184184 \times 184). Combining the multiscale features with simple superpixel averaging yields 80.4%80.4\% pixel accuracy in 0.70.7 seconds on a 4-core Intel i7, outperforming prior complex models while running orders of magnitude faster. Incorporating a CRF on gPb contours achieves 81.4%81.4\% pixel accuracy and state-of-the-art 76.0%76.0\% class accuracy.

  8. Knowl 8 — Scene Parsing Benchmarks on SIFT Flow and Barcelona Datasets

    data/table

    The multiscale convolutional system was benchmarked on larger-scale datasets: SIFT Flow (2,688 images across 33 semantic classes; 2,488 training and 200 test) and Barcelona (14,871 training images and 279 test images of Barcelona street scenes across 170 semantic classes).

    SIFT Flow Method Pixel Acc. Class Acc.
    Liu et al. 2009 74.75% -
    Tighe et al. 2010 76.9% 29.4%
    raw multiscale net (balanced frequencies) 67.9% 45.9%
    multiscale net + superpixels (balanced frequencies) 71.9% 50.8%
    multiscale net + cover (balanced frequencies) 72.3% 50.8%
    multiscale net + cover (natural frequencies) 78.5% 29.6%
    Barcelona Method Pixel Acc. Class Acc.
    Tighe et al. 2010 66.9% 7.6%
    raw multiscale net (balanced frequencies) 37.8% 12.1%
    multiscale net + superpixels (balanced frequencies) 44.1% 12.4%
    multiscale net + cover (balanced frequencies) 46.4% 12.5%
    multiscale net + cover (natural frequencies) 67.8% 9.5%

    On SIFT Flow, training with natural class frequencies combined with a 10-level Felzenszwalb superpixel optimal cover achieves 78.5%78.5\% pixel accuracy, establishing a new record. When trained with balanced frequencies, the system achieves 50.8%50.8\% average per-class accuracy, substantially exceeding Tighe et al.'s 29.4%29.4\%. On Barcelona (170 classes), the optimal cover achieves 67.8%67.8\% pixel accuracy (natural frequencies) and 12.5%12.5\% class accuracy (balanced frequencies), outperforming existing methods in both metrics.

  9. Knowl 9 — Impact of Class Frequency Balancing on Semantic Recognition versus Pixel Accuracy

    empirical result

    Training the multiscale convolutional feature extractor using balanced class sampling (where each class is presented with equal frequency during SGD) versus natural frequency sampling creates a fundamental trade-off between semantic recognition of small/rare objects and aggregate pixel accuracy:

    1. Rare Class Discrimination: Balancing class occurrences trains the network to model the conditional likelihood P(Fi∣a)P(F_i \mid a) rather than the prior-dominated posterior, significantly boosting average per-class accuracy (e.g., improving SIFT Flow per-class accuracy from 29.6%29.6\% to 50.8%50.8\%, and Barcelona per-class accuracy from 9.5%9.5\% to 12.5%12.5\%).
    2. Pixel-Wise Accuracy Penalty: Because common background classes (e.g., sky, road, buildings) comprise the vast majority of image pixels, ignoring natural priors reduces total pixel-wise accuracy (e.g., decreasing SIFT Flow pixel accuracy from 78.5%78.5\% to 72.3%72.3\%, and Barcelona pixel accuracy from 67.8%67.8\% to 46.4%46.4\%).
    3. Overfitting on Sparse Datasets: In datasets with severe class imbalance and few examples per rare class (such as the 170-class Barcelona dataset), frequency balancing leads to rapid overfitting on underrepresented categories.

  10. Knowl 10 — Diminishing Utility of Complex Graphical Models with Wide Multiscale Context

    empirical result

    Conventional scene parsing pipelines rely heavily on complex inference over conditional random fields (CRFs) or Markov random fields (MRFs) to enforce spatial consistency and prevent semantic aberrations (e.g., sky appearing inside buildings or persons floating in sky).

    However, incorporating wide contextual receptive fields (184×184184 \times 184 pixels) directly into a feed-forward multiscale convolutional network causes the learned representations to already internalize long-range scene co-occurrence and spatial relationships. As a result:

    1. Standalone feedforward multiscale predictions reach competitive accuracies (78.8%78.8\% on Stanford Background) without any post-processing.
    2. Simple majority voting within superpixels improves accuracy to 80.4%80.4\% while adding only 0.10.1s of computation.
    3. Complex CRF inference on gPb contours provides only a minor accuracy gain (81.4%81.4\% vs 80.4%80.4\%) while increasing computational latency by roughly two orders of magnitude (60.560.5s vs 0.70.7s).

    This demonstrates that feed-forward convolutional architectures with large multi-scale windows largely subsume the contextual consistency role traditionally played by graphical model inference.

Coverage note — None omitted. All primary contributions—including the multiscale ConvNet architecture, shape-invariant spatial grid feature pooling, optimal purity cover algorithm, CRF and superpixel post-processing schemes, training procedures, class balancing trade-offs, and empirical benchmarks across Stanford Background, SIFT Flow, and Barcelona datasets—are fully represented.

References

  1. 1.P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 33(5):898–916, 2011.
  2. 2.Y. Boykov and M. P. Jolly. Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. In Proceedings of International Conference of Computer Vision (ICCV), volume 1, pages 105–112, 2001.
  3. 3.Y. Boykov and V. Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell., 26(9):1124–1137, 2004.
  4. 4.Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell., 23(11):1222–1239, 2001.
  5. 5.J. Carreira and C. Sminchisescu. CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012.
  6. 6.D. Ciresan, U. Meier, J. Masci, and J. Schmidhuber. A committee of neural networks for traffic sign classification. In Neural Networks (IJCNN), The 2011 International Joint Conference on, pages 1918–1921. IEEE, 2011.
  7. 7.C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Scene parsing with multiscale feature learning, purity trees, and optimal covers. In Proceedings of the International Conference on Machine Learning (ICML), June 2012.
  8. 8.C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Scene parsing with multiscale feature learning, purity trees, and optimal covers. CoRR, abs/1202.2160, February 2012.
  9. 9.C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello. Hardware accelerated convolutional neural networks for synthetic vision systems. In International Symposium on Circuits and Systems (ISCAS’10), Paris, May 2010. IEEE.
  10. 10.C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, and Y. LeCun. Neuflow: A runtime reconfigurable dataflow processor for vision. In Proceedings of the Fifth IEEE Workshop on Embedded Computer Vision. IEEE, 2011.
  11. 11.P. Felzenszwalb and D. Huttenlocher. Efficient graph-based image segmentation. International Journal of Computer Vision, 59:167–181, 2004.
  12. 12.L. R. Ford and D. R. Fulkerson. A simple algorithm for finding maximal network flows and an application to the hitchcock problem. Technical report, RAND Corp., Santa Monica, 1955.
  13. 13.B. Fulkerson, A. Vedaldi, and S. Soatto. Class segmentation and object localization with superpixel neighborhoods. In ICCV, pages 670–677. IEEE, 2009.
  14. 14.C. Garcia and M. Delakis. Convolutional face finder: A neural architecture for fast and robust face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004.
  15. 15.S. Gould, R. Fulton, and D. Koller. Decomposing a scene into geometric and semantically consistent regions. IEEE International Conference on Computer Vision, pages 1–8, Sept. 2009.
  16. 16.S. Gould, J. Rodgers, D. Cohen, G. Elidan, and D. Koller. Multi-class segmentation with relative location prior. Int. J. Comput. Vision, 80(3):300–316, Dec. 2008.
  17. 17.D. Grangier, L. Bottou, and R. Collobert. Deep Convolutional Networks for Scene Parsing. In ICML 2009 Deep Learning Workshop, 2009.
  18. 18.R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In Proc. Computer Vision and Pattern Recognition Conference (CVPR’06). IEEE Press, 2006.
  19. 19.X. He and R. Zemel. Learning hybrid models for image annotation with partially labeled data. Advances in Neural Information Processing Systems, 2008.
  20. 20.V. Jain, J. F. Murray, F. Roth, S. Turaga, V. Zhigulin, K. Briggman, M. Helmstaedter, W. Denk, and S. H. Seung. Supervised learning of image restoration with convolutional networks. In ICCV, 2007.
  21. 21.K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In Proc. International Conference on Computer Vision (ICCV’09). IEEE, 2009.
  22. 22.K. Kavukcuoglu, M. Ranzato, R. Fergus, and Y. LeCun. Learning invariant features through topographic filter maps. In Proc. International Conference on Computer Vision and Pattern Recognition. IEEE, 2009.
  23. 23.K. Kavukcuoglu, M. Ranzato, and Y. LeCun. Fast inference in sparse coding algorithms with applications to object recognition. Technical report, Courant Institute of Mathematical Sciences, New York University, 2008. Tech Report CBLL-TR-2008-12-01.
  24. 24.K. Kavukcuoglu, P. Sermanet, Y. Boureau, K. Gregor, M. Mathieu, and Y. LeCun. Learning convolutional feature hierachies for visual recognition. In Advances in Neural Information Processing Systems (NIPS 2010), volume 23, 2010.
  25. 25.M. Kumar and D. Koller. Efficiently selecting regions for scene understanding. In Computer Vision and Pattern Recognition (CVPR), pages 3217–3224. IEEE, 2010.
  26. 26.Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Handwritten digit recognition with a back-propagation network. In NIPS’89, 1990.
  27. 27.Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998.
  28. 28.Y. LeCun, L. Bottou, G. Orr, and K. Muller. Efficient backprop. In G. Orr and M. K., editors, Neural Networks: Tricks of the trade. Springer, 1998.
  29. 29.H. Lee, R. Grosse, R. Ranganath, and Y. Ng, Andrew. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proc. of International Conference on Machine Learning (ICML’09), 2009.
  30. 30.V. Lempitsky, A. Vedaldi, and A. Zisserman. A pylon model for semantic segmentation. In Advances in Neural Information Processing Systems, 2011.
  31. 31.C. Liu, J. Yuen, and A. Torralba. Nonparametric scene parsing: Label transfer via dense scene alignment. Artificial Intelligence, 2009.
  32. 32.D. Munoz, J. Bagnell, and M. Hebert. Stacked hierarchical labeling. ECCV 2010, Jan 2010.
  33. 33.L. Najman and M. Schmitt. Geodesic saliency of watershed contours and hierarchical segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 18(12):1163–1173, December 1996.
  34. 34.F. Ning, D. Delhomme, Y. LeCun, F. Piano, L. Bottou, and P. Barbano. Toward automatic phenotyping of developing embryos from videos. IEEE Trans. on Image Processing, 2005. Special issue on Molecular & Cellular Bioimaging.
  35. 35.M. Osadchy, Y. LeCun, and M. Miller. Synergistic face detection and pose estimation with energy-based models. Journal of Machine Learning Research, 8:1197–1215, 2007.
  36. 36.C. Pantofaru, C. Schmid, and M. Hebert. Object recognition by integrating multiple image segmentations. In ECCV 2008, 10th European Conference on Computer Vision, Marseille, France, pages 481–494, 2008.
  37. 37.M. Ranzato, F. Huang, Y. Boureau, and Y. LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In Proc. of Computer Vision and Pattern Recognition. IEEE, 2007.
  38. 38.B. Russell, A. Torralba, C. Liu, R. Fergus, and W. Freeman. Object recognition by scene alignment. In Neural Advances in Neural Information, 2007.
  39. 39.C. Russell, P. H. S. Torr, and P. Kohli. Associative hierarchical CRFs for object class image segmentation. In Proc. ICCV, 2009.
  40. 40.H. Schulz and S. Behnke. Learning object-class segmentation with convolutional neural networks. In 11th European Symposium on Artificial Neural Networks (ESANN), 2012.
  41. 41.J. Shotton, J. M. Winn, C. Rother, and A. Criminisi. TextonBoost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In A. Leonardis, H. Bischof, and A. Pinz, editors, ECCV (1), volume 3951 of Lecture Notes in Computer Science, pages 1–15. Springer, 2006.
  42. 42.P. Simard, D. Steinkraus, and J. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, volume 2, pages 958–962, 2003.
  43. 43.R. Socher, C. C. Lin, A. Y. Ng, and C. D. Manning. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In Proceedings of the 26th International Conference on Machine Learning (ICML), 2011.
  44. 44.J. Tighe and S. Lazebnik. Superparsing: scalable nonparametric image parsing with superpixels. ECCV, pages 352–365, 2010.
  45. 45.A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR, pages 1521–1528. IEEE, 2011.
  46. 46.S. Turaga, K. Briggman, M. Helmstaedter, W. Denk, and H. Seung. Maximin affinity learning of image segmentation. NIPS, Jan 2009.
  47. 47.R. Vaillant, C. Monrocq, and Y. LeCun. Original approach for the localisation of objects in images. IEE Proc on Vision, Image, and Signal Processing, 141(4):245–250, August 1994.

Citation

MLA
Farabet, C., et al. “Learning Hierarchical Features for Scene Labeling”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, 2013, pp. 1915–29, https://doi.org/10.1109/TPAMI.2012.231.
APA
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013). Learning Hierarchical Features for Scene Labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1915–1929. https://doi.org/10.1109/TPAMI.2012.231
Chicago
Farabet, C., C. Couprie, L. Najman, and Y. LeCun. 2013. “Learning Hierarchical Features for Scene Labeling”. IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (8): 1915–29. https://doi.org/10.1109/TPAMI.2012.231.
Harvard
Farabet, C. et al. (2013) “Learning Hierarchical Features for Scene Labeling”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), pp. 1915–1929. Available at: https://doi.org/10.1109/TPAMI.2012.231.
Vancouver
1. Farabet C, Couprie C, Najman L, LeCun Y (2013) Learning Hierarchical Features for Scene Labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 35:1915–1929

BibTeX

@article{Farabet_2013, title={Learning Hierarchical Features for Scene Labeling}, volume={35}, ISSN={2160-9292}, url={http://dx.doi.org/10.1109/TPAMI.2012.231}, DOI={10.1109/tpami.2012.231}, number={8}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, publisher={Institute of Electrical and Electronics Engineers (IEEE)}, author={Farabet, Clement and Couprie, Camille and Najman, Laurent and LeCun, Yann}, year={2013}, month=Aug, pages={1915–1929} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF