Back

Per-Pixel Classification is Not All You Need for Semantic Segmentation

Bowen ChengAlexander G. SchwingAlexander Kirillov

article2021NeurIPS2,168 citations

Proposes MaskFormer, a unified mask classification framework that replaces standard per-pixel approaches to achieve state-of-the-art accuracy across both semantic and panoptic segmentation benchmarks.

Listen

Modern computer vision divides image segmentation into two distinct paradigms: semantic segmentation, which assigns a category label to each pixel, and instance-level or panoptic segmentation, which detects individual object regions using mask classification. This division forces engineering and research teams to maintain separate architectures, loss functions, and training pipelines for tasks that share fundamental visual goals. As real-world computer vision systems expand to large-scale vocabularies with hundreds or thousands of categories, standard per-pixel classification models face significant computational overhead and performance degradation.

The article demonstrates that mask classification is a sufficiently general paradigm to solve both semantic- and instance-level segmentation using a single unified model architecture, loss function, and training pipeline. It introduces MaskFormer, a framework that converts standard segmentation backbones into mask classification systems by predicting a set of binary masks alongside a single global category probability for each mask.

To evaluate this approach, the authors conducted comprehensive experiments across five semantic segmentation benchmarks—ADE20K, COCO-Stuff-10K, Cityscapes, Mapillary Vistas, and the 847-class ADE20K-Full—as well as standard panoptic segmentation benchmarks including COCO and ADE20K Panoptic. The method pairs convolutional or vision transformer backbones with a lightweight pixel decoder and a transformer decoder, using bipartite matching and combined focal and dice losses to optimize predictions directly without requiring bounding box supervision.

The findings show that MaskFormer consistently outperforms or matches established per-pixel baselines while reducing computational complexity. Key results demonstrate that the performance advantage grows substantially as the number of categories increases: on ADE20K-Full (847 classes), MaskFormer exceeds the per-pixel baseline by 3.5 mean Intersection-over-Union (mIoU), while matching baseline performance on smaller vocabularies like Cityscapes (19 classes). When equipped with advanced vision transformer backbones, MaskFormer sets state-of-the-art results on ADE20K semantic segmentation (55.6 mIoU) and COCO panoptic segmentation (52.7 Panoptic Quality), outperforming prior specialized models while reducing parameters by 10% and computation by approximately 40%.

These results indicate that modern vision systems do not need fragmented architectures to handle different segmentation tasks. Transitioning to mask classification reduces development maintenance costs, optimizes hardware memory during large-vocabulary training, and delivers superior fine-grained region recognition. MaskFormer eliminates the need for complex auxiliary loss designs and bounding-box intermediate stages that previously complicated panoptic models.

Organizations developing image understanding pipelines should consider adopting unified mask classification architectures, particularly for complex environments with large category sets. Future engineering efforts should explore improved pixel decoder designs to further enhance boundary precision on datasets with fewer categories, where per-pixel mask alignment remains the primary challenge. Confidence in these conclusions is high across diverse benchmarks, though practitioners should note that high-resolution edge quality on limited-vocabulary datasets may require careful decoder selection.

MetaUniversity of Illinois Urbana-Champaignclassification modelssemanticsclass labelsSegmentationClassi CationMaskFormer
arXiv: 2107.06278facebookresearch/MaskFormer
  • Paper: Segment Anything, Alexander M. Kirillov et al. (2023). This paper naturally extends the source by scaling universal mask classification into a promptable foundation model capable of zero-shot segmentation across arbitrary inputs.
  • Paper: SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers, Enze Xie et al. (2021). This paper builds directly on the source's push for unified mask-based paradigms by introducing a transformer-driven segmentation design optimized for efficiency and performance.
  • Paper: Masked Autoencoders Are Scalable Vision Learners, Kaiming He et al. (2022). This work follows the source by demonstrating how self-supervised vision learners can further enhance downstream transfer learning for complex mask classification tasks.
Cover for Per-Pixel Classification is Not All You Need for Semantic Segmentation

Abstract

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

Topics

classification modelssemanticsclass labels

Keywords

SegmentationClassi CationMaskFormerSemantic SegmentationMaskMask Classi CationPer-PixelPer-Pixel Classi CationPanoptic SegmentationModel

Table of Contents

  • 1 Introduction
  • 2 Related Works
  • 3 From Per-Pixel to Mask Classification
  • 3.1 Per-pixel classification formulation
  • 3.2 Mask classification formulation
  • 3.3 MaskFormer
  • 3.4 Mask-classification inference
  • 4 Experiments
  • 4.1 Implementation details
  • 4.2 Training settings
  • 4.3 Main results
  • 4.4 Ablation studies
  • 5 Discussion
  • 6 Conclusion
  • A Datasets description
  • A.1 Semantic segmentation datasets
  • A.2 Panoptic segmentation datasets
  • B Semantic segmentation results
  • C Panoptic segmentation results
  • D Additional ablation studies
  • E Visualization
  • References

Knowls

  1. Knowl 1 — MaskFormer Architecture for Mask Classification

    model/method

    MaskFormer is a neural network architecture that formulates semantic, instance, and panoptic segmentation tasks as a unified set-based mask classification problem. The model consists of three core components:

    1. Pixel-Level Module: Takes an image of dimension H×W×3H \times W \times 3. A convolutional or vision transformer backbone extracts image feature representations F∈RCF×HS×WS\mathcal{F} \in \mathbb{R}^{C_\mathcal{F} \times \frac{H}{S} \times \frac{W}{S}} (with stride S=32S=32). A lightweight Feature Pyramid Network (FPN)-style pixel decoder progressively upsamples and fuses multi-scale features to produce dense per-pixel embeddings Epixel∈RCE×H×W\mathcal{E}_{pixel} \in \mathbb{R}^{C_\mathcal{E} \times H \times W} (where CE=256C_\mathcal{E} = 256).

    2. Transformer Module: A standard Transformer decoder with LL layers (default L=6L=6) takes NN learnable positional query embeddings (default N=100N=100) and the backbone image features F\mathcal{F} to compute NN per-segment embedding vectors Q∈RCQ×N\mathcal{Q} \in \mathbb{R}^{C_Q \times N} in parallel.

    3. Segmentation Module:

    • Classification Head: A linear classifier followed by a softmax activation maps each segment embedding in Q\mathcal{Q} to a probability distribution pi∈ΔK+1p_i \in \Delta^{K+1} across KK semantic categories plus an auxiliary "no object" label (∅\emptyset).
    • Mask Embedding Head: A Multi-Layer Perceptron (MLP) with 2 hidden layers of 256 channels converts Q\mathcal{Q} into NN mask embedding vectors Emask∈RCE×N\mathcal{E}_{mask} \in \mathbb{R}^{C_\mathcal{E} \times N}.
    • Binary Mask Prediction: Binary masks mi∈[0,1]H×Wm_i \in [0, 1]^{H \times W} are computed via a dot product between mask embeddings and per-pixel embeddings followed by a sigmoid activation: mi[h,w]=sigmoid(Emask[:,i]T⋅Epixel[:,h,w])m_i[h, w] = \text{sigmoid}\left( \mathcal{E}_{mask}[:, i]^T \cdot \mathcal{E}_{pixel}[:, h, w] \right) Masks are not constrained to be mutually exclusive across queries via softmax, enabling overlapping segment predictions.

  2. Knowl 2 — Loss Function and Bipartite Matching for Mask Classification

    equation

    MaskFormer supervises its set of NN predicted pairs z={(pi,mi)}i=1Nz = \{(p_i, m_i)\}_{i=1}^N (where pi∈ΔK+1p_i \in \Delta^{K+1} is a class distribution and mi∈[0,1]H×Wm_i \in [0, 1]^{H \times W} is a binary mask) against a set of NgtN_{gt} ground-truth segments zgt={(cjgt,mjgt)}j=1Ngtz^{gt} = \{(c_j^{gt}, m_j^{gt})\}_{j=1}^{N_{gt}} (where cjgt∈{1,…,K}c_j^{gt} \in \{1, \dots, K\} and mjgt∈{0,1}H×Wm_j^{gt} \in \{0, 1\}^{H \times W}). The ground-truth set is padded with "no object" tokens ∅\emptyset to size NN assuming N≥NgtN \ge N_{gt}.

    A bipartite matching permutation σ∈SN\sigma \in \mathfrak{S}_N is computed using the Hungarian algorithm by minimizing the pairwise matching cost: σ=arg⁡min⁡π∈SN∑j=1NLmatch(zπ(j),zjgt)\sigma = \arg\min_{\pi \in \mathfrak{S}_N} \sum_{j=1}^N \mathcal{L}_{match}(z_{\pi(j)}, z_j^{gt}) where the matching cost between prediction ziz_i and ground-truth segment zjgtz_j^{gt} is defined directly over class and mask probabilities: Lmatch(zi,zjgt)=−pi(cjgt)+Lmask(mi,mjgt)\mathcal{L}_{match}(z_i, z_j^{gt}) = -p_i(c_j^{gt}) + \mathcal{L}_{mask}(m_i, m_j^{gt}) If cjgt=∅c_j^{gt} = \emptyset, the matching cost is −pi(∅)-p_i(\emptyset).

    Given the optimal assignment σ\sigma, the model is trained end-to-end with the mask classification loss: Lmask-cls(z,zgt)=∑j=1N[−log⁡pσ(j)(cjgt)+Icjgt≠∅Lmask(mσ(j),mjgt)]\mathcal{L}_{mask\text{-}cls}(z, z^{gt}) = \sum_{j=1}^N \left[ -\log p_{\sigma(j)}(c_j^{gt}) + \mathbb{I}_{c_j^{gt} \neq \emptyset} \mathcal{L}_{mask}(m_{\sigma(j)}, m_j^{gt}) \right] where I\mathbb{I} is the indicator function, the classification loss for the ∅\emptyset label is weighted by 0.10.1, and the binary mask loss Lmask\mathcal{L}_{mask} is a linear combination of focal loss and dice loss: Lmask(m,mgt)=λfocalLfocal(m,mgt)+λdiceLdice(m,mgt)\mathcal{L}_{mask}(m, m^{gt}) = \lambda_{focal}\mathcal{L}_{focal}(m, m^{gt}) + \lambda_{dice}\mathcal{L}_{dice}(m, m^{gt}) with hyperparameters λfocal=20.0\lambda_{focal} = 20.0 and λdice=1.0\lambda_{dice} = 1.0.

  3. Knowl 3 — Inference Strategies for Mask Classification Models

    model/method

    MaskFormer produces NN prediction pairs {(pi,mi)}i=1N\{(p_i, m_i)\}_{i=1}^N, where pi∈ΔK+1p_i \in \Delta^{K+1} is a probability distribution over KK categories plus ∅\emptyset, and mi∈[0,1]H×Wm_i \in [0, 1]^{H \times W} is a binary mask. MaskFormer utilizes two inference strategies depending on the task and evaluation metric:

    1. General Inference (Panoptic and Instance Segmentation): Assigns each pixel location [h,w][h, w] to the prediction pair index i∗i^* that maximizes the joint product of predicted class probability and mask probability: i∗=arg⁡max⁡i:ci≠∅pi(ci)⋅mi[h,w],where ci=arg⁡max⁡c∈{1,…,K,∅}pi(c)i^* = \arg\max_{i: c_i \neq \emptyset} p_i(c_i) \cdot m_i[h, w], \quad \text{where } c_i = \arg\max_{c \in \{1, \dots, K, \emptyset\}} p_i(c) Pixels assigned to the same pair ii form an individual segment labeled with class cic_i. In panoptic segmentation, false positives are suppressed by filtering out predictions with pi(ci)<0.8p_i(c_i) < 0.8 and removing segments where over 80%80\% of their binary mask area (thresholded at mi>0.5m_i > 0.5) is occluded by other predictions.

    2. Semantic Inference (Marginalization for Semantic Segmentation): Directly computes per-pixel class probabilities by marginalizing over all NN probability-mask pairs via matrix multiplication without hard pixel-to-query assignment: c∗[h,w]=arg⁡max⁡c∈{1,…,K}∑i=1Npi(c)⋅mi[h,w]c^*[h, w] = \arg\max_{c \in \{1, \dots, K\}} \sum_{i=1}^N p_i(c) \cdot m_i[h, w] The "no object" label ∅\emptyset is excluded from the arg⁡max⁡\arg\max. This marginalization blends mask predictions across queries and improves mean Intersection-over-Union (mIoU) and mask quality (SQStSQ^{St}) compared to hard assignment.

  4. Knowl 4 — ADE20K Semantic Segmentation Performance of MaskFormer

    empirical result

    MaskFormer sets state-of-the-art results for semantic segmentation on the ADE20K validation set across convolutional and vision transformer backbones while using fewer parameters and FLOPs than per-pixel classification models.

    Method Backbone Crop Size mIoU (s.s.) mIoU (m.s.) #Params FLOPs
    OCRNet R101c 520×520520 \times 520 - 45.3 - -
    DeepLabV3+ R50c 512×512512 \times 512 44.0 44.9 44M 177G
    DeepLabV3+ R101c 512×512512 \times 512 45.5 46.4 63M 255G
    MaskFormer R50 512×512512 \times 512 44.5±0.544.5 \pm 0.5 46.7±0.646.7 \pm 0.6 41M 53G
    MaskFormer R101 512×512512 \times 512 45.5±0.545.5 \pm 0.5 47.2±0.247.2 \pm 0.2 60M 73G
    MaskFormer R101c 512×512512 \times 512 46.0±0.146.0 \pm 0.1 48.1±0.248.1 \pm 0.2 60M 80G
    SETR ViT-L†^\dagger 512×512512 \times 512 - 50.3 308M -
    Swin-UperNet Swin-T 512×512512 \times 512 - 46.1 60M 236G
    Swin-UperNet Swin-S 512×512512 \times 512 - 49.3 81M 259G
    Swin-UperNet Swin-B†^\dagger 640×640640 \times 640 - 51.6 121M 471G
    Swin-UperNet Swin-L†^\dagger 640×640640 \times 640 - 53.5 234M 647G
    MaskFormer Swin-T 512×512512 \times 512 46.7±0.746.7 \pm 0.7 48.8±0.648.8 \pm 0.6 42M 55G
    MaskFormer Swin-S 512×512512 \times 512 49.8±0.449.8 \pm 0.4 51.0±0.451.0 \pm 0.4 63M 79G
    MaskFormer Swin-B 640×640640 \times 640 51.1±0.251.1 \pm 0.2 52.3±0.452.3 \pm 0.4 102M 195G
    MaskFormer Swin-B†^\dagger 640×640640 \times 640 52.7±0.452.7 \pm 0.4 53.9±0.253.9 \pm 0.2 102M 195G
    MaskFormer Swin-L†^\dagger 640×640640 \times 640 54.1±0.254.1 \pm 0.2 55.6±0.155.6 \pm 0.1 212M 375G

    Note: †\dagger marks backbones pre-trained on ImageNet-22K. (s.s.) and (m.s.) denote single-scale and multi-scale inference respectively.

    On the ADE20K test set, MaskFormer with Swin-L achieves 79.36%79.36\% pixel accuracy, 49.67 mIoU49.67\text{ mIoU}, and a test score of 64.5164.51, outperforming Swin-UperNet (47.07 mIoU47.07\text{ mIoU}, 62.7562.75 score) and SETR (45.03 mIoU45.03\text{ mIoU}, 61.6961.69 score).

  5. Knowl 5 — Panoptic Segmentation Performance on COCO and ADE20K

    empirical result

    MaskFormer solves panoptic segmentation using the identical model, loss, and training procedure used for semantic segmentation, outperforming bounding-box-based and complex auxiliary-loss-based models.

    On COCO panoptic validation (133 classes: 80 things, 53 stuff):

    Method Backbone PQ PQTh\text{PQ}^\text{Th} PQSt\text{PQ}^\text{St} SQ RQ #Params
    DETR R50 + 6 Enc 43.4 48.2 36.3 79.3 53.8 -
    MaskFormer (DETR post-proc) R50 + 6 Enc 45.6 50.0 39.0 80.2 55.8 -
    MaskFormer R50 + 6 Enc 46.5 51.0 39.8 80.4 56.8 45M
    DETR R101 + 6 Enc 45.1 50.5 37.0 79.9 55.5 -
    MaskFormer R101 + 6 Enc 47.6 52.5 40.3 80.7 58.0 64M
    Max-DeepLab Max-S 48.4 53.0 41.5 - - 62M
    Max-DeepLab Max-L 51.1 57.0 42.2 - - 451M
    MaskFormer Swin-T 47.7 51.7 41.7 80.4 58.3 42M
    MaskFormer Swin-S 49.7 54.4 42.6 80.9 60.4 63M
    MaskFormer Swin-B 51.1 56.3 43.2 81.4 61.8 102M
    MaskFormer Swin-B†^\dagger 51.8 56.9 44.1 81.4 62.6 102M
    MaskFormer Swin-L†^\dagger 52.7 58.5 44.0 81.8 63.5 212M

    Note: †\dagger marks ImageNet-22K pretraining.

    On the COCO panoptic test-dev benchmark, MaskFormer (Swin-L) attains 53.3 PQ53.3\text{ PQ} (59.1 PQTh,44.5 PQSt59.1\text{ PQ}^\text{Th}, 44.5\text{ PQ}^\text{St}), exceeding Max-DeepLab (51.3 PQ51.3\text{ PQ}). On the ADE20K panoptic validation set, MaskFormer achieves 34.7 PQ34.7\text{ PQ} with R50 and 35.7 PQ35.7\text{ PQ} with R101, surpassing Auto-Panoptic (32.4 PQ32.4\text{ PQ}) and BGRNet (31.8 PQ31.8\text{ PQ}).

  6. Knowl 6 — Performance Scaling of MaskFormer vs Per-Pixel Baselines with Vocabulary Size

    empirical result

    The performance gain of MaskFormer over per-pixel classification models scales directly with the number of semantic categories in the dataset.

    MaskFormer is compared with two ablation baselines:

    1. PerPixelBaseline: Uses the same pixel-level module (backbone + pixel decoder) to predict per-pixel class logits directly under cross-entropy loss.
    2. PerPixelBaseline+: Extends PerPixelBaseline with the Transformer decoder and mask embedding MLP to compute per-pixel logits via inner product EmaskTEpixel\mathcal{E}_{mask}^T \mathcal{E}_{pixel} supervised with per-pixel cross-entropy loss (differing from MaskFormer strictly in the loss and matching formulation).
    Cityscapes (19) ADE20K (150) COCO-Stuff (171) ADE20K-Full (847)
    Model mIoU PQSt\text{PQ}^\text{St} mIoU PQSt\text{PQ}^\text{St} mIoU PQSt\text{PQ}^\text{St} mIoU PQSt\text{PQ}^\text{St}
    PerPixelBaseline 77.4 58.9 39.2 21.6 32.4 15.5 12.4 5.8
    PerPixelBaseline+ 78.5 60.2 41.9 28.3 34.2 24.6 13.9 9.0
    MaskFormer 78.5 63.1 44.5 33.4 37.1 28.9 17.4 11.9
    Δ\Delta (vs Baseline+) +0.0 +2.9 +2.6 +5.1 +2.9 +4.3 +3.5 +2.9

    Note: ResNet-50 backbone is used for ADE20K, COCO-Stuff-10K, and ADE20K-Full; ResNet-101 is used for Cityscapes.

    On Cityscapes (19 categories), MaskFormer matches the per-pixel baseline in mIoU (78.578.5). As category count expands to 150 (ADE20K), 171 (COCO-Stuff-10K), and 847 (ADE20K-Full), MaskFormer's advantage grows to +2.6 mIoU+2.6\text{ mIoU}, +2.9 mIoU+2.9\text{ mIoU}, and +3.5 mIoU+3.5\text{ mIoU}, respectively. On ADE20K-Full, MaskFormer also reduces GPU training memory from 26,698 MB26,698\text{ MB} to 6,529 MB6,529\text{ MB} because the number of queries N=100N=100 is decoupled from the class vocabulary size K=847K=847.

  7. Knowl 7 — Ablation of Mask Classification Formulation and Matching Schemes

    empirical result

    Ablation experiments on ADE20K validation (150 categories, ResNet-50 backbone, 150 queries) verify that the performance improvements in semantic segmentation stem from mask classification rather than loss formulations:

    1. Loss Formulations: Supervising PerPixelBaseline+ with a combination of focal loss and dice loss instead of cross-entropy produces no change in segmentation performance.
    2. Fixed Matching vs Per-Pixel Classification: MaskFormer trained with fixed matching (MaskFormer-fixed, where N=K=150N = K = 150 and prediction ii is statically mapped to category index ii) achieves 43.7 mIoU43.7\text{ mIoU} and 30.3 PQSt30.3\text{ PQ}^\text{St}, outperforming PerPixelBaseline+ (41.9 mIoU41.9\text{ mIoU} and 28.3 PQSt28.3\text{ PQ}^\text{St}) by +1.8 mIoU+1.8\text{ mIoU}. This demonstrates that predicting a set of binary masks paired with global class labels is inherently more effective than classifying pixels individually.
    3. Fixed vs Bipartite Matching: Switching from fixed matching (N=150N=150) to bipartite matching (N=150N=150, MaskFormer-bipartite) further improves performance to 44.2 mIoU44.2\text{ mIoU} (+0.5+0.5) and 33.4 PQSt33.4\text{ PQ}^\text{St} (+3.1+3.1). Bipartite matching provides higher accuracy and removes the constraint that the number of queries NN must equal the category vocabulary size KK.

  8. Knowl 8 — Superiority of Mask-Based Matching and Shared Pixel Decoder over DETR

    empirical result

    MaskFormer eliminates the bounding box head from Transformer set prediction models and replaces per-query upsampling with a shared per-image pixel decoder, leading to higher segmentation quality and reduced computational cost.

    1. Mask-Based Matching vs Box-Based Matching: On COCO panoptic validation (ResNet-50 with 6 encoder layers):
    Method Matching Type PQ PQTh\text{PQ}^\text{Th} PQSt\text{PQ}^\text{St}
    DETR Box-based 43.4 48.2 36.3
    MaskFormer Box-based 43.7 49.2 35.3
    MaskFormer Mask-based 46.5 51.0 39.8

    Matching predictions to ground truth via binary mask loss (Lmask\mathcal{L}_{mask}) yields a +2.8 PQ+2.8\text{ PQ} improvement over box-based matching (46.546.5 vs 43.7 PQ43.7\text{ PQ}), with a large gain in stuff classes (PQSt\text{PQ}^\text{St} increases from 35.335.3 to 39.839.8). This indicates that bounding boxes introduce spatial ambiguity for unbounded, non-compact stuff segments.

    1. Mask Head Efficiency: DETR generates low-resolution query attention maps and applies independent upsampling convolutions to each query separately, incurring a cost of O(N)O(N) with respect to query count NN. MaskFormer computes a single dense per-pixel embedding Epixel\mathcal{E}_{pixel} via a shared FPN pixel decoder and obtains masks through an inner product with Emask\mathcal{E}_{mask}. The mask head in DETR is therefore NN times more computationally expensive than in MaskFormer.

  9. Knowl 9 — Impact of Query Count and Decoder Layers in Semantic vs Panoptic Tasks

    empirical result

    Empirical analyses reveal how query capacity and decoder depth behave differently across semantic and panoptic segmentation:

    1. Number of Queries (NN):
    ADE20K (150) COCO-Stuff (171) ADE20K-Full (847)
    # Queries mIoU PQSt\text{PQ}^\text{St} mIoU PQSt\text{PQ}^\text{St} mIoU PQSt\text{PQ}^\text{St}
    PerPixelBaseline+ 41.9 28.3 34.2 24.6 13.9 9.0
    20 42.9 32.6 35.0 27.6 14.1 10.8
    50 43.9 32.7 35.5 27.9 15.4 11.1
    100 44.5 33.4 37.1 28.9 16.0 11.9
    150 44.2 33.4 37.0 28.9 15.5 11.5
    300 43.5 32.3 36.1 29.1 14.2 10.3
    1000 35.4 26.7 34.4 27.6 8.0 5.8

    N=100N=100 queries achieves optimal performance across datasets regardless of category count. Even N=20N=20 queries outperforms per-pixel classification because average images contain few simultaneous classes (8.2 on ADE20K, 6.6 on COCO-Stuff, 9.1 on ADE20K-Full), and queries can dynamically bind to different semantic categories across images.

    1. Decoder Depth and Self-Attention:
    • In semantic segmentation, a single Transformer decoder layer (L=1L=1) attains 43.0 mIoU43.0\text{ mIoU} and 31.1 PQSt31.1\text{ PQ}^\text{St} on ADE20K (outperforming 6-layer PerPixelBaseline+ at 41.9 mIoU41.9\text{ mIoU}). Removing self-attention across 6 decoder layers retains strong performance (44.6 mIoU44.6\text{ mIoU}).
    • In panoptic segmentation, stacking multiple decoder layers with self-attention is essential to de-duplicate instance proposals (1 layer yields 31.9 PQ31.9\text{ PQ}, 6 layers yield 34.7 PQ34.7\text{ PQ}, and removing self-attention reduces PQ to 32.632.6).

  10. Knowl 10 — Unified Query Behavior in Semantic vs Panoptic Ground-Truth Supervision

    theoretical result

    MaskFormer handles semantic, instance, and panoptic segmentation with identical architecture, loss formulation, and optimization, modifying its query specialization solely based on the ground-truth target format:

    1. Semantic Ground Truth: All pixels belonging to a class across an image are labeled under a single binary mask mgtm^{gt}. Under Hungarian matching and binary mask loss (Lmask\mathcal{L}_{mask}), this compels a single query to predict a combined mask encompassing all instances of that category in the image.
    2. Panoptic / Instance Ground Truth: Individual instances of the same category are annotated as distinct binary masks mjgtm^{gt}_j. The bipartite matching penalty assigns different queries to different instances, forcing queries to isolate and segment separate objects independently.

    This demonstrates that specialized task-specific network designs (such as separate bounding box heads for "things" and dense convolutional heads for "stuff") or complex auxiliary objectives (such as instance discrimination or center regression) are unnecessary for unified segmentation.

Coverage note — None was omitted; all main architectural components, mathematical losses, matching and inference formulations, primary empirical benchmarks across semantic and panoptic datasets, and ablations on queries, layers, and matching strategies are fully covered.

References

  1. 1.COCO + Places Challenges 2017. https://places-coco2017.github.io/, 2016. 11
  2. 2.Pablo Arbelaez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In CVPR, 2014. 2
  3. 3.Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In CVPR, 2018. 2, 5, 11, 12
  4. 4.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 1, 2, 3, 4, 5, 6, 8, 9, 10, 13
  5. 5.Joao Carreira, Rui Caseiro, Jorge Batista, and Cristian Sminchisescu. Semantic segmentation with second-order pooling. In ECCV, 2012. 1, 2
  6. 6.Joao Carreira and Cristian Sminchisescu. CPMC: Automatic object segmentation using constrained parametric min-cuts. PAMI, 2011. 2
  7. 7.Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. PAMI, 2018. 2, 6
  8. 8.Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587, 2017. 2, 6, 7
  9. 9.Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 1, 6, 7, 12
  10. 10.Bowen Cheng, Liang-Chieh Chen, Yunchao Wei, Yukun Zhu, Zilong Huang, Jinjun Xiong, Thomas S Huang, Wen-Mei Hwu, and Honghui Shi. SPGNet: Semantic prediction guidance for scene parsing. In ICCV, 2019.
  11. 11.Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang- Chieh Chen. Panoptic-DeepLab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020. 6, 12
  12. 12.Francois Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 12
  13. 13.Dorin Comaniciu and Peter Meer. Robust Analysis of Feature Spaces: Color Image Segmentation. In CVPR, 1997. 2
  14. 14.MMSegmentation Contributors. MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020. 6, 7
  15. 15.Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benen- son, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2, 5, 11
  16. 16.Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object and stuff segmenta- tion. In CVPR, 2015. 2
  17. 17.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 2, 7
  18. 18.Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes challenge: A retrospective. IJCV, 2015. 2, 5
  19. 19.Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 2
  20. 20.Bharath Hariharan, Pablo Arbelaez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014. 1, 3
  21. 21.Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 1, 3, 4, 8
  22. 22.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6, 7, 8, 13
  23. 23.Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CCNet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 2, 6
  24. 24.Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic segmentation. In CVPR, 2019. 2, 3, 5, 6, 11
  25. 25.Scott Konishi and Alan Yuille. Statistical Cues for Domain Specific Image Segmentation with Performance Analysis. In CVPR, 2000. 2
  26. 26.Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017. 6
  27. 27.Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In ICCV, 2017. 4, 6
  28. 28.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2, 5, 11
  29. 29.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2103.14030, 2021. 2, 4, 6, 7, 11, 12, 15
  30. 30.Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015. 1, 2
  31. 31.Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6
  32. 32.Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. 13
  33. 33.Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016. 4, 6
  34. 34.Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In CVPR, 2017. 2, 5, 11
  35. 35.Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015. 6
  36. 36.Jianbo Shi and Jitendra Malik. Normalized Cuts and Image Segmentation. PAMI, 2000. 2
  37. 37.Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv:2105.05633, 2021. 2, 4
  38. 38.Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmenta- tion. arXiv:2005.10821, 2020. 12
  39. 39.Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, 2020. 3
  40. 40.Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. IJCV, 2013. 2
  41. 41.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 1, 2, 3, 4
  42. 42.Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. MaX-DeepLab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021. 2, 3, 4, 8, 13
  43. 43.Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 2
  44. 44.Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. NeurIPS, 2020. 3
  45. 45.Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018. 6
  46. 46.Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https: //github.com/facebookresearch/detectron2, 2019. 6
  47. 47.Yangxin Wu, Gengwei Zhang, Yiming Gao, Xiajun Deng, Ke Gong, Xiaodan Liang, and Liang Lin. Bidirectional graph reasoning network for panoptic segmentation. In CVPR, 2020. 13
  48. 48.Yangxin Wu, Gengwei Zhang, Hang Xu, Xiaodan Liang, and Liang Lin. Auto-panoptic: Cooperative multi-component architecture search for panoptic segmentation. In NeurIPS, 2020. 13
  49. 49.Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018. 7, 11
  50. 50.Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, 2020. 6, 7, 12
  51. 51.Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. OCNet: Object context for semantic segmentation. IJCV, 2021. 2
  52. 52.Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. 1, 2, 6
  53. 53.Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021. 2, 4, 7, 11
  54. 54.Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing challenge 2016. http://sceneparsing.csail.mit.edu/index_challenge.html, 2016. 5, 11
  55. 55.Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017. 2, 5, 11, 14

Citation

MLA
Cheng, B., et al. “Per-Pixel Classification Is Not All You Need for Semantic Segmentation”. arXiv, 2021, http://arxiv.org/abs/2107.06278v2.
APA
Cheng, B., Schwing, A. G., & Kirillov, A. (2021). Per-Pixel Classification is Not All You Need for Semantic Segmentation. arXiv. http://arxiv.org/abs/2107.06278v2
Chicago
Cheng, B., A. G. Schwing, and A. Kirillov. 2021. “Per-Pixel Classification Is Not All You Need for Semantic Segmentation”. arXiv. http://arxiv.org/abs/2107.06278v2.
Harvard
Cheng, B., Schwing, A.G. and Kirillov, A. (2021) “Per-Pixel Classification is Not All You Need for Semantic Segmentation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2107.06278v2.
Vancouver
1. Cheng B, Schwing AG, Kirillov A (2021) Per-Pixel Classification is Not All You Need for Semantic Segmentation. arXiv

BibTeX

@article{cheng2021per,
  title = {Per-Pixel Classification is Not All You Need for Semantic Segmentation},
  author = {Cheng, Bowen and Schwing, Alexander G. and Kirillov, Alexander},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2107.06278v2},
  eprint = {2107.06278}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors