Back

Segmenter: Transformer for Semantic Segmentation

Robin StrudelRicardo GarciaIvan LaptevCordelia Schmid

article2021ICCV2,025 citations

Introduces Segmenter, an end-to-end Vision Transformer architecture for semantic segmentation that captures global context throughout the network using a mask transformer decoder, surpassing convolutional baselines on the ADE20K and Pascal Context benchmarks.

Listen

Semantic image segmentation—the process of labeling every individual pixel in a scene with its object or background category—is essential for technologies such as autonomous driving, robotics, augmented reality, and medical imaging. Historically, the dominant machine learning approaches relied on convolutional networks, which process visual information through localized filters. Because identifying an ambiguous local region often depends on understanding the entire visual scene, standard models struggle to capture broader context without relying on complex, specialized modifications.

The article evaluates whether a purely attention-based transformer architecture, which models global interactions across an entire image from the start without any convolutions, can outperform traditional convolutional networks on standard semantic segmentation benchmarks.

To test this, the authors developed Segmenter, an end-to-end framework based on the Vision Transformer. The approach divides an image into distinct patches, transforms them into input tokens, and contextualizes them globally using a transformer encoder. The encodings are subsequently converted into pixel-level predictions using either a simple point-wise linear decoder or an advanced mask transformer decoder that dynamically interacts with class-specific representations. The framework was evaluated across three widely recognized benchmark datasets: ADE20K, Pascal Context, and Cityscapes, utilizing models initialized with large-scale image classification pre-training.

The findings establish that pure transformer models provide superior segmentation accuracy and global scene coherence compared to traditional architectures. On the complex ADE20K dataset, the largest Segmenter configuration achieved a mean Intersection over Union of 53.63%, surpassing the best convolutional alternatives by more than 5%. On Pascal Context, the method achieved 59.04%, outperforming prior state-of-the-art methods by nearly 3%. The evaluation showed that patch size is a critical operational lever; smaller patch sizes substantially improve fine detail and small-object detection, though at higher computational cost. Furthermore, the specialized mask transformer decoder consistently boosted performance across all configurations compared to basic linear decoding, and large-scale pre-training proved indispensable, as training from scratch yielded severely degraded accuracy.

These results indicate that self-attention mechanisms can capture adaptive, long-range scene context more effectively than fixed convolutional filters, delivering significant gains in labeling large, coherent objects and complex environments. For decision-makers, this highlights a viable path toward simpler, unified visual recognition pipelines. However, organizations must balance accuracy against computational budget. While models with large patch sizes offer high inference throughput exceeding 400 images per second, models configured for fine-grained accuracy require considerably more memory and processing power during both training and deployment.

Organizations adopting these models should select configurations based on runtime constraints, utilizing smaller patch sizes when boundary precision is paramount and larger patches when low latency is required. Technical teams should ensure adequate pre-trained foundation models are leveraged, as fine-tuning with fewer than 8,000 domain-specific training images causes notable degradation. Future efforts should focus on memory-efficient attention techniques to enable very small patch sizes on large models and extend the mask-based decoding framework to unified panoptic and instance segmentation tasks.

Confidence in these findings is high given the comprehensive ablations across standard benchmarks, though practitioners should note that convolutional baselines still demonstrated slightly sharper boundary definition in tight clusters of small objects, and performance remains heavily contingent on robust upstream pre-training.

INRIAclass labelsencoder-decoderimage classificationSegmentationTransformerSemantic Segmentation
arXiv: 2105.05633rstrudel/segmenter
Cover for Segmenter: Transformer for Semantic Segmentation

Abstract

Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embeddings with a point-wise linear decoder or a mask transformer decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes.

Topics

class labelsencoder-decoderimage classificationimage patchesobject segmentationsegmentation methodsobject detectionsemantic informationimage features

Keywords

SegmentationTransformerSemantic SegmentationSizeSegmenterPatchImageModelsDeeplabv3Patch Size

Table of Contents

  • 1 Introduction
  • 2 Related work
  • 3 Our approach: Segmenter
  • 3.1 Encoder
  • 3.2 Decoder
  • 4 Experimental results
  • 4.1 Datasets and metrics
  • 4.2 Implementation details
  • 4.3 Ablation study
  • 4.4 Comparison with state of the art
  • 5 Conclusion
  • 6 Acknowledgements
  • References
  • A ImageNet pre-training
  • B Attention maps and class embeddings
  • C Qualitative results

Knowls

  1. Knowl 1 — Segmenter Framework Architecture

    model/method

    Segmenter is a fully transformer-based encoder-decoder architecture designed for semantic image segmentation that maps an image directly to pixel-level category annotations without using convolutional feature extraction backbones.

    Given an input image x∈RH×W×Cx \in \mathbb{R}^{H \times W \times C} with height HH, width WW, and CC channels, the model divides xx into a sequence of N=HW/P2N = HW / P^2 non-overlapping patches of size P×PP \times P. The patches are linearly projected into DD-dimensional token embeddings and summed with learnable 1D position embeddings. A Vision Transformer (ViT) encoder comprising LL self-attention layers processes these tokens to produce contextualized patch encodings zL∈RN×Dz_L \in \mathbb{R}^{N \times D}.

    A transformer-based decoder (either a point-wise linear decoder or a multi-layer mask transformer) decodes zLz_L into patch-level category representations, which are subsequently reshaped into a 2D spatial grid R(H/P)×(W/P)×K\mathbb{R}^{(H/P) \times (W/P) \times K}, bilinearly upsampled to the original image resolution RH×W×K\mathbb{R}^{H \times W \times K}, and normalized via softmax along the class dimension to yield per-pixel class probabilities for KK classes. The network is trained end-to-end using standard pixel-wise cross-entropy loss.

  2. Knowl 2 — Segmenter Vision Transformer Encoder Formulation

    model/method

    The encoder of Segmenter converts raw image patches into contextualized patch representations using a sequence of standard transformer layers.

    An input image x∈RH×W×Cx \in \mathbb{R}^{H \times W \times C} is partitioned into N=HW/P2N = HW/P^2 flattened patch vectors x=[x1,…,xN]∈RN×(P2C)x = [x_1, \ldots, x_N] \in \mathbb{R}^{N \times (P^2 C)}, where (P,P)(P, P) is the patch spatial resolution. A learnable linear projection matrix E∈RD×(P2C)E \in \mathbb{R}^{D \times (P^2 C)} maps each patch into a DD-dimensional embedding space: x0=[Ex1,…,ExN]∈RN×Dx_0 = [Ex_1, \ldots, Ex_N] \in \mathbb{R}^{N \times D}. Learnable 1D position embeddings pos=[pos1,…,posN]∈RN×D\text{pos} = [\text{pos}_1, \ldots, \text{pos}_N] \in \mathbb{R}^{N \times D} are added to retain spatial positions, producing the input token sequence z0=x0+posz_0 = x_0 + \text{pos}.

    The sequence z0z_0 is processed through LL consecutive transformer layers. For each layer i∈{1,…,L}i \in \{1, \ldots, L\}, intermediate representations are computed via multi-headed self-attention (MSA) and a two-layer point-wise MLP block with Layer Normalization (LN) and residual connections:

    ai−1=MSA(LN(zi−1))+zi−1a_{i-1} = \text{MSA}(\text{LN}(z_{i-1})) + z_{i-1}

    zi=MLP(LN(ai−1))+ai−1z_i = \text{MLP}(\text{LN}(a_{i-1})) + a_{i-1}

    Multi-headed self-attention maps tokens to queries Q∈RN×dQ \in \mathbb{R}^{N \times d}, keys K∈RN×dK \in \mathbb{R}^{N \times d}, and values V∈RN×dV \in \mathbb{R}^{N \times d} using learned linear projections, computing attention weights across the entire sequence:

    MSA(Q,K,V)=softmax(QKTd)V\text{MSA}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V

    The encoder output zL=[zL,1,…,zL,N]∈RN×Dz_L = [z_{L,1}, \ldots, z_{L,N}] \in \mathbb{R}^{N \times D} provides globally contextualized patch tokens for decoding.

  3. Knowl 3 — Segmenter Mask Transformer Decoder

    model/method

    The Mask Transformer decoder in Segmenter introduces a set of KK learnable class embeddings cls=[cls1,…,clsK]∈RK×D\text{cls} = [\text{cls}_1, \ldots, \text{cls}_K] \in \mathbb{R}^{K \times D}, where each embedding is randomly initialized and assigned to one of the KK semantic classes.

    The class embeddings cls\text{cls} and the patch encodings zL∈RN×Dz_L \in \mathbb{R}^{N \times D} from the encoder are concatenated and processed jointly through an MM-layer transformer decoder (with M=2M=2 layers of identical token and hidden dimension as the encoder). The decoder produces updated L2L_2-normalized patch representations zM′∈RN×Dz'_M \in \mathbb{R}^{N \times D} and updated class embeddings c∈RK×Dc \in \mathbb{R}^{K \times D}.

    The set of KK class mask logits is generated by computing the scalar product between the patch representations and class embeddings:

    Masks(zM′,c)=zM′cT∈RN×K\text{Masks}(z'_M, c) = z'_M c^T \in \mathbb{R}^{N \times K}

    Each mask sequence is reshaped into a 2D map smask∈R(H/P)×(W/P)×Ks_{\text{mask}} \in \mathbb{R}^{(H/P) \times (W/P) \times K}, bilinearly upsampled to the original image dimensions RH×W×K\mathbb{R}^{H \times W \times K}, softmax-normalized over the class dimension, and layer-normalized to yield final pixel-wise class probability distributions s∈RH×W×Ks \in \mathbb{R}^{H \times W \times K} such that ∑k=1Ksi,j,k=1\sum_{k=1}^K s_{i,j,k} = 1 for all (i,j)∈H×W(i, j) \in H \times W.

    Because the class embeddings interact with all patch tokens through self-attention, the class embeddings act as dynamically generated convolutional filters that adapt to image content.

  4. Knowl 4 — Segmenter Point-wise Linear Decoder

    model/method

    The linear decoder serves as a minimalist baseline for Segmenter. Given the sequence of contextualized patch encodings zL∈RN×Dz_L \in \mathbb{R}^{N \times D} produced by the encoder, a point-wise linear projection layer directly maps each patch embedding to KK class logits:

    zlin=zLWlin+blin∈RN×Kz_{\text{lin}} = z_L W_{\text{lin}} + b_{\text{lin}} \in \mathbb{R}^{N \times K}

    where Wlin∈RD×KW_{\text{lin}} \in \mathbb{R}^{D \times K} and blin∈RKb_{\text{lin}} \in \mathbb{R}^K. The resulting sequence zlinz_{\text{lin}} is unflattened and reshaped into a 2D grid slin∈R(H/P)×(W/P)×Ks_{\text{lin}} \in \mathbb{R}^{(H/P) \times (W/P) \times K}, bilinearly interpolated to the original image size s∈RH×W×Ks \in \mathbb{R}^{H \times W \times K}, and passed through a softmax function across the category channel to obtain per-pixel class probabilities.

  5. Knowl 5 — Stochastic Depth vs. Dropout Regularization for Segmentation Transformers

    empirical result

    When fine-tuning Vision Transformers for semantic segmentation, stochastic depth regularization (randomly dropping transformer layers during forward passes) consistently improves performance, whereas standard dropout (randomly dropping tokens or activations) degrades segmentation accuracy.

    Ablation on the ADE20K validation set using a Seg-S/16 model (ViT-Small backbone, 16×1616 \times 16 patch size) demonstrates this effect across combinations of dropout rates and stochastic depth drop rates:

    Dropout Stochastic Depth
    0.0 0.1 0.2
    0.0 45.01 45.37 45.10
    0.1 42.02 42.30 41.14
    0.2 36.49 36.63 35.67

    Setting stochastic depth to 0.1 without dropout yields a +0.36%+0.36\% mean IoU gain over the unregularized baseline (45.37% vs. 45.01%), whereas applying dropout rates of 0.1 or 0.2 reduces mean IoU substantially by 3.07%3.07\% to 8.88%8.88\%.

  6. Knowl 6 — Impact of Patch Size and Model Capacity on Segmentation Accuracy and Speed

    data/table

    Input patch size PP dictates the sequence length N=HW/P2N = HW/P^2 processed by the transformer self-attention blocks. Decreasing PP significantly enhances segmentation detail and mean IoU at the cost of quadratic attention complexity and reduced throughput (frames per second).

    Evaluation of Segmenter variants on the ADE20K validation set under single-scale (SS) inference:

    Method Backbone Patch size Im/sec ImNet acc. mIoU (SS)
    Seg-Ti/16 ViT-Ti 16 396 78.6 39.03
    Seg-S/32 ViT-S 32 1032 80.5 40.64
    Seg-S/16 ViT-S 16 196 83.7 45.37
    Seg-B†^{\dagger}/16 DeiT-B 16 92 85.2 47.08
    Seg-B/32 ViT-B 32 516 83.3 43.07
    Seg-B/16 ViT-B 16 92 86.0 48.06
    Seg-B/8 ViT-B 8 7 85.7 49.54
    Seg-L/16 ViT-L 16 33 87.1 50.71

    Reducing the patch size from 32 to 16 improves mIoU by 4.99%4.99\% on Seg-B (43.07%→48.06%43.07\% \to 48.06\%). Further reducing the patch size to 8 achieves 49.54%49.54\% mIoU, narrowing the gap to Seg-L/16 (50.71%50.71\%) to just 1.17%1.17\% without adding parameters, although throughput drops from 92 to 7 images per second on an NVIDIA V100 GPU.

  7. Knowl 7 — Impact of Pre-training and Fine-tuning Dataset Size on Transformer Segmentation

    empirical result

    Pre-training on large-scale classification data is essential for vision transformer performance in semantic segmentation, and segmentation fine-tuning exhibits severe performance degradation when target dataset size is restricted.

    1. Pre-training Requirement: When training Seg-S/16 on ADE20K from scratch with random weight initialization (truncated normal distribution), the model achieves only 4.42%4.42\% mIoU with AdamW (using cosine scheduling and 16K16\text{K} iteration warmup) and 12.51%12.51\% mIoU with SGD. In contrast, pre-training the backbone on ImageNet-21k yields 34.77%34.77\% mIoU with AdamW and 45.37%45.37\% mIoU with SGD, representing a +32.86%+32.86\% mIoU improvement over training from scratch.

    2. Training Data Size Sensitivity: Subsampling the ADE20K training set for Seg-S/16 demonstrates a steep performance drop when training with fewer than 8K8\text{K} images:

    Dataset Size 4k 8k 12k 16k 20k
    mIoU (SS) 38.31 41.87 43.42 44.61 45.37

  8. Knowl 8 — State-of-the-Art Semantic Segmentation on ADE20K

    data/table

    Segmenter outperforms fully convolutional networks (FCNs) and concurrent vision transformers on the ADE20K validation set across single-scale (SS) and multi-scale (+MS) inference protocols.

    Method Backbone Im/sec mIoU +MS
    OCR HRNetV2-W48 83 - 45.66
    ACNet ResNet-101 - - 45.90
    DNL ResNet-101 - - 45.97
    DRANet ResNet-101 - - 46.18
    CPNet ResNet-101 - - 46.27
    DeepLabv3+ ResNet-101 76 45.47 46.35
    DeepLabv3+ ResNeSt-101 15 46.47 47.27
    DeepLabv3+ ResNeSt-200 - - 48.36
    SETR-L MLA ViT-L/16 34 48.64 50.28
    Swin-L UperNet Swin-L/16 34 52.10 53.50
    Seg-B†^{\dagger}/16 DeiT-B/16 77 47.08 48.05
    Seg-B†^{\dagger}-Mask/16 DeiT-B/16 76 48.70 50.08
    Seg-L/16 ViT-L/16 33 50.71 52.25
    Seg-L-Mask/16 ViT-L/16 31 51.82 53.63

    Seg-L-Mask/16 attains 53.63%53.63\% mIoU (+MS), exceeding the leading convolutional model (DeepLabv3+ ResNeSt-200 at 48.36%48.36\%) by 5.27%5.27\% mIoU and outperforming SETR-L MLA (50.28%50.28\%) and Swin-L UperNet (53.50%53.50\%).

  9. Knowl 9 — Benchmark Performance on Pascal Context and Cityscapes

    data/table

    Segmenter achieves state-of-the-art results on Pascal Context and competitive performance on Cityscapes under multi-scale (MS) evaluation.

    Pascal Context Validation Cityscapes Validation
    Method Backbone mIoU (MS) Method Backbone mIoU (MS)
    DeepLabv3+ ResNet-101 48.5 PSANet ResNet-101 79.1
    DANet ResNet-101 52.6 DeepLabv3+ Xception-71 79.6
    CPNet ResNet-101 53.9 ANN ResNet-101 79.9
    ACNet ResNet-101 54.1 MDEQ MDEQ 80.3
    DNL HRNetV2-W48 55.3 DeepLabv3+ ResNeSt-101 80.4
    DRANet ResNet-101 55.4 DNL ResNet-101 80.5
    OCR HRNetV2-W48 56.2 CCNet ResNet-101 81.3
    SETR-L MLA ViT-L/16 55.8 Panoptic-Deeplab Xception-71 81.5
    Seg-B†^{\dagger}/16 DeiT-B/16 53.9 DeepLabv3+ ResNeSt-200 82.7
    Seg-B†^{\dagger}-Mask/16 DeiT-B/16 55.0 SETR-L PUP ViT-L/16 82.2
    Seg-L/16 ViT-L/16 56.5 Seg-B†^{\dagger}/16 DeiT-B/16 80.5
    Seg-L-Mask/16 ViT-L/16 59.0 Seg-L-Mask/16 ViT-L/16 81.3

    On Pascal Context (59 semantic classes + background), Seg-L-Mask/16 achieves 59.0%59.0\% mIoU, outperforming OCR HRNetV2-W48 by 2.8%2.8\% and SETR-L MLA by 3.2%3.2\%. On Cityscapes, Seg-L-Mask/16 (using a 1-layer decoder due to memory constraints at 768×768768 \times 768 resolution) reaches 81.3%81.3\% mIoU.

  10. Knowl 10 — Object-Size Breakdown and Structural Trade-offs Between Segmenter and FCNs

    empirical result

    Segmenter models achieve their largest performance advantages over Fully Convolutional Networks (FCNs) on large objects, due to the global receptive field enabled by all-layer self-attention, while convolutional networks remain competitive on fine local boundaries of small instances.

    Evaluation of mean IoU on ADE20K validation categorized by object size:

    Method Decoder Small Medium Large mIoU (SS)
    DeepLabv3+ ResNeSt-101 UNet 37.85 50.89 50.67 46.47
    Seg-B/32 Linear 31.95 47.82 49.44 43.07
    Seg-B-Mask/32 Mask 32.29 49.44 50.82 44.19
    Seg-B†^{\dagger}/16 Linear 38.31 50.91 52.08 47.10
    Seg-B†^{\dagger}-Mask/16 Mask 40.49 51.37 54.24 48.70
    Seg-B/16 Linear 39.57 51.32 53.28 48.06
    Seg-B-Mask/16 Mask 40.16 52.61 52.66 48.48
    Seg-B/8 Linear 41.43 54.35 52.85 49.54
    Seg-L/16 Linear 42.08 54.67 55.39 50.71
    Seg-L-Mask/16 Mask 42.02 54.83 57.06 51.30

    Seg-L-Mask/16 improves over DeepLabv3+ ResNeSt-101 by +6.39%+6.39\% on large objects (57.06%57.06\% vs. 50.67%50.67\%) and generates more coherent segment labels across occluded regions. However, DeepLabv3+ produces sharper boundary delineations around small objects compared to Seg-B/16 unless smaller patch sizes (such as 8×88 \times 8, which achieves 41.43%41.43\% on small objects) are employed.

Coverage note — None was omitted; all key architectural components, decoding strategies, regularizations, ablations, benchmark evaluations, and empirical comparative analyses are fully covered.

References

  1. 1.Md Amirul Islam, Mrigank Rochan, Neil D. B. Bruce, and Yang Wang. Gated feedback refinement network for dense image labeling. In CVPR, 2017. 2
  2. 2.Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. ViViT: A video vision transformer. In ICCV, 2021. 3
  3. 3.Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint, 2016. 6
  4. 4.V. Badrinarayanan, A. Kendall, and R. Cipolla. SegNet: A deep convolutional encoder-decoder architecture for image segmentation. TPAMI, 2017. 2
  5. 5.Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. In NeurIPS, 2020. 8
  6. 6.Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint, 2021. 3
  7. 7.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with Transformers. In ECCV, 2020. 2, 3, 4
  8. 8.Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI, 2018. 2
  9. 9.Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint, 2017. 2
  10. 10.Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 1, 2, 5, 7, 8
  11. 11.Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. PLMR, 2020. 2
  12. 12.Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-Deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020. 8
  13. 13.MMSegmentation Contributors. MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020. 5, 7, 8
  14. 14.Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2, 4
  15. 15.Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pages 3008–3017. IEEE, 2020. 4
  16. 16.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2
  17. 17.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. 6
  18. 18.Carl Doersch, Ankush Gupta, and Andrew Zisserman. CrossTransformers: spatially-aware few-shot transfer. In NeurIPS, 2020. 2
  19. 19.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 2, 4, 5, 6
  20. 20.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. Vision transformer models. https://console.cloud.google.com/storage/browser/vit_models, 2021. 5
  21. 21.C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchical features for scene labeling. TPAMI, 2013. 2
  22. 22.J. Fu, J. Liu, J. Jiang, Y. Li, Y. Bao, and H. Lu. Scene segmentation with Dual Relation-Aware attention Network. TNNLS, 2020. 1, 2, 7, 8
  23. 23.Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual Attention Network for scene segmentation. In CVPR, 2019. 1, 2, 8
  24. 24.Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu. Adaptive Context Network for scene parsing. ICCV, 2019. 7, 8
  25. 25.Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. In NeurIPS, 2018. 5
  26. 26.Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. Adaptive Pyramid Context Network for semantic segmentation. In CVPR, 2019. 8
  27. 27.Mohammad Hesam Hesamian, Wenjing Jia, Xiangjian He, and Paul Kennedy. Deep learning techniques for medical image segmentation: Achievements and challenges. JDI, 2019. 1
  28. 28.Zhang-Wei Hong, Chen Yu-Ming, Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, Hsuan-Kung Yang, Brian Hsi-Lin Ho, Chih-Chieh Tu, Yueh-Chuan Chang, Tsu-Ching Hsiao, Hsin-Wei Hsiao, Sih-Pin Lai, and Chun-Yi Lee. Virtual-to-real: Learning to control in visual semantic segmentation. IJCAI, 2018. 1
  29. 29.Chia-Wei Hsieh, Chieh-Yun Chen, Chien-Lung Chou, Hong-Han Shuai, Jiaying Liu, and Wen-Huang Cheng. Fashionon: Semantic-guided image-based virtual try-on with detailed human and clothing information. In ACM MM, 2019. 1
  30. 30.Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016. 4, 5
  31. 31.Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CCNet: Criss-Cross attention for semantic segmentation. In ICCV, 2019. 1, 8
  32. 32.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 6
  33. 33.Chen Liang-Chieh, George Papandreou, Iasonas Kokkinos, kevin murphy, and Alan Yuille. Semantic image segmentation with deep convolutional nets and fully connected CRFs. In ICLR, 2015. 2, 5
  34. 34.Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. RefineNet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR, 2017. 2
  35. 35.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint, 2021. 3, 5, 7
  36. 36.Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully Convolutional Networks for semantic segmentation. In CVPR, 2015. 2
  37. 37.Shervin Minaee, Yuri Boykov, F. Porikli, Antonio J. Plaza, N. Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. TPAMI, 2021. 1
  38. 38.Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan L. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014. 2, 4
  39. 39.F. Ning, D. Delhomme, Yann LeCun, F. Piano, Leon Bottou, and Paolo Emilio Barbano. Toward automatic phenotyping of developing embryos from videos. TIP, 2005. 2
  40. 40.Pedro Pinheiro and Ronan Collobert. Recurrent Convolutional Neural Networks for scene labeling. In ICML, 2014. 2
  41. 41.Tobias Pohlen, Alexander Hermans, Markus Mathias, and Bastian Leibe. Full-resolution residual networks for semantic segmentation in street scenes. In CVPR, 2017. 2
  42. 42.H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 1951. 5
  43. 43.O. Ronneberger, P.Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 2
  44. 44.Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully Convolutional Networks for semantic segmentation. TPAMI, 2017. 1
  45. 45.M. Siam, S. Elkerdawy, M. Jagersand, and S. Yogamani. Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges. In ITSC, 2017. 1
  46. 46.Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014. 4, 5
  47. 47.Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers. arXiv preprint, 2021. 4, 5
  48. 48.Farhana Sultana, Abu Sufian, and Paramartha Dutta. Evolution of image segmentation using deep convolutional neural network: A survey. Knowledge-Based Systems, 2020. 1
  49. 49.Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training Data-Efficient image Transformers and distillation through attention. arXiv preprint, 2020. 2, 4, 6
  50. 50.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 2, 3
  51. 51.Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. arXiv preprint, 2020. 2
  52. 52.Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang-Chieh Chen. MaX-DeepLab: End-to-end panoptic segmentation with mask transformers. arXiv preprint, 2020. 4
  53. 53.Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial-Deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV, 2020. 2
  54. 54.Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 2
  55. 55.Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. In NeurIPS, 2020. 4
  56. 56.Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2020. 5
  57. 57.Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled Non-local Neural Networks. In ECCV, 2020. 1, 2, 7, 8
  58. 58.Changqian Yu, Jingbo Wang, Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang. Context Prior for scene segmentation. In CVPR, 2020. 1, 2, 7, 8
  59. 59.Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016. 2
  60. 60.Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-Contextual Representations for semantic segmentation. In ECCV, 2020. 1, 7, 8
  61. 61.Yuhui Yuan and Jingdong Wang. OCNet: Object Context Network for scene parsing. arXiv preprint, 2018. 2
  62. 62.Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. 4
  63. 63.Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Mueller, R. Manmatha, Mu Li, and Alexander J. Smola. Resnest: Split-attention networks. arXiv preprint, 2020. 7
  64. 64.H. Zhang, H. Zhang, C. Wang, and J. Xie. Co-occurrent features in semantic segmentation. In CVPR, 2019. 1, 8
  65. 65.Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid Scene Parsing Network. In CVPR, 2017. 1, 2
  66. 66.Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: Point-wise Spatial Attention Network for scene parsing. In ECCV, September 2018. 1, 2, 8
  67. 67.Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequence perspective with Transformers. arXiv preprint, 2020. 3, 7, 8
  68. 68.Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. IJCV, 2019. 2, 4
  69. 69.Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In ICCV, 2019. 8

Citation

MLA
Strudel, R., et al. “Segmenter: Transformer for Semantic Segmentation”. arXiv, 2021, http://arxiv.org/abs/2105.05633v3.
APA
Strudel, R., Garcia, R., Laptev, I., & Schmid, C. (2021). Segmenter: Transformer for Semantic Segmentation. arXiv. http://arxiv.org/abs/2105.05633v3
Chicago
Strudel, R., R. Garcia, I. Laptev, and C. Schmid. 2021. “Segmenter: Transformer for Semantic Segmentation”. arXiv. http://arxiv.org/abs/2105.05633v3.
Harvard
Strudel, R. et al. (2021) “Segmenter: Transformer for Semantic Segmentation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2105.05633v3.
Vancouver
1. Strudel R, Garcia R, Laptev I, Schmid C (2021) Segmenter: Transformer for Semantic Segmentation. arXiv

BibTeX

@article{strudel2021segmenter,
  title = {Segmenter: Transformer for Semantic Segmentation},
  author = {Strudel, Robin and Garcia, Ricardo and Laptev, Ivan and Schmid, Cordelia},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2105.05633v3},
  eprint = {2105.05633}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/