Back

CvT: Introducing Convolutions to Vision Transformers

Haiping WuBin XiaoNoel CodellaMengchen LiuXiyang DaiLu YuanLei Zhang

article2021ICCV2,526 citations

Presents CvT, a hybrid architecture that incorporates convolutional token embeddings and projections into Vision Transformers to achieve superior ImageNet accuracy with fewer parameters and FLOPs while eliminating the need for positional encodings.

Listen

Vision transformers have shown strong results on image classification when trained on massive datasets, but they lag behind convolutional neural networks on smaller datasets because they lack built-in properties such as shift and scale invariance. This gap matters now because many practical vision applications still rely on datasets of moderate size, and models that combine high accuracy with low computational cost are needed for deployment.

The article set out to test whether strategically adding convolutions to the vision transformer architecture could deliver the accuracy and efficiency advantages of both designs. Researchers created the Convolutional vision Transformer (CvT) and evaluated it through large-scale image classification experiments on ImageNet-1k and ImageNet-22k, plus transfer to several downstream datasets.

The approach uses a multi-stage hierarchical structure. Each stage begins with a convolutional token embedding that overlaps patches and reduces spatial resolution while increasing feature depth, followed by convolutional transformer blocks that replace the standard linear projections for attention with depth-wise separable convolutions. Experiments compared multiple CvT variants against ResNets, prior vision transformers, and concurrent transformer models, measuring top-1 accuracy, parameter count, and FLOPs.

CvT-13 reached 81.6 percent top-1 accuracy on ImageNet-1k with 20 million parameters and 4.5 gigaFLOPs, outperforming larger ResNet-152 and several recent transformer models while using fewer resources. CvT-21 improved this to 82.5 percent. When pretrained on ImageNet-22k and fine-tuned, the wider CvT-W24 model achieved 87.7 percent top-1 accuracy. Removing positional encodings caused no accuracy drop, and the models transferred strongly to CIFAR, Oxford Pets, and Flowers datasets.

These results show that the hybrid design closes much of the performance gap between transformers and convolutional networks while remaining lightweight and simpler to adapt to variable image resolutions. The efficiency gains and removal of positional encodings reduce both training and inference costs and broaden applicability to tasks that require flexible input sizes.

Further gains appear possible by applying neural architecture search to stride and expansion ratios, and by testing the architecture on dense prediction tasks such as detection and segmentation. Additional validation on even larger or more diverse datasets would strengthen before widespread deployment.

The main limitation is that all primary results come from ImageNet-scale classification; behavior on very different domains or extremely high-resolution inputs remains untested. Confidence in the reported accuracy and efficiency improvements is high because the experiments include extensive ablations, multiple model scales, and direct comparisons on public benchmarks.

McGill UniversityMicrosoftconvolutional neural networksencoder-decoderarchitecture designsConvolutional Vision TransformerConvolutional Token EmbeddingConvolutional Projection
arXiv: 2103.15808leoxiaobin/CvT
  • Paper: A ConvNet for the 2020s, Zhuang Liu et al. (2022). This work explores the reverse trajectory of hybrid transformer-convolution designs by modernizing pure ConvNets with architectural principles drawn from Vision Transformers.
  • Paper: MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer, Sachin Mehta et al. (2021). This paper builds on the synergy between convolutions and transformers to create lightweight, mobile-friendly hybrid vision backbones.
  • Paper: Swin Transformer V2: Scaling Up Capacity and Resolution, Ze Liu et al. (2022). This work extends hierarchical vision transformer architectures to extreme parameter scales and ultra-high input resolutions.
  • Paper: Visual Prompt Tuning, Menglin Jia et al. (2022). This paper introduces parameter-efficient visual prompt tuning for vision transformer architectures, eliminating the need for expensive full fine-tuning across downstream vision tasks.
Cover for CvT: Introducing Convolutions to Vision Transformers

Abstract

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at \url{this https URL}.

Topics

convolutional neural networksencoder-decoderarchitecture designsimage classificationtransfer learningvision systemssignal distortion

Keywords

Convolutional Vision TransformerConvolutional Token EmbeddingConvolutional ProjectionHierarchical TransformerPositional EmbeddingImageNet‑1kImageNet‑22kFLOPsMulti‑Stage DesignDepth‑wise Separable Convolution

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Convolutional vision Transformer
  • 3.1 Convolutional Token Embedding
  • 3.2 Convolutional Projection for Attention
  • 3.2.1 Implementation Details
  • 3.2.2 Efficiency Considerations
  • 3.3 Methodological Discussions
  • 4 Experiments
  • 4.1 Setup
  • 4.2 Comparison to state of the art
  • 4.3 Downstream task transfer
  • 4.4 Ablation Study
  • 5 Conclusion
  • References

Knowls

  1. Knowl 1 — Hierarchical Pipeline of the Convolutional Vision Transformer (CvT)

    model/method

    The Convolutional Vision Transformer (CvT) introduces convolutional operations into the Vision Transformer (ViT) architecture via a multi-stage hierarchical design structured into three sequential stages.

    Each stage i∈{1,2,3}i \in \{1, 2, 3\} comprises two core components:

    1. A Convolutional Token Embedding layer that reshapes the incoming 1D sequence of tokens into a 2D spatial feature grid, performs an overlapping 2D convolution with stride, and applies layer normalization before flattening back to 1D. This simultaneously decreases spatial sequence length and increases channel feature dimensions across stages.

    2. A stack of Convolutional Transformer Blocks, where the conventional position-wise linear projections for the Query (QQ), Key (KK), and Value (VV) in Multi-Head Self-Attention (MHSA) are replaced by depth-wise separable convolutional projections.

    A learnable classification token ([CLS]) is introduced solely at the final stage (Stage 3). The output classification token from Stage 3 is passed through a Multi-Layer Perceptron (MLP) Head to generate class predictions. Explicit positional encodings are omitted entirely throughout the network.

  2. Knowl 2 — Convolutional Token Embedding

    model/method

    The Convolutional Token Embedding layer replaces standard non-overlapping patch embeddings by applying an overlapping 2D convolution to model local spatial context and implement spatial downsampling.

    Given an input 2D image or a 2D-reshaped token map from stage i−1i-1, denoted as xi−1∈RHi−1×Wi−1×Ci−1x_{i-1} \in \mathbb{R}^{H_{i-1} \times W_{i-1} \times C_{i-1}}, the embedding layer applies a 2D convolution f(⋅)f(\cdot) with kernel size s×ss \times s, stride s−os - o (where oo is the overlap parameter), and padding pp to yield a new token feature map f(xi−1)∈RHi×Wi×Cif(x_{i-1}) \in \mathbb{R}^{H_i \times W_i \times C_i}. The output spatial dimensions are:

    Hi=⌊Hi−1+2p−ss−o⌋+1,Wi=⌊Wi−1+2p−ss−o⌋+1H_i = \left\lfloor \frac{H_{i-1} + 2p - s}{s - o} \right\rfloor + 1, \quad W_i = \left\lfloor \frac{W_{i-1} + 2p - s}{s - o} \right\rfloor + 1

    The resulting tensor f(xi−1)f(x_{i-1}) is flattened into a 2D sequence of shape (HiWi)×Ci(H_i W_i) \times C_i and normalized with Layer Normalization before entering the stage's Transformer blocks.

  3. Knowl 3 — Convolutional Projection for Multi-Head Self-Attention

    model/method

    The Convolutional Projection replaces standard position-wise linear projections used to compute Query (QQ), Key (KK), and Value (VV) matrices in Vision Transformer Multi-Head Self-Attention (MHSA).

    For a 1D token sequence xi∈RT×Cx_i \in \mathbb{R}^{T \times C} at layer ii (where TT is the number of tokens and CC is the channel dimension), the tokens are first reshaped into a 2D spatial grid Reshape2D(xi)∈RH×W×C\text{Reshape2D}(x_i) \in \mathbb{R}^{H \times W \times C}. A depth-wise separable 2D convolution with kernel size s×ss \times s is then applied, followed by flattening:

    xiq/k/v=Flatten(Conv2d(Reshape2D(xi),s))x_i^{q/k/v} = \text{Flatten}\left(\text{Conv2d}(\text{Reshape2D}(x_i), s)\right)

    Here, Conv2d\text{Conv2d} is implemented sequentially as:

    Depth-wise Conv2d⟶BatchNorm2d⟶Point-wise Conv2d\text{Depth-wise Conv2d} \longrightarrow \text{BatchNorm2d} \longrightarrow \text{Point-wise Conv2d}

    Using depth-wise separable convolutions adds only s2Cs^2 C parameters and O(s2CT)\mathcal{O}(s^2 C T) FLOPs per projection compared to linear projection, avoiding the s2C2s^2 C^2 parameters and O(s2C2T)\mathcal{O}(s^2 C^2 T) FLOPs required by standard 2D convolutions.

  4. Knowl 4 — Squeezed Convolutional Projection via Key-Value Subsampling

    model/method

    To reduce the computational complexity of the Multi-Head Self-Attention (MHSA) module, the Convolutional Projection employs strided depth-wise separable convolutions to spatially subsample the Key (KK) and Value (VV) representations while maintaining the resolution of the Query (QQ) representation.

    In the default squeezed convolutional projection:

    • Query projection uses kernel size s×ss \times s with stride=1\text{stride} = 1.
    • Key and Value projections use kernel size s×ss \times s with stride=2\text{stride} = 2.

    Setting stride=2\text{stride} = 2 for Key and Value reduces the sequence length of KK and VV by a factor of 4 (2×22 \times 2 spatial downsampling), which reduces the computational complexity of the subsequent attention matrix multiplication and softmax operations by 4×4\times with negligible loss in accuracy.

  5. Knowl 5 — Elimination of Explicit Positional Encodings in Vision Transformers

    empirical result

    Because the combination of Convolutional Token Embeddings and Convolutional Projections introduces translation equivariance and 2D local context modeling into every stage of the network, explicit positional encodings (either learnable or sinusoidal) can be eliminated entirely without degrading model accuracy.

    On the ImageNet-1k classification benchmark using CvT-13 (20M parameters):

    • CvT-13 without positional encodings achieves 81.6% top-1 accuracy.
    • CvT-13 with positional encodings added at every stage achieves 81.5% top-1 accuracy.
    • CvT-13 with positional encodings at only the first stage or only the last stage achieves 81.4% top-1 accuracy.

    In contrast, removing positional encodings from DeiT-S (22M parameters) leads to a 1.8% drop in top-1 accuracy (from 79.8% down to 78.0%). Omitting positional encodings allows CvT models to natively process variable-resolution input images during inference without requiring positional encoding interpolation.

  6. Knowl 6 — CvT Architectural Configurations: CvT-13, CvT-21, and CvT-W24

    data/table

    The CvT architecture is instantiated across three primary configurations varying in depth (number of Transformer blocks per stage) and channel width (DiD_i). All models employ a 3-stage hierarchy, an MLP expansion ratio Ri=4R_i = 4 for all stages i∈{1,2,3}i \in \{1, 2, 3\}, and evaluate on default 224×224224 \times 224 input image resolutions.

    Layer / Metric CvT-13 CvT-21 CvT-W24
    Stage 1 Conv. Embed. 7×7,64,stride 47 \times 7, 64, \text{stride } 4 7×7,64,stride 47 \times 7, 64, \text{stride } 4 7×7,192,stride 47 \times 7, 192, \text{stride } 4
    Stage 1 Conv. Proj. 3×3,643 \times 3, 64 3×3,643 \times 3, 64 3×3,1923 \times 3, 192
    Stage 1 MHSA H1=1,D1=64H_1=1, D_1=64 H1=1,D1=64H_1=1, D_1=64 H1=3,D1=192H_1=3, D_1=192
    Stage 1 Blocks 1 1 2
    Stage 2 Conv. Embed. 3×3,192,stride 23 \times 3, 192, \text{stride } 2 3×3,192,stride 23 \times 3, 192, \text{stride } 2 3×3,768,stride 23 \times 3, 768, \text{stride } 2
    Stage 2 Conv. Proj. 3×3,1923 \times 3, 192 3×3,1923 \times 3, 192 3×3,7683 \times 3, 768
    Stage 2 MHSA H2=3,D2=192H_2=3, D_2=192 H2=3,D2=192H_2=3, D_2=192 H2=12,D2=768H_2=12, D_2=768
    Stage 2 Blocks 2 4 2
    Stage 3 Conv. Embed. 3×3,384,stride 23 \times 3, 384, \text{stride } 2 3×3,384,stride 23 \times 3, 384, \text{stride } 2 3×3,1024,stride 23 \times 3, 1024, \text{stride } 2
    Stage 3 Conv. Proj. 3×3,3843 \times 3, 384 3×3,3843 \times 3, 384 3×3,10243 \times 3, 1024
    Stage 3 MHSA H3=6,D3=384H_3=6, D_3=384 H3=6,D3=384H_3=6, D_3=384 H3=16,D3=1024H_3=16, D_3=1024
    Stage 3 Blocks 10 16 20
    Parameters (M) 19.98 31.54 276.70
    FLOPs (G) 4.53 7.13 60.86

    HiH_i denotes the number of self-attention heads and DiD_i denotes the embedding feature dimension in stage ii.

  7. Knowl 7 — ImageNet Classification Performance of CvT Models

    data/table

    CvT achieves higher classification accuracy on ImageNet-1k, ImageNet Real, and ImageNet V2 compared to standard CNNs and contemporary Vision Transformers, while requiring fewer parameters and lower FLOP counts.

    Network Params (M) Input Size FLOPs (G) ImageNet Top-1 (%) Real Top-1 (%) V2 Top-1 (%)
    ResNet-50 25 2242224^2 4.1 76.2 82.5 63.3
    ResNet-152 60 2242224^2 11.0 78.3 84.1 67.0
    DeiT-S 22 2242224^2 4.6 79.8 85.7 68.5
    DeiT-B 86 2242224^2 17.6 81.8 86.7 71.5
    PVT-Small 25 2242224^2 3.8 79.8 – –
    PVT-Large 61 2242224^2 9.8 81.7 – –
    T2T-ViTt-14 22 2242224^2 6.1 80.7 – –
    TNT-S 24 2242224^2 5.2 81.3 – –
    CvT-13 20 2242224^2 4.5 81.6 86.7 70.4
    CvT-21 32 2242224^2 7.1 82.5 87.2 71.3
    CvT-13 (384) 20 3842384^2 16.3 83.0 87.9 71.9
    CvT-21 (384) 32 3842384^2 24.9 83.3 87.7 71.9

    CvT-21 exceeds DeiT-B by 0.7% on ImageNet top-1 accuracy while reducing parameters by 63% and FLOPs by 60%. CvT-13 outperforms ResNet-152 by 3.3% top-1 accuracy with one-third of the parameters.

  8. Knowl 8 — ImageNet-22k Pretraining and Downstream Task Transfer Performance

    data/table

    When pretrained on the larger ImageNet-22k dataset (14M images, 22k classes) and fine-tuned to ImageNet-1k or downstream datasets at 384×384384 \times 384 resolution, CvT architectures outperform larger Transformer (ViT) and CNN (BiT) models.

    Model Params (M) ImageNet-1k Top-1 CIFAR-10 CIFAR-100 Pets Flowers-102
    BiT-M (4802480^2) 928 85.4 98.91 92.17 94.46 99.30
    ViT-B/16 86 84.0 98.95 91.67 94.43 99.38
    ViT-L/16 307 85.2 99.16 93.44 94.73 99.61
    ViT-H/16 632 85.1 99.27 93.82 94.82 99.51
    CvT-13 20 83.3 98.83 91.11 93.25 99.50
    CvT-21 32 84.9 99.16 92.88 94.03 99.62
    CvT-W24 277 87.7 99.39 94.09 94.73 99.72

    CvT-W24 pretrained on ImageNet-22k achieves 87.7% top-1 accuracy on ImageNet-1k, surpassing ViT-L/16 (85.2%) by 2.5% with fewer parameters (277M vs 307M) and competitive FLOPs (193.2G vs 191.1G), while achieving the highest accuracy on CIFAR-10 (99.39%), CIFAR-100 (94.09%), and Oxford Flowers-102 (99.72%).

  9. Knowl 9 — Ablation of Convolutional Projection Across Stages

    empirical result

    Progressively replacing standard position-wise linear projections with Convolutional Projections across the stages of CvT-13 demonstrates continuous performance improvements on ImageNet-1k top-1 accuracy:

    • Position-wise Linear Projection in all 3 stages (no Conv Proj): 80.6% top-1 accuracy.
    • Convolutional Projection in Stage 1 only (Linear in Stages 2 and 3): 80.8% top-1 accuracy.
    • Convolutional Projection in Stages 1 and 2 (Linear in Stage 3): 81.0% top-1 accuracy.
    • Convolutional Projection in all 3 stages: 81.6% top-1 accuracy.

    Replacing linear projections with depth-wise separable convolutional projections in every stage provides a cumulative +1.0% top-1 accuracy improvement over pure linear projections.

  10. Knowl 10 — Effect of Key and Value Subsampling Stride in Convolutional Projection

    empirical result

    In the Convolutional Projection module of CvT-13 (20M parameters), using a strided convolution for Key (KK) and Value (VV) projections provides a favorable computation-accuracy trade-off on ImageNet-1k:

    • With stride=1\text{stride} = 1 for Key and Value projections: 82.3% ImageNet top-1 accuracy at 6.55 G FLOPs.
    • With stride=2\text{stride} = 2 for Key and Value projections: 81.6% ImageNet top-1 accuracy at 4.53 G FLOPs.

    Setting stride=2\text{stride} = 2 decreases computational complexity by 30.8% (2.02 G FLOPs reduction) with an accuracy reduction of 0.7%, making stride=2\text{stride} = 2 the preferred default for computational efficiency.

Coverage note — Details of the neural architecture search (CvT-13-NAS) search configuration were omitted as they represent a standard search variation rather than the primary architectural contribution.

References

  1. 1.Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
  2. 2.Lucas Beyer, Olivier J H'enaff, Alexander Kolesnikov, Xiaohua Zhai, and Aaron van den Oord. Are we done with ¨ imagenet? arXiv preprint arXiv:2006.07159, 2020.
  3. 3.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  4. 4.Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint arXiv:2012.00364, 2020.
  5. 5.François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  6. 6.Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? arXiv preprint arXiv:2102.10882, 2021.
  7. 7.Xiyang Dai, Dongdong Chen, Mengchen Liu, Yinpeng Chen, and Lu YUan. Da-nas: Data adapted pruning for efficient neural architecture search. In European Conference on Computer Vision, 2020.
  8. 8.Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:2011.09094, 2020.
  9. 9.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  10. 10.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
  11. 11.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  12. 12.Nico Engel, Vasileios Belagiannis, and Klaus Dietmayer. Point transformer. arXiv preprint arXiv:011.00931, 2020.
  13. 13.Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.
  14. 14.Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112, 2021.
  15. 15.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  16. 16.Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  17. 17.Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. arXiv preprint arXiv:1904.11491, 2019.
  18. 18.Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint arXiv:1912.11370, 6(2):8, 2019.
  19. 19.Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
  20. 20.Yann Lecun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient-based learning. In Contour and Grouping in Computer Vision. Springer, 1999.
  21. 21.Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  22. 22.Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  23. 23.Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
  24. 24.Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055–4064. PMLR, 2018.
  25. 25.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
  26. 26.Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389–5400. PMLR, 2019.
  27. 27.Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101.11605, 2021.
  28. 28.Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris Kitani. Rethinking transformer-based set prediction for object detection. arXiv preprint arXiv:2011.10881, 2020.
  29. 29.Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105–6114. PMLR, 2019.
  30. 30.Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877, 2020.
  31. 31.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
  32. 32.Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  33. 33.Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. arXiv preprint arXiv:2012.00759, 2020.
  34. 34.Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122, 2021.
  35. 35.Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  36. 36.Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint arXiv:2011.14503, 2020.
  37. 37.Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, and Yunhai Tong. Evolving attention with residual convolutions. arXiv preprint arXiv:2102.12895, 2021.
  38. 38.Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430, 2019.
  39. 39.Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. arXiv preprint arXiv:2004.11886, 2020.
  40. 40.Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5791–5800, 2020.
  41. 41.Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
  42. 42.Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint spatial-temporal transformations for video inpainting. In European Conference on Computer Vision, pages 528–543. Springer, 2020.
  43. 43.Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:2011.09315, 2020.
  44. 44.Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  45. 45.Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.

Citation

MLA
Wu, H., et al. “CvT: Introducing Convolutions to Vision Transformers”. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 22–31, https://doi.org/10.1109/ICCV48922.2021.00009.
APA
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021). CvT: Introducing Convolutions to Vision Transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 22–31. https://doi.org/10.1109/ICCV48922.2021.00009
Chicago
Wu, H., B. Xiao, N. Codella, et al. 2021. “CvT: Introducing Convolutions to Vision Transformers”. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 22–31. https://doi.org/10.1109/ICCV48922.2021.00009.
Harvard
Wu, H. et al. (2021) “CvT: Introducing Convolutions to Vision Transformers”, 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp. 22–31. Available at: https://doi.org/10.1109/ICCV48922.2021.00009.
Vancouver
1. Wu H, Xiao B, Codella N, Liu M, Dai X, Yuan L, Zhang L (2021) CvT: Introducing Convolutions to Vision Transformers. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, pp 22–31

BibTeX

@inproceedings{Wu_2021, title={CvT: Introducing Convolutions to Vision Transformers}, url={http://dx.doi.org/10.1109/ICCV48922.2021.00009}, DOI={10.1109/iccv48922.2021.00009}, booktitle={2021 IEEE/CVF International Conference on Computer Vision (ICCV)}, publisher={IEEE}, author={Wu, Haiping and Xiao, Bin and Codella, Noel and Liu, Mengchen and Dai, Xiyang and Yuan, Lu and Zhang, Lei}, year={2021}, month=Oct, pages={22–31} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/