A Survey on Vision Transformer

Kai HanYunhe WangHanting ChenXinghao ChenJianyuan GuoZhenhua LiuYehui TangAn XiaoChunjing XuYixing Xu

article2020TPAMI3,913 citations

Categorizes vision transformer models across high-level, low-level, and video processing tasks, analyzing their architectural trade-offs against convolutional networks and detailing efficient designs for real-world deployment.

Listen

A survey on visual transformers reviews the rapid adoption of transformer architectures, originally developed for natural language processing, across a wide range of computer vision tasks. The work addresses the challenge that the fast proliferation of new models has made it difficult for researchers and practitioners to track progress, compare approaches, or identify promising directions amid growing interest in alternatives to convolutional and recurrent networks.

The document sets out to deliver a structured overview of transformer-based models for vision by grouping them according to primary application areas, while also examining the underlying self-attention mechanism and methods for improving efficiency.

The authors conducted a literature review that focuses on representative early, pioneering, or influential works published through mid-2023. They draw on published results from standard benchmarks such as ImageNet for classification, COCO for detection, and various video and low-level vision datasets, and they organize findings into categories that include backbone networks, high- and mid-level vision, low-level vision, video processing, multi-modal tasks, and model compression techniques.

The review shows that pure transformer models such as ViT achieve competitive or superior accuracy to CNNs on image classification when pre-trained on large datasets, with top-1 accuracy on ImageNet reaching 88 percent after JFT-300M pre-training. Hybrid designs that combine convolutions with attention improve locality and training stability while retaining global modeling benefits. In object detection, set-prediction frameworks such as DETR and its deformable variant match or exceed established CNN detectors with fewer hand-crafted components, though they initially require longer training. Similar gains appear in segmentation, pose estimation, image generation, and video tasks, where transformers capture long-range spatial and temporal dependencies more directly than prior architectures. Efficiency methods, including pruning, distillation, quantization, and sparse attention, reduce memory and compute demands but still lag behind the most compact CNNs.

These results indicate that transformers can serve as a flexible backbone or end-to-end solution for many vision problems, often with reduced reliance on vision-specific inductive biases. The shift enables more unified modeling across tasks and modalities, yet it also raises questions about data efficiency, robustness, and deployment on resource-limited devices.

Further work is needed to design task-specific architectures, develop large-scale pre-training strategies comparable to those in language models, and create more effective compression techniques. Hybrid CNN-transformer models currently offer the strongest accuracy-efficiency trade-offs, but purely attention-based or even simpler feed-forward designs may prove viable with continued scaling.

The survey is necessarily selective and cannot encompass every preprint; the field continues to evolve quickly, so some performance numbers and architectural details may already be superseded by newer results. The synthesis rests on the cited empirical evidence and provides a reliable high-level map for decision makers evaluating transformer adoption.

Huawei Peking University University of Sydney attention mechanism computer vision encoder-decoder vision transformer Image Processing Transformer transformer-based object detectors

arXiv: 2012.12556

Paper: Attention Is All You Need, Ashish Vaswani et al. (2017). Reading the original foundational paper on the Transformer architecture is essential for understanding the self-attention mechanisms that vision transformers build upon.
Paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy et al. (2021). This seminal work introducing the Vision Transformer (ViT) directly establishes the core methodology and baseline model that the survey reviews and categorizes.

Paper: Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions, Wenhai Wang et al. (2021). This paper extends the survey's discussed vision transformer concepts by introducing a hierarchical pyramid architecture to overcome resolution limits in dense prediction tasks.
Paper: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, Ze Liu et al. (2021). This work continues the evolution of vision transformers by developing shifted-window self-attention, serving as a direct architectural continuation of the models reviewed in the survey.

Cover for A Survey on Vision Transformer

Abstract

Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism. Thanks to its strong representation capabilities, researchers are looking at ways to apply transformer to computer vision tasks. In a variety of visual benchmarks, transformer-based models perform similar to or better than other types of networks such as convolutional and recurrent neural networks. Given its high performance and less need for vision-specific inductive bias, transformer is receiving more and more attention from the computer vision community. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages. The main categories we explore include the backbone network, high/mid-level vision, low-level vision, and video processing. We also include efficient transformer methods for pushing transformer into real device-based applications. Furthermore, we also take a brief look at the self-attention mechanism in computer vision, as it is the base component in transformer. Toward the end of this paper, we discuss the challenges and provide several further research directions for vision transformers.

Topics

attention mechanism computer vision encoder-decoder natural language processing recurrent neural networks video processing convolutional neural networks

Keywords

vision transformer Image Processing Transformer transformer-based object detectors Feature Pyramid Transformer Texture transformer Point Transformer Adaptive Clustering Transformer spatiotemporal transformer Shuffle Transformer computer vision tasks

A Survey on Visual Transformer
1 INTRODUCTION
2 FORMULATION OF TRANSFORMER
2.1 Self-Attention
2.2 Other Key Concepts in Transformer
3 VISION TRANSFORMER
3.1 Backbone for Representation Learning
3.1.1 Pure Transformer
3.1.2 Transformer with Convolution
3.1.3 Self-supervised Representation Learning
3.1.4 Discussions
3.2 High/Mid-level Vision
3.2.1 Generic Object Detection
3.2.2 Segmentation
3.2.3 Pose Estimation
3.2.4 Other Tasks
3.2.5 Discussions
3.3 Low-level Vision
3.3.1 Image Generation
3.3.2 Image Processing
3.4 Video Processing
3.4.1 High-level Video Processing
3.4.2 Low-level Video Processing
3.4.3 Discussions
3.5 Multi-Modal Tasks
3.6 Efficient Transformer
3.6.1 Pruning and Decomposition
3.6.2 Knowledge Distillation
3.6.3 Quantization
3.6.4 Compact Architecture Design
4 CONCLUSIONS AND DISCUSSIONS
4.1 Challenges
4.2 Future Prospects
A3. Self-attention for Computer Vision

Knowls

Knowl 1 — Categorization and Structural Taxonomy of Vision Transformers
model/method
Vision transformer architectures are classified across five core computer vision domains based on architectural topology and downstream tasks:
1. Backbone Networks for Representation Learning: Categorized into pure transformers (e.g., ViT, DeiT, Swin, TNT, PVT) and hybrid CNN-Transformer architectures (e.g., BoTNet, CeiT, CMT, VOLO), trained via supervised classification or self-supervised learning (e.g., iGPT, MoCo v3, MAE, SimMIM).
2. High- and Mid-Level Vision: Encompasses generic object detection (direct set-prediction frameworks such as DETR, Deformable DETR, and transformer-backbone detectors), segmentation (panoptic segmentation like Max-DeepLab, instance segmentation like VisTR and ISTR, semantic segmentation like SETR and SegFormer, and medical segmentation like Swin-Unet), pose estimation (e.g., METRO, HOT-Net, Hand-Transformer), tracking (TransTrack, TransT), re-identification (TransReID), and 3D point cloud understanding (Point Transformer, PCT).
3. Low-Level Vision: Divided into generative models (Image Transformer, TransGAN, ViTGAN, Taming Transformer, SceneFormer) and image restoration/processing (Texture Transformer Network for Super-Resolution [TTSR], Image Processing Transformer [IPT]).
4. Video Processing: Comprises high-level video understanding (action recognition, video retrieval, video object detection, untrimmed video multi-task learning) and low-level video processing (frame/video synthesis with ConvTransformer, video inpainting with spatial-temporal transformers).
5. Multi-Modal and Efficient Transformers: Encompasses cross-modal representation learning across text, audio, and vision (e.g., CLIP, DALL-E, CogView, UniT) as well as model compression and acceleration strategies (pruning, low-rank decomposition, knowledge distillation, quantization, and compact neural architecture search).
Knowl 2 — Vision Transformer (ViT) Architecture and Patch Embedding Formulation
model/method
Vision Transformer (ViT) applies a standard transformer encoder directly to non-overlapping image patches for image classification. Given an input 2D image $X \in \mathbb{R}^{h \times w \times c}$ , where $(h, w)$ denotes spatial resolution and $c$ denotes the channel dimension, the image is reshaped into a sequence of flattened 2D patches $X_p \in \mathbb{R}^{n \times (p^2 \cdot c)}$ , where $(p, p)$ is the patch resolution and $n = hw/p^2$ is the effective sequence length.

A trainable linear projection maps each flattened patch vector to a model dimension $d$ , yielding patch embeddings. A learnable class token embedding ( $[\text{class}]$ ) is prepended to the sequence, and 1D learnable positional embeddings are added to all token embeddings to preserve spatial information. The full sequence of length $n + 1$ is then processed by stacked standard transformer encoder layers consisting of Multi-Head Self-Attention (MSA) and Feed-Forward Networks (FFN) with Pre-Layer Normalization (Pre-LN) and residual connections:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V$

$\text{FFN}(X) = W_2 \sigma(W_1 X)$

where $W_1 \in \mathbb{R}^{d_h \times d}$ , $W_2 \in \mathbb{R}^{d \times d_h}$ , and $\sigma$ is the GELU non-linear activation. The state of the $[\text{class}]$ token at the final layer output is fed into an MLP head to compute classification logits.

Knowl 3 — ImageNet Performance Benchmark of CNNs versus Pure and Hybrid Vision Transformers

data/table

ImageNet-1K classification benchmark comparing representative Convolutional Neural Networks (CNNs), pure Vision Transformers, and hybrid CNN + Transformer models evaluated with $224 \times 224$ input resolution. Throughput is measured on an NVIDIA Tesla V100 GPU in PyTorch.

Model	Params (M)	FLOPs (B)	Throughput (image/s)	Top-1 (%)
CNN
ResNet-50	25.6	4.1	1226	79.1
ResNet-101	44.7	7.9	753	79.9
ResNet-152	60.2	11.5	526	80.8
EfficientNet-B0	5.3	0.39	2694	77.1
EfficientNet-B1	7.8	0.70	1662	79.1
EfficientNet-B2	9.2	1.0	1255	80.1
EfficientNet-B3	12.0	1.8	732	81.6
EfficientNet-B4	19.0	4.2	349	82.9
Pure Transformer
DeiT-Ti	5.0	1.3	2536	72.2
DeiT-S	22.0	4.6	940	79.8
DeiT-B	86.0	17.6	292	81.8
T2T-ViT-14	21.5	5.2	764	81.5
T2T-ViT-19	39.2	8.9	464	81.9
T2T-ViT-24	64.1	14.1	312	82.3
PVT-Small	24.5	3.8	820	79.8
PVT-Medium	44.2	6.7	526	81.2
PVT-Large	61.4	9.8	367	81.7
TNT-S	23.8	5.2	428	81.5
TNT-B	65.6	14.1	246	82.9
CPVT-S	23.0	4.6	930	80.5
CPVT-B	88.0	17.6	285	82.3
Swin-T	29.0	4.5	755	81.3
Swin-S	50.0	8.7	437	83.0
Swin-B	88.0	15.4	278	83.3
CNN + Transformer
Twins-SVT-S	24.0	2.9	1059	81.7
Twins-SVT-B	56.0	8.6	469	83.2
Twins-SVT-L	99.2	15.1	288	83.7
Shuffle-T	29.0	4.6	791	82.5
Shuffle-S	50.0	8.9	450	83.5
Shuffle-B	88.0	15.6	279	84.0
CMT-S	25.1	4.0	563	83.5
CMT-B	45.7	9.3	285	84.5
VOLO-D1	27.0	6.8	481	84.2
VOLO-D2	59.0	14.1	244	85.2
VOLO-D3	86.0	20.6	168	85.4
VOLO-D4	193.0	43.8	100	85.7
VOLO-D5	296.0	69.0	64	86.1

The data shows that pure vision transformers without convolutions match or exceed standard ResNet baselines when equipped with adequate regularization or data-efficient distillation (e.g., DeiT). Combining convolutional modules with transformer self-attention layers (hybrid models such as CMT and VOLO) achieves the best trade-off between computational cost and Top-1 accuracy, with VOLO-D5 reaching 86.1%.

Knowl 4 — End-to-End Object Detection with Transformers (DETR) and Hungarian Loss
model/method
DETR (DEtection TRansformer) reformulates object detection as a direct set-prediction problem, replacing hand-crafted anchors and Non-Maximum Suppression (NMS) with an end-to-end transformer pipeline. Feature maps extracted by a CNN backbone are flattened, augmented with fixed positional encodings, and processed by a transformer encoder. A transformer decoder receives $N$ learned positional embeddings (termed object queries) and attends to the encoder features, decoding $N$ predicted bounding boxes and class labels in parallel.

To align predicted objects $\hat{y} = \{\hat{y}_i\}_{i=1}^N$ with ground-truth objects $y = \{(c_i, b_i)\}_{i=1}^N$ (padded with background $\emptyset$ tokens to size $N$ ), DETR computes an optimal bipartite matching permutation $\hat{\sigma} \in \mathfrak{S}_N$ via the Hungarian algorithm. The network parameters are optimized by minimizing the Hungarian loss:

$\mathcal{L}_{\text{Hungarian}}(y, \hat{y}) = \sum_{i=1}^N \left[ -\log \hat{p}_{\hat{\sigma}(i)}(c_i) + \mathbb{1}_{\{c_i \neq \emptyset\}} \mathcal{L}_{\text{box}}(b_i, \hat{b}_{\hat{\sigma}(i)}) \right]$

where $c_i$ is the ground-truth target class, $\hat{p}_{\hat{\sigma}(i)}(c_i)$ is the predicted probability for class $c_i$ , $b_i \in [0, 1]^4$ and $\hat{b}_{\hat{\sigma}(i)}$ denote normalized bounding box coordinates and sizes, and $\mathcal{L}_{\text{box}}$ is a linear combination of $\ell_1$ loss and Generalized IoU (GIoU) loss.

Knowl 5 — COCO Object Detection Benchmark Comparison for Transformer Detectors

data/table

Object detection benchmark on the COCO 2017 validation dataset comparing CNN baselines, CNN backbones with transformer prediction heads, transformer backbones with CNN detection heads, and pure transformer detectors. Running speed (FPS) is evaluated on an NVIDIA Tesla V100 GPU.

Method	Epochs	AP	$\text{AP}_{50}$	$\text{AP}_{75}$	$\text{AP}_S$	$\text{AP}_M$	$\text{AP}_L$	Params (M)	GFLOPs
CNN based
FCOS	36	41.0	59.8	44.1	26.2	44.6	52.2	-	177
Faster R-CNN + FPN	109	42.0	62.1	45.5	26.6	45.4	53.4	42	180
CNN Backbone + Transformer Head
DETR	500	42.0	62.4	44.2	20.5	45.8	61.1	41	86
DETR-DC5	500	43.3	63.1	45.9	22.5	47.3	61.1	41	187
Deformable DETR	50	46.2	65.2	50.0	28.8	49.2	61.7	40	173
TSP-FCOS	36	43.1	62.3	47.0	26.6	46.8	55.9	-	189
TSP-RCNN	96	45.0	64.5	49.6	29.7	47.7	58.0	-	188
SMCA	108	45.6	65.5	49.1	25.9	49.3	62.6	-	-
Efficient DETR	36	45.1	63.1	49.1	28.3	48.4	59.0	35	210
UP-DETR (150 ep)	150	40.5	60.8	42.6	19.0	44.4	60.0	41	-
UP-DETR (300 ep)	300	42.8	63.0	45.3	20.8	47.1	61.7	41	-
Transformer Backbone + CNN Head
ViT-B/16-FRCNN (IN-21k)	21	36.6	56.3	39.3	17.4	40.0	55.5	-	-
ViT-B/16-FRCNN (1.3B)	21	37.8	57.4	40.1	17.8	41.4	57.3	-	-
PVT-Small + RetinaNet	12	40.4	61.3	43.0	25.0	42.9	55.7	34.2	118
Twins-SVT-S + RetinaNet	12	43.0	64.2	46.3	28.0	46.4	57.5	34.3	104
Swin-T + RetinaNet	12	41.5	62.1	44.2	25.1	44.9	55.5	38.5	118
Swin-T + ATSS	36	47.2	66.5	51.3	-	-	-	36	215
Pure Transformer based
PVT-Small + DETR	50	34.7	55.7	35.4	12.0	36.4	56.7	40	-
TNT-S + DETR	50	38.2	58.9	39.4	15.5	41.1	58.8	39	-
YOLOS-Ti	300	30.0	-	-	-	-	-	6.5	21
YOLOS-S	150	37.6	57.6	39.2	15.9	40.2	57.3	28	179
YOLOS-B	150	42.0	62.2	44.5	19.5	45.3	62.1	127	537

The table demonstrates that standard DETR requires long training schedules (500 epochs) and suffers from weak performance on small objects ( $\text{AP}_S = 20.5$ ). Deformable DETR resolves both limitations by restricting attention to local reference sampling points, reaching $46.2$ AP in only 50 epochs ( $10\times$ training acceleration). Swin-T paired with ATSS achieves the highest overall accuracy among compact backbones ( $47.2$ AP).

Knowl 6 — Self-Supervised Learning Paradigms for Vision Transformers: Generative and Contrastive
model/method
Self-supervised pre-training enables vision transformers to learn generalizable visual representations without manual annotations via two main formulations:
1. Generative Pixel Prediction (iGPT / MAE): Images are serialized into token sequences $x = (x_1, \dots, x_n)$ via clustered palette indices or patch embeddings. Training optimizes the Auto-Regressive (AR) negative log-likelihood:
$\mathcal{L}_{\text{AR}} = \mathbb{E}_{x \sim X} [-\log p(x)] = \mathbb{E}_{x \sim X} \left[ -\sum_{i=1}^n \log p(x_{\pi_i} \mid x_{\pi_1}, \dots, x_{\pi_{i-1}}, \theta) \right]$

with raster permutation $\pi_i = i$ , or the BERT-style masked prediction loss across randomly masked indices $M \subset [1, n]$ (with sampling probability $0.15$ ):

$\mathcal{L}_{\text{BERT}} = \mathbb{E}_{x \sim X} \mathbb{E}_{M} \left[ \sum_{i \in M} -\log p(x_i \mid x_{[1, n] \setminus M}) \right]$
1. Contrastive Representation Learning (MoCo v3): Two augmented views of an image generate query representation $q$ from online encoder $f_q$ and key representation $k$ from momentum encoder $f_k$ . The model minimizes the InfoNCE contrastive loss:
$\mathcal{L}_q = -\log \frac{\exp(q \cdot k^+ / \tau)}{\exp(q \cdot k^+ / \tau) + \sum_{k^-} \exp(q \cdot k^- / \tau)}$

where $k^+$ is the positive key from the same image, $\{k^-\}$ are negative keys from other batch images, and $\tau$ is a temperature hyperparameter. To address severe training instability in self-supervised ViT, MoCo v3 freezes the random linear patch projection layer during training.
Knowl 7 — Image Processing Transformer (IPT) for Multi-Task Low-Level Vision
model/method
The Image Processing Transformer (IPT) provides a unified architecture for solving multiple low-level image processing tasks—such as super-resolution ( $\times 2, \times 4$ ), denoising, and deraining—using a single shared model.

IPT utilizes a multi-head, multi-tail structure wrapped around a shared transformer encoder-decoder backbone. Degraded input images pass through task-specific heads that extract shallow feature representations. These features are unrolled into patch sequences and paired with task-specific embeddings before being fed into the shared encoder-decoder. The transformer encoder-decoder models long-range spatial context across patches, and its output is reshaped and channeled into corresponding task-specific tails for final image reconstruction.

IPT is pre-trained on ImageNet using synthetically corrupted images (degraded with added noise, rain streaks, or downsampling) with the original clean images as the reconstruction target, combined with contrastive self-supervised losses. When fine-tuned on target downstream tasks, IPT achieves substantial performance gains over CNN baselines (up to 2 dB improvement in image denoising).
Knowl 8 — Equivalence of Transformer Self-Attention to Non-Local Filtering Operations in Computer Vision
theoretical result
The self-attention mechanism in transformers is functionally equivalent to non-local filtering operations used for long-range spatial modeling in computer vision. Given an input feature map $X \in \mathbb{R}^{n \times d}$ , where $n = h \times w$ is the number of spatial positions and $d$ is channel dimension, a general non-local filtering operation computes output position $y_i \in \mathbb{R}^{1 \times d}$ as:

$y_i = \frac{1}{C(x_i)} \sum_{\forall j} f(x_i, x_j) g(x_j)$

where $f(x_i, x_j)$ evaluates pairwise feature similarity, $g(x_j)$ represents the input signal at position $j$ , and $C(x_i) = \sum_{\forall j} f(x_i, x_j)$ is a normalization factor.

When linear embedding projections are defined as $\theta(X) = X W_\theta$ , $\phi(X) = X W_\phi$ , and $g(X) = X W_g$ with $W_\theta, W_\phi \in \mathbb{R}^{d \times d_k}$ and $W_g \in \mathbb{R}^{d \times d_v}$ , and similarity is modeled by the exponential dot product $f(x_i, x_j) = \exp(\theta(x_i) \phi(x_j)^\top)$ , the operation in matrix notation becomes:

$Y = \text{softmax}(X W_\theta W_\phi^\top X^\top) X W_g$

Setting $Q = X W_q$ , $K = X W_k$ , and $V = X W_v$ with $W_q = W_\theta$ , $W_k = W_\phi$ , and $W_v = W_g$ directly recovers scaled dot-product attention:

$Y = \text{softmax}(Q K^\top) V = \text{Attention}(Q, K, V)$

In vision models, the non-local/self-attention block is wrapped with a linear projection $W^o$ and a residual shortcut: $Z = Y W^o + X$ . Setting $W^o = 0$ at initialization allows the module to be inserted into any architecture without altering its initial behavior.

Knowl 9 — Transformer Model Compression and Acceleration Benchmarks

data/table

Quantitative comparison of transformer compression and acceleration methods across model pruning, matrix decomposition, knowledge distillation, and network quantization.

Model	Compression Type	#Layers	Parameters	Speedup
$\text{BERT}_{\text{BASE}}$	Baseline	12	110M	$\times 1.0$
ALBERT	Decomposition	12	12M	$\times 5.6$
BERT-of-Theseus	Architecture Design	6	66M	$\times 1.94$
Q-BERT	Quantization	12	-	-
Q8BERT	Quantization (8-bit)	12	-	-
TinyBERT	Distillation	4	14.5M	$\times 9.4$
DistilBERT	Distillation	6	6.6M	$\times 1.63$
BERT-PKD	Distillation	3–6	45.7–67M	$\times 3.73$ – $\times 1.64$
MobileBERT	Distillation	24	25.3M	$\times 4.0$
Patient Distillation (PD)	Distillation	6	67.5M	$\times 2.0$

Compression techniques address the quadratic memory and latency costs ( $O(N^2)$ attention complexity) of vision and language transformers:

Pruning & Decomposition: Redundant attention heads and transformer layers are removed based on sensitivity/importance scores, while parameter matrices are factorized via low-rank decomposition (e.g., ALBERT achieves $\times 5.6$ parameter reduction).
Knowledge Distillation: Student networks mimic teacher representations across self-attention maps, hidden state embeddings, and output distributions (e.g., TinyBERT achieves a $\times 9.4$ inference speedup with 14.5M parameters).
Quantization: Network weights and feature representations are quantized to low-bit representations (such as 8-bit or 4-bit) without degrading task performance.
Efficient Attention Mechanisms: Linear attention and sparse graph approximations reduce the computation of self-attention from $O(N^2)$ to $O(N)$ with respect to sequence length $N$ .

Knowl 10 — Fundamental Limitations and Open Research Challenges of Vision Transformers
limitation
Vision transformers exhibit several fundamental constraints compared to traditional Convolutional Neural Networks (CNNs):
1. Absence of Vision-Specific Inductive Biases: Pure transformers lack inherent translational equivariance and local spatial inductive biases. Consequently, they underperform CNNs on small- to mid-sized training datasets and rely strictly on massive datasets (e.g., ImageNet-21k, JFT-300M) or heavy data augmentation and distillation (e.g., DeiT) to reach competitive generalization.
2. High Computational Complexity for Dense Predictions: Standard full self-attention scales quadratically $O(N^2)$ with the number of image tokens $N = hw/p^2$ . Processing high-resolution inputs in dense prediction tasks (e.g., object detection, semantic segmentation, image generation) incurs extreme FLOP and memory overhead (e.g., base ViT requires 18B FLOPs per image compared to ~600M FLOPs for lightweight CNNs such as GhostNet).
3. Training Instability and Hyperparameter Sensitivity: Vision transformers are sensitive to the choice of optimizers (often failing to train stably with SGD without convolutional stems), learning rate schedules, and layer normalization configurations.
4. Limited Architectural Specialization: Direct adaptations of NLP encoder structures (such as columnar ViT) transfer suboptimally to dense multi-scale vision tasks unless coupled with hierarchical pyramids, shifted windows, or hybrid convolutional blocks.

Coverage note — All primary taxonomies, mathematical formulations (ViT patch embedding, DETR bipartite matching loss, iGPT/MoCo v3 self-supervised losses, non-local attention equivalence), quantitative benchmark comparisons (ImageNet classification, COCO object detection, transformer compression), and synthesized analyses of challenges/limitations have been extracted. Individual task-specific papers summarized in passing throughout the survey (such as specific lane detectors, tracking algorithms, or NLP language models from related work) are omitted in favor of the foundational models and primary benchmark results.

References

1.F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.
2.F. ROSENBLATT. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Technical report, 1961.
3.Y. LeCun et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
4.A. Krizhevsky et al. Imagenet classification with deep convolutional neural networks. In NeurIPS, pp. 1097–1105, 2012.
5.D. E. Rumelhart et al. Learning internal representations by error propagation. Technical report, 1985.
6.S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
7.D. Bahdanau et al. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
8.A. Parikh et al. A decomposable attention model for natural language inference. In EMNLP, 2016.
9.A. Vaswani et al. Attention is all you need. In NeurIPS, 2017.
10.J. Devlin et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
11.T. B. Brown et al. Language models are few-shot learners. In NeurIPS, 2020.
12.K. He et al. Deep residual learning for image recognition. In CVPR, pp. 770–778, 2016.
13.S. Ren et al. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
14.M. Chen et al. Generative pretraining from pixels. In ICML, 2020.
15.A. Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
16.N. Carion et al. End-to-end object detection with transformers. In ECCV, 2020.
17.X. Zhu et al. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
18.S. Zheng et al. Rethinking semantic segmentation from a sequence-tosequence perspective with transformers. In CVPR, 2021.
19.H. Chen et al. Pre-trained image processing transformer. In CVPR, 2021.
20.L. Zhou et al. End-to-end dense video captioning with masked transformer. In CVPR, pp. 8739–8748, 2018.
21.S. Ullman et al. High-level vision: Object recognition and visual cognition, volume 2. MIT press Cambridge, MA, 1996.
22.R. Kimchi et al. Perceptual organization in vision: Behavioral and neural perspectives. Psychology Press, 2003.
23.J. Zhu et al. Top-down saliency detection via contextual pooling. Journal of Signal Processing Systems, 74(1):33–46, 2014.
24.J. Long et al. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
25.H. Wang et al. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR, pp. 5463–5474, 2021.
26.R. B. Fisher. Cvonline: The evolving, distributed, non-proprietary, online compendium of computer vision. Retrieved January 28, 2006 from http://homepages. inf. ed. ac. uk/rbf/CVonline, 2008.
27.N. Parmar et al. Image transformer. In ICML, 2018.
28.Y. Zeng et al. Learning joint spatial-temporal transformations for video inpainting. In ECCV, pp. 528–543. Springer, 2020.
29.K. Han et al. Transformer in transformer. In NeurIPS, 2021.
30.H. Cao et al. Swin-unet: Unet-like pure transformer for medical image segmentation. arXiv:2105.05537, 2021.
31.X. Chen et al. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
32.K. He et al. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000–16009, 2022.
33.Z. Dai et al. UP-DETR: unsupervised pre-training for object detection with transformers. In CVPR, 2021.
34.Y. Wang et al. End-to-end video instance segmentation with transformers. In CVPR, 2021.
35.L. Huang et al. Hand-transformer: Non-autoregressive structured modeling for 3d hand pose estimation. In ECCV, pp. 17–33, 2020.
36.L. Huang et al. Hot-net: Non-autoregressive transformer for 3d handobject pose estimation. In ACM MM, pp. 3136–3145, 2020.
37.K. Lin et al. End-to-end human pose and mesh reconstruction with transformers. In CVPR, 2021.
38.P. Esser et al. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
39.Y. Jiang et al. Transgan: Two transformers can make one strong gan. In NeurIPS, 2021.
40.F. Yang et al. Learning texture transformer network for image superresolution. In CVPR, pp. 5791–5800, 2020.
41.A. Radford et al. Learning transferable visual models from natural language supervision. arXiv:2103.00020, 2021.
42.A. Ramesh et al. Zero-shot text-to-image generation. In ICML, 2021.
43.M. Ding et al. Cogview: Mastering text-to-image generation via transformers. In NeurIPS, 2021.
44.OpenAI. Gpt-4 technical report, 2023.
45.P. Michel et al. Are sixteen heads really better than one? In NeurIPS, pp. 14014–14024, 2019.
46.X. Jiao et al. TinyBERT: Distilling BERT for natural language understanding. In Findings of EMNLP, pp. 4163–4174, 2020.
47.G. Prato et al. Fully quantized transformer for machine translation. In Findings of EMNLP, 2020.
48.Z.-H. Jiang et al. Convbert: Improving bert with span-based dynamic convolution. NeurIPS, 33, 2020.
49.J. Gehring et al. Convolutional sequence to sequence learning. In ICML, pp. 1243–1252. PMLR, 2017.
50.P. Shaw et al. Self-attention with relative position representations. In NAACL, pp. 464–468, 2018.
51.D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv:1606.08415, 2016.
52.J. L. Ba et al. Layer normalization. arXiv:1607.06450, 2016.
53.A. Baevski and M. Auli. Adaptive input representations for neural language modeling. In ICLR, 2019.
54.Q. Wang et al. Learning deep transformer models for machine translation. In ACL, pp. 1810–1822, 2019.
55.S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
56.S. Shen et al. Powernorm: Rethinking batch normalization in transformers. In ICML, 2020.
57.J. Xu et al. Understanding and improving layer normalization. In NeurIPS, 2019.
58.T. Bachlechner et al. Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1352–1361. PMLR, 2021.
59.B. Wu et al. Visual transformers: Token-based image representation and processing for computer vision. arXiv:2006.03677, 2020.
60.H. Touvron et al. Training data-efficient image transformers & distillation through attention. In ICML, 2020.
61.Z. Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
62.C.-F. Chen et al. Regionvit: Regional-to-local attention for vision transformers. arXiv:2106.02689, 2021.
63.X. Chu et al. Twins: Revisiting the design of spatial attention in vision transformers. arXiv:2104.13840, 2021.
64.H. Lin et al. Cat: Cross attention in vision transformer. arXiv, 2021.
65.X. Dong et al. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv:2107.00652, 2021.
66.Z. Huang et al. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv:2106.03650, 2021.
67.J. Fang et al. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. arXiv:2105.15168, 2021.
68.L. Yuan et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
69.D. Zhou et al. Deepvit: Towards deeper vision transformer. arXiv, 2021.
70.P. Wang et al. Kvt: k-nn attention for boosting vision transformers. arXiv:2106.00515, 2021.
71.D. Zhou et al. Refiner: Refining self-attention for vision transformers. arXiv:2106.03714, 2021.
72.A. El-Nouby et al. Xcit: Cross-covariance image transformers. arXiv:2106.09681, 2021.
73.W. Wang et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
74.S. Sun* et al. Visual parser: Representing part-whole hierarchies with transformers. arXiv:2107.05790, 2021.
75.H. Fan et al. Multiscale vision transformers. arXiv:2104.11227, 2021.
76.Z. Zhang et al. Nested hierarchical transformer: Towards accurate, dataefficient and interpretable visual understanding. In AAAI, 2022.
77.Z. Pan et al. Less is more: Pay less attention in vision transformers. In AAAI, 2022.
78.Z. Pan et al. Scalable visual transformers with hierarchical pooling. In ICCV, 2021.
79.B. Heo et al. Rethinking spatial dimensions of vision transformers. In ICCV, 2021.
80.C.-F. Chen et al. Crossvit: Cross-attention multi-scale vision transformer for image classification. In ICCV, 2021.
81.Z. Wang et al. Uformer: A general u-shaped transformer for image restoration. arXiv:2106.03106, 2021.
82.X. Zhai et al. Scaling vision transformers. arXiv:2106.04560, 2021.
83.X. Su et al. Vision transformer architecture search. arXiv, 2021.
84.M. Chen et al. Autoformer: Searching transformers for visual recognition. In ICCV, pp. 12270–12280, 2021.
85.B. Chen et al. Glit: Neural architecture search for global and local image transformer. In ICCV, pp. 12–21, 2021.
86.X. Chu et al. Conditional positional encodings for vision transformers. arXiv:2102.10882, 2021.
87.K. Wu et al. Rethinking and improving relative position encoding for vision transformer. In ICCV, 2021.
88.H. Touvron et al. Going deeper with image transformers. arXiv:2103.17239, 2021.
89.Y. Tang et al. Augmented shortcuts for vision transformers. In NeurIPS, 2021.
90.I. Tolstikhin et al. Mlp-mixer: An all-mlp architecture for vision. arXiv:2105.01601, 2021.
91.L. Melas-Kyriazi. Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet. arXiv:2105.02723, 2021.
92.M.-H. Guo et al. Beyond self-attention: External attention using two linear layers for visual tasks. arXiv:2105.02358, 2021.
93.H. Touvron et al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv:2105.03404, 2021.
94.M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
95.J. Guo et al. Cmt: Convolutional neural networks meet vision transformers. arXiv:2107.06263, 2021.
96.L. Yuan et al. Volo: Vision outlooker for visual recognition. arXiv:2106.13112, 2021.
97.H. Wu et al. Cvt: Introducing convolutions to vision transformers. arXiv:2103.15808, 2021.
98.K. Yuan et al. Incorporating convolution designs into visual transformers. arXiv:2103.11816, 2021.
99.Y. Li et al. Localvit: Bringing locality to vision transformers. arXiv:2104.05707, 2021.
100.B. Graham et al. Levit: a vision transformer in convnet’s clothing for faster inference. In ICCV, 2021.
101.A. Srinivas et al. Bottleneck transformers for visual recognition. In CVPR, 2021.
102.Z. Chen et al. Visformer: The vision-friendly transformer. arXiv, 2021.
103.T. Xiao et al. Early convolutions help transformers see better. In NeurIPS, volume 34, 2021.
104.G. E. Hinton and R. S. Zemel. Autoencoders, minimum description length, and helmholtz free energy. NIPS, 6:3–10, 1994.
105.P. Vincent et al. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096–1103, 2008.
106.A. v. d. Oord et al. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05328, 2016.
107.D. Pathak et al. Context encoders: Feature learning by inpainting. In CVPR, pp. 2536–2544, 2016.
108.Z. Li et al. Mst: Masked self-supervised transformer for visual representation. In NeurIPS, 2021.
109.H. Bao et al. Beit: Bert pre-training of image transformers. arXiv:2106.08254, 2021.
110.A. Radford et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
111.Z. Xie et al. Simmim: A simple framework for masked image modeling. In CVPR, pp. 9653–9663, 2022.
112.Z. Xie et al. Self-supervised learning with swin transformers. arXiv:2105.04553, 2021.
113.C. Li et al. Efficient self-supervised vision transformers for representation learning. arXiv:2106.09785, 2021.
114.K. He et al. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
115.J. Beal et al. Toward transformer-based object detection. arXiv:2012.09958, 2020.
116.Z. Yuan et al. Temporal-channel transformer for 3d lidar-based video object detection for autonomous driving. IEEE TCSVT, 2021.
117.X. Pan et al. 3d object detection with pointformer. In CVPR, 2021.
118.R. Liu et al. End-to-end lane shape prediction with transformers. In WACV, 2021.
119.S. Yang et al. Transpose: Keypoint localization via transformer. In ICCV, 2021.
120.D. Zhang et al. Feature pyramid transformer. In ECCV, 2020.
121.C. Chi et al. Relationnet++: Bridging visual representations for object detection via transformer decoder. NeurIPS, 2020.
122.Z. Sun et al. Rethinking transformer-based set prediction for object detection. In ICCV, pp. 3611–3620, 2021.
123.M. Zheng et al. End-to-end object detection with adaptive clustering transformer. In BMVC, 2021.
124.T. Ma et al. Oriented object detection with transformer. arXiv:2106.03146, 2021.
125.P. Gao et al. Fast convergence of detr with spatially modulated coattention. In ICCV, 2021.
126.Z. Yao et al. Efficient detr: Improving end-to-end object detector with dense prior. arXiv:2104.01318, 2021.
127.Z. Tian et al. Fcos: Fully convolutional one-stage object detection. In ICCV, pp. 9627–9636, 2019.
128.Y. Fang et al. You only look at one sequence: Rethinking transformer in vision through object detection. In NeurIPS, 2021.
129.T.-Y. Lin et al. Focal loss for dense object detection. In ICCV, 2017.
130.Z. Cai and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018.
131.A. Bar et al. Detreg: Unsupervised pretraining with region priors for object detection. arXiv:2106.04550, 2021.
132.J. Hu et al. Istr: End-to-end instance segmentation with transformers. arXiv:2105.00637, 2021.
133.Z. Yang et al. Associating objects with transformers for video object segmentation. In NeurIPS, 2021.
134.S. Wu et al. Fully transformer networks for semantic image segmentation. arXiv:2106.04108, 2021.
135.B. Dong et al. Solq: Segmenting objects by learning queries. In NeurIPS, 2021.
136.R. Strudel et al. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
137.E. Xie et al. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
138.J. M. J. Valanarasu et al. Medical transformer: Gated axial-attention for medical image segmentation. In MICCAI, 2021.
139.T. Prangemeier et al. Attention-based transformers for instance segmentation of cells in microstructures. In International Conference on Bioinformatics and Biomedicine, pp. 700–707. IEEE, 2020.
140.C. R. Qi et al. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pp. 652–660, 2017.
141.C. R. Qi et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS, 30:5099–5108, 2017.
142.S. Hampali et al. Handsformer: Keypoint transformer for monocular 3d pose estimation ofhands and object in interaction. arXiv, 2021.
143.Y. Li et al. Tokenpose: Learning keypoint tokens for human pose estimation. In ICCV, 2021.
144.W. Mao et al. Tfpose: Direct human pose estimation with transformers. arXiv:2103.15320, 2021.
145.T. Jiang et al. Skeletor: Skeletal transformers for robust body-pose estimation. In CVPR, 2021.
146.Y. Li et al. Test-time personalization with a transformer for human pose estimation. Advances in Neural Information Processing Systems, 34, 2021.
147.M. Lin et al. Detr for pedestrian detection. arXiv:2012.06785, 2020.
148.L. Tabelini et al. Polylanenet: Lane estimation via deep polynomial regression. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 6150–6156. IEEE, 2021.
149.L. Liu et al. Condlanenet: a top-to-down lane detection framework based on conditional convolution. arXiv:2105.05003, 2021.
150.P. Xu et al. A survey of scene graph: Generation and application. IEEE Trans. Neural Netw. Learn. Syst, 2020.
151.J. Yang et al. Graph r-cnn for scene graph generation. In ECCV, 2018.
152.S. Sharifzadeh et al. Classification by attention: Scene graph classification with prior knowledge. In AAAI, 2021.
153.S. Sharifzadeh et al. Improving Visual Reasoning by Exploiting The Knowledge in Texts. arXiv:2102.04760, 2021.
154.C. Raffel et al. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67, 2020.
155.N. Wang et al. Transformer meets tracker: Exploiting temporal context for robust visual tracking. In CVPR, 2021.
156.M. Zhao et al. TrTr: Visual Tracking with Transformer. arXiv:2105.03817 [cs], May 2021. arXiv: 2105.03817.
157.X. Chen et al. Transformer tracking. In CVPR, 2021.
158.P. Sun et al. TransTrack: Multiple Object Tracking with Transformer. arXiv:2012.15460 [cs], May 2021. arXiv: 2012.15460.
159.S. He et al. TransReID: Transformer-based object re-identification. In ICCV, 2021.
160.X. Liu et al. A video is worth three views: Trigeminal transformers for video-based person re-identification. arXiv:2104.01745, 2021.
161.T. Zhang et al. Spatiotemporal transformer for video-based person reidentification. arXiv:2103.16469, 2021.
162.N. Engel et al. Point transformer. IEEE Access, 9:134826–134840, 2021.
163.M.-H. Guo et al. Pct: Point cloud transformer. Computational Visual Media, 7(2):187–199, 2021.
164.H. Zhao et al. Point transformer. In ICCV, pp. 16259–16268, 2021.
165.K. Lee et al. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04589, 2021.
166.A. v. d. Oord et al. Neural discrete representation learning. arXiv, 2017.
167.J. Ho et al. Denoising diffusion probabilistic models. volume 33, pp. 6840–6851, 2020.
168.A. Ramesh et al. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
169.R. Rombach et al. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684–10695, 2022.
170.X. Wang et al. Sceneformer: Indoor scene generation with transformers. In 3DV, pp. 106–115. IEEE, 2021.
171.Z. Liu et al. Convtransformer: A convolutional transformer network for video frame synthesis. arXiv:2011.10185, 2020.
172.R. Girdhar et al. Video action transformer network. In CVPR, 2019.
173.H. Liu et al. Two-stream transformer networks for video-based face alignment. T-PAMI, 40(11):2546–2554, 2017.
174.J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
175.S. Lohit et al. Temporal transformer networks: Joint learning of invariant and discriminative time warping. In CVPR, 2019.
176.M. Fayyaz and J. Gall. Sct: Set constrained temporal transformer for set supervised action segmentation. In 2020 CVPR, pp. 501–510, 2020.
177.W. Choi et al. What are they doing?: Collective activity classification using spatio-temporal relationship among people. In ICCVW, 2009.
178.K. Gavrilyuk et al. Actor-transformers for group activity recognition. In CVPR, pp. 839–848, 2020.
179.J. Shao et al. Temporal context aggregation for video retrieval with contrastive learning. In WACV, 2021.
180.V. Gabeur et al. Multi-modal transformer for video retrieval. In ECCV, pp. 214–229, 2020.
181.Y. Chen et al. Memory enhanced global-local aggregation for video object detection. In CVPR, pp. 10337–10346, 2020.
182.J. Yin et al. Lidar-based online 3d video object detection with graphbased message passing and spatiotemporal transformer attention. In 2020 CVPR, pp. 11495–11504, 2020.
183.H. Seong et al. Video multitask transformer network. In ICCVW, 2019.
184.K. M. Schatz et al. A recurrent transformer network for novel view action synthesis. In ECCV (27), pp. 410–426, 2020.
185.C. Sun et al. Videobert: A joint model for video and language representation learning. In ICCV, pp. 7464–7473, 2019.
186.L. H. Li et al. Visualbert: A simple and performant baseline for vision and language. arXiv:1908.03557, 2019.
187.W. Su et al. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR, 2020.
188.Y.-S. Chuang et al. Speechbert: Cross-modal pre-trained language model for end-to-end spoken question answering. In Interspeech, 2020.
189.R. Hu and A. Singh. Unit: Multimodal multitask learning with a unified transformer. In ICCV, 2021.
190.S. Prasanna et al. When bert plays the lottery, all tickets are winning. In EMNLP, 2020.
191.J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018.
192.Y. Tang et al. Patch slimming for efficient vision transformers. arXiv:2106.02852, 2021.
193.M. Zhu et al. Vision transformer pruning. arXiv:2104.08500, 2021.
194.Z. Liu et al. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
195.Z. Lan et al. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020.
196.C. Xu et al. Bert-of-theseus: Compressing bert by progressive module replacing. In EMNLP, pp. 7859–7869, 2020.
197.S. Shen et al. Q-bert: Hessian based ultra low precision quantization of bert. In AAAI, pp. 8815–8821, 2020.
198.O. Zafrir et al. Q8bert: Quantized 8bit bert. arXiv:1910.06188, 2019.
199.V. Sanh et al. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv:1910.01108, 2019.
200.S. Sun et al. Patient knowledge distillation for bert model compression. In EMNLP-IJCNLP, pp. 4323–4332, 2019.
201.Z. Sun et al. Mobilebert: a compact task-agnostic bert for resourcelimited devices. In ACL, pp. 2158–2170, 2020.
202.I. Turc et al. Well-read students learn better: The impact of student initialization on knowledge distillation. arXiv:1908.08962, 2019.
203.X. Qiu et al. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, pp. 1–26, 2020.
204.A. Fan et al. Reducing transformer depth on demand with structured dropout. In ICLR, 2020.
205.L. Hou et al. Dynabert: Dynamic bert with adaptive width and depth. NeurIPS, 33, 2020.
206.Z. Wang et al. Structured pruning of large language models. In EMNLP, pp. 6151–6162, 2020.
207.G. Hinton et al. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.
208.C. Buciluă et al. Model compression. In SIGKDD, pp. 535–541, 2006.
209.J. Ba and R. Caruana. Do deep nets really need to be deep? NIPS, 2014.
210.S. Mukherjee and A. H. Awadallah. Xtremedistil: Multi-stage distillation for massive multilingual models. In ACL, pp. 2221–2234, 2020.
211.W. Wang et al. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv:2002.10957, 2020.
212.S. I. Mirzadeh et al. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
213.D. Jia et al. Efficient vision transformers via fine-grained manifold distillation. arXiv:2107.01378, 2021.
214.V. Vanhoucke et al. Improving the speed of neural networks on cpus. In NIPS Workshop, 2011.
215.Z. Yang et al. Searching for low-bit weights in quantized neural networks. In NeurIPS, 2020.
216.E. Park and S. Yoo. Profit: A novel training method for sub-4-bit mobilenet models. In ECCV, pp. 430–446. Springer, 2020.
217.J. Fromm et al. Riptide: Fast end-to-end binarized neural networks. Proceedings of Machine Learning and Systems, 2:379–389, 2020.
218.Y. Bai et al. Proxquant: Quantized neural networks via proximal operators. In ICLR, 2019.
219.A. Bhandare et al. Efficient 8-bit quantization of transformer neural machine language translation model. arXiv:1906.00532, 2019.
220.C. Fan. Quantized transformer. Technical report, Stanford Univ., 2019.
221.K. Shridhar et al. End to end binarized neural networks for text classification. In SustaiNLP, 2020.
222.R. Cheong and R. Daniel. transformers. zip: Compressing transformers with pruning and quantization. Technical report, 2019.
223.Z. Zhao et al. An investigation on different underlying quantization schemes for pre-trained language models. In NLPCC, 2020.
224.Z. Liu et al. Post-training quantization for vision transformer. In NeurIPS, 2021.
225.Z. Wu et al. Lite transformer with long-short range attention. In ICLR, 2020.
226.Z. Geng et al. Is attention better than matrix decomposition? In ICLR, 2020.
227.Y. Guo et al. Nat: Neural architecture transformer for accurate and compact architectures. In NeurIPS, pp. 737–748, 2019.
228.D. So et al. The evolved transformer. In ICML, pp. 5877–5886, 2019.
229.C. Li et al. Bossnas: Exploring hybrid cnn-transformers with blockwisely self-supervised neural architecture search. In ICCV, 2021.
230.A. Katharopoulos et al. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, 2020.
231.C. Yun et al. o(n) connections are expressive enough: Universal approximability of sparse transformers. In NeurIPS, 2020.
232.M. Zaheer et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
233.D. A. Spielman and S.-H. Teng. Spectral sparsification of graphs. SIAM Journal on Computing, 40(4), 2011.
234.F. Chung and L. Lu. The average distances in random graphs with given expected degrees. PNAS, 99(25):15879–15882, 2002.
235.A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
236.X. Zhai et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv:1910.04867, 2019.
237.Y. Cheng et al. Robust neural machine translation with doubly adversarial inputs. In ACL, 2019.
238.W. E. Zhang et al. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM TIST, 11(3):1–41, 2020.
239.K. Mahmood et al. On the robustness of vision transformers to adversarial examples. arXiv:2104.02610, 2021.
240.X. Mao et al. Towards robust vision transformer. arXiv, 2021.
241.S. Serrano and N. A. Smith. Is attention interpretable? In ACL, 2019.
242.S. Wiegreffe and Y. Pinter. Attention is not not explanation. In EMNLPIJCNLP, 2019.
243.H. Chefer et al. Transformer interpretability beyond attention visualization. In CVPR, pp. 782–791, 2021.
244.R. Livni et al. On the computational efficiency of training neural networks. In NeurIPS, 2014.
245.B. Neyshabur et al. Towards understanding the role of overparametrization in generalization of neural networks. In ICLR, 2019.
246.K. Han et al. Ghostnet: More features from cheap operations. In CVPR, pp. 1580–1589, 2020.
247.K. Han et al. Model rubik’s cube: Twisting resolution, depth and width for tinynets. NeurIPS, 33, 2020.
248.T. Chen et al. Diannao: a small-footprint high-throughput accelerator for ubiquitous machine-learning. In ASPLOS, pp. 269–284, 2014.
249.H. Liao et al. Davinci: A scalable architecture for neural network computing. In 2019 IEEE Hot Chips 31 Symposium (HCS), 2019.
250.A. Jaegle et al. Perceiver: General perception with iterative attention. In ICML, volume 139, pp. 4651–4664. PMLR, 18–24 Jul 2021.
251.A. Jaegle et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.
252.X. Wang et al. Non-local neural networks. In CVPR, pp. 7794–7803, 2018.
253.A. Buades et al. A non-local algorithm for image denoising. In CVPR, pp. 60–65, 2005.
254.J. Chung et al. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555, 2014.
255.M. Joshi et al. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77, 2020.
256.Y. Liu et al. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692, 2019.
257.Y. Zhu et al. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pp. 19–27, 2015.
258.A. Radford et al. Improving language understanding by generative pretraining, 2018.
259.Z. Yang et al. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, pp. 5753–5763, 2019.
260.K. Clark et al. Electra: Pre-training text encoders as discriminators rather than generators. arXiv:2003.10555, 2020.
261.L. Dong et al. Unified language model pre-training for natural language understanding and generation. In NeurIPS, pp. 13063–13075, 2019.
262.M. Lewis et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv:1910.13461, 2019.
263.Z. Zhang et al. Ernie: Enhanced language representation with informative entities. arXiv:1905.07129, 2019.
264.M. E. Peters et al. Knowledge enhanced contextual word representations. arXiv:1909.04164, 2019.
265.J. Lee et al. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020.
266.I. Beltagy et al. Scibert: A pretrained language model for scientific text. arXiv:1903.10676, 2019.
267.K. Huang et al. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv:1904.05342, 2019.
268.J. Ba et al. Multiple object recognition with visual attention. In ICLR, 2014.
269.V. Mnih et al. Recurrent models of visual attention. NeurIPS, pp. 2204–2212, 2014.
270.K. Xu et al. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048–2057, 2015.
271.F. Wang et al. Residual attention network for image classification. In CVPR, pp. 3156–3164, 2017.
272.S. Jetley et al. Learn to pay attention. In ICLR, 2018.
273.K. Han et al. Attribute-aware attention model for fine-grained representation learning. In ACM MM, pp. 2040–2048, 2018.
274.P. Ramachandran et al. Stand-alone self-attention in vision models. In NeurIPS, 2019.
275.Q. Guan et al. Diagnose like a radiologist: Attention guided convolutional neural network for thorax disease classification. In arXiv:1801.09927, 2018.
276.J. Hu et al. Squeeze-and-excitation networks. In CVPR, pp. 7132–7141, 2018.
277.H. Zhao et al. Psanet: Point-wise spatial attention network for scene parsing. In ECCV, pp. 267–283, 2018.
278.Y. Yuan et al. Ocnet: Object context for semantic segmentation. International Journal of Computer Vision, pp. 1–24, 2021.
279.J. Fu et al. Dual attention network for scene segmentation. In CVPR, pp. 3146–3154, 2019.
280.H. Zhang et al. Co-occurrent features in semantic segmentation. In CVPR, pp. 548–557, 2019.
281.F. Zhang et al. Acfnet: Attentional class feature network for semantic segmentation. In ICCV, pp. 6798–6807, 2019.
282.X. Li et al. Expectation-maximization attention networks for semantic segmentation. In ICCV, pp. 9167–9176, 2019.
283.J. He et al. Adaptive pyramid context network for semantic segmentation. In CVPR, pp. 7519–7528, 2019.
284.O. Oktay et al. Attention u-net: Learning where to look for the pancreas. 2018.
285.Y. Wang et al. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In CVPR, pp. 12275–12284, 2020.
286.X. Li et al. Global aggregation then local distribution in fully convolutional networks. In BMVC, 2019.
287.Y. Chen et al. Aˆ 2-nets: Double attention networks. NeurIPS, pp. 352–361, 2018.
288.L. Zhang et al. Dual graph convolutional network for semantic segmentation. In BMVC, 2019.
289.K. Yue et al. Compact generalized non-local network. In NeurIPS, pp. 6510–6519, 2018.
290.Z. Huang et al. Ccnet: Criss-cross attention for semantic segmentation. In ICCV, pp. 603–612, 2019.
291.L. Huang et al. Interlaced sparse self-attention for semantic segmentation. arXiv:1907.12273, 2019.
292.Y. Li and A. Gupta. Beyond grids: Learning graph representations for visual recognition. NeurIPS, pp. 9225–9235, 2018.
293.S. Kumaar et al. Cabinet: Efficient context aggregation network for low-latency semantic segmentation. arXiv:2011.00993, 2020.
294.X. Liang et al. Symbolic graph reasoning meets convolutions. NeurIPS, pp. 1853–1863, 2018.
295.Y. Chen et al. Graph-based global reasoning networks. In CVPR, pp. 433–442, 2019.
296.T.-Y. Lin et al. Microsoft coco: Common objects in context. In ECCV, pp. 740–755, 2014.
297.Y. Cao et al. Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In ICCV Workshops, 2019.
298.W. Li et al. Object detection based on an adaptive attention mechanism. Scientific Reports, pp. 1–13, 2020.
299.T.-I. Hsieh et al. One-shot object detection with co-attention and coexcitation. In NeurIPS, pp. 2725–2734, 2019.
300.Q. Fan et al. Few-shot object detection with attention-rpn and multirelation detector. In CVPR, pp. 4013–4022, 2020.
301.H. Perreault et al. Spotnet: Self-attention multi-task network for object detection. In 2020 17th Conference on Computer and Robot Vision (CRV), pp. 230–237, 2020.
302.X.-T. Vo et al. Bidirectional non-local networks for object detection. In International Conference on Computational Collective Intelligence, pp. 491–501, 2020.
303.H. Hu et al. Relation networks for object detection. In CVPR, pp. 3588–3597, 2018.
304.K. Zhang et al. Learning enhanced resolution-wise features for human pose estimation. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 2256–2260, 2020.
305.Y. Chang et al. The same size dilated attention network for keypoint detection. In International Conference on Artificial Neural Networks, pp. 471–483, 2019.
306.A. Johnston and G. Carneiro. Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume. In CVPR, pp. 4756–4765, 2020.
307.Y. Chen et al. Attention-based context aggregation network for monocular depth estimation. International Journal of Machine Learning and Cybernetics, pp. 1583–1596, 2021.
308.S. Aich et al. Bidirectional attention network for monocular depth estimation. In ICRA, 2021.

Citation

MLA

Han, K., et al. “A Survey on Vision Transformer”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, 2023, pp. 87–110, https://doi.org/10.1109/TPAMI.2022.3152247.

APA

Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., Yang, Z., Zhang, Y., & Tao, D. (2023). A Survey on Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 87–110. https://doi.org/10.1109/TPAMI.2022.3152247

Chicago

Han, K., Y. Wang, H. Chen, et al. 2023. “A Survey on Vision Transformer”. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (1): 87–110. https://doi.org/10.1109/TPAMI.2022.3152247.

Harvard

Han, K. et al. (2023) “A Survey on Vision Transformer”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), pp. 87–110. Available at: https://doi.org/10.1109/TPAMI.2022.3152247.

Vancouver

1. Han K, Wang Y, Chen H, et al (2023) A Survey on Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence 45:87–110

BibTeX

@article{Han_2023, title={A Survey on Vision Transformer}, volume={45}, ISSN={1939-3539}, url={http://dx.doi.org/10.1109/TPAMI.2022.3152247}, DOI={10.1109/tpami.2022.3152247}, number={1}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, publisher={Institute of Electrical and Electronics Engineers (IEEE)}, author={Han, Kai and Wang, Yunhe and Chen, Hanting and Chen, Xinghao and Guo, Jianyuan and Liu, Zhenhua and Tang, Yehui and Xiao, An and Xu, Chunjing and Xu, Yixing and Yang, Zhaohui and Zhang, Yiman and Tao, Dacheng}, year={2023}, month=Jan, pages={87–110} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

A Survey on Vision Transformer

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Categorization and Structural Taxonomy of Vision Transformers

Knowl 2 — Vision Transformer (ViT) Architecture and Patch Embedding Formulation

Knowl 3 — ImageNet Performance Benchmark of CNNs versus Pure and Hybrid Vision Transformers

Knowl 4 — End-to-End Object Detection with Transformers (DETR) and Hungarian Loss

Knowl 5 — COCO Object Detection Benchmark Comparison for Transformer Detectors

Knowl 6 — Self-Supervised Learning Paradigms for Vision Transformers: Generative and Contrastive

Knowl 7 — Image Processing Transformer (IPT) for Multi-Task Low-Level Vision

Knowl 8 — Equivalence of Transformer Self-Attention to Non-Local Filtering Operations in Computer Vision

Knowl 9 — Transformer Model Compression and Acceleration Benchmarks

Knowl 10 — Fundamental Limitations and Open Research Challenges of Vision Transformers

References

Citation

BibTeX

Access the Paper