Video Swin Transformer

Ze LiuJia NingYue CaoYixuan WeiZheng ZhangStephen LinHan Hu

article2021CVPR2,279 citations

Extends the Swin Transformer architecture to video recognition via localized 3D spatio-temporal attention, establishing state-of-the-art accuracy on action recognition benchmarks with vastly superior parameter and data efficiency compared to global transformer models.

Listen

First-person (egocentric) video understanding is critical for emerging applications in augmented reality, robotics, and assistive technology. A major challenge in this domain is enabling computer vision systems to accurately recognize when a person alters the physical state of an object and to pinpoint the exact moment of irreversible change. The article evaluates advanced vision transformer models on two standardized tasks using the Ego4D Hands and Objects benchmark: classifying whether an object state change occurred in eight-second video clips, and precisely localizing the temporal "point-of-no-return"—the frame where the change becomes irreversible.

To tackle these tasks, the researchers developed a Deformable Swin Transformer for state change classification. This approach incorporates deformable attention mechanisms into three-dimensional video windows, restricting focus to a sparse set of key points rather than entire spatial-temporal patches, which lowers computational complexity. The classification model was evaluated across tens of thousands of video clips against leading convolutional and transformer baselines. For temporal localization, the team conducted statistical frequency analysis on the dataset and implemented a standard Video Swin Transformer architecture using a fixed 16-frame uniform sampling strategy.

The findings show that the proposed Deformable Swin Transformer achieved the highest performance in object state change classification, reaching 69.8% validation accuracy and 67.7% test accuracy, outperforming the challenge's convolutional baseline (68.7%) and standard transformer models. For temporal localization, statistical analysis revealed a strong dataset bias: the point-of-no-return clustered heavily around 45% of the clip duration. A simple rule-based heuristic that always placed the event at the 45% mark beat both the official center-frame baseline (1.06-second test error) and a recurrent neural network model (0.76-second error) by achieving a 0.67-second test error. The trained Video Swin Transformer achieved the best overall result, lowering test error to 0.66 seconds.

These results demonstrate that vision transformers with deformable attention can improve visual recognition performance while mitigating model complexity. However, the discovery that a rigid 45% heuristic outperforms established neural network baselines highlights a significant risk: benchmark datasets can contain structural biases that artificially inflate performance without true visual comprehension. In practice, models exposed to these biases struggle when critical events occur at non-standard times, such as the very beginning or end of a sequence.

For future development, decision-makers and engineering teams should incorporate deformable attention mechanisms to optimize compute efficiency in video processing pipelines. Concurrently, dataset curators and model validators must address dataset collection biases by balancing event timing and improving annotation quality. Because models remain prone to high errors when handling subtle physical interactions or edge-case event timings, real-world deployment should proceed cautiously with targeted testing across diverse, unaligned video data.

Google Universidad de los Andes action recognition encoder-decoder factorization Video Video Swin Video Swin Transformer

arXiv: 2106.13230 SwinTransformer/Video-Swin-Transformer

Paper: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, Ze Liu et al. (2021). Reading the Swin Transformer paper first is essential because Video Swin Transformer directly adapts its hierarchical shifted-window attention mechanism to the video domain.

Paper: Swin Transformer V2: Scaling Up Capacity and Resolution, Ze Liu et al. (2022). Swin Transformer V2 naturally continues this work by scaling up the capacity and resolution of Swin-based architectures for advanced vision tasks.

Abstract

The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built on Transformer layers that globally connect patches across the spatial and temporal dimensions. In this paper, we instead advocate an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the Swin Transformer designed for the image domain, while continuing to leverage the power of pre-trained image models. Our approach achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including on action recognition (84.9 top-1 accuracy on Kinetics-400 and 86.1 top-1 accuracy on Kinetics-600 with ~20x less pre-training data and ~3x smaller model size) and temporal modeling (69.6 top-1 accuracy on Something-Something v2). The code and models will be made publicly available at this https URL.

Topics

action recognition encoder-decoder factorization trade video streams architecture designs

Keywords

Video Video Swin Video Swin Transformer Frame PNR Baseline Error State Change Ego4D Clips

1. Video Swin Transformer
1.1. Deformable Swin Transformer
2. Egocentric video understanding
2.1. Object State Change Classification
2.1.1 Experimental Setting
2.1.2 Results
2.2. Point-of-No-Return Temporal Localization
2.2.1 Baseline
2.2.2 Experimental Setting
2.2.3 Results
3. Conclusion
References

Knowls

Knowl 1 — Deformable Swin Transformer for Video Understanding
model/method
In 3D Video Swin Transformers, computing self-attention within 3D spatiotemporal sub-volume patches incurs substantial computational complexity. The Deformable Swin Transformer replaces standard full self-attention within 3D patches with deformable attention mechanisms. Under this mechanism, each input query position attends to a sparse set of only $N$ sampled points rather than all tokens in the 3D patch. In practice, setting $N = 4$ maintains classification accuracy while substantially reducing computational overhead. The transformer attention modules are trained from scratch, while the multi-layer perceptrons (MLPs) within the transformer blocks are initialized from weights pretrained on Kinetics-600.

Knowl 2 — Performance Comparison on Ego4D Object State Change Classification

data/table

The Object State Change Classification (OSCC) task on the Ego4D Hands and Objects benchmark requires identifying whether an object undergoes a state change within an 8-second egocentric video clip. The dataset consists of 41,000 training clips, 28,000 validation clips, and 28,000 test clips. Models are trained using the AdamW optimizer with a learning rate of $3 \times 10^{-4}$ for 30 epochs.

The standard Video Swin Transformer outperforms both the I3D ResNet-50 CNN baseline and MViT by 0.8% in validation accuracy. The Deformable Swin Transformer achieves the highest validation accuracy of 69.8% (and 67.7% on the Ego4D test set).

Knowl 3 — Normalized Temporal Bias and Constant Fraction Baseline for Ego4D Point-of-No-Return Localization
empirical result
In the Ego4D Hands and Objects Point-of-No-Return (PNR) temporal localization benchmark, the normalized PNR frame index is defined as:

$r_{\text{PNR}} = \frac{t_{\text{PNR}}}{T}$

where $t_{\text{PNR}}$ is the timestamp or frame index of the state change point, and $T$ is the total clip duration (or frame count). Frequency analysis across the training and validation splits reveals a pronounced dataset bias where ground-truth PNR frames are heavily concentrated around the 0.45 fraction ( $r_{\text{PNR}} \approx 0.45$ ) of the video clip duration.

A constant heuristic baseline that predicts the PNR frame as:

$\hat{t}_{\text{PNR}} = 0.45 \times T$

achieves an absolute temporal localization error of 0.63 seconds on the validation set and 0.67 seconds on the test set. This simple constant prior outperforms the official challenge baselines, which include selecting the center frame (1.03s val / 1.06s test) and a Bi-directional LSTM baseline (0.79s val / 0.76s test).

Knowl 4 — Comparative Evaluation on Ego4D Point-of-No-Return Temporal Localization

data/table

The Point-of-No-Return (PNR) temporal localization task in Ego4D requires predicting the exact frame/timestamp at which an object state change occurs in clips containing a state change. The dataset comprises 20,000 training clips, 13,000 validation clips, and 28,000 test clips. Performance is measured using the absolute temporal localization error in seconds (lower is better).

The Video Swin Transformer outperforms all baselines, achieving an absolute temporal localization error of 0.61 seconds on the validation split and 0.66 seconds on the test split.

Knowl 5 — Uniform Frame Sampling and Timestamp Decoding for PNR Temporal Localization
algorithm
To train and evaluate video transformers for temporal point localization under variable video durations, frames are uniformly downsampled to a fixed temporal resolution and subsequently decoded back to continuous video time.
Input: Video clip $V$ with duration $D$ (in seconds) or total frames $T$ , input sample size $K = 16$
Output: Predicted Point-of-No-Return timestamp $\hat{t}_{\text{PNR}}$ (in seconds)
1. Sample $K$ frames uniformly from $V$ spanning from $0$ to $D$
2. Calculate standard sampling rate: $R = \frac{D}{K}$
3. Pass the $K$ -frame sequence through the Video Swin Transformer
4. Obtain the predicted sequence frame index $k^* \in [0, K - 1]$
5. Decode continuous timestamp: $\hat{t}_{\text{PNR}} = k^* \times R$
6. return $\hat{t}_{\text{PNR}}$
Knowl 6 — Failure Modes in Egocentric Object State Change and Point-of-No-Return Tasks
limitation
Video Swin Transformer models for egocentric video understanding exhibit specific failure patterns across tasks:
1. Object State Change Classification: Models frequently misclassify non-state-altering object manipulations as positive state changes when intense interaction occurs (e.g., changing containers, such as taking meat out of a bag or pouring dirt into a bucket). They also struggle with subtle state changes (such as throwing wood on the ground) or ambiguous ground-truth labels (e.g., welded metal labeled as negative).
2. Point-of-No-Return Localization: Due to the severe prior bias in the dataset toward the middle of the clip ( $0.45 \times \text{duration}$ ), the model's error distribution exhibits a strong V-shape centered near 0.45. Predictions cluster toward the center, leading to large temporal errors (over 4 seconds) whenever the actual state change happens near the extreme boundaries (the very beginning or very end) of a video clip.

Coverage note — None. All primary contributions, models, benchmarks, empirical results, and failure analyses presented in the paper have been extracted.

References

1.Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018. 1
2.Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 1, 2, 3
3.Ego4D. Ego4d object state change classification. https://eval.ai/web/challenges/challenge-page/1627/overview, Mar 2022. 1
4.Ego4D. Ego4d pnr temporal localization challenge. https://eval.ai/web/challenges/challenge-page/1622/overview, Mar 2022. 1
5.Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021. 1, 2
6.Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. arXiv preprint arXiv:2110.07058, 3, 2021. 1, 2, 3
7.Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Bidirectional lstm networks for improved phoneme classification and recognition. In International conference on artificial neural networks, pages 799–804. Springer, 2005. 3
8.Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. 1
9.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021. 1
10.Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106.13230, 2021. 1, 2, 3

Citation

MLA

Liu, Z., et al. “Video Swin Transformer”. arXiv, 2021, http://arxiv.org/abs/2106.13230v1.

APA

Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021). Video Swin Transformer. arXiv. http://arxiv.org/abs/2106.13230v1

Chicago

Liu, Z., J. Ning, Y. Cao, et al. 2021. “Video Swin Transformer”. arXiv. http://arxiv.org/abs/2106.13230v1.

Harvard

Liu, Z. et al. (2021) “Video Swin Transformer”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2106.13230v1.

Vancouver

1. Liu Z, Ning J, Cao Y, Wei Y, Zhang Z, Lin S, Hu H (2021) Video Swin Transformer. arXiv

BibTeX

@article{liu2021video,
  title = {Video Swin Transformer},
  author = {Liu, Ze and Ning, Jia and Cao, Yue and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Hu, Han},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2106.13230v1},
  eprint = {2106.13230}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Method	Validation Accuracy (%)
I3D ResNet-50	68.7
MViT	68.7
Video Swin Transformer	69.5
Deformable Swin Transformer (Ours)	69.8

Method	Validation Error (s)	Test Error (s)
Always Center Frame	1.03	1.06
Bi-directional LSTM	0.79	0.76
Constant 0.45 Baseline	0.63	0.67
Video Swin Transformer	0.61	0.66