MetaFormer is Actually What You Need for Vision

Weihao YuRomy Mi LuoPan ZhouChenyang SiYichen ZhouXinchao WangJiashi FengShuicheng Yan

article2021CVPR1,437 citations

Demonstrates that the general macro-architecture of Vision Transformers matters more than self-attention by showing that replacing attention with simple spatial pooling outperforms standard transformer and MLP baselines with significantly fewer parameters.

Listen

Recent advancements in computer vision have heavily favored transformer-based architectures, largely attributing their breakthrough performance to complex, attention-based token-mixing mechanisms. However, developing increasingly complex token mixers has made these models computationally demanding, raising critical questions about what architectural component actually drives their success. Understanding these core drivers is vital for organizations seeking to optimize high-performance vision models under tight computational and memory constraints.

The article demonstrates that the overall foundational structure—termed "MetaFormer"—is the primary driver of performance, rather than the specific, sophisticated token-mixing components. By abstracting the standard transformer architecture to isolate the underlying layout from the token-mixing operator, the article evaluates whether a model can achieve competitive visual recognition capabilities using only a minimal, non-parametric pooling function.

To test this hypothesis, the authors developed "PoolFormer," a model that adopts the MetaFormer architecture while replacing complex attention mechanisms entirely with simple spatial average pooling. The authors conducted extensive empirical evaluations across standard computer vision benchmarks, including ImageNet-1K for image classification (comprising approximately 1.3 million training images), COCO for object detection and instance segmentation, and ADE20K for semantic segmentation, training models using standard 300-epoch schedules and modern data augmentation techniques.

The findings show that PoolFormer consistently achieves state-of-the-art efficiency and performance. On ImageNet-1K, PoolFormer-M36 achieved 82.1% top-1 accuracy, outperforming the well-tuned DeiT-B transformer baseline by 0.3% and the ResMLP-B24 baseline by 1.1%, while requiring 35% to 52% fewer parameters and 50% to 62% fewer multiply-accumulate operations. When evaluated on downstream tasks such as object detection, instance segmentation, and semantic segmentation, PoolFormer backbones consistently exceeded the accuracy of established convolutional and transformer alternatives like ResNet and PVT. Furthermore, ablation experiments revealed that MetaFormer maintains competitive accuracy even when using basic identity mappings or static random matrices as token mixers, whereas removing residual connections or channel multi-layer perceptrons causes performance to collapse completely.

These results significantly alter the conventional understanding of vision transformers by showing that the overarching architecture provides the baseline competence, while specific token mixers offer incremental refinement. For industry practitioners, this insight means that high-accuracy vision systems can be deployed with significantly lower computational overhead, reducing hardware costs, memory footprint, and execution latency in production environments.

The article recommends that researchers and system designers shift their primary focus from inventing complex token-mixing layers toward optimizing the broader MetaFormer architecture and exploring hybrid models. For instance, combining pooling layers in early stages with attention mechanisms in later stages achieved 81.0% accuracy on ImageNet-1K using only 16.5 million parameters—about seven times fewer parameters than comparable multi-layer perceptron models. While these results establish high confidence across standard supervised image benchmarks, future work remains necessary to validate the architecture across self-supervised learning, natural language processing, and varied transfer learning environments.

National University of Singapore Sea AI Lab architecture designs encoder-decoder multilayer perceptrons MetaFormer architecture PoolFormer-based models Vision Transformer

arXiv: 2111.11418 sail-sg/poolformer

Paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy et al. (2021). It introduces the standard Vision Transformer backbone whose general macro-architecture MetaFormer analyzes and abstracts.
Paper: MLP-Mixer: An all-MLP Architecture for Vision, I. Tolstikhin et al. (2021). It demonstrates that attention can be replaced with spatial MLPs, providing the key motivating observation that inspires MetaFormer's token-mixer hypothesis.
Paper: Training data-efficient image transformers & distillation through attention, Hugo Touvron et al. (2021). It provides the DeiT baseline architecture and training strategies directly benchmarked and compared against PoolFormer.
Paper: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, Ze Liu et al. (2021). It establishes the hierarchical multi-stage vision transformer design that MetaFormer adopts for structured vision backbones.
Paper: Transformers in Vision: A Survey, Salman Khan et al. (2021). It offers a comprehensive survey of early vision transformer designs and components that provides essential context for MetaFormer's structural abstraction.

Paper: A ConvNet for the 2020s, Zhuang Liu et al. (2022). It continues the exploration of modernizing macro-architectures without attention by applying transformer-style structural designs to pure ConvNets.
Paper: ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders, Sanghyun Woo et al. (2023). It builds upon modern non-transformer structural backbones like ConvNeXt by integrating masked autoencoder pre-training and global response normalization.

Cover for MetaFormer is Actually What You Need for Vision

Abstract

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in Transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the Transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in Transformers with an embarrassingly simple spatial pooling operator to conduct only basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1% top-1 accuracy, surpassing well-tuned Vision Transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 50%/62% fewer MACs. The effectiveness of PoolFormer verifies our hypothesis and urges us to initiate the concept of "MetaFormer", a general architecture abstracted from Transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent Transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design. Code is available at this https URL.

Topics

architecture designs encoder-decoder multilayer perceptrons computer vision applications convolutional neural networks vision systems

Keywords

MetaFormer architecture PoolFormer-based models Vision Transformer image classification semantic segmentation object detection convolutional neural networks Layer Normalization

1 Introduction
2 Related work
3 Method
3.1 MetaFormer
3.2 PoolFormer
4 Experiments
4.1 Image classification
4.2 Object detection and instance segmentation
4.3 Semantic segmentation
4.4 Ablation studies
5 Conclusion and future work
References
A Detailed hyper-parameters on ImageNet-1K
B Training for longer epochs
C Qualitative results
D Comparison between Layer Normalization and Modified Layer Normalization
E Code in PyTorch

Knowls

Knowl 1 — MetaFormer General Architecture
model/method
MetaFormer is an architectural abstraction derived from the Transformer encoder that leaves the token mixer unspecified while maintaining the overall macro-architecture, normalization, and channel multi-layer perceptrons (MLPs).

Given an input image $I$ , the architecture first computes an initial token representation via an input embedding module:

$X = \mathrm{InputEmb}(I)$

where $X \in \mathbb{R}^{N \times C}$ denotes the sequence of $N$ token embeddings with channel dimension $C$ .

The tokens then pass through repeated MetaFormer blocks. Each block contains two sequential residual sub-blocks:
1. A token mixing sub-block to propagate information across spatial token positions:
$Y = \mathrm{TokenMixer}(\mathrm{Norm}(X)) + X$

where $\mathrm{Norm}(\cdot)$ denotes a normalization layer (such as Layer Normalization or Modified Layer Normalization) and $\mathrm{TokenMixer}(\cdot)$ represents any module that communicates information across spatial tokens.
1. A channel MLP sub-block with non-linear activation:
$Z = \sigma(\mathrm{Norm}(Y)W_1)W_2 + Y$

where $W_1 \in \mathbb{R}^{C \times rC}$ and $W_2 \in \mathbb{R}^{rC \times C}$ are learnable weight matrices with MLP expansion ratio $r$ , and $\sigma(\cdot)$ is a non-linear activation function (such as GELU or ReLU).

MetaFormer acts as a meta-framework: specifying $\mathrm{TokenMixer}$ as multi-head self-attention yields standard Vision Transformers, specifying it as spatial fully connected layers yields MLP-like vision models (such as ResMLP or MLP-Mixer), and specifying it as average pooling yields PoolFormer.
Knowl 2 — Pooling as a Parameter-Free Token Mixer
equation
In the PoolFormer model, the token mixer in the MetaFormer architecture is replaced with a non-parametric average pooling operator. For a channel-first feature tensor $T \in \mathbb{R}^{C \times H \times W}$ , where $C$ is the number of channels, $H$ is the height, and $W$ is the width, the pooling token mixer is defined as:

$T'_{:, i, j} = \frac{1}{K \times K} \sum_{p=1}^{K} \sum_{q=1}^{K} T_{:, i+p-\frac{K+1}{2}, j+q-\frac{K+1}{2}} - T_{:, i, j}$

where $K$ is the pooling kernel size (default $K=3$ with stride 1 and padding $\lfloor K/2 \rfloor$ ).

The subtraction of the input tensor $T_{:, i, j}$ is incorporated because the surrounding MetaFormer sub-block includes an additive residual connection $Y = \mathrm{TokenMixer}(\mathrm{Norm}(X)) + X$ . Adding $X$ back to $T'$ produces a standard local average pooling operation over a $K \times K$ neighborhood.

The computational complexity of this pooling token mixer scales linearly with sequence length ( $\mathcal{O}(C \cdot H \cdot W)$ ) and introduces zero learnable parameters.

Knowl 3 — PoolFormer Hierarchical Network Architecture and Model Scaling

model/method

PoolFormer adopts a 4-stage hierarchical pyramid structure similar to convolutional networks and hierarchical vision transformers. Given an input image of size $3 \times H \times W$ , the token resolutions across the four stages are $\frac{H}{4} \times \frac{W}{4}$ , $\frac{H}{8} \times \frac{W}{8}$ , $\frac{H}{16} \times \frac{W}{16}$ , and $\frac{H}{32} \times \frac{W}{32}$ .

Patch embeddings are applied at the beginning of each stage. Stage 1 uses a $7 \times 7$ convolution with stride 4, while Stages 2, 3, and 4 use $3 \times 3$ convolutions with stride 2. In each PoolFormer block, the pooling token mixer uses kernel size $3 \times 3$ with stride 1, and the channel MLP expansion ratio is set to $r=4$ .

For a total of $L$ PoolFormer blocks across the network, the distribution of blocks across stages $[1, 2, 3, 4]$ follows the ratio $[L/6, L/6, L/2, L/6]$ .

The architecture is scaled into five standard model configurations across two embedding dimension groups (Small with embedding dimensions $[64, 128, 320, 512]$ and Medium with $[96, 192, 384, 768]$ ):

Model	Blocks per Stage	Embedding Dimensions	Parameters (M)	MACs (G)
PoolFormer-S12	[2, 2, 6, 2]	[64, 128, 320, 512]	11.9	1.8
PoolFormer-S24	[4, 4, 12, 4]	[64, 128, 320, 512]	21.4	3.4
PoolFormer-S36	[6, 6, 18, 6]	[64, 128, 320, 512]	30.8	5.0
PoolFormer-M36	[6, 6, 18, 6]	[96, 192, 384, 768]	56.1	8.8
PoolFormer-M48	[8, 8, 24, 8]	[96, 192, 384, 768]	73.4	11.6

Knowl 4 — Modified Layer Normalization for Channel-First Vision Tensors
model/method
Modified Layer Normalization (MLN) adapts standard Layer Normalization for channel-first 4D feature tensors $X \in \mathbb{R}^{B \times C \times H \times W}$ , where $B$ is the batch size, $C$ is the channel dimension, and $H, W$ are the spatial dimensions.

While vanilla Layer Normalization computes the mean and variance across the channel dimension $C$ independently at each spatial location $(h, w)$ , Modified Layer Normalization computes the mean $\mu$ and variance $\sigma^2$ across both the channel and spatial dimensions simultaneously for each sample $b$ :

$\mu_b = \frac{1}{C \cdot H \cdot W} \sum_{c=1}^C \sum_{h=1}^H \sum_{w=1}^W X_{b, c, h, w}$

$\sigma_b^2 = \frac{1}{C \cdot H \cdot W} \sum_{c=1}^C \sum_{h=1}^H \sum_{w=1}^W (X_{b, c, h, w} - \mu_b)^2$

$X'_{b, c, h, w} = \frac{X_{b, c, h, w} - \mu_b}{\sqrt{\sigma_b^2 + \epsilon}} \cdot \gamma_c + \beta_c$

where $\gamma, \beta \in \mathbb{R}^C$ are learnable affine parameters along the channel dimension, and $\epsilon$ is a small constant for numerical stability.

This operation is equivalent to Group Normalization with the number of groups set to $1$ (nn.GroupNorm(1, C) in PyTorch).

Knowl 5 — PoolFormer ImageNet-1K Classification Performance

data/table

PoolFormer models trained from scratch on ImageNet-1K for 300 epochs achieve competitive performance, outperforming well-tuned Vision Transformer (ViT, DeiT), MLP-like (ResMLP, MLP-Mixer), and convolutional (RSB-ResNet) baselines of comparable or larger compute and parameter budgets.

Model	Architecture Type	Params (M)	MACs (G)	Top-1 Acc (%)
RSB-ResNet-18	ConvNet	12	1.8	70.6
RSB-ResNet-34	ConvNet	22	3.7	75.5
RSB-ResNet-50	ConvNet	26	4.1	79.8
RSB-ResNet-101	ConvNet	45	7.9	81.3
RSB-ResNet-152	ConvNet	60	11.6	81.8
DeiT-S	Transformer (Attention)	22	4.6	79.8
DeiT-B	Transformer (Attention)	86	17.5	81.8
PVT-Small	Transformer (Attention)	25	3.8	79.8
PVT-Medium	Transformer (Attention)	44	6.7	81.2
PVT-Large	Transformer (Attention)	61	9.8	81.7
MLP-Mixer-B/16	Spatial MLP	59	12.7	76.4
ResMLP-S24	Spatial MLP	30	6.0	79.4
ResMLP-B24	Spatial MLP	116	23.0	81.0
gMLP-S	Spatial MLP	20	4.5	79.6
gMLP-B	Spatial MLP	73	15.8	81.6
PoolFormer-S12	MetaFormer (Pooling)	12	1.8	77.2
PoolFormer-S24	MetaFormer (Pooling)	21	3.4	80.3
PoolFormer-S36	MetaFormer (Pooling)	31	5.0	81.4
PoolFormer-M36	MetaFormer (Pooling)	56	8.8	82.1
PoolFormer-M48	MetaFormer (Pooling)	73	11.6	82.5

PoolFormer-S24 achieves 80.3% top-1 accuracy, surpassing DeiT-S (79.8%) with 35% fewer MACs (3.4G vs. 4.6G) and ResMLP-S24 (79.4%) with 43% fewer parameters and 76% fewer MACs. PoolFormer-M36 achieves 82.1% accuracy, exceeding DeiT-B (81.8%) and ResMLP-B24 (81.0%) with 35% to 52% fewer parameters and 50% to 62% fewer MACs.

Knowl 6 — PoolFormer Downstream Performance on Dense Vision Tasks

data/table

When used as a backbone feature extractor for dense computer vision tasks, PoolFormer consistently outperforms CNN (ResNet, ResNeXt) and Transformer (PVT) baselines on COCO object detection/instance segmentation and ADE20K semantic segmentation.

On COCO val2017 with a $1\times$ training schedule (12 epochs):

Backbone	Detector	Params (M)	$\text{AP}^{\text{box}}$	$\text{AP}_{50}^{\text{box}}$	$\text{AP}^{\text{mask}}$
ResNet-18	RetinaNet	21.3	31.8	49.6	–
PoolFormer-S12	RetinaNet	21.7	36.2	56.2	–
ResNet-50	RetinaNet	37.7	36.3	55.3	–
PoolFormer-S24	RetinaNet	31.1	38.9	59.7	–
ResNet-101	RetinaNet	56.7	38.5	57.8	–
PoolFormer-S36	RetinaNet	40.6	39.5	60.5	–
ResNet-18	Mask R-CNN	31.2	34.0	54.0	31.2
PoolFormer-S12	Mask R-CNN	31.6	37.3	59.0	34.6
ResNet-50	Mask R-CNN	44.2	38.0	58.6	34.4
PoolFormer-S24	Mask R-CNN	41.0	40.1	62.2	37.0
ResNet-101	Mask R-CNN	63.2	40.4	61.1	36.4
PoolFormer-S36	Mask R-CNN	50.5	41.0	63.1	37.7

On ADE20K semantic segmentation using Semantic FPN (evaluated by mean Intersection over Union, mIoU):

Backbone	Params (M)	mIoU (%)
ResNet-18	15.5	32.9
PVT-Tiny	17.0	35.7
PoolFormer-S12	15.7	37.2
ResNet-50	28.5	36.7
PVT-Small	28.2	39.8
PoolFormer-S24	23.2	40.3
ResNet-101	47.5	38.8
ResNeXt-101-32x4d	47.1	39.7
PVT-Medium	48.0	41.6
PoolFormer-S36	34.6	42.0
PVT-Large	65.1	42.1
PoolFormer-M36	59.8	42.4
ResNeXt-101-64x4d	86.4	40.2
PoolFormer-M48	77.1	42.7

PoolFormer-S12 achieves 36.2 AP in RetinaNet object detection (vs. 31.8 for ResNet-18) and 37.2% mIoU in ADE20K segmentation (vs. 32.9% for ResNet-18 and 35.7% for PVT-Tiny).

Knowl 7 — Ablation of MetaFormer Components, Token Mixers, and Normalization

empirical result

Ablation studies on ImageNet-1K using PoolFormer-S12 (baseline top-1 accuracy: 77.2%, 11.9M parameters, 1.8G MACs) evaluate the impact of individual architectural components:

Ablation Aspect	Variant	Params (M)	MACs (G)	Top-1 Acc (%)
Baseline	PoolFormer-S12	11.9	1.8	77.2
Token Mixer	Identity Mapping	11.9	1.8	74.3
	Global Random Matrix ( $W_R \in \mathbb{R}^{N \times N}$ , frozen)	11.9 (+21M frozen)	3.3	75.8
	Depthwise Convolution	11.9	1.8	78.1
	Pooling Kernel Size $3 \to 5$	11.9	1.8	77.2
	Pooling Kernel Size $3 \to 7$	11.9	1.8	77.1
	Pooling Kernel Size $3 \to 9$	11.9	1.8	76.8
Normalization	Modified Layer Normalization $\to$ Layer Normalization	11.9	1.8	76.5
	Modified Layer Normalization $\to$ Batch Normalization	11.9	1.8	76.4
	Modified Layer Normalization $\to$ None	11.9	1.8	46.1
Activation	GELU $\to$ ReLU	11.9	1.8	76.4
	GELU $\to$ SiLU	11.9	1.8	77.2
Other Components	Residual Connections $\to$ None	11.9	1.8	0.1
	Channel MLP $\to$ None	2.5	0.2	5.7

Key takeaways:

MetaFormer macro-structure dominates: Even with identity mapping (no token mixing) or a frozen random matrix as the token mixer, the MetaFormer model attains 74.3% and 75.8% accuracy, respectively.
Indispensability of residuals and channel MLPs: Completely removing residual connections drops accuracy to 0.1%, and removing channel MLPs drops accuracy to 5.7%.
Normalization preference: Modified Layer Normalization outperforms vanilla Layer Normalization and Batch Normalization by +0.7% and +0.8%, respectively.

Knowl 8 — Hybrid Stage MetaFormer Architectures

empirical result

Because pooling is computationally efficient ( $\mathcal{O}(N)$ ) and can process high-resolution sequences whereas self-attention and spatial fully connected layers ( $\mathcal{O}(N^2)$ ) capture global context, hybrid MetaFormer architectures stack pooling token mixers in the initial stages and attention or spatial FC mixers in later stages where sequence length is reduced.

ImageNet-1K performance of 4-stage hybrid variants based on the S12 architecture:

Mixer Configuration across Stages [1, 2, 3, 4]	Params (M)	MACs (G)	Top-1 Acc (%)
(PoolFormer-S12)	11.9	1.8	77.2
[Pool, Pool, Pool, Attention]	14.0	1.9	78.3
[Pool, Pool, Attention, Attention]	16.5	2.5	81.0
[Pool, Pool, Pool, SpatialFC]	11.9	1.8	77.5
[Pool, Pool, SpatialFC, SpatialFC]	12.2	1.9	77.9

The hybrid variant [Pool, Pool, Attention, Attention] achieves 81.0% top-1 accuracy with 16.5M parameters and 2.5G MACs, matching the 81.0% accuracy of ResMLP-B24 while using $7.0\times$ fewer parameters (16.5M vs. 116M) and $9.2\times$ fewer MACs (2.5G vs. 23.0G).

Knowl 9 — PoolFormer Training Scheme and Hyperparameters
experimental setup
PoolFormer models are trained from scratch on ImageNet-1K (1.28M training images, 1,000 classes) with input resolution $224 \times 224$ for 300 epochs.
- Optimizer: AdamW with $\beta_1 = 0.9, \beta_2 = 0.999$ , $\epsilon = 10^{-8}$ , and weight decay of 0.05.
- Learning rate schedule: 5 warmup epochs followed by a cosine learning rate decay schedule. Peak learning rate scales linearly with batch size: $\text{lr} = 10^{-3} \times \frac{\text{batch\_size}}{1024}$ . The default setup uses batch size 4096 with peak learning rate $4 \times 10^{-3}$ .
- Data Augmentation: Mixup ( $\alpha = 0.8$ ), CutMix ( $\alpha = 1.0$ ), Cutmix-Mixup switch probability 0.5, RandAugment, and Random Erasing probability 0.25. Repeated Augmentation is disabled.
- Regularization: Label smoothing is set to 0.1. Standard dropout is set to 0. Stochastic depth is used with linear drop rates $d_r \in [0.1, 0.4]$ scaling with depth (0.1 for S12/S24, 0.2 for S36, 0.3 for M36, 0.4 for M48). LayerScale is initialized to $\epsilon = 10^{-5}$ for S12/S24 and $\epsilon = 10^{-6}$ for S36/M36/M48.

Coverage note — None was omitted; all key definitions, network configurations, mathematical formulations, experimental benchmarks, ablation results, and training hyperparameters are covered.

References

1.Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
2.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
3.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213–229. Springer, 2020.
4.Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
5.Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
6.Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691–1703. PMLR, 2020.
7.Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107.10224, 2021.
8.Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. Aˆ 2-nets: Double attention networks. Advances in Neural Information Processing Systems, 31:352–361, 2018.
9.Franc¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
10.MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020.
11.Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
12.Stéphane d’Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103.10697, 2021.
13.Stephane D’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2286–2296. PMLR, 18–24 Jul 2021.
14.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
15.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
16.Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arXiv preprint arXiv:2103.03404, 2021.
17.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
18.Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3–11, 2018.
19.fvcore Contributors. fvcore. https://github.com/facebookresearch/fvcore, 2021.
20.Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
21.Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers. arXiv preprint arXiv:2107.06263, 2021.
22.Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112, 2021.
23.Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
24.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
25.Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
26.Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mlp-like architecture for visual recognition. arXiv preprint arXiv:2106.12368, 2021.
27.Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646–661. Springer, 2016.
28.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR, 2015.
29.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
30.Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollar. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6399–6408, 2019.
31.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097–1105, 2012.
32.James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824, 2021.
33.Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
34.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
35.Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv preprint arXiv:2105.08050, 2021.
36.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, October 2021.
37.Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
38.Franck Mamalet and Christophe Garcia. Simplifying convnets for fast learning. In International Conference on Artificial Neural Networks, pages 58–65. Springer, 2012.
39.Andre Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro Aguiar, and Mario Figueiredo. Sparse and continuous attention mechanisms. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20989–21001. Curran Associates, Inc., 2020.
40.Pedro Henrique Martins, Zita Marinho, and Andre FT Martins. ∞-former: Infinite memory transformer. arXiv preprint arXiv:2109.00301, 2021.
41.Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, 2010.
42.Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations, 2022.
43.Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? arXiv preprint arXiv:2108.08810, 2021.
44.Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone selfattention in vision models. Advances in Neural Information Processing Systems, 32, 2019.
45.Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. arXiv preprint arXiv:2107.00645, 2021.
46.Frank Rosenblatt. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Technical report, Cornell Aeronautical Lab Inc Buffalo NY, 1961.
47.David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
48.Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
49.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
50.Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
51.Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
52.Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.03404, 2021.
53.Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
54.Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herve Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.17239, 2021.
55.Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12894–12904, 2021.
56.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
57.Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 568–578, October 2021.
58.Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
59.Ross Wightman, Hugo Touvron, and Herve Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476, 2021.
60.Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808, 2021.
61.Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10033–10041, 2021.
62.Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
63.Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 558–567, October 2021.
64.Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019.
65.Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
66.Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13001–13008, 2020.
67.Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.
68.Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, and Jiashi Feng. Refiner: Refining self-attention for vision transformers. arXiv preprint arXiv:2106.03714, 2021.

Citation

MLA

Yu, W., et al. “MetaFormer Is Actually What You Need for Vision”. arXiv, 2021, http://arxiv.org/abs/2111.11418v3.

APA

Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., & Yan, S. (2021). MetaFormer Is Actually What You Need for Vision. arXiv. http://arxiv.org/abs/2111.11418v3

Chicago

Yu, W., M. Luo, P. Zhou, et al. 2021. “MetaFormer Is Actually What You Need for Vision”. arXiv. http://arxiv.org/abs/2111.11418v3.

Harvard

Yu, W. et al. (2021) “MetaFormer Is Actually What You Need for Vision”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2111.11418v3.

Vancouver

1. Yu W, Luo M, Zhou P, Si C, Zhou Y, Wang X, Feng J, Yan S (2021) MetaFormer Is Actually What You Need for Vision. arXiv

BibTeX

@article{yu2021metaformer,
  title = {MetaFormer Is Actually What You Need for Vision},
  author = {Yu, Weihao and Luo, Mi and Zhou, Pan and Si, Chenyang and Zhou, Yichen and Wang, Xinchao and Feng, Jiashi and Yan, Shuicheng},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2111.11418v3},
  eprint = {2111.11418}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE