Back

Transformer in Transformer

Kai HanAn XiaoEnhua WuJianyuan GuoChunjing XuYunhe Wang

article2021NeurIPS2,284 citations

Introduces the Transformer in Transformer architecture to model fine-grained interactions within local image sub-patches alongside global patch representations, improving image classification accuracy with minimal computational overhead.

Listen

Modern computer vision models are rapidly shifting from traditional convolutional networks to transformer-based architectures that rely on self-attention mechanisms. However, standard vision transformers divide images into relatively coarse patches, treating them as simple sequences. This coarse division fails to capture fine-grained internal details and subtle color variations, creating a performance bottleneck when recognizing complex objects across different scales.

The article aims to demonstrate that embedding a secondary, fine-grained sub-transformer within standard vision transformer architectures significantly enhances visual feature extraction and overall recognition accuracy with minimal added computational cost.

To achieve this, the researchers introduced the Transformer-in-Transformer architecture, which conceptualizes coarse image patches as visual sentences and subdivides each patch into smaller sub-patches termed visual words. The system uses an inner transformer block to model local word interactions and an outer transformer block to model broader sentence interactions, fusing the representations layer by layer. The authors evaluated this design on the ImageNet benchmark containing over 1.2 million training images, as well as downstream tasks including image classification, object detection on the COCO dataset, and semantic segmentation on ADE20K.

The evaluation yielded several key findings. First, the small variant of the model achieved an 81.5% top-1 accuracy on ImageNet, outperforming comparable state-of-the-art vision transformers by 1.7% while requiring only an 8% increase in parameters and a 14% increase in computational operations. Second, across downstream tasks, the architecture consistently outperformed baseline transformers, raising object detection performance by 3.5 Average Precision on COCO and semantic segmentation by 1.0% mean Intersection over Union on ADE20K. Third, ablation analyses confirmed that modeling both global patch positions and local word positions is vital, as omitting position information resulted in an accuracy drop of up to 1.0%. Finally, throughput tests showed that selectively applying fine-grained blocks achieves high processing speeds (over 800 images per second) while maintaining strong accuracy.

These findings indicate that internal patch structures contain valuable visual cues that standard transformers discard. By capturing these local details without a severe computational penalty, the proposed method establishes a more effective balance between accuracy, model size, and execution speed. Although it outperforms standard convolutional baselines like ResNet, pure transformer architectures still trail specialized convolutional designs like EfficientNet in peak computational efficiency, highlighting that local feature modeling in transformers remains an active optimization area.

For engineering and product teams deploying computer vision systems, adopting nested transformer architectures offers a practical route to higher accuracy in visual classification, object detection, and segmentation. Practitioners should evaluate replacing standard vision transformer backbones with this nested approach, taking advantage of hybrid configurations—such as mixing nested blocks with standard transformer blocks—to meet strict latency requirements in production environments.

Readers should note that while the empirical evidence is strong across multiple standardized vision benchmarks, training these models requires substantial computational resources, including specialized graphics hardware, which carries associated energy and financial costs. Further evaluation in domain-specific, real-world deployment scenarios is recommended before full-scale operational rollout.

HuaweiInstitute of Software, Chinese Academy of SciencesUniversity of Macauattention mechanismcomputational efficiencyencoder-decoderTransformer in Transformervision transformervisual transformer
arXiv: 2103.00112huawei-noah/CV-Backbones
Cover for Transformer in Transformer

Abstract

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16×\times16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4×\times4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at this https URL, and the MindSpore code is available at this https URL.

Topics

attention mechanismcomputational efficiencyencoder-decoderarchitecture designspart of speech

Keywords

Transformer in Transformervision transformervisual transformerinner transformerouter transformerposition encodingsimage classificationobject detectionImageNet datasetlayer normalization

Table of Contents

  • 1 Introduction
  • 2 Approach
  • 2.1 Preliminaries
  • 2.2 Transformer in Transformer
  • 2.3 Complexity Analysis
  • 2.4 Network Architecture
  • 3 Experiments
  • 3.1 Datasets and Experimental Settings
  • 3.2 TNT on ImageNet
  • 3.3 Ablation Studies
  • 3.4 Visualization
  • 3.5 Transfer Learning
  • 4 Conclusion
  • A Appendix
  • A.1 Visualization of Attention Maps
  • A.2 Exploring SE module in TNT
  • A.3 Object Detection with Faster RCNN
  • References

Knowls

  1. Knowl 1 — Transformer-iN-Transformer (TNT) Architecture and Dual Data Flow

    model/method

    The Transformer-iN-Transformer (TNT) architecture models visual data at two nested granularities to capture both global patch interactions and local sub-patch structures:

    1. Visual Sentences and Visual Words: An input 2D image is partitioned uniformly into nn non-overlapping patches of resolution p×pp \times p, termed visual sentences. Each visual sentence is further subdivided into mm smaller sub-patches of spatial size s×ss \times s (where p=m⋅sp = \sqrt{m} \cdot s), termed visual words.
    2. Dual-Path Data Flow:
      • Inner Transformer Block (TinT_{in}): Operates locally on the sequence of mm visual words within each visual sentence to model fine-grained, pixel-level relationships and extract local features using multi-head self-attention (MSA) and a multi-layer perceptron (MLP). The inner block parameters are shared across all nn sentences in the same layer.
      • Linear Projection / Feature Fusion: The transformed word embeddings belonging to a visual sentence are vectorized and projected via a learnable linear layer into the sentence embedding domain, where they are added element-wise into the corresponding sentence token.
      • Outer Transformer Block (ToutT_{out}): Operates globally across the sequence of nn visual sentence embeddings (plus an appended learnable classification token ZclassZ_{\text{class}}) to capture long-range contextual relationships across the entire image.

    By stacking LL such TNT blocks, the network iteratively refines representations at both the sub-patch and patch levels, passing the final ZclassZ_{\text{class}} token to a fully-connected classification head.

  2. Knowl 2 — Mathematical Formulation of the TNT Block

    model/method

    Given an image decomposed into nn visual sentences X=[X1,X2,…,Xn]∈Rn×p×p×3X = [X^1, X^2, \dots, X^n] \in \mathbb{R}^{n \times p \times p \times 3}, each sentence XiX^i is partitioned into mm visual words Xi=[xi,1,xi,2,…,xi,m]X^i = [x^{i,1}, x^{i,2}, \dots, x^{i,m}] with xi,j∈Rs×s×3x^{i,j} \in \mathbb{R}^{s \times s \times 3}.

    1. Initialization and Position Encodings: Visual words are linearly projected into cc-dimensional word embeddings and augmented with a shared learnable 1D word position encoding Eword∈Rm×cE_{\text{word}} \in \mathbb{R}^{m \times c}: Y0i=[y0i,1,y0i,2,…,y0i,m]+Eword,y0i,j=FC(Vec(xi,j))∈RcY_0^i = [y_0^{i,1}, y_0^{i,2}, \dots, y_0^{i,m}] + E_{\text{word}}, \quad y_0^{i,j} = \text{FC}(\text{Vec}(x^{i,j})) \in \mathbb{R}^c where Vec(⋅)\text{Vec}(\cdot) denotes vectorization and FC\text{FC} is a linear layer. Visual sentence representations Z0=[Zclass,Z01,…,Z0n]∈R(n+1)×dZ_0 = [Z_{\text{class}}, Z_0^1, \dots, Z_0^n] \in \mathbb{R}^{(n+1) \times d} (initialized to zero) are augmented with learnable 1D sentence position encodings Esentence∈R(n+1)×dE_{\text{sentence}} \in \mathbb{R}^{(n+1) \times d}: Z0←Z0+EsentenceZ_0 \leftarrow Z_0 + E_{\text{sentence}}

    2. Block Operations at Layer l∈{1,2,…,L}l \in \{1, 2, \dots, L\}:

    • Inner Transformer Block (TinT_{in}) on visual words for each sentence i∈{1,…,n}i \in \{1, \dots, n\}: Yl′i=Yl−1i+MSA(LN(Yl−1i))Y_l^{'i} = Y_{l-1}^i + \text{MSA}(\text{LN}(Y_{l-1}^i)) Yli=Yl′i+MLP(LN(Yl′i))Y_l^i = Y_l^{'i} + \text{MLP}(\text{LN}(Y_l^{'i})) where MSA\text{MSA} is multi-head self-attention, LN\text{LN} is layer normalization, and MLP\text{MLP} consists of two linear layers with GELU non-linearity.

    • Word-to-Sentence Aggregation: Zl−1i=Zl−1i+FC(Vec(Yli))Z_{l-1}^i = Z_{l-1}^i + \text{FC}(\text{Vec}(Y_l^i)) where Vec(Yli)∈Rmc\text{Vec}(Y_l^i) \in \mathbb{R}^{mc} and FC:Rmc→Rd\text{FC}: \mathbb{R}^{mc} \to \mathbb{R}^d matches the sentence dimension dd.

    • Outer Transformer Block (ToutT_{out}) on sentence embeddings: Zl′=Zl−1+MSA(LN(Zl−1))Z_l' = Z_{l-1} + \text{MSA}(\text{LN}(Z_{l-1})) Zl=Zl′+MLP(LN(Zl′))Z_l = Z_l' + \text{MLP}(\text{LN}(Z_l'))

    The layer mapping is compactly denoted as Yl,Zl=TNT(Yl−1,Zl−1)Y_l, Z_l = \text{TNT}(Y_{l-1}, Z_{l-1}).

  3. Knowl 3 — Computational and Parameter Complexity of TNT Versus Standard Vision Transformer

    equation

    For a sequence length of nn, embedding dimension dd, MLP expansion ratio r=4r=4, and key/query/value dimensions equal to dd, a standard vision transformer (ViT) block has computational and parameter complexity: FLOPsT=2nd(6d+n)\text{FLOPs}_T = 2nd(6d + n) ParamsT=12d2\text{Params}_T = 12d^2

    A TNT block comprises an inner transformer TinT_{in} (processing nn sequences of length mm with dimension cc), a word-to-sentence linear mapping (dimension mc→dmc \to d), and an outer transformer ToutT_{out} (sequence length nn, dimension dd). Its computational and parameter complexity are: FLOPsTNT=2nmc(6c+m)+nmcd+2nd(6d+n)\text{FLOPs}_{\text{TNT}} = 2nmc(6c + m) + nmcd + 2nd(6d + n) ParamsTNT=12c2+mcd+12d2\text{Params}_{\text{TNT}} = 12c^2 + mcd + 12d^2

    Because the word embedding dimension is small (c≪dc \ll d) and the sub-patch count matches typical patch counts (O(m)≈O(n)O(m) \approx O(n)), the overhead is minimal. For instance, in TNT-S (d=384,n=196,c=24,m=16d=384, n=196, c=24, m=16), standard block FLOPs FLOPsT=376M\text{FLOPs}_T = 376\text{M} and FLOPsTNT=429M\text{FLOPs}_{\text{TNT}} = 429\text{M} (a 1.14×1.14\times ratio), while parameters increase by 1.08×1.08\times.

  4. Knowl 4 — TNT Architecture Configurations (TNT-Ti, TNT-S, TNT-B)

    model/method

    TNT architectures are built with a depth of L=12L=12 layers, an image patch size of p×p=16×16p \times p = 16 \times 16, and m=4×4=16m = 4 \times 4 = 16 sub-patches per patch (sub-patch size s×s=4×4s \times s = 4 \times 4). The MLP expansion ratio is set to r=4r=4 for both inner and outer transformer blocks.

    Model Depth Inner transformer Outer transformer Params FLOPs
    dim cc #heads MLP rr dim dd #heads MLP rr (M) (B)
    TNT-Ti 12 12 2 4 192 3 4 6.1 1.4
    TNT-S 12 24 4 4 384 6 4 23.8 5.2
    TNT-B 12 40 4 4 640 10 4 65.6 14.1

    Parameters and FLOPs are reported for standard input resolution 224×224224 \times 224 (n=14×14=196n = 14 \times 14 = 196 visual sentences).

  5. Knowl 5 — ImageNet-1K Classification Performance of TNT Models

    empirical result

    TNT models were trained from scratch on ImageNet ILSVRC-2012 (1.2M training images, 1,000 classes) for 300 epochs using AdamW, cosine learning rate schedule, batch size 1024, Rand-Augment, Random Erasing, Mixup, CutMix, DropPath (0.1), and repeated augmentation at 224×224224 \times 224 resolution.

    Model Resolution Params (M) FLOPs (B) Top-1 (%) Top-5 (%)
    DeiT-Ti 2242×24 5.7 1.3 72.2 -
    TNT-Ti 2242×24 6.1 1.4 73.9 91.9
    DeiT-S 2242×24 22.1 4.6 79.8 -
    PVT-Small 2242×24 24.5 3.8 79.8 -
    T2T-ViT_t-14 2242×24 21.5 5.2 80.7 -
    TNT-S 2242×24 23.8 5.2 81.5 95.7
    ViT-B/16 3843×84 86.4 55.5 77.9 -
    DeiT-B 2242×24 86.4 17.6 81.8 -
    T2T-ViT_t-24 2242×24 63.9 13.2 82.2 -
    TNT-B 2242×24 65.6 14.1 82.9 96.3

    TNT-S achieves 81.5% Top-1 accuracy, outperforming the baseline DeiT-S by 1.7% with comparable computational budget. TNT-B achieves 82.9% Top-1 accuracy, outperforming DeiT-B by 1.1% while using 24% fewer parameters and 20% fewer FLOPs.

  6. Knowl 6 — GPU Inference Throughput and Hybrid TNT Layer Configurations

    empirical result

    Because shallow layers capture detailed local structure while deep layers extract abstract semantics, replacing inner transformer blocks in deeper layers with vanilla transformer blocks reduces latency with minimal accuracy loss. Throughput is measured on an NVIDIA V100 GPU in PyTorch at batch size with 224×224224 \times 224 images on ImageNet:

    Model Indices of TNT blocks FLOPs (B) Throughput (images/s) Top-1 (%)
    DeiT-S - 4.6 907 79.8
    PVT-Small - 3.8 820 79.8
    PVT-Medium - 6.7 526 81.2
    TNT-S [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] 5.2 428 81.5
    TNT-S-1 [1, 4, 8, 12] 4.8 668 81.4
    TNT-S-2 [1, 6, 12] 4.7 704 81.3
    TNT-S-3 [1, 6] 4.7 757 81.1
    TNT-S-4 [1] 4.6 822 80.8

    TNT-S-1 retains inner transformer blocks in only 4 out of 12 layers, improving throughput by 56%56\% (from 428 to 668 images/s) while preserving 81.4% Top-1 accuracy (only a 0.1% decrease from full TNT-S).

  7. Knowl 7 — Ablation Studies on Position Encodings, Inner Attention Heads, and Word Granularity

    empirical result

    Ablation experiments conducted on the TNT-S architecture on ImageNet evaluate three design choices:

    1. Sentence-Level vs. Word-Level Position Encodings:

      • Both sentence-level and word-level encodings: 81.5% Top-1
      • Sentence-level only: 80.8% Top-1 (−0.7%-0.7\% drop)
      • Word-level only: 80.7% Top-1 (−0.8%-0.8\% drop)
      • No position encodings: 80.5% Top-1 (−1.0%-1.0\% drop) This confirms that global spatial coordinates and local sub-patch relative coordinates are complementary.

    2. Number of Heads in Inner Transformer Block:

      • 1 head: 81.0%
      • 2 heads: 81.4%
      • 4 heads: 81.5%
      • 6 heads: 81.3%
      • 8 heads: 81.1% A moderate number of heads (2 to 4) yields optimal local feature aggregation.

    3. Visual Word Granularity (mm) and Dimension (cc):

      • m=64m = 64 sub-patches (2×22 \times 2 size), c=6c = 6: 23.8M params, 5.1B FLOPs, 81.0% Top-1
      • m=16m = 16 sub-patches (4×44 \times 4 size), c=24c = 24: 23.8M params, 5.2B FLOPs, 81.5% Top-1
      • m=4m = 4 sub-patches (8×88 \times 8 size), c=96c = 96: 25.1M params, 6.0B FLOPs, 81.1% Top-1 m=16m=16 (4×44 \times 4 sub-patches) achieves the optimal trade-off between representation granularity and capacity.

  8. Knowl 8 — Transfer Learning on Downstream Image Classification Benchmarks

    empirical result

    Pre-trained TNT-S and TNT-B models fine-tuned at 384×384384 \times 384 resolution (with bicubic position encoding interpolation) evaluate transferability across generic (CIFAR-10, CIFAR-100) and fine-grained (Oxford 102 Flowers, Oxford-IIIT Pets, iNaturalist 2019) benchmarks:

    Model Params (M) ImageNet CIFAR10 CIFAR100 Flowers Pets iNat-19
    ViT-B/16384 86.4 77.9 98.1 87.1 89.5 93.8 -
    DeiT-B384 86.4 83.1 99.1 90.8 98.4 - -
    TNT-S384 23.8 83.1 98.7 90.1 98.8 94.7 81.4
    TNT-B384 65.6 83.9 99.1 91.1 99.0 95.0 83.2

    TNT-S (23.8M23.8\text{M} params) matches the ImageNet top-1 accuracy of DeiT-B (86.4M86.4\text{M} params) at 83.1% and surpasses it on Oxford Flowers (98.8% vs 98.4%). TNT-B achieves 83.2% top-1 on the challenging iNaturalist 2019 dataset, outperforming Grafit RegNetY-8GF (80.0%) and Grafit ResNet-50 (75.9%).

  9. Knowl 9 — Dense Prediction Performance on COCO Object Detection and ADE20K Semantic Segmentation

    empirical result

    TNT backbones transfer effectively to dense downstream computer vision tasks:

    1. Object Detection with DETR on COCO 2017 val (trained with AdamW for 50 epochs, with 2×22\times 2 average pooling on backbone features to match standard feature scales):

      • ResNet-50: 41M params, 32.3 AP
      • DeiT-S: 38M params, 33.9 AP
      • PVT-Small: 40M params, 34.7 AP
      • TNT-S: 39M params, 38.2 AP (+3.5+3.5 AP over PVT-Small, +4.3+4.3 AP over DeiT-S)

    2. Object Detection with Faster R-CNN + FPN on COCO 2017 minival (12 epochs schedule, multi-scale features constructed from layers 3, 6, 9, 12 with stride convolution/deconvolution layers):

      • ResNet-50: 41.5M params, 37.4 AP
      • DeiT-S: 46.4M params, 39.9 AP
      • TNT-S: 48.1M params, 41.5 AP (+1.6+1.6 AP over DeiT-S, +4.1+4.1 AP over ResNet-50)

    3. Semantic Segmentation with Trans2Seg on ADE20K val (trained with AdamW for 40k steps, 512×512512 \times 512 crop, single-scale testing):

      • ResNet-50: 56.1M params, 79.3G FLOPs, 39.7% mIoU
      • DeiT-S: 30.3M params, 27.2G FLOPs, 40.5% mIoU
      • PVT-Small: 32.1M params, 31.6G FLOPs, 42.6% mIoU
      • TNT-S: 32.1M params, 30.4G FLOPs, 43.6% mIoU (+1.0%+1.0\% over PVT-Small, +3.1%+3.1\% over DeiT-S)

  10. Knowl 10 — Efficiency Gap Between Pure Visual Transformers and Depthwise Convolutional Architectures

    limitation

    While TNT significantly outperforms standard pure vision transformers (ViT, DeiT, PVT, T2T-ViT) and standard convolutional networks (ResNet, RegNet), pure transformer architectures remain inferior in parameter and FLOP efficiency compared to highly optimized CNN architectures utilizing depthwise separable convolutions, such as EfficientNet. Specifically, EfficientNet-B3 achieves 81.6% ImageNet Top-1 accuracy with only 12.0M parameters and 1.8B FLOPs, whereas TNT-S requires 23.8M parameters and 5.2B FLOPs to achieve 81.5% Top-1 accuracy.

Coverage note — None was omitted; the extracted knowls comprehensively capture the TNT architecture, formal equations, complexity analysis, model configurations, ImageNet benchmark results, inference throughput analysis, ablation studies, transfer learning, dense prediction tasks, and architectural limitations.

References

  1. 1.Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  2. 2.Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
  3. 3.Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
  4. 4.Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
  5. 5.Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
  6. 6.Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
  7. 7.Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? arXiv preprint arXiv:2102.10882, 2021.
  8. 8.Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020.
  9. 9.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
  10. 10.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  11. 11.Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. arXiv preprint arXiv:2012.12556, 2020.
  12. 12.Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, 2020.
  13. 13.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
  14. 14.Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  15. 15.Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR, 2020.
  16. 16.Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018.
  17. 17.Huawei. Mindspore. https://www.mindspore.cn/, 2020.
  18. 18.Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
  19. 19.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1097–1105, 2012.
  20. 20.Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648, 2016.
  21. 21.Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
  22. 22.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755, 2014.
  23. 23.Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  24. 24.Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NeurIPS, 2019.
  25. 25.Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  26. 26.Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages 3498–3505. IEEE, 2012.
  27. 27.Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
  28. 28.Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
  29. 29.Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.
  30. 30.Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
  31. 31.Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
  32. 32.Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
  33. 33.Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, and Yunhe Wang. Augmented shortcuts for vision transformers. arXiv preprint arXiv:2106.15941, 2021.
  34. 34.Yehui Tang, Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. In CVPR, pages 5018–5028, 2021.
  35. 35.Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
  36. 36.Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning fine-grained image representations with coarse labels. arXiv preprint arXiv:2011.12982, 2020.
  37. 37.Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  38. 38.Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, pages 8769–8778, 2018.
  39. 39.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
  40. 40.Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
  41. 41.Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794–7803, 2018.
  42. 42.Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu, Ding Liang, and Ping Luo. Segmenting transparent object in the wild with transformer. In IJCAI, 2021.
  43. 43.Yixing Xu, Yunhe Wang, Kai Han, Yehui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. Renas: Relativistic evaluation of neural architecture search. In CVPR, pages 4411–4420, 2021.
  44. 44.Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
  45. 45.Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
  46. 46.Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
  47. 47.Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, 2021.
  48. 48.Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, volume 34, pages 13001–13008, 2020.
  49. 49.Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
  50. 50.Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.

Citation

MLA
Han, K., et al. “Transformer in Transformer”. Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 15908–19, https://proceedings.neurips.cc/paper_files/paper/2021/file/854d9fca60b4bd07f9bb215d59ef5561-Paper.pdf.
APA
Han, K., Xiao, A., Wu, E., Guo, J., XU, C., & Wang, Y. (2021). Transformer in Transformer. Advances in Neural Information Processing Systems, 34, 15908–15919. https://proceedings.neurips.cc/paper_files/paper/2021/file/854d9fca60b4bd07f9bb215d59ef5561-Paper.pdf
Chicago
Han, K., A. Xiao, E. Wu, J. Guo, C. XU, and Y. Wang. 2021. “Transformer in Transformer”. Advances in Neural Information Processing Systems 34: 15908–19. https://proceedings.neurips.cc/paper_files/paper/2021/file/854d9fca60b4bd07f9bb215d59ef5561-Paper.pdf.
Harvard
Han, K. et al. (2021) “Transformer in Transformer”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 15908–15919. Available at: https://proceedings.neurips.cc/paper_files/paper/2021/file/854d9fca60b4bd07f9bb215d59ef5561-Paper.pdf.
Vancouver
1. Han K, Xiao A, Wu E, Guo J, XU C, Wang Y (2021) Transformer in Transformer. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 15908–15919

BibTeX

@inproceedings{han2021transformer,
  title = {Transformer in Transformer},
  author = {Han, Kai and Xiao, An and Wu, Enhua and Guo, Jianyuan and XU, Chunjing and Wang, Yunhe},
  year = {2021},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {34},
  pages = {15908-15919},
  url = {https://proceedings.neurips.cc/paper_files/paper/2021/file/854d9fca60b4bd07f9bb215d59ef5561-Paper.pdf}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors