Back

Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

Lianghui ZhuBencheng LiaoQian ZhangXinlong WangWenyu LiuXinggang Wang

article2024ICML2,176 citations

Introduces Vision Mamba, a bidirectional state space backbone that replaces self-attention to achieve linear-time visual representation learning, outperforming standard vision transformers across core visual benchmarks while cutting GPU memory use by over 86% on high-resolution images.

Listen

Modern computer vision heavily relies on vision transformers to achieve high accuracy across complex visual recognition tasks. However, these transformer models depend on self-attention mechanisms whose computational and memory costs grow quadratically with the length of the visual sequence. This creates a severe performance bottleneck when processing high-resolution images, limiting operational throughput and inflating hardware infrastructure costs.

To resolve this bottleneck, the article evaluated whether self-attention can be eliminated in generic visual backbones by introducing Vision Mamba (Vim). Vim adapts modern selective state space models—initially developed for linear-time language modeling—into a bidirectional architecture designed specifically for sequential visual representation without relying on conventional attention mechanisms or hand-crafted spatial biases.

The authors designed Vim by flattening images into patch sequences, marking them with position embeddings, and processing them using forward and backward state space pathways optimized for hardware memory bandwidth. The model was empirically evaluated across multiple standardized benchmarks, including supervised image classification on ImageNet-1K (1.28 million training images), semantic segmentation on ADE20K, and object detection and instance segmentation on COCO 2017. The evaluation benchmarked both task accuracy and hardware efficiency against established baseline models such as Data-efficient Image Transformers (DeiT).

The evaluation produced four primary findings. First, Vim consistently surpassed DeiT across multiple model scales on ImageNet classification; for instance, the tiny variant achieved 76.1% top-1 accuracy compared to DeiT's 72.2%, rising to 78.3% after long-sequence fine-tuning. Second, Vim delivered major hardware efficiency gains at high resolutions: when processing 1248×1248 images, Vim operated 2.8 times faster than DeiT and reduced GPU memory consumption by 86.8%. Third, on dense downstream tasks, Vim-Tiny outperformed DeiT-Tiny on the COCO dataset by 1.3 box average precision points and 1.1 mask average precision points, showing particular strength on large objects due to improved long-range contextual modeling. Finally, in semantic segmentation on ADE20K, Vim achieved higher mean intersection over union than DeiT while matching traditional convolutional ResNet-101 performance with nearly half the parameter footprint.

These findings indicate that bidirectional state space models can match or exceed transformer-level visual understanding while maintaining subquadratic computational speed and linear memory scaling. For technical leadership and deployment teams, this translates into significantly lower memory footprint, reduced cloud computing costs, and the ability to process high-resolution visual inputs end-to-end without resorting to complex, performance-limiting windowing approximations.

Organizations handling high-resolution visual workflows—such as medical pathology, remote sensing, and long-form video—should consider piloting bidirectional state space architectures as drop-in alternatives to conventional vision transformers. Future technical development should prioritize exploring large-scale unsupervised pretraining (such as masked image modeling) and integrating Vim into multimodal vision-language architectures.

While empirical results on standard benchmarks show high confidence, current evaluations are primarily bounded by supervised training regimes and standard model sizes up to roughly 100 million parameters. Further validation is warranted at extreme foundation-model scales and across specialized industry domains before broad production replacement.

Beijing Academy of Artificial IntelligenceHorizon RoboticsHuazhong University of Science and Technologyclassificationencoder-decodergpuVision MambaBidirectional State Space Modelvisual representation learning
arXiv: 2401.09417hustvl/Vim
Cover for Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

Abstract

Recently the state space models (SSMs) with efficient hardware-aware designs, i.e., the Mamba deep learning model, have shown great potential for long sequence modeling. Meanwhile building efficient and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data is challenging for SSMs due to the position-sensitivity of visual data and the requirement of global context for visual understanding. In this paper, we show that the reliance on self-attention for visual representation learning is not necessary and propose a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. On ImageNet classification, COCO object detection, and ADE20k semantic segmentation tasks, Vim achieves higher performance compared to well-established vision transformers like DeiT, while also demonstrating significantly improved computation & memory efficiency. For example, Vim is 2.8×\times faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on images with a resolution of 1248×\times1248. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images and it has great potential to be the next-generation backbone for vision foundation models. Code is available at this https URL.

Topics

classificationencoder-decodergpuhigh resolution imageimage sequenceinferenceobject detectionrepresentation learningsemanticsstate space

Keywords

Vision MambaBidirectional State Space Modelvisual representation learningvision foundation modelsvision transformerslong-form video understandingdense prediction tasksposition embeddings

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Method
  • 3.1 Preliminaries
  • 3.2 Vision Mamba
  • 3.3 Vim Block
  • 3.4 Architecture Details
  • 3.5 Efficiency Analysis
  • 4 Experiment
  • 4.1 Image Classification
  • 4.2 Semantic Segmentation
  • 4.3 Object Detection and Instance Segmentation
  • 4.4 Ablation Study
  • 5 Conclusion and Future Work
  • References
  • A Visualization
  • B Additional Setting
  • C Extended Comparison on Hierarchical Architecture

Knowls

  1. Knowl 1 — Vision Mamba (Vim) Architecture and Token Sequence Formulation

    model/method

    Vision Mamba (Vim) is a pure state space model (SSM) designed as a generic backbone for visual representation learning without relying on self-attention mechanisms.

    Given an input 2D image t∈RH×W×Ct \in \mathbb{R}^{H \times W \times C}, where (H,W)(H, W) is the spatial resolution and CC is the number of channels, the image is partitioned into J=(H⋅W)/P2J = (H \cdot W) / P^2 non-overlapping patches xp∈RJ×(P2⋅C)x_p \in \mathbb{R}^{J \times (P^2 \cdot C)} with patch size P×PP \times P (default P=16P = 16). The flattened patches xpx_p are linearly projected to dimension DD using a learnable projection matrix W∈R(P2⋅C)×DW \in \mathbb{R}^{(P^2 \cdot C) \times D}, prepended/interleaved with a learnable class token tclst_{\text{cls}}, and summed with 1D learnable position embeddings Epos∈R(J+1)×DE_{\text{pos}} \in \mathbb{R}^{(J+1) \times D} to form the input token sequence T0T_0:

    T0=[tcls;tp1W;tp2W;… ;tpJW]+EposT_0 = [t_{\text{cls}}; t_p^1 W; t_p^2 W; \dots; t_p^J W] + E_{\text{pos}}

    The token sequence is processed through a stack of LL bidirectional Vim blocks:

    Tl=Vim(Tl−1)+Tl−1,l∈{1,…,L}T_l = \text{Vim}(T_{l-1}) + T_{l-1}, \quad l \in \{1, \dots, L\}

    For classification, the output class token TL0T_L^0 from the final layer is normalized and passed through a multi-layer perceptron (MLP) head to produce the final prediction vector p^\hat{p}:

    f=Norm(TL0),p^=MLP(f)f = \text{Norm}(T_L^0), \quad \hat{p} = \text{MLP}(f)

    Standard model configurations set block count L=24L = 24 and SSM state dimension N=16N = 16. The tiny variant (Vim-Ti) uses hidden state dimension D=192D = 192 and expanded state dimension E=384E = 384. The small variant (Vim-S) uses D=384D = 384 and E=768E = 768.

  2. Knowl 2 — Vim Block Processing with Bidirectional Selective State Spaces

    algorithm

    The Vim block processes a sequence of visual tokens bidirectionally to enable position-aware and globally context-aware feature extraction. The complete layer transformation is defined by the following algorithm:

    Algorithm: Vim Block Forward Pass
    Input: Token sequence Tl−1∈RB×M×DT_{l-1} \in \mathbb{R}^{B \times M \times D}, where BB is batch size, MM is sequence length, and DD is hidden dimension
    Output: Token sequence Tl∈RB×M×DT_l \in \mathbb{R}^{B \times M \times D}
    Tl−1′←Norm(Tl−1)T'_{l-1} \leftarrow \text{Norm}(T_{l-1})
    x←Linearx(Tl−1′)x \leftarrow \text{Linear}^x(T'_{l-1}) # Projected to dimension EE
    z←Linearz(Tl−1′)z \leftarrow \text{Linear}^z(T'_{l-1}) # Gating branch of dimension EE
    for direction o∈{forward,backward}o \in \{\text{forward}, \text{backward}\} do
        xo′←SiLU(Conv1do(x))x'_o \leftarrow \text{SiLU}(\text{Conv1d}_o(x))
        Bo←LinearoB(xo′)B_o \leftarrow \text{Linear}_o^B(x'_o) # Shape: (B,M,N)(B, M, N)
        Co←LinearoC(xo′)C_o \leftarrow \text{Linear}_o^C(x'_o) # Shape: (B,M,N)(B, M, N)
        Δo←log⁡(1+exp⁡(LinearoΔ(xo′)+ParameteroΔ))\Delta_o \leftarrow \log(1 + \exp(\text{Linear}_o^\Delta(x'_o) + \text{Parameter}_o^\Delta)) # Shape: (B,M,E)(B, M, E)
        Aˉo←exp⁡(Δo⊗ParameteroA)\bar{A}_o \leftarrow \exp(\Delta_o \otimes \text{Parameter}_o^A) # Discretized evolution parameter
        Bˉo←Δo⊗Bo\bar{B}_o \leftarrow \Delta_o \otimes B_o # Discretized projection parameter
        
        ho←0B×E×Nh_o \leftarrow 0_{B \times E \times N}
        yo←0B×M×Ey_o \leftarrow 0_{B \times M \times E}
        for i←0i \leftarrow 0 to M−1M-1 do
            ho←Aˉo[:,i,:,:]⊙ho+Bˉo[:,i,:,:]⊙xo′[:,i,:,None]h_o \leftarrow \bar{A}_o[:, i, :, :] \odot h_o + \bar{B}_o[:, i, :, :] \odot x'_o[:, i, :, \text{None}]
            yo[:,i,:]←ho⊗Co[:,i,:]y_o[:, i, :] \leftarrow h_o \otimes C_o[:, i, :]
        end for
    end for
    yforward′←yforward⊙SiLU(z)y'_{\text{forward}} \leftarrow y_{\text{forward}} \odot \text{SiLU}(z)
    ybackward′←ybackward⊙SiLU(z)y'_{\text{backward}} \leftarrow y_{\text{backward}} \odot \text{SiLU}(z)
    Tl←LinearT(yforward′+ybackward′)+Tl−1T_l \leftarrow \text{Linear}^T(y'_{\text{forward}} + y'_{\text{backward}}) + T_{l-1}
    return TlT_l

  3. Knowl 3 — Computational Complexity: SSM vs. Multi-Head Self-Attention

    theoretical result

    Given an input sequence T∈R1×M×DT \in \mathbb{R}^{1 \times M \times D}, where MM is the sequence length, DD is the hidden state dimension, E=2DE = 2D is the expanded inner dimension, and NN is the fixed SSM dimension (set to 16 by default), the theoretical computational complexity Ω\Omega of standard global self-attention compared to the selective State Space Model (SSM) layer in Vim is:

    Ω(self-attention)=4MD2+2M2D\Omega(\text{self-attention}) = 4MD^2 + 2M^2D

    Ω(SSM)=3M(2D)N+M(2D)N=8MDN\Omega(\text{SSM}) = 3M(2D)N + M(2D)N = 8MDN

    Standard self-attention scales quadratically with sequence length MM (O(M2D)O(M^2 D)), whereas the selective SSM layer scales linearly with sequence length MM (O(MDN)O(MDN)). Consequently, Vision Mamba exhibits subquadratic-time computation and linear memory complexity with respect to image token count.

  4. Knowl 4 — Hardware-Aware Memory and IO Optimization in Vision Mamba

    model/method

    To prevent the IO bottlenecks and out-of-memory errors typical of continuous/discrete SSM recurrent loops on GPUs, Vim employs hardware-aware memory optimizations:

    1. SRAM/HBM IO Reduction: A naive SSM implementation requires High Bandwidth Memory (HBM) IO on the order of O(BMEN)O(BMEN) for intermediate state transfers, where BB is batch size, MM is sequence length, EE is expanded state dimension, and NN is SSM dimension. Vim loads continuous parameters (Δo,Ao,Bo,Co)(\Delta_o, A_o, B_o, C_o) requiring O(BME+EN)O(BME + EN) bytes from slow HBM into fast GPU SRAM, discretizes to obtain Aˉo,Bˉo∈RB×M×E×N\bar{A}_o, \bar{B}_o \in \mathbb{R}^{B \times M \times E \times N} directly within SRAM, performs the recurrent scan entirely in SRAM, and writes back only the final output yo∈RB×M×Ey_o \in \mathbb{R}^{B \times M \times E} to HBM.

    2. Gradient and Activation Recomputation: To minimize GPU memory overhead during long sequence processing, Vim avoids storing intermediate states of size (B,M,E,N)(B, M, E, N) or intermediate activation maps (activation and convolution outputs). Instead, these activations and intermediate SSM states are recomputed on-the-fly during the backward pass.

  5. Knowl 5 — ImageNet-1K Classification Performance Across Model Scales

    data/table

    Vision Mamba models were evaluated on the ImageNet-1K benchmark (1.28M training images, 50K validation images, 1,000 classes) trained for 300 epochs at 224×224224 \times 224 resolution, and further evaluated with long-sequence fine-tuning (denoted by †\dagger, using a patch stride of 8 for 30 epochs).

    Method Image Size #Params ImageNet Top-1 Acc. (%)
    ResNet-18 2242224^2 12M 69.8
    ResNet-50 2242224^2 25M 76.2
    ResNet-101 2242224^2 45M 77.4
    ResNet-152 2242224^2 60M 78.3
    ResNeXt50-32x4d 2242224^2 25M 77.6
    RegNetY-4GF 2242224^2 21M 80.0
    ViT-B/16 3842384^2 86M 77.9
    ViT-L/16 3842384^2 307M 76.5
    DeiT-Ti 2242224^2 6M 72.2
    DeiT-S 2242224^2 22M 79.8
    DeiT-B 2242224^2 86M 81.8
    S4ND-ViT-B 2242224^2 89M 80.4
    Vim-Ti 2242224^2 7M 76.1
    Vim-Ti†^\dagger 2242224^2 7M 78.3 (+2.2)
    Vim-S 2242224^2 26M 80.3
    Vim-S†^\dagger 2242224^2 26M 81.4 (+1.1)
    Vim-B 2242224^2 98M 81.9
    Vim-B†^\dagger 2242224^2 98M 83.2 (+1.3)

    Vim-Ti outperforms DeiT-Ti by 3.9% top-1 accuracy (76.1% vs. 72.2%) with similar parameter scale. Vim-S surpasses ResNet-50 by 4.1% (80.3% vs. 76.2%) and DeiT-S by 0.5%. Long sequence fine-tuning yields further improvements across all Vim scales.

  6. Knowl 6 — Inference Speed and GPU Memory Scaling with Image Resolution

    empirical result

    When comparing tiny-size models (Vim-Ti vs. DeiT-Ti) across varying input image resolutions:

    1. At low resolution (512×512512 \times 512), Vim-Ti and DeiT-Ti exhibit comparable inference frame rates (FPS) and memory footprints (4.56 GB vs. 3.32 GB for backbone inference).
    2. As the image resolution scales to 1248×12481248 \times 1248 (corresponding to 6,084 tokens per image), DeiT-Ti incurs an out-of-memory (OOM) error during batch inference, while Vim-Ti runs 2.8×2.8\times faster than DeiT-Ti and consumes 86.8% less GPU memory (11.14 GB vs. 40.09 GB at 1024×10241024 \times 1024, and DeiT OOM at 1248×12481248 \times 1248).
    3. When integrated with a Feature Pyramid Network (FPN) for downstream vision tasks, Vim-Ti requires 73.2% less GPU memory and achieves 2.8×2.8\times higher throughput at 1248×12481248 \times 1248 resolution.

  7. Knowl 7 — ADE20K Semantic Segmentation Performance with UperNet

    data/table

    Semantic segmentation performance was evaluated on the ADE20K validation set (150 semantic categories) using the UperNet framework at an evaluation input scale of shorter side 512.

    Method Backbone Image Size #Params Val mIoU (%)
    DeepLab v3+ ResNet-101 5122512^2 63M 44.1
    UperNet ResNet-50 5122512^2 67M 41.2
    UperNet ResNet-101 5122512^2 86M 44.9
    UperNet DeiT-Ti 5122512^2 11M 39.2
    UperNet DeiT-S 5122512^2 43M 44.0
    UperNet Vim-Ti 5122512^2 13M 41.0
    UperNet Vim-S 5122512^2 46M 44.9

    Vim-Ti surpasses DeiT-Ti by 1.8 mIoU (41.0 vs. 39.2) and Vim-S surpasses DeiT-S by 0.9 mIoU (44.9 vs. 44.0). Moreover, Vim-S achieves the same validation mIoU (44.9) as the ResNet-101 backbone while requiring nearly 2×2\times fewer parameters (46M vs. 86M).

  8. Knowl 8 — COCO Object Detection and Instance Segmentation Performance

    data/table

    Object detection and instance segmentation performance were evaluated on the COCO 2017 validation dataset using the Cascade Mask R-CNN framework with 1024×10241024 \times 1024 image inputs.

    Backbone APbox\text{AP}^{\text{box}} AP50box\text{AP}^{\text{box}}_{50} AP75box\text{AP}^{\text{box}}_{75} APsbox\text{AP}^{\text{box}}_s APmbox\text{AP}^{\text{box}}_m APlbox\text{AP}^{\text{box}}_l
    DeiT-Ti 44.4 63.0 47.8 26.1 47.4 61.8
    Vim-Ti 45.7 63.9 49.6 26.1 49.0 63.2
    Backbone APmask\text{AP}^{\text{mask}} AP50mask\text{AP}^{\text{mask}}_{50} AP75mask\text{AP}^{\text{mask}}_{75} APsmask\text{AP}^{\text{mask}}_s APmmask\text{AP}^{\text{mask}}_m APlmask\text{AP}^{\text{mask}}_l
    DeiT-Ti 38.1 59.9 40.5 18.1 40.5 58.4
    Vim-Ti 39.2 60.9 41.7 18.2 41.8 60.2

    Vim-Ti operates directly on full 1024×10241024 \times 1024 sequences without 2D window attention priors, surpassing DeiT-Ti (which uses ViTDet window attention) by +1.3 APbox\text{AP}^{\text{box}} and +1.1 APmask\text{AP}^{\text{mask}}, with distinct gains on large objects (+1.4 APlbox\text{AP}^{\text{box}}_l and +1.8 APlmask\text{AP}^{\text{mask}}_l).

  9. Knowl 9 — Ablation of Bidirectional SSM Design Variants

    data/table

    Ablation experiments evaluated the bidirectional design choices in Vim on ImageNet-1K classification and ADE20K semantic segmentation (using a 2-layer Segmenter head) without class tokens.

    Bidirectional Strategy ImageNet Top-1 Acc. (%) ADE20K mIoU (%)
    None 73.2 32.3
    Bidirectional Layer 70.9 33.6
    Bidirectional SSM 72.8 33.2
    Bidirectional SSM + Conv1d 73.9 35.9
    • None: Unidirectional Mamba forward pass only.
    • Bidirectional Layer: Alternating forward and backward blocks in pairs.
    • Bidirectional SSM: Forward and backward SSM branches within each block without backward convolution.
    • Bidirectional SSM + Conv1d: Forward and backward 1D convolutions preceding forward and backward SSM branches.

    Combining bidirectional SSM with directional 1D convolutions delivers the highest ImageNet top-1 accuracy (73.9%) and segmentation mIoU (35.9%), outperforming the unidirectional baseline by +0.7% on classification and +3.6 mIoU on segmentation.

  10. Knowl 10 — Ablation of Token Classification Strategies

    data/table

    Ablation on ImageNet-1K classification evaluates different sequence aggregation mechanisms in Vim:

    Classification Strategy ImageNet Top-1 Acc. (%)
    Mean pool 73.9
    Max pool 73.4
    Head class token 75.2
    Double class token 74.3
    Middle class token 76.1
    • Mean pool: Global average pooling over output tokens of the final layer.
    • Max pool: Classification head applied per-token followed by max pooling.
    • Head class token: Learnable class token appended at the beginning of the patch sequence.
    • Double class token: Class tokens at both head and tail of the patch sequence.
    • Middle class token: Class token inserted at the sequence midpoint.

    Placing the class token at the middle of the patch sequence achieves the highest accuracy (76.1%), leveraging both the recurrent SSM structure and the central object spatial bias of ImageNet.

Coverage note — Omitted the extended Hier-Vim appendix experiment (Table 6 and Table 7) as it represents a Swin-style hybrid baseline extension rather than the core Vim non-hierarchical architecture, and omitted generic training hyperparameter restatements already standard in DeiT.

References

  1. 1.Bao, H., Dong, L., Piao, S., and Wei, F. Beit: BERT pre-training of image transformers. In ICLR, 2022. URL https://openreview.net/forum?id=p-BhZSz59o4.
  2. 2.Baron, E., Zimerman, I., and Wolf, L. 2-d ssm: A general spatial layer for visual transformers. arXiv preprint arXiv:2306.06635, 2023.
  3. 3.Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Tas¸ırlar, S. Introducing our multimodal models, 2023. URL https://www.adept.ai/blog/fuyu-8b.
  4. 4.Cai, Z. and Vasconcelos, N. Cascade r-cnn: High quality object detection and instance segmentation. TPAMI, 2019.
  5. 5.Caron, M., Touvron, H., Misra, I., J'egou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
  6. 6.Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  7. 7.Choromanski, K. M., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J. Q., Mohiuddin, A., Kaiser, L., Belanger, D. B., Colwell, L. J., and Weller, A. Rethinking attention with performers. In ICLR, 2021. URL https://openreview.net/forum?id=Ua6zuk0WRH.
  8. 8.Dai, Z., Liu, H., Le, Q. V., and Tan, M. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 34, 2021.
  9. 9.Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  10. 10.Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  11. 11.Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023.
  12. 12.Ding, X., Zhang, X., Han, J., and Ding, G. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In CVPR, 2022.
  13. 13.Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., and Guo, B. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
  14. 14.Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
  15. 15.d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., and Sagun, L. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021.
  16. 16.Fang, J., Xie, L., Wang, X., Zhang, X., Liu, W., and Tian, Q. Msg-transformer: Exchanging local spatial information by manipulating messenger tokens. In CVPR, 2022.
  17. 17.Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
  18. 18.Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with state space models. In ICLR, 2023. URL https://openreview.net/forum?id=COZDy0WYGg.
  19. 19.Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.-Y., Cubuk, E. D., Le, Q. V., and Zoph, B. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021.
  20. 20.Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
  21. 21.Gu, A., Goel, K., and R'e, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021a.
  22. 22.Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and R'e, C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In NeurIPS, 2021b.
  23. 23.Gu, A., Goel, K., Gupta, A., and R'e, C. On the parameterization and initialization of diagonal state space models. In NeurIPS, 2022.
  24. 24.Gupta, A., Gu, A., and Berant, J. Diagonal state spaces are as effective as structured state spaces. In NeurIPS, 2022.
  25. 25.He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
  26. 26.Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, 2017.
  27. 27.Islam, M. M. and Bertasius, G. Long movie clip classification with state-space video models. In ECCV, 2022.
  28. 28.Islam, M. M., Hasan, M., Athrey, K. S., Braskich, T., and Bertasius, G. Efficient movie scene detection using state-space transformers. In CVPR, 2023.
  29. 29.Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
  30. 30.Kalman, R. E. A new approach to linear filtering and prediction problems. 1960.
  31. 31.Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  32. 32.Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In ICLR, 2020. URL https://openreview.net/forum?id=rkgNKkHtvB.
  33. 33.Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
  34. 34.LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  35. 35.Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022a.
  36. 36.Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  37. 37.Li, Y., Cai, T., Zhang, Y., Chen, D., and Dey, D. What makes convolutional models great on long sequence modeling? In ICLR, 2022b.
  38. 38.Li, Y., Mao, H., Girshick, R., and He, K. Exploring plain vision transformer backbones for object detection. In ECCV, 2022c.
  39. 39.Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll'ar, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, 2014.
  40. 40.Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
  41. 41.Liu, S., Chen, T., Chen, X., Chen, X., Xiao, Q., Wu, B., K'arkk 'ainen, T., Pechenizkiy, M., Mocanu, D., and Wang, Z. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:2207.03620, 2022a.
  42. 42.Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024.
  43. 43.Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
  44. 44.Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In CVPR, 2022b.
  45. 45.Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In ICLR, 2019.
  46. 46.Ma, J., Li, F., and Wang, B. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722, 2024.
  47. 47.Mehta, H., Gupta, A., Cutkosky, A., and Neyshabur, B. Long range language modeling via gated state spaces. In ICLR, 2023. URL https://openreview.net/forum?id=5MkYIYCbva.
  48. 48.Nguyen, E., Goel, K., Gu, A., Downs, G., Shah, P., Dao, T., Baccus, S., and R'e, C. S4nd: Modeling images and videos as multidimensional signals with state spaces. In NeurIPS, 2022.
  49. 49.Qin, Z., Yang, S., and Zhong, Y. Hierarchically gated recurrent neural network for sequence modeling. In NeurIPS, 2023. URL https://openreview.net/forum?id=P1TCHxJwLB.
  50. 50.Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
  51. 51.Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., and Doll'ar, P. Designing network design spaces. In CVPR, 2020.
  52. 52.Rao, Y., Zhao, W., Zhu, Z., Lu, J., and Zhou, J. Global filter networks for image classification. Advances in neural information processing systems, 34:980–993, 2021.
  53. 53.Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  54. 54.Smith, J. T., De Mello, S., Kautz, J., Linderman, S., and Byeon, W. Convolutional state space models for long-range spatiotemporal modeling. In NeurIPS, 2023a.
  55. 55.Smith, J. T., Warrington, A., and Linderman, S. Simplified state space layers for sequence modeling. In ICLR, 2023b. URL https://openreview.net/forum?id=Ai8Hw3AXqks.
  56. 56.Strudel, R., Garcia, R., Laptev, I., and Schmid, C. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
  57. 57.Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language modelss. arXiv preprint arXiv:2307.08621, 2023.
  58. 58.Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In CVPR, 2015.
  59. 59.Tan, M. and Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
  60. 60.Tan, M. and Le, Q. Efficientnetv2: Smaller models and faster training. In ICML, 2021.
  61. 61.Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architecture for vision. In NeurIPS, 2021.
  62. 62.Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J'egou, H. Training data-efficient image transformers & distillation through attention. In ICML, 2021a.
  63. 63.Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J'egou, H. Training data-efficient image transformers & distillation through attention. In ICML, 2021b.
  64. 64.Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al. Resmlp: Feedforward networks for image classification with data-efficient training. TPAMI, 2022.
  65. 65.Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. Deep high-resolution representation learning for visual recognition. TPAMI, 2020a.
  66. 66.Wang, J., Yan, J. N., Gu, A., and Rush, A. M. Pretraining without attention. arXiv preprint arXiv:2212.10544, 2022.
  67. 67.Wang, J., Zhu, W., Wang, P., Yu, X., Liu, L., Omar, M., and Hamid, R. Selective structured state-spaces for long-form video understanding. In CVPR, 2023a.
  68. 68.Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020b.
  69. 69.Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
  70. 70.Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2023b.
  71. 71.Wang, W., Ma, S., Xu, H., Usuyama, N., Ding, J., Poon, H., and Wei, F. When an image is worth 1,024 x 1,024 words: A case study in computational pathology. arXiv preprint arXiv:2312.03558, 2023c.
  72. 72.Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. Cvt: Introducing convolutions to vision transformers. In ICCV, 2021.
  73. 73.Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified perceptual parsing for scene understanding. In ECCV, 2018a.
  74. 74.Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Unified perceptual parsing for scene understanding. In ECCV, 2018b.
  75. 75.Xie, S., Girshick, R., Doll'ar, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
  76. 76.Xing, Z., Ye, T., Yang, Y., Liu, G., and Zhu, L. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560, 2024.
  77. 77.Yan, J. N., Gu, J., and Rush, A. M. Diffusion models without attention. arXiv preprint arXiv:2311.18257, 2023.
  78. 78.Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., and Gao, J. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021.
  79. 79.Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819–10829, 2022.
  80. 80.Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., and Torralba, A. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.

Citation

MLA
Zhu, L., et al. “Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model”. arXiv, 2024, http://arxiv.org/abs/2401.09417v3.
APA
Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., & Wang, X. (2024). Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. arXiv. http://arxiv.org/abs/2401.09417v3
Chicago
Zhu, L., B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang. 2024. “Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model”. arXiv. http://arxiv.org/abs/2401.09417v3.
Harvard
Zhu, L. et al. (2024) “Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2401.09417v3.
Vancouver
1. Zhu L, Liao B, Zhang Q, Wang X, Liu W, Wang X (2024) Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. arXiv

BibTeX

@article{zhu2024vision,
  title = {Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model},
  author = {Zhu, Lianghui and Liao, Bencheng and Zhang, Qian and Wang, Xinlong and Liu, Wenyu and Wang, Xinggang},
  year = {2024},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2401.09417v3},
  eprint = {2401.09417}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/