Back

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Peng GaoShijie GengRenrui ZhangTeli MaRongyao FangYongfeng ZhangHongsheng LiYu Qiao

article2021IJCV1,903 citations

Proposes CLIP-Adapter, a parameter-efficient fine-tuning strategy that adds residual bottleneck layers to pre-trained vision-language features, outperforming prompt-tuning methods in few-shot visual classification.

Listen

Adapting large, pretrained vision-language foundation models to specialized tasks often requires heavy computational resources or extensive manual tuning. While recent efforts focus heavily on prompt tuning—optimizing automated text instructions using small sample sets—this approach can be computationally demanding to train and slow to run during practical deployment. The article evaluates an alternative adaptation technique called CLIP-Adapter, which uses lightweight feature adapters to adjust pretrained visual and language representations for image classification tasks using only a few training examples.

The authors conducted a series of computational experiments across 11 standard image classification benchmarks, evaluating performance across few-shot settings ranging from 1 to 16 labeled examples per category. CLIP-Adapter freezes the underlying foundation model and appends compact two-layer bottleneck networks after the visual and text encoders. It then dynamically blends newly learned features with the original pretrained embeddings using a residual connection, avoiding full-model retraining and heavy gradient calculations.

The findings show that CLIP-Adapter consistently outperforms zero-shot baselines, traditional linear classifiers, and advanced prompt-tuning methods across all 11 evaluated datasets. On benchmark tests, it achieved higher classification accuracy than prompt tuning while training about 16 times faster and providing approximately 29 times faster inference speed. Specialized, fine-grained visual categories saw dramatic performance gains of up to 51% over zero-shot baselines, whereas generic datasets saw modest, steady improvements. Additionally, using an adapter solely on the visual branch proved superior to text-only or combined adapters, and the method maintained higher accuracy under significant dataset distribution shifts.

These results demonstrate that parameter-efficient fine-tuning via lightweight feature adapters is a practical and superior alternative to prompt tuning for vision-language models. For engineering and technology leaders, this approach reduces computational costs, cuts development cycle times, and provides robust edge or real-time inference without sacrificing accuracy. Organizations deploying visual AI can leverage existing pretrained knowledge while customizing models on minimal annotated data.

Decision-makers should prioritize lightweight feature adapter architectures over complex prompt engineering pipelines when adapting vision-language models to specialized domain tasks. Future development efforts should evaluate extending these residual adapter mechanisms to broader multimodal applications, such as video and 3D perception, and test combinations with feature-caching techniques. However, users should note that optimal performance currently relies on calibrating a blending ratio between old and new knowledge, with domain-specific tasks requiring a higher mix of new features compared to generic visual tasks.

Centre for Perceptual and Interactive Intelligence (CPII)Rutgers UniversityShanghai Artificial Intelligence LaboratoryThe Chinese University of Hong KongclassificationengineeringoptimizationCLIP-Adaptervision-language modelsfeature adapters
arXiv: 2110.04544gaopengcuhk/CLIP-Adapter
  • Paper: Learning Transferable Visual Models From Natural Language Supervision, Alec Radford et al.. Introduces the CLIP vision-language foundation model whose pre-trained visual and textual representations provide the direct architecture and baseline for CLIP-Adapter.
  • Paper: Learning to Prompt for Vision-Language Models, Kaiyang Zhou et al. (2021). Presents Context Optimization (CoOp) for continuous prompt tuning in CLIP, which serves as the primary baseline and motivation that CLIP-Adapter aims to outperform via feature adapters.
  • Paper: Parameter-Efficient Transfer Learning for NLP, Neil Houlsby et al. (2019). Establishes the foundational bottleneck adapter architecture for parameter-efficient transfer learning that CLIP-Adapter adopts and adapts to multimodal vision-language representations.
  • Paper: Conditional Prompt Learning for Vision-Language Models, Kaiyang Zhou et al. (2022). Extends parameter-efficient adaptation of CLIP by conditioning prompt learning on visual instances, addressing the generalization and base-to-new category limitations explored in early adapter and prompt work.
  • Paper: Visual Prompt Tuning, Menglin Jia et al. (2022). Investigates prompt tuning specifically on the visual branch of Transformer backbones as an alternative parameter-efficient transfer paradigm to feature adaptation.
  • Paper: Towards a Unified View of Parameter-Efficient Transfer Learning, Junxian He et al. (2022). Synthesizes adapters, prompt tuning, and prefix tuning into a unified mathematical and empirical framework across parameter-efficient transfer learning methods.
Cover for CLIP-Adapter: Better Vision-Language Models with Feature Adapters

Abstract

Large-scale contrastive vision-language pre-training has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in \cite{radford2021learning} to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions.~To avoid non-trivial prompt engineering, context optimization \cite{zhou2021coop} has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples.~In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning.~While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pre-trained features.~As a consequence, CLIP-Adapter is able to outperform context optimization while maintains a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach. Code is released at t this https URL.

Topics

classificationengineeringoptimizationoptimization problemsprompt engineeringrepresentation learningvisual representationsvocabularynetwork interfacevision systems

Keywords

CLIP-Adaptervision-language modelsfeature adaptersfew-shot learnersvisual classification taskslearnable promptsimage-text pairs

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Our Approach
  • 3.1 Classifier Weight Generation for Few-Shot Learning
  • 3.2 CLIP-Adapter
  • 3.3 Variants of CLIP-Adapter
  • 4 Experiments
  • 4.1 Experimental Setups
  • 4.2 Comparison on Few-Shot Learning
  • 4.2.1 Baseline Models
  • 4.2.2 Performance Comparison & Analysis
  • 4.2.3 Efficiency Comparison & Analysis
  • 4.2.4 Observation on Optimal Residual Ratio
  • 4.2.5 Variants with Text Adapter
  • 4.2.6 Where to Insert CLIP-Adapter?
  • 4.2.7 Comparison with other Adapter Methods
  • 4.2.8 Comparison with ELEVATER Benchmark Baselines
  • 4.3 Visualization of Learned Manifold
  • 4.4 Ablation Studies
  • 5 Conclusions and Future Work
  • A Appendix
  • References

Knowls

  1. Knowl 1 — CLIP-Adapter Architecture and Residual Feature Blending

    model/method

    CLIP-Adapter adapts prompt-fixed, pretrained vision-language models (such as CLIP) to downstream few-shot classification tasks by appending lightweight feature adapters to the output representations of the frozen visual and textual backbones rather than fine-tuning the full backbone or optimizing continuous prompt vectors.

    The architecture maintains the entire pretrained CLIP visual encoder (e.g., ResNet-50 or ViT) and textual encoder (Transformer) completely frozen. At the output of the encoders, lightweight bottleneck adapters—each comprising two linear layers with an intermediate ReLU activation and a dimension reduction factor—are attached. To avoid catastrophic forgetting of zero-shot representations and prevent overfitting on small training samples, CLIP-Adapter employs residual-style feature blending: the newly learned feature from the adapter is scaled by a residual ratio and added directly to the original frozen representation.

    During training, gradients only propagate through the bottleneck adapter layers, which eliminates the need to calculate gradients through the large pretrained backbones and significantly accelerates convergence while reducing GPU memory consumption.

  2. Knowl 2 — Mathematical Formulation and Contrastive Objective of CLIP-Adapter

    equation

    Given an input image II, a frozen visual backbone computes the zero-shot image feature embedding f∈RDf \in \mathbb{R}^D, where DD is the embedding dimension. For KK target categories with names {Ci}i=1K\{C_i\}_{i=1}^K, hand-crafted text prompt templates HH are passed through a frozen text tokenizer and textual encoder to produce prototype classifier weight vectors W=[W1,…,WK]∈RD×KW = [W_1, \dots, W_K] \in \mathbb{R}^{D \times K}, where Wi∈RDW_i \in \mathbb{R}^D.

    The visual adapter Av(⋅)A_v(\cdot) and textual adapter At(⋅)A_t(\cdot) each comprise two linear bottleneck transformations:

    Av(f)=ReLU(fTW1v)W2vA_v(f) = \text{ReLU}(f^T W_1^v) W_2^v

    At(W)=ReLU(WTW1t)W2tA_t(W) = \text{ReLU}(W^T W_1^t) W_2^t

    where W1v,W1t∈RD×DmidW_1^v, W_1^t \in \mathbb{R}^{D \times D_{\text{mid}}} and W2v,W2t∈RDmid×DW_2^v, W_2^t \in \mathbb{R}^{D_{\text{mid}} \times D} are learnable weights with intermediate bottleneck dimensionality Dmid=D/4D_{\text{mid}} = D/4.

    The newly adapted features are blended with the original frozen representations via residual connections using residual ratios α,β∈[0,1]\alpha, \beta \in [0, 1]:

    f⋆=αAv(f)T+(1−α)ff^\star = \alpha A_v(f)^T + (1 - \alpha) f

    W⋆=βAt(W)T+(1−β)WW^\star = \beta A_t(W)^T + (1 - \beta) W

    The predicted probability pip_i that image II belongs to class ii is calculated using a softmax function with temperature τ>0\tau > 0:

    pi=exp⁡((Wi⋆)Tf⋆/τ)∑j=1Kexp⁡((Wj⋆)Tf⋆/τ)p_i = \frac{\exp((W_i^\star)^T f^\star / \tau)}{\sum_{j=1}^K \exp((W_j^\star)^T f^\star / \tau)}

    The parameters θ={W1v,W2v,W1t,W2t}\theta = \{W_1^v, W_2^v, W_1^t, W_2^t\} are trained over NN labeled few-shot examples using the contrastive cross-entropy loss:

    Lθ=−1N∑i=1Nlog⁡exp⁡((Wi⋆)Tfi⋆/τ)∑j=1Nexp⁡((Wj⋆)Tfi⋆/τ)\mathcal{L}_\theta = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp((W_i^\star)^T f_i^\star / \tau)}{\sum_{j=1}^N \exp((W_j^\star)^T f_i^\star / \tau)}

  3. Knowl 3 — Efficiency and Accuracy Trade-offs on 16-Shot ImageNet Classification

    data/table

    The table below compares training time, learnable parameter count, training GPU memory footprint, per-sample inference latency, and top-1 classification accuracy across adaptation methods on the ImageNet dataset using a 16-shot per class setup with a ResNet-50 visual backbone on a single NVIDIA A100 GPU:

    Models Train Time Parameters GPU Mem. Infer Speed Accuracy
    Zero-shot CLIP 0 0 2227 MiB 10.2ms 55.41%
    Linear Probe CLIP 13min 1.02M - - 53.44%
    CoOp 14h 40min 0.02M 7193 MiB 299.6ms 60.46%
    CLIP-Adapter 50min 0.52M 2227 MiB 10.6ms 61.33%

    CLIP-Adapter achieves the highest top-1 classification accuracy (61.33%), surpassing Zero-shot CLIP by 5.92%, Linear Probe CLIP by 7.89%, and Context Optimization (CoOp) by 0.87%.

    Compared to CoOp, which must backpropagate gradients through the entire text encoder to update continuous soft prompt tokens (requiring 7193 MiB GPU memory and 14h 40min training time), CLIP-Adapter only backpropagates through the lightweight bottleneck adapter layers. This reduces training time by approximately 17.6×17.6\times (50 min), lowers GPU memory consumption to 2227 MiB (a 3.2×3.2\times reduction), and speeds up inference by 28.3×28.3\times (10.6 ms vs 299.6 ms) because the prompt text embeddings are precomputed once.

  4. Knowl 4 — Few-Shot Classification Performance Across 11 Visual Recognition Datasets

    empirical result

    CLIP-Adapter was evaluated against Zero-shot CLIP, Linear Probe CLIP, and CoOp across 11 visual classification benchmarks (ImageNet, StanfordCars, UCF101, Caltech101, Flowers102, SUN397, DTD, EuroSAT, FGVCAircraft, OxfordPets, Food101) across 1, 2, 4, 8, and 16 shots per class.

    CLIP-Adapter consistently outperforms all baselines across all shot counts on average. The performance margin is especially large in extreme low-data regimes (1-shot and 2-shot setups), where Linear Probe CLIP suffers from severe overfitting and fails to match Zero-shot CLIP (e.g., on OxfordPets, 1-shot and 2-shot margins between CLIP-Adapter and Linear Probe CLIP are 53.6% and 42.16%; on ImageNet, 37.17% and 27.58%).

    Under 16-shot training, the absolute performance improvements of CLIP-Adapter over Zero-shot CLIP are largest on specialized fine-grained datasets:

    • EuroSAT: +51.32%
    • Flowers102: +31.89%
    • DTD: +25.43%
    • StanfordCars: +24.89%
    • FGVCAircraft: +21.96%
    • UCF101: +15.45%
    • SUN397: +11.90%
    • Caltech101: +9.01%
    • ImageNet: +5.83%
    • OxfordPets: +4.19%
    • Food101: +2.02%

  5. Knowl 5 — Comparison of Visual, Textual, and Joint Feature Adapters

    empirical result

    Three structural variants of CLIP-Adapter were evaluated across four representative datasets (the fine-grained datasets DTD and EuroSAT, and the generic datasets Caltech101 and ImageNet) using learnable residual parameters α\alpha and β\beta:

    1. Visual Adapter only: DTD 66.06%, EuroSAT 82.85%, Caltech101 93.43%, ImageNet 61.33%.
    2. Text Adapter only: DTD 63.63%, EuroSAT 81.85%, Caltech101 92.78%, ImageNet 60.06%.
    3. Visual + Text Adapter: DTD 64.78%, EuroSAT 82.74%, Caltech101 93.10%, ImageNet 60.70%.

    Key observations:

    • Both the visual adapter and the textual adapter individually provide substantial accuracy gains over Zero-shot CLIP.
    • The visual adapter alone consistently outperforms the text adapter alone across all datasets, demonstrating that adapting visual features is more critical for few-shot image recognition because the domain shift between pretraining and target data is larger in the visual manifold than in the textual class name space.
    • Activating visual and textual adapters concurrently yields lower accuracy than using the visual adapter alone across all evaluated datasets, showing that learning both adapters simultaneously captures redundant information or causes gradient conflicts.

  6. Knowl 6 — Impact of Adapter Insertion Depth in Vision Transformers

    data/table

    The table below evaluates the placement of the visual adapter across different intermediate layers and the final output layer of a 12-layer Vision Transformer (ViT-B/16) backbone on 16-shot ImageNet classification:

    Insert Layers All 12 10 8 6 4 2 0
    Accuracy (%) 67.67 70.88 71.85 70.55 70.03 70.14 69.43 69.05
    GPU Memory (GiB) 5.98 2.22 2.65 2.89 3.23 4.46 4.88 5.14
    Parameters (M) 5.20 0.52 0.78 0.78 0.78 0.78 0.78 0.78

    Attaching a single adapter at the final layer (Layer 12) achieves 70.88% accuracy with the smallest GPU memory requirement (2.22 GiB) and fewest parameters (0.52M). Inserting adapters across all 12 transformer layers introduces 5.20M parameters and increases memory footprint to 5.98 GiB, but degrades accuracy to 67.67% due to overfitting on few-shot training data. Inserting adapters at earlier layers (Layers 0 to 8) requires higher memory (2.89–5.14 GiB) to backpropagate gradients through deep transformer blocks and generally yields lower classification performance.

  7. Knowl 7 — Effect of Residual Ratio on Domain Adaptation to Specialized versus Generic Tasks

    data/table

    The table below details the effect of varying the residual weighting ratio α\alpha (from α=0.0\alpha=0.0 to α=1.0\alpha=1.0) on 16-shot classification accuracy for DTD (fine-grained texture domain) and ImageNet (generic object domain) using a ResNet-50 visual adapter:

    Ratio α\alpha 0 0.2 0.4 0.6 0.8 1.0
    DTD (%) 40.72 54.59 64.84 66.06 65.96 63.79
    ImageNet (%) 60.46 61.33 61.17 60.77 59.79 59.05

    Key conclusions:

    • At α=0\alpha = 0, the model reduces to Zero-shot CLIP (no newly learned features are used).
    • At α=1.0\alpha = 1.0, the model discards original CLIP embeddings and relies entirely on adapter output (CLIP-Adapter without residual connection), leading to degraded accuracy (63.79% on DTD, 59.05% on ImageNet) due to overfitting.
    • The optimal residual ratio correlates directly with the semantic gap between pretraining data and target downstream data: specialized fine-grained datasets (DTD, EuroSAT) require a larger residual ratio (optimal α≈0.6−0.8\alpha \approx 0.6 - 0.8) to incorporate more newly learned task knowledge, whereas broad generic datasets (ImageNet, Caltech101) achieve optimal performance with a smaller residual ratio (optimal α≈0.2\alpha \approx 0.2), prioritizing preservation of general zero-shot representations.

  8. Knowl 8 — Bottleneck Layer Hidden Dimension Ablation in CLIP-Adapter

    data/table

    The table below presents the 16-shot classification performance on DTD and ImageNet when varying the intermediate bottleneck dimension DmidD_{\text{mid}} of the visual adapter relative to the original CLIP feature dimension DD (where D=1024D=1024 for ResNet-50):

    Dimension DD D/2D/2 D/4D/4 D/8D/8 D/16D/16 D/32D/32
    DTD (%) 65.03 65.62 66.06 64.93 63.75 63.50
    ImageNet (%) 59.78 60.03 61.33 60.06 60.02 59.45

    A bottleneck dimension of Dmid=D/4D_{\text{mid}} = D/4 (256 dimensions) achieves the best classification accuracy on both datasets (66.06% on DTD, 61.33% on ImageNet). Larger intermediate dimensions (D,D/2D, D/2) increase parameter count and lead to overfitting, while overly narrow dimensions (D/8D/8 down to D/32D/32) compress feature representations too heavily, impairing task-specific expressiveness.

  9. Knowl 9 — Out-of-Distribution Robustness Under Distribution Shift

    data/table

    The table below evaluates models trained on 16-shot ImageNet (source) when tested directly on four out-of-distribution target datasets (ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R):

    Datasets Source Target
    ImageNet ImageNetV2 ImageNet-Sketch ImageNet-A ImageNet-R
    Zero-Shot CLIP 55.41 48.08 31.67 18.63 53.45
    Linear Probe CLIP 53.44 43.40 17.63 11.66 32.63
    CoOp 60.46 52.17 31.14 19.62 53.31
    CLIP-Adapter 61.33 52.67 32.04 20.12 54.75

    CLIP-Adapter achieves the highest out-of-distribution generalization accuracy across all target shifts (52.67% on ImageNetV2, 32.04% on ImageNet-Sketch, 20.12% on ImageNet-A, and 54.75% on ImageNet-R), outperforming Zero-shot CLIP, Linear Probe CLIP, and CoOp. In contrast to Linear Probe CLIP, which drops substantially under distribution shifts (e.g., dropping to 17.63% on ImageNet-Sketch and 11.66% on ImageNet-A), CLIP-Adapter preserves CLIP's zero-shot robustness through its residual feature blending mechanism.

  10. Knowl 10 — Comparison of Full Backbone Fine-Tuning versus Adapter-Only Tuning

    data/table

    The table below reports 16-shot ImageNet classification accuracy and training time when fine-tuning different components of CLIP with ResNet-50 backbone (where checkmarks indicate components unfrozen during training):

    Visual Encoder Textual Encoder Adapter Accuracy Train Time
    - - ✓ 61.33% 50min
    ✓ - ✓ 60.07% 1h 20min
    - ✓ ✓ 57.88% 3h+
    ✓ ✓ ✓ 52.78% 3h+

    Freezing both the visual and textual encoders and updating only the lightweight bottleneck adapter achieves the highest accuracy (61.33%) in the shortest training time (50 min). Unfreezing the visual encoder drops accuracy to 60.07% while increasing training time to 1h 20min. Unfreezing the textual encoder causes a steeper drop to 57.88% (3h+). Unfreezing the full network (visual encoder, textual encoder, and adapter) results in severe overfitting on the few-shot training set, causing accuracy to collapse to 52.78% (lower than zero-shot CLIP at 55.41%).

  11. Knowl 11 — Comparison with Alternative Parameter-Efficient Adapter Architectures and Baselines

    empirical result

    CLIP-Adapter was compared against established parameter-efficient fine-tuning adapter architectures and ELEVATER benchmark baselines on 16-shot ImageNet classification using a ViT-B/16 backbone:

    1. Traditional adapter baselines:

      • CLIP-Adapter: 70.88% accuracy, 2.22 GiB GPU memory, 0.52M parameters.
      • Houlsby adapter: 70.16% accuracy, 6.74 GiB GPU memory, 12.84M parameters.
      • He (Parallel Adapter): 70.84% accuracy, 6.06 GiB GPU memory, 9.62M parameters. CLIP-Adapter matches or exceeds the performance of traditional adapters while using 18.5×18.5\times to 24.7×24.7\times fewer parameters and roughly 3×3\times less GPU memory because it avoids dense per-layer insertions throughout the transformer backbone.

    2. ELEVATER benchmark projection baselines:

      • Random-Init with Two-Projection (R-2P): 68.94%
      • Language-Init with Two-Projection (L-2P): 69.77%
      • Language-Init with One-Projection (L-1P): 70.38%
      • CoOp: 70.16%
      • CLIP-Adapter: 70.88% CLIP-Adapter outperforms all projection baselines, demonstrating that residual feature blending is superior to training non-residual random or language-initialized linear projections.

  12. Knowl 12 — Effect of Prompt Styles on CLIP-Adapter Few-Shot Performance

    empirical result

    An ablation study on 16-shot ImageNet classification evaluated the effect of different textual prompt designs when used alongside CLIP-Adapter's visual adapter:

    • Single Hard Prompt (default template: "a photo of a {class}"): 61.33% accuracy.
    • Hard Prompt Ensemble (averaging embeddings across 7 hand-crafted templates, such as "itap of a {class}", "a bad photo of the {class}", "art of the {class}", "a photo of the small {class}"): 61.68% accuracy.
    • Hard + Soft Prompt Mixture (combining hand-crafted templates with learnable continuous prompt tokens): 59.69% accuracy.

    Ensembling discrete hand-crafted prompts provides a 0.35% boost over a single prompt template, whereas combining continuous learnable soft prompts with adapters degrades performance by 1.64%, indicating that fixed discrete natural language prompts provide a more reliable and robust prior for adapter fine-tuning.

Coverage note — None was omitted; all contributed methodology, equations, experimental comparisons, ablation studies, and architectural variants are covered.

References

  1. 1.Alayrac JB, Donahue J, Luc P, et al (2022) Flamingo: a visual language model for few-shot learning. In: Oh AH, Agarwal A, Belgrave D, et al (eds) Advances in Neural Information Processing Systems
  2. 2.Anderson P, He X, Buehler C, et al (2018) Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR
  3. 3.Bossard L, Guillaumin M, Van Gool L (2014) Food-101–mining discriminative components with random forests. In: European conference on computer vision, Springer, pp 446–461
  4. 4.Brown T, Mann B, Ryder N, et al (2020) Language models are few-shot learners. In: NeurIPS
  5. 5.Carion N, Massa F, Synnaeve G, et al (2020) End-to-end object detection with transformers. In: ECCV
  6. 6.Chen YC, Li L, Yu L, et al (2020) Uniter: Learning universal image-text representations. In: ECCV
  7. 7.Cimpoi M, Maji S, Kokkinos I, et al (2014) Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3606–3613
  8. 8.Conneau A, Khandelwal K, Goyal N, et al (2020) Unsupervised cross-lingual representation learning at scale. In: ACL
  9. 9.Deng J, Dong W, Socher R, et al (2009) Imagenet: A large-scale hierarchical image database. In: CVPR
  10. 10.Devlin J, Chang MW, Lee K, et al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT
  11. 11.Dong L, Yang N, Wang W, et al (2019) Unified language model pre-training for natural language understanding and generation. In: NeurIPS
  12. 12.Dosovitskiy A, Beyer L, Kolesnikov A, et al (2021) An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR
  13. 13.Fei-Fei L, Fergus R, Perona P (2004) Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: 2004 conference on computer vision and pattern recognition workshop, IEEE, pp 178–178
  14. 14.Gao P, Jiang Z, You H, et al (2019) Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In: CVPR
  15. 15.Gao P, Lu J, Li H, et al (2021) Container: Context aggregation network. In: NeurIPS
  16. 16.Gao P, Zheng M, Wang X, et al (2021) Fast convergence of detr with spatially modulated co-attention. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3621–3630
  17. 17.Gao T, Fisch A, Chen D (2021) Making pre-trained language models better few-shot learners. In: ACL-IJCNLP
  18. 18.Gu Y, Han X, Liu Z, et al (2022) Ppt: Pretrained prompt tuning for few-shot learning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 8410–8423
  19. 19.He J, Zhou C, Ma X, et al (2022) Towards a unified view of parameter-efficient transfer learning. In: International Conference on Learning Representations
  20. 20.He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: CVPR
  21. 21.Helber P, Bischke B, Dengel A, et al (2019) Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7):2217–2226
  22. 22.Hendrycks D, Basart S, Mu N, et al (2021) The many faces of robustness: A critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 8340–8349
  23. 23.Hendrycks D, Zhao K, Basart S, et al (2021) Natural adversarial examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 15,262–15,271
  24. 24.Houlsby N, Giurgiu A, Jastrzebski S, et al (2019) Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning, PMLR, pp 2790–2799
  25. 25.Howard AG, Zhu M, Chen B, et al (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:170404861
  26. 26.Hu S, Zhang Z, Ding N, et al (2022) Sparse structure search for parameter-efficient tuning. arXiv preprint arXiv:220607382
  27. 27.Jia C, Yang Y, Xia Y, et al (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In: ICML
  28. 28.Jia M, Tang L, Chen BC, et al (2022) Visual prompt tuning. In: ECCV, pp 709–727
  29. 29.Jiang Z, Xu FF, Araki J, et al (2020) How can we know what language models know? Transactions of the Association for Computational Linguistics 8:423–438
  30. 30.Kim JH, Jun J, Zhang BT (2018) Bilinear attention networks. In: NIPS
  31. 31.Krause J, Stark M, Deng J, et al (2013) 3d object representations for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops, pp 554–561
  32. 32.Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: NIPS
  33. 33.Lester B, Al-Rfou R, Constant N (2021) The power of scale for parameter-efficient prompt tuning. In: EMNLP
  34. 34.Li C, Liu H, Li LH, et al (2022) ELEVATER: A benchmark and toolkit for evaluating language-augmented visual models. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track
  35. 35.Li J, Selvaraju R, Gotmare A, et al (2021) Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems 34:9694–9705
  36. 36.Li X, Yin X, Li C, et al (2020) Oscar: Object-semantics aligned pre-training for vision-language tasks. In: ECCV
  37. 37.Li XL, Liang P (2021) Prefix-tuning: Optimizing continuous prompts for generation. In: ACL
  38. 38.Lian D, Zhou D, Feng J, et al (2022) Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems 35:109–123
  39. 39.Lin Z, Geng S, Zhang R, et al (2022) Frozen clip models are efficient video learners. ECCV 2022
  40. 40.Liu P, Yuan W, Fu J, et al (2023) Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55(9):1–35
  41. 41.Liu X, Zheng Y, Du Z, et al (2021) Gpt understands, too. arXiv preprint arXiv:210310385
  42. 42.Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: CVPR
  43. 43.Lu J, Batra D, Parikh D, et al (2019) Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS
  44. 44.Van der Maaten L, Hinton G (2008) Visualizing data using t-sne. Journal of machine learning research 9(11)
  45. 45.Maji S, Rahtu E, Kannala J, et al (2013) Fine-grained visual classification of aircraft. arXiv preprint arXiv:13065151
  46. 46.Mao M, Zhang R, Zheng H, et al (2021) Dual-stream network for visual recognition. Advances in Neural Information Processing Systems 34:25,346–25,358
  47. 47.Nilsback ME, Zisserman A (2008) Automated flower classification over a large number of classes. In: 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, IEEE, pp 722–729
  48. 48.Parkhi OM, Vedaldi A, Zisserman A, et al (2012) Cats and dogs. In: 2012 IEEE conference on computer vision and pattern recognition, IEEE, pp 3498–3505
  49. 49.Radford A, Wu J, Child R, et al (2019) Language models are unsupervised multitask learners. OpenAI blog
  50. 50.Radford A, Kim JW, Hallacy C, et al (2021) Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, PMLR, pp 8748–8763
  51. 51.Recht B, Roelofs R, Schmidt L, et al (2019) Do imagenet classifiers generalize to imagenet? In: International Conference on Machine Learning, PMLR, pp 5389–5400
  52. 52.Ren S, He K, Girshick R, et al (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: NIPS
  53. 53.Shin T, Razeghi Y, Logan IV RL, et al (2020) Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In: EMNLP
  54. 54.Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR
  55. 55.Soomro K, Zamir AR, Shah M (2012) Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:12120402
  56. 56.Sun T, Shao Y, Qian H, et al (2022) Black-box tuning for language-model-as-a-service. In: International Conference on Machine Learning, PMLR, pp 20,841–20,855
  57. 57.Sung YL, Cho J, Bansal M (2022) Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems 35:12,991–13,005
  58. 58.Sung YL, Cho J, Bansal M (2022) Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5227–5237
  59. 59.Tan H, Bansal M (2019) Lxmert: Learning cross-modality encoder representations from transformers. In: EMNLP-IJCNLP
  60. 60.Touvron H, Cord M, Douze M, et al (2021) Training data-efficient image transformers & distillation through attention. In: ICML
  61. 61.Tsimpoukelli M, Menick JL, Cabi S, et al (2021) Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34:200–212
  62. 62.Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: NIPS
  63. 63.Wang H, Ge S, Lipton Z, et al (2019) Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems 32
  64. 64.Wang W, Bao H, Dong L, et al (2022) Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:220810442
  65. 65.Wang Z, Yu J, Yu AW, et al (2022) SimVLM: Simple visual language model pretraining with weak supervision. In: International Conference on Learning Representations
  66. 66.Wortsman M, Ilharco G, Kim JW, et al (2022) Robust fine-tuning of zero-shot models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7959–7971
  67. 67.Xiao J, Hays J, Ehinger KA, et al (2010) Sun database: Large-scale scene recognition from abbey to zoo. In: 2010 IEEE computer society conference on computer vision and pattern recognition, IEEE, pp 3485–3492
  68. 68.Yao Y, Zhang A, Zhang Z, et al (2021) Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:210911797
  69. 69.Yao Y, Chen Q, Zhang A, et al (2022) PEVL: Position-enhanced pre-training and prompt tuning for vision-language models. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp 11,104–11,117
  70. 70.Yu Z, Yu J, Cui Y, et al (2019) Deep modular co-attention networks for visual question answering. In: CVPR
  71. 71.Zhang R, Zeng Z, Guo Z (2022) Can language understand depth? ACM MM 2022
  72. 72.Zhang R, Zhang W, Fang R, et al (2022) Tip-adapter: Training-free adaption of clip for few-shot classification. In: ECCV 2022. Springer Nature Switzerland
  73. 73.Zhang R, Hu X, Li B, et al (2023) Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. CVPR 2023
  74. 74.Zhang R, Wang L, Qiao Y, et al (2023) Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. CVPR 2023
  75. 75.Zhou K, Yang J, Loy CC, et al (2022) Learning to prompt for vision-language models. International Journal of Computer Vision pp 1–12
  76. 76.Zhu X, Zhang R, He B, et al (2022) Pointclip v2: Adapting clip for powerful 3d open-world learning. ICCV 2023
  77. 77.Zhu X, Zhang R, He B, et al (2023) Not all features matter: Enhancing few-shot clip with adaptive prior refinement. ICCV 2023

Citation

MLA
Gao, P., et al. “CLIP-Adapter: Better Vision-Language Models with Feature Adapters”. arXiv, 2021, http://arxiv.org/abs/2110.04544v2.
APA
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2021). CLIP-Adapter: Better Vision-Language Models with Feature Adapters. arXiv. http://arxiv.org/abs/2110.04544v2
Chicago
Gao, P., S. Geng, R. Zhang, et al. 2021. “CLIP-Adapter: Better Vision-Language Models with Feature Adapters”. arXiv. http://arxiv.org/abs/2110.04544v2.
Harvard
Gao, P. et al. (2021) “CLIP-Adapter: Better Vision-Language Models with Feature Adapters”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2110.04544v2.
Vancouver
1. Gao P, Geng S, Zhang R, Ma T, Fang R, Zhang Y, Li H, Qiao Y (2021) CLIP-Adapter: Better Vision-Language Models with Feature Adapters. arXiv

BibTeX

@article{gao2021clip,
  title = {CLIP-Adapter: Better Vision-Language Models with Feature Adapters},
  author = {Gao, Peng and Geng, Shijie and Zhang, Renrui and Ma, Teli and Fang, Rongyao and Zhang, Yongfeng and Li, Hongsheng and Qiao, Yu},
  year = {2021},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2110.04544v2},
  eprint = {2110.04544}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF