Back

Conditional Prompt Learning for Vision-Language Models

Kaiyang ZhouJingkang YangChen Change LoyZiwei Liu

article2022CVPR2,433 citations

Proposes Conditional Context Optimization (CoCoOp), an instance-conditional prompt learning approach that prevents vision-language models like CLIP from overfitting to training classes and substantially improves generalization to unseen categories and new domains.

Listen

Adapting large, pre-trained vision-language foundation models like CLIP to specialized downstream tasks is critical as these models grow in scale. Fine-tuning an entire model of hundreds of millions of parameters is computationally prohibitive and risks degrading pre-trained representations. While continuous prompt learning methods such as Context Optimization (CoOp) replace manual prompt engineering by learning static context vectors from a few labeled images, they suffer from severe overfitting. Specifically, static prompts optimize heavily for the seen training categories, causing a catastrophic drop in accuracy when encountering new, unseen categories within the same visual domain.

To address this generalization failure, the article evaluates whether conditioning prompt vectors directly on each visual input can prevent overfitting to training categories while preserving high downstream performance. The authors developed Conditional Context Optimization (CoCoOp), which enhances static prompt learning with a lightweight neural network called a Meta-Net. This network processes an input image to produce a dynamic conditioning token that adjusts the learned context vectors for each specific image instance. The authors conducted extensive experimental evaluations using a ViT-B/16 CLIP base across 11 diverse recognition benchmarks, testing base-to-new category generalization, cross-dataset transfer, and domain generalization across visual shifts.

The findings demonstrate that dynamic, instance-conditional prompts substantially outperform static baselines in generalization settings. On unseen classes within the 11 datasets, CoCoOp improves average accuracy from 63.22% (achieved by CoOp) to 71.69%, significantly narrowing the performance gap with manual prompts while sustaining an average harmonic mean accuracy of 75.83% across both base and new classes. Although base-class accuracy drops slightly compared to CoOp (from 82.69% to 80.47%), this modest decline of about 2% is far outweighed by double-digit gains on unseen classes in several benchmarks. Additionally, when transferring prompts trained on ImageNet to 10 distinct target datasets, CoCoOp consistently outperforms CoOp (achieving a 65.74% average target accuracy versus 63.88%) and delivers superior robustness against out-of-distribution domain shifts across specialized ImageNet variants.

These results establish that dynamic instance conditioning prevents prompt learning from collapsing into narrow category biases, making adapted vision-language models more reliable for real-world deployment where test data distributions shift. For organizations building on top of foundation models, conditional prompt learning lowers operational risk and deployment failure rates by maintaining strong zero-shot capabilities alongside specialized fine-tuning.

Decision-makers and engineering teams seeking parameter-efficient adaptation should adopt conditional prompting architectures over static prompt tuning, particularly in dynamic environments where unseen classes frequently appear. However, because generating instance-specific prompts requires evaluating the text encoder for each image individually, training CoCoOp consumes significant graphics memory and necessitates smaller batch sizes. Before deploying at enterprise scale, teams should conduct pilot implementations and explore optimized architectures to improve training and inference throughput. Overall confidence in the empirical results is high across standard vision benchmarks, though caution is warranted for specialized domains such as fine-grained aircraft classification, where learning-based prompts still lag behind heavily engineered manual baselines.

Nanyang Technological UniversityS-Labneural networksoptimizationoptimization problemsLatexit sha1 base64CoOpLearning
arXiv: 2203.05557KaiyangZhou/CoOp
Cover for Conditional Prompt Learning for Vision-Language Models

Abstract

With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) introduces the concept of prompt learning -- a recent trend in NLP -- to the vision domain for adapting pre-trained vision-language models. Specifically, CoOp turns context words in a prompt into a set of learnable vectors and, with only a few labeled images for learning, can achieve huge improvements over intensively-tuned manual prompts. In our study we identify a critical problem of CoOp: the learned context is not generalizable to wider unseen classes within the same dataset, suggesting that CoOp overfits base classes observed during training. To address the problem, we propose Conditional Context Optimization (CoCoOp), which extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp's static prompts, our dynamic prompts adapt to each instance and are thus less sensitive to class shift. Extensive experiments show that CoCoOp generalizes much better than CoOp to unseen classes, even showing promising transferability beyond a single dataset; and yields stronger domain generalization performance as well. Code is available at this https URL.

Topics

neural networksoptimizationoptimization problemsstatistical language models

Keywords

Latexit sha1 base64CoOpLearningCoCoOpModelsPromptClassesPrompt LearningCLIPContext

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Methodology
  • 3.1 Reviews of CLIP and CoOp
  • 3.2 CoCoOp: Conditional Context Optimization
  • 4 Experiments
  • 4.1 Generalization From Base to New Classes
  • 4.2 Cross-Dataset Transfer
  • 4.3 Domain Generalization
  • 4.4 Further Analysis
  • 5 Limitations
  • 6 Discussion and Conclusion
  • A Results on DOSCO-2k
  • References

Knowls

  1. Knowl 1 — Conditional Context Optimization Architecture

    model/method

    Conditional Context Optimization (CoCoOp) is a parameter-efficient prompt learning method for vision-language models like CLIP. To address the problem of static context vectors overfitting to training classes, CoCoOp conditions prompt tokens dynamically on each input image instance.

    CoCoOp introduces two learnable components while keeping the underlying vision-language model encoders frozen:

    1. A sequence of MM learnable continuous context vectors V={v1,v2,…,vM}V = \{v_1, v_2, \dots, v_M\}, each having the same embedding dimension as the word tokens.
    2. A lightweight neural network called Meta-Net, denoted hθ(⋅)h_\theta(\cdot) and parameterized by θ\theta, which takes as input the image feature vector xx generated by the image encoder and produces a conditional meta-token vector π=hθ(x)\pi = h_\theta(x).

    The Meta-Net is designed as a two-layer bottleneck network (Linear-ReLU-Linear) where the intermediate hidden layer reduces the input feature dimension by a factor of 1616.

    For a specific input instance xx, the dynamic context vectors are generated by adding the instance-specific meta-token π\pi to each base context vector:

    vm(x)=vm+π,m∈{1,2,…,M}v_m(x) = v_m + \pi, \quad m \in \{1, 2, \dots, M\}

    The prompt for the ii-th class category is constructed by concatenating these conditioned context vectors with the pre-trained word embedding cic_i of the ii-th class name:

    ti(x)={v1(x),v2(x),…,vM(x),ci}t_i(x) = \{v_1(x), v_2(x), \dots, v_M(x), c_i\}

    During training, the base context vectors {vm}m=1M\{v_m\}_{m=1}^M and Meta-Net parameters θ\theta are updated jointly via cross-entropy loss, while the pre-trained image and text encoders remain frozen.

  2. Knowl 2 — Conditional Prompt Classification Probability Formulation

    equation

    In Conditional Context Optimization (CoCoOp), classification probabilities are computed dynamically per image instance. For an image feature vector x∈Rdx \in \mathbb{R}^d and a set of KK candidate classes, the predicted probability p(y∣x)p(y|x) for class label y∈{1,…,K}y \in \{1, \dots, K\} is given by:

    p(y∣x)=exp⁡(sim(x,g(ty(x)))/τ)∑i=1Kexp⁡(sim(x,g(ti(x)))/τ)p(y|x) = \frac{\exp(\text{sim}(x, g(t_y(x))) / \tau)}{\sum_{i=1}^K \exp(\text{sim}(x, g(t_i(x))) / \tau)}

    where:

    • sim(u,v)=u⋅v∥u∥2∥v∥2\text{sim}(u, v) = \frac{u \cdot v}{\|u\|_2 \|v\|_2} is the cosine similarity between feature vectors uu and vv.
    • g(⋅)g(\cdot) is the frozen pre-trained text encoder (e.g., Transformer).
    • τ\tau is a learned temperature parameter from the pre-trained vision-language model.
    • ti(x)={v1+hθ(x),v2+hθ(x),…,vM+hθ(x),ci}t_i(x) = \{v_1 + h_\theta(x), v_2 + h_\theta(x), \dots, v_M + h_\theta(x), c_i\} is the instance-conditional prompt for category ii, where hθ(x)h_\theta(x) is the output of the Meta-Net, vmv_m are learnable base context vectors, and cic_i is the word embedding of the ii-th class name.

  3. Knowl 3 — Base-to-New Class Generalization Across 11 Visual Recognition Datasets

    data/table

    To evaluate generalization under class shift, models are trained on 16 shots per class using only half of the classes (base classes) and evaluated on both the base classes and the remaining unseen half (new classes). The performance trade-off is quantified using the harmonic mean H=2⋅Base⋅NewBase+NewH = \frac{2 \cdot \text{Base} \cdot \text{New}}{\text{Base} + \text{New}}. Models use a ViT-B/16 backbone and context length M=4M=4.

    Dataset CLIP (Zero-Shot) CoOp CoCoOp
    Base New H Base New H Base New H
    ImageNet 72.43 68.14 70.22 76.47 67.88 71.92 75.98 70.43 73.10
    Caltech101 96.84 94.00 95.40 98.00 89.81 93.73 97.96 93.81 95.84
    OxfordPets 91.17 97.26 94.12 93.67 95.29 94.47 95.20 97.69 96.43
    StanfordCars 63.37 74.89 68.65 78.12 60.40 68.13 70.49 73.59 72.01
    Flowers102 72.08 77.80 74.83 97.60 59.67 74.06 94.87 71.75 81.71
    Food101 90.10 91.22 90.66 88.33 82.26 85.19 90.70 91.29 90.99
    FGVCAircraft 27.19 36.29 31.09 40.44 22.30 28.75 33.41 23.71 27.74
    SUN397 69.36 75.35 72.23 80.60 65.89 72.51 79.74 76.86 78.27
    DTD 53.24 59.90 56.37 79.44 41.18 54.24 77.01 56.00 64.85
    EuroSAT 56.48 64.05 60.03 92.19 54.74 68.69 87.49 60.04 71.21
    UCF101 70.53 77.50 73.85 84.69 56.05 67.46 82.33 73.45 77.64
    Average 69.34 74.22 71.70 82.69 63.22 71.66 80.47 71.69 75.83

    While static prompt learning (CoOp) improves base class accuracy from 69.34% to 82.69%, it suffers a substantial accuracy drop on new classes (falling to 63.22%, which is 11.00% lower than zero-shot CLIP). CoCoOp boosts new class accuracy to 71.69% (+8.47% over CoOp) while largely maintaining base class performance (80.47%), achieving the highest overall harmonic mean of 75.83%.

  4. Knowl 4 — Cross-Dataset Prompt Transferability

    data/table

    To evaluate the general transferability of prompt representations across different recognition tasks, models are trained on all 1,000 classes of ImageNet (16 shots per class, ViT-B/16 backbone) and directly tested on 10 other datasets without further adaptation.

    Method Source Target Datasets Target
    ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 Average
    CoOp 71.51 93.70 89.14 64.51 68.71 85.30 18.47 64.15 41.92 46.39 66.55 63.88
    CoCoOp 71.02 94.43 90.14 65.32 71.88 86.06 22.94 67.36 45.73 45.37 68.21 65.74
    Gain (Δ\Delta) -0.49 +0.73 +1.00 +0.81 +3.17 +0.76 +4.47 +3.21 +3.81 -1.02 +1.66 +1.86

    CoCoOp achieves an average target accuracy of 65.74% across the 10 datasets, outperforming CoOp (63.88%) by +1.86%. Performance gains are especially pronounced on fine-grained and specialized tasks with large domain distances from ImageNet, such as FGVCAircraft (+4.47%), DTD (+3.81%), SUN397 (+3.21%), and Flowers102 (+3.17%).

  5. Knowl 5 — Out-of-Distribution Domain Generalization Performance

    data/table

    Domain generalization performance is assessed by training models on 16-shot ImageNet (source) and evaluating them on four ImageNet target variants featuring distinct natural distribution shifts: ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R (all using a ViT-B/16 backbone).

    Method Learnable? Source (ImageNet) Target (Out-of-Distribution)
    ImageNetV2 ImageNet-Sketch ImageNet-A ImageNet-R
    CLIP (Zero-Shot) No 66.73 60.83 46.15 47.77 73.96
    CoOp Yes 71.51 64.20 47.99 49.71 75.21
    CoCoOp Yes 71.02 64.07 48.75 50.63 76.18

    Both prompt tuning methods significantly outperform zero-shot CLIP across all target domains. Compared to static CoOp, dynamic CoCoOp achieves higher domain generalization accuracy on ImageNet-Sketch (48.75% vs 47.99%), ImageNet-A (50.63% vs 49.71%), and ImageNet-R (76.18% vs 75.21%), while performing comparably on ImageNetV2 (64.07% vs 64.20%).

  6. Knowl 6 — Base and New Class Incremental Recognition

    empirical result

    In a generalized zero-shot / class-incremental setting where the label space during inference is expanded to simultaneously include all base classes and new classes, models are trained only on 16 shots per base class. Accuracy is averaged over all 11 evaluation datasets:

    • Zero-shot CLIP achieves 65.22%.
    • CoOp achieves 65.55% (+0.33% over zero-shot CLIP).
    • CoCoOp achieves 69.13% (+3.91% over zero-shot CLIP, +3.58% over CoOp).

    CoOp loses competitiveness because its static prompts bias predictions toward base classes observed during training. CoCoOp's input-conditional prompts prevent this seen-class bias, maintaining strong discriminatory capacity when evaluating base and novel classes concurrently.

  7. Knowl 7 — Parameter Scaling vs. Dynamic Conditioning in CoCoOp

    data/table

    To verify whether CoCoOp's gains are caused by the additional capacity of the Meta-Net rather than the input-conditioning mechanism, CoOp is trained with an increased number of static context tokens (M=60M=60) to match CoCoOp's parameter count (evaluated on ImageNet, 16-shot base class training, ViT-B/16 backbone).

    Model # params Base New H
    CoOp (M=4M=4) 2,048 76.47 67.88 71.92
    CoOp (M=60M=60) 30,720 76.16 65.34 70.34
    CoCoOp (M=4M=4 + Meta-Net) 34,816 75.98 70.43 73.10

    Increasing the number of static context tokens in CoOp from 4 to 60 lowers new class accuracy from 67.88% to 65.34% and reduces the harmonic mean from 71.92% to 70.34%. In contrast, CoCoOp achieves 70.43% on new classes and 73.10% harmonic mean, confirming that instance-conditional dynamism, rather than expanded parameter budget, drives the generalization performance.

  8. Knowl 8 — Sensitivity to Context Length and Initialization

    empirical result

    Ablation studies on the 11 benchmark datasets analyze prompt initialization and context length in base-to-new generalization:

    1. Initialization: Initializing base context vectors with pre-trained word embeddings from "a photo of a" outperforms random Gaussian initialization N(0,0.022)\mathcal{N}(0, 0.02^2) across both base and new classes on average.
    2. Context Length: In base-to-new generalization with random initialization across context lengths M∈{4,8,16}M \in \{4, 8, 16\}, base class performance remains nearly identical, whereas new class performance monotonically increases with longer context lengths (M=16M=16 yielding the highest accuracy on unseen classes).

  9. Knowl 9 — Computational and Unseen-Class Limitations of CoCoOp

    limitation

    CoCoOp presents two main limitations:

    1. Training Efficiency and Memory: Because CoCoOp generates instance-specific prompts ti(x)t_i(x) for each image xx, the text encoder must perform an independent forward pass for every single image. For a batch of size NN with KK classes, text encoder computations scale as O(N⋅K)O(N \cdot K), compared to CoOp which performs a single O(K)O(K) forward pass for the entire batch. Consequently, training CoCoOp requires a batch size of 1 to prevent excessive GPU memory consumption and training delays.
    2. Performance Gap with Zero-Shot Prompts: On 7 of the 11 tested datasets (Caltech101, StanfordCars, Flowers102, FGVCAircraft, DTD, EuroSAT, UCF101), CoCoOp's accuracy on unseen classes remains lower than zero-shot CLIP with manually engineered prompts.

  10. Knowl 10 — Contextual Domain Shift Evaluation on the DOSCO-2k Benchmark

    data/table

    The DOSCO-2k benchmark evaluates domain generalization under automated, contextual domain shifts across seven recognition datasets (P-Air, P-Cars, P-Ctech, P-Ins, P-Mam, P-Pets, P-UCF) using 2,000 images per dataset (1,600 training, 400 validation). Models are trained on the training split and evaluated on unseen domain test splits across four visual backbones:

    Backbone Method P-Air P-Cars P-Ctech P-Ins P-Mam P-Pets P-UCF Avg
    ResNet-50 CLIP 16.1 56.1 86.7 62.7 59.7 84.0 60.6 60.9
    CoOp 22.1 60.7 89.4 66.3 61.6 83.8 69.2 64.7
    CoCoOp 20.1 59.8 90.4 67.9 63.8 87.6 69.1 65.5
    ResNet-101 CLIP 17.5 63.2 89.5 62.4 62.2 84.2 61.3 62.9
    CoOp 24.6 68.2 92.0 68.3 65.4 88.2 72.7 68.5
    CoCoOp 22.5 65.2 93.3 69.9 67.5 88.6 71.5 68.4
    ViT-B/32 CLIP 18.2 60.1 91.6 61.3 61.8 85.5 61.3 62.8
    CoOp 24.0 63.0 93.6 67.3 65.7 88.5 74.5 68.1
    CoCoOp 19.5 60.4 93.8 69.8 67.3 88.5 72.7 67.4
    ViT-B/16 CLIP 24.4 64.9 92.6 67.5 67.9 87.4 66.1 67.2
    CoOp 32.4 72.4 94.7 73.2 72.1 90.1 78.2 73.3
    CoCoOp 30.4 68.7 94.8 73.5 73.6 91.6 76.3 72.7

    Both CoOp and CoCoOp substantially outperform zero-shot CLIP on all network backbones under contextual domain shift (e.g., +4.6% to +6.1% average accuracy gain on ResNet-50 and ViT-B/16). CoCoOp achieves higher accuracy than CoOp on 4 out of the 7 individual datasets across backbones.

Coverage note — None was omitted; all key theoretical formulations, empirical results across the three primary benchmarks (base-to-new, cross-dataset transfer, domain generalization), ablation studies, and limitations from the paper are represented.

References

  1. 1.Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  2. 2.Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In ECCV, 2014.
  3. 3.Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In ECCV, 2016.
  4. 4.Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
  5. 5.Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
  6. 6.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  7. 7.Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
  8. 8.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
  9. 9.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  10. 10.Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write a classifier: Zero-shot learning using purely textual descriptions. In ICCV, 2013.
  11. 11.Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR-W, 2004.
  12. 12.Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. NeurIPS, 2013.
  13. 13.Andreas Furst, Elisabeth Rumetshofer, Viet Tran, Hubert Ramsauer, Fei Tang, Johannes Lehner, David Kreil, Michael Kopp, Gunter Klambauer, Angela Bitto-Nemling, et al. Cloob: Modern hopfield networks with infoloob outperform clip. arXiv preprint arXiv:2110.11316, 2021.
  14. 14.Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021.
  15. 15.Tianyu Gao, Adam Fisch, and Danqi Chen. Making pretrained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020.
  16. 16.Lluis Gomez, Yash Patel, Marc¸al Rusinol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning of visual features through embedding images into text topic spaces. In CVPR, 2017.
  17. 17.Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
  18. 18.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
  19. 19.Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  20. 20.Olivier J. Henaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
  21. 21.Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
  22. 22.Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
  23. 23.Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute-based attention. In CVPR, 2020.
  24. 24.Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
  25. 25.Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? ACL, 2020.
  26. 26.Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016.
  27. 27.Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. arXiv preprint arXiv:2112.04478, 2021.
  28. 28.Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV-W, 2013.
  29. 29.Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In ICCV, 2015.
  30. 30.Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
  31. 31.Ang Li, Allan Jabri, Armand Joulin, and Laurens van der Maaten. Learning visual n-grams from web data. In ICCV, 2017.
  32. 32.Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
  33. 33.Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021.
  34. 34.Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586, 2021.
  35. 35.Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  36. 36.Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
  37. 37.German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
  38. 38.Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
  39. 39.Fabio Petroni, Tim Rocktaschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? In EMNLP, 2019.
  40. 40.Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
  41. 41.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019.
  42. 42.Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
  43. 43.Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
  44. 44.Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, 2020.
  45. 45.Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D Manning, and Andrew Y Ng. Zero-shot learning through cross-modal transfer. In NeurIPS, 2013.
  46. 46.Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
  47. 47.Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In NeurIPS, 2020.
  48. 48.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
  49. 49.Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
  50. 50.Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019.
  51. 51.Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. TIST, 2019.
  52. 52.Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018.
  53. 53.Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01903, 2021.
  54. 54.Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In CVPR, 2017.
  55. 55.Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
  56. 56.Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797, 2021.
  57. 57.Kai Yi, Xiaoqian Shen, Yunhao Gou, and Mohamed Elhoseiny. Exploring hierarchical graph representation for large-scale zero-shot image classification. arXiv preprint arXiv:2203.01386, 2022.
  58. 58.Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. arXiv preprint arXiv:2112.02413, 2021.
  59. 59.Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020.
  60. 60.Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [mask]: Learning vs. learning to recall. In NAACL, 2021.
  61. 61.Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  62. 62.Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint arXiv:2103.02503, 2021.
  63. 63.Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134, 2021.
  64. 64.Kaiyang Zhou, Yuanhan Zhang, Yuhang Zang, Jingkang Yang, Chen Change Loy, and Ziwei Liu. On-device domain generalization. arXiv preprint arXiv:2209.07521, 2022.

Citation

MLA
Zhou, K., et al. “Conditional Prompt Learning for Vision-Language Models”. arXiv, 2022, http://arxiv.org/abs/2203.05557v2.
APA
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022). Conditional Prompt Learning for Vision-Language Models. arXiv. http://arxiv.org/abs/2203.05557v2
Chicago
Zhou, K., J. Yang, C. C. Loy, and Z. Liu. 2022. “Conditional Prompt Learning for Vision-Language Models”. arXiv. http://arxiv.org/abs/2203.05557v2.
Harvard
Zhou, K. et al. (2022) “Conditional Prompt Learning for Vision-Language Models”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2203.05557v2.
Vancouver
1. Zhou K, Yang J, Loy CC, Liu Z (2022) Conditional Prompt Learning for Vision-Language Models. arXiv

BibTeX

@article{zhou2022conditional,
  title = {Conditional Prompt Learning for Vision-Language Models},
  author = {Zhou, Kaiyang and Yang, Jingkang and Loy, Chen Change and Liu, Ziwei},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2203.05557v2},
  eprint = {2203.05557}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE