Visual Prompt Tuning

Menglin JiaLuming TangBor-Chun ChenClaire CardieSerge BelongieBharath HariharanSer-Nam Lim

article2022ECCV2,874 citations

Proposes Visual Prompt Tuning (VPT), a parameter-efficient adaptation technique that prepends learnable tokens to frozen vision transformers, outperforming full fine-tuning across downstream recognition tasks while updating less than one percent of model parameters.

Listen

Large pre-trained vision Transformers deliver strong results on recognition tasks but adapting them to new problems typically requires full fine-tuning. This stores a separate copy of the entire backbone for every task, an increasingly expensive proposition as models grow to hundreds of millions of parameters.

The article evaluates whether a far lighter approach, Visual Prompt Tuning (VPT), can match or exceed full fine-tuning. VPT freezes the backbone and adds fewer than 1 % extra trainable parameters as task-specific prompt tokens inserted into the input sequence of each Transformer layer.

The authors test both shallow and deep variants of VPT on 24 classification tasks spanning fine-grained recognition and the VTAB-1k benchmark, using supervised and self-supervised ViT and Swin backbones. They compare against full fine-tuning and seven established parameter-efficient baselines, measuring accuracy, total storage across tasks, and behavior across data scales and model sizes. Additional experiments cover semantic segmentation and ConvNet backbones.

VPT-deep surpasses full fine-tuning on 20 of the 24 classification tasks while requiring only 1.18× versus 24× total parameters across all tasks. It also outperforms every other efficient baseline on every task group. Gains hold across data regimes, model scales, and hierarchical Transformers, and remain competitive on segmentation. Random prompt initialization and prompts placed in early layers work best.

These results show that visual prompting can deliver higher accuracy at dramatically lower storage cost than conventional adaptation, making multi-task deployment of large vision models practical. The approach is especially attractive when storage or serving many task-specific models is the bottleneck.

Organizations facing many downstream tasks should pilot VPT-deep on representative workloads and compare storage and accuracy trade-offs against full fine-tuning and bias tuning. Further work is needed to understand why gains are smaller or absent with certain self-supervised pre-training objectives and to extend prompting to additional dense-prediction and video tasks.

The main limitations are that prompt length must be tuned per task, results with self-supervised backbones are inconclusive, and the method offers smaller relative gains on smaller ConvNets. remains high for supervised ViT models on classification benchmarks but lower for other settings until more data are collected.

Cornell University Meta University of Copenhagen encoder-decoder language model large language models Visual Prompt Tuning VPT parameter‑efficient tuning

arXiv: 2203.12119 KMnP/vpt

Paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy et al. (2021). Introduces the Vision Transformer (ViT) architecture that serves as the primary frozen backbone evaluated throughout Visual Prompt Tuning.
Paper: The Power of Scale for Parameter-Efficient Prompt Tuning, Brian Lester et al. (2021). Establishes continuous prompt tuning for Transformer language models, providing the direct conceptual foundation adapted to visual inputs in VPT.
Paper: Prefix-Tuning: Optimizing Continuous Prompts for Generation, Xiang Lisa Li et al. (2021). Proposes prepending continuous, trainable prefix tokens across Transformer layers, directly inspiring the deep variant of Visual Prompt Tuning.
Paper: Learning to Prompt for Vision-Language Models, Kaiyang Zhou et al. (2021). Demonstrates prompt learning for vision-language models like CLIP, motivating the extension of prompt tuning directly into vision-only backbones.
Paper: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, Ze Liu et al. (2021). Introduces the hierarchical Swin Transformer architecture, which VPT adopts as a key backbone to evaluate prompting across vision Transformer designs.
Paper: Parameter-Efficient Transfer Learning for NLP, Neil Houlsby et al. (2019). Establishes parameter-efficient transfer learning with bottleneck adapters in Transformers, serving as a primary baseline and motivation for VPT.
Paper: Towards a Unified View of Parameter-Efficient Transfer Learning, Junxian He et al. (2022). Presents a unified structural taxonomy connecting adapters, prefix tuning, and prompt tuning in frozen Transformers.
Paper: Training data-efficient image transformers & distillation through attention, Hugo Touvron et al. (2021). Establishes data-efficient training and tokenization recipes for Vision Transformers (DeiT) used extensively in VPT's experiments.
Paper: Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, Pengfei Liu et al. (2021). Provides a comprehensive taxonomy of continuous and discrete prompt learning techniques foundational to visual prompting methods.
Paper: Emerging Properties in Self-Supervised Vision Transformers, Mathilde Caron et al. (2021). Introduces DINO self-supervised Vision Transformers, which VPT examines to analyze how pre-training objectives affect prompt tuning efficacy.

Paper: Conditional Prompt Learning for Vision-Language Models, Kaiyang Zhou et al. (2022). Extends static prompt learning in vision models by conditioning prompt tokens on specific visual inputs to enhance generalization.

Abstract

The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost.

Topics

encoder-decoder language model large language models tuning method parameter-efficient tuning control parameters

Keywords

Visual Prompt Tuning VPT parameter‑efficient tuning freeze backbone full fine‑tune prompt length shallow prompt deep prompt storage cost downstream task

1 Introduction
2 Related Work
3 Approach
3.1 Preliminaries
3.2 Visual-Prompt Tuning (VPT)
VPT-Shallow.
VPT-Deep.
Storing Visual Prompts.
4 Experiments
4.1 Experiment Setup
4.2 Main Results
VPT on different downstream data size.
VPT on different backbone scales.
VPT on hierarchical Transformers.
4.3 Ablation on Model Design Variants
Prompt Location.
Prompt Length.
Prompt Depth.
Final Output.
5 Analysis and Discussion
Visualization.
Apply VPT to more vision tasks.
Apply VPT to more pre-training methods.
Apply VPT to ConvNets.
6 Conclusion
0.A Implementation Details
0.A.1 Classification Experiments
VPT.
Adapter.
Augmentation and other hyper-parameters.
Datasets and pre-trained backbones specifications.
0.A.2 Semantic Segmentation Experiments
0.B Extended Analysis
Effect of expanding input sequence length.
Sharing prompts.
Prompt initialization.
Prompt depth vs. prompt length.
Combine VPT with Bias Tuning.
Prompt ensembling.
Test of statistical significance.
Effect of different fine-tuning hyper-parameters.
Effect of image resolution.
Empirical computational cost.
0.C Further Discussion
VPT vs. Adversarial Reprogramming (AR).
Visual prompt vs. textual prompt.
0.D Supplementary Results
Numerical results of .
Per-task results on training data ablations.
More t-SNE visualizations.
References

Knowls

Knowl 1 — Visual Prompt Tuning Formulation for Vision Transformers
model/method
Visual Prompt Tuning (VPT) adapts a pre-trained, frozen Vision Transformer (ViT) backbone to downstream vision tasks by introducing a small set of task-specific continuous learnable parameters (prompts) directly into the input sequence of Transformer layers, optimized jointly with a task-specific linear classification head.

For a standard ViT with $N$ layers, an input image is partitioned into $m$ fixed-size patches $\{I_j \in \mathbb{R}^{3 \times h \times w}\}_{j=1}^m$ . Each patch is projected into a $d$ -dimensional embedding space with positional encodings:

$e_0^j = \mathrm{Embed}(I_j) \in \mathbb{R}^d, \quad j = 1, 2, \ldots, m$

Let $E_i = \{e_i^j \in \mathbb{R}^d\}_{j=1}^m$ denote the collection of patch embeddings input to Transformer layer $L_{i+1}$ , and let $x_i \in \mathbb{R}^d$ denote the $[ ext{CLS}]$ token embedding. VPT defines two architectural variants:
1. VPT-Shallow: $p$ learnable continuous prompt tokens $P = \{p^k \in \mathbb{R}^d\}_{k=1}^p$ are prepended to the input sequence of the first Transformer layer $L_1$ only:
$[x_1, Z_1, E_1] = L_1([x_0, P, E_0])$

$[x_i, Z_i, E_i] = L_i([x_{i-1}, Z_{i-1}, E_{i-1}]), \quad i = 2, 3, \ldots, N$

$y = \mathrm{Head}(x_N)$

where $Z_i \in \mathbb{R}^{p \times d}$ represents the transformed prompt features at layer $i$ , and $[\,\cdot\,, \,\cdot\,]$ denotes sequence-dimension concatenation.
1. VPT-Deep: Independent learnable prompt tokens $P_{i-1} = \{p_{i-1}^k \in \mathbb{R}^d\}_{k=1}^p$ are introduced at every layer $L_i$ ( $i = 1, \ldots, N$ ). The prompt features produced by the previous layer are discarded and replaced by new prompts:
$[x_i, \_, E_i] = L_i([x_{i-1}, P_{i-1}, E_{i-1}]), \quad i = 1, 2, \ldots, N$

$y = \mathrm{Head}(x_N)$

During downstream fine-tuning, all pre-trained Transformer backbone weights are frozen; only the prompts $P$ (or $\{P_i\}_{i=0}^{N-1}$ ) and the parameters of $\mathrm{Head}$ are updated. For a ViT-Base backbone ( $N=12, d=768, 85.8\text{M}$ parameters) with $p=50$ , VPT-Shallow adds $p \times d = 0.038\text{M}$ ( $0.04\%$ ) parameters, and VPT-Deep adds $N \times p \times d = 0.46\text{M}$ ( $0.53\%$ ) parameters per task.

Knowl 2 — Benchmark Evaluation of Visual Prompt Tuning on ViT-B/16

data/table

Visual Prompt Tuning was evaluated on an ImageNet-21k pre-trained ViT-B/16 backbone across 24 downstream visual recognition tasks: 5 Fine-Grained Visual Classification (FGVC) benchmarks (CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, Stanford Cars) and the 19 tasks of the Visual Task Adaptation Benchmark (VTAB-1k), grouped into Natural (7 tasks), Specialized (4 tasks), and Structured (8 tasks).

Method	Total params	Scope	FGVC	VTAB-1k
				Natural	Specialized	Structured
Full	$24.02\times$	Backbone	88.54	75.88	83.36	47.64
Linear	$1.02\times$	Head	79.32 (0)	68.93 (1)	77.16 (1)	26.84 (0)
Partial-1	$3.00\times$	Head	82.63 (0)	69.44 (2)	78.53 (0)	34.17 (0)
MLP-3	$1.35\times$	Head	79.80 (0)	67.80 (2)	72.83 (0)	30.62 (0)
Sidetune	$3.69\times$	Backbone	78.35 (0)	58.21 (0)	68.12 (0)	23.41 (0)
Bias	$1.05\times$	Backbone	88.41 (3)	73.30 (3)	78.25 (0)	44.09 (2)
Adapter	$1.23\times$	Backbone	85.66 (2)	70.39 (4)	77.11 (0)	33.43 (0)
VPT-Shallow (Ours)	$1.04\times$	Input	84.62 (1)	76.81 (4)	79.66 (0)	46.98 (4)
VPT-Deep (Ours)	$1.18\times$	Input	89.11 (4)	78.48 (6)	82.43 (2)	54.98 (8)

Total params represents the parameter storage multiplier needed to store adapted models for all 24 downstream tasks relative to a single backbone. Values in parentheses denote task win counts against full fine-tuning (Full).

Key empirical findings:

VPT-Deep outperforms Full fine-tuning on 3 of the 4 benchmark categories (FGVC, VTAB-Natural, VTAB-Structured) and achieves superior performance in 20 of the 24 individual tasks, while reducing per-task storage overhead to less than $1\%$ of the backbone parameters ( $1.18\times$ vs. $24.02\times$ total storage for 24 tasks).
VPT-Deep outperforms all other parameter-efficient adaptation baselines (Linear, Partial-1, MLP-3, Sidetune, Bias, and Adapter) across all 4 benchmark groups.
VPT-Shallow provides strong parameter efficiency, adding only $0.04\times$ parameter overhead across 24 tasks while outperforming all head-oriented baselines on every benchmark group.

Knowl 3 — Scaling of Visual Prompt Tuning Across Backbone Architectures and Model Sizes

empirical result

Visual Prompt Tuning generalizes effectively across model scales and architectural paradigms:

ViT Model Capacity Scaling (ViT-B, ViT-L, ViT-H): When evaluating on VTAB-1k across ViT-Base (85.8M parameters), ViT-Large (307M parameters), and ViT-Huge (630M parameters), the performance advantage of VPT-Deep over Full fine-tuning and parameter-efficient baselines (Linear probing, Bias tuning, Adapter) is maintained or expanded at larger scales on the Natural and Structured groups, while delivering performance comparable to Full fine-tuning on the Specialized group.
Hierarchical Vision Transformers (Swin-B): VPT extends directly to hierarchical Vision Transformers such as Swin-B (86.7M parameters pre-trained on ImageNet-21k), which utilize shifted local window multi-head self-attention. Prompts are attended to within local windows and bypassed during patch merging stages. On the 19 VTAB-1k tasks:

Method	Total params	Natural (7)	Specialized (4)	Structured (8)
Full	$19.01\times$	79.10	86.21	59.65
Linear	$1.01\times$	73.52 (5)	80.77 (0)	33.52 (0)
MLP-3	$1.47\times$	73.56 (5)	75.21 (0)	35.69 (0)
Partial-1	$3.77\times$	73.11 (4)	81.70 (0)	34.96 (0)
Bias	$1.06\times$	74.19 (2)	80.14 (0)	42.42 (0)
VPT-Shallow (Ours)	$1.01\times$	79.85 (6)	82.45 (0)	37.75 (0)
VPT-Deep (Ours)	$1.05\times$	76.78 (6)	84.53 (0)	53.35 (0)

Both VPT variants substantially outperform all other parameter-efficient baselines on Swin-B. On VTAB-Natural, VPT-Shallow achieves $79.85\%$ average accuracy, exceeding Full fine-tuning ( $79.10\%$ ).

Knowl 4 — Scaling Dynamics of Visual Prompt Tuning Across Training Data Sizes
empirical result
When varying the fraction of downstream training data from $10\%$ to $100\%$ across five Fine-Grained Visual Classification (FGVC) benchmarks (CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, Stanford Cars) using a pre-trained ViT-B/16 backbone:
1. Low-Data Regime ( $10\%$ – $40\%$ ): All parameter-efficient tuning methods (VPT-Deep, VPT-Shallow, Linear probing, Bias tuning, and Adapters) substantially outperform Full fine-tuning, as Full fine-tuning overfits severely when task data is scarce.
2. High-Data Regime ( $40\%$ – $100\%$ ): As training data volume increases, the performance advantage of Linear probing and Adapters reverses, falling behind Full fine-tuning. In contrast, VPT-Deep consistently maintains higher accuracy than Full fine-tuning across all data regimes up to $100\%$ data. Bias tuning tracks VPT-Deep's trend but consistently underperforms VPT-Deep across the entire spectrum.
Knowl 5 — Design Choices for Prompt Location, Insertion Space, and Layer Depth
empirical result
Ablations on supervised ImageNet-21k pre-trained ViT-Base across VTAB-1k establish the following design principles for prompt tuning in vision:
1. Latent Space vs. Pixel Space: Prepending prompt tokens as continuous vectors in the latent embedding space (after the patch embedding projection) achieves $78.5\%$ (Natural), $82.4\%$ (Specialized), and $55.0\%$ (Structured) for VPT-Deep. Inserting prompts as learnable pixels before patch projection (Prepend-pixel) drops shallow prompt accuracy by $6.9\%$ on VTAB-Natural ( $71.5\%$ vs. $76.8\%$ ). Concatenating learnable pixels as extra image channels (Concat-channel) degrades accuracy severely (dropping by up to 30 points on Natural to $47.2\%$ ).
2. Prepending vs. Element-Wise Addition: Adding prompts element-wise to patch embeddings without increasing sequence length (Add) achieves competitive accuracy on Natural ( $77.5\%$ Deep / $74.0\%$ Shallow) but lags behind default prepending across all VTAB groups ( $80.2\%$ Specialized, $43.2\%$ Structured for Deep).
3. Prompt Depth and Layer Order: Inserting prompts across all layers ( $1 \to 12$ ) produces optimal results. When inserting prompts into subsets of layers, bottom-to-top insertion (starting from layer 1 closest to input, e.g., $1 \to 3, 1 \to 6$ ) yields significantly higher accuracy than top-to-bottom insertion (starting from layer 12 closest to output, e.g., $9 \to 12, 6 \to 12$ ), showing that conditioning earlier Transformer layers is substantially more impactful than conditioning later layers.
4. Final Output Feature Selection: Using the final $[ ext{CLS}]$ token embedding ( $x_N$ ) or average-pooled patch embeddings ( $E_N$ ) yields equivalent performance (e.g., $82.4\%$ vs. $82.3\%$ on Specialized). However, average-pooling over prompt token outputs ( $Z_N$ ) or globally over all tokens degrades accuracy by up to 8 percentage points.
Knowl 6 — Prompt Parameter Sharing, Initialization, and Interaction with Bias Tuning
empirical result
Analysis of prompt architectural variations on ViT-Base across the three VTAB-1k subgroups reveals:
1. Prompt Parameter Sharing:
  - Inter-layer sharing (Shared-inter): Sharing the exact same prompt parameters across all $N$ Transformer layers performs slightly better than independent layer prompts ( $78.9\%$ Natural, $83.7\%$ Specialized, $55.3\%$ Structured vs. $78.5\%, 82.4\%, 55.0\%$ for default VPT-Deep) while using similar parameter overhead ( $1.14\times$ vs. $1.13\times$ ), facilitated by selecting a slightly longer optimal prompt length $p$ (average length $64.58$ vs. $60.94$ ).
  - Intra-layer sharing (Shared-intra): Sharing prompt embeddings within each layer performs worse than layer-independent prompts ( $78.1\%, 82.0\%, 47.6\%$ ).
  - Global sharing (Shared-all): Sharing a single prompt vector across all positions and layers degrades performance significantly ( $77.9\%, 82.1\%, 39.0\%$ ).
2. Prompt Initialization: Standard random initialization (Xavier uniform) matches or outperforms downstream class-prototype initialization (where prompts are initialized with downstream training class-averaged final $[ ext{CLS}]$ embeddings): $78.5\%$ vs. $77.5\%$ on Natural, $82.4\%$ vs. $78.8\%$ on Specialized, and $55.0\%$ vs. $48.4\%$ on Structured for VPT-Deep.
3. Hybrid Tuning with Bias Parameters (VPT + Bias): Jointly optimizing prompts and backbone bias parameters while freezing all other weights (VPT-Deep + Bias) consistently degrades performance relative to VPT-Deep alone on all VTAB groups (Natural: $77.64\%$ vs. $78.48\%$ ; Specialized: $82.22\%$ vs. $82.43\%$ ; Structured: $53.87\%$ vs. $54.98\%$ ), indicating that prompt tuning and bias tuning interfere rather than complement each other.

Knowl 7 — Visual Prompt Tuning for Dense Prediction: Semantic Segmentation with SETR

empirical result

Visual Prompt Tuning was evaluated on dense scene parsing using the ADE20K benchmark (150 classes, 20,210 train images, 2,000 validation images) using the SETR-PUP architecture with an ImageNet-21k pre-trained ViT-Large/16 encoder.

Prompts are injected only into the ViT encoder backbone; prompt tokens are discarded before feeding patch embeddings into the Progressive Upsampling (PUP) convolutional decoder. Only the PUP decoder head and the prompts are optimized during fine-tuning.

Backbone	ViT-L/16					ResNet-101
Method	Full	Head Only	Bias	VPT-Deep	VPT + Bias	Full
mIoU (SS)	48.31	35.12	43.40	42.11	44.04	45.47
mIoU (MS)	50.07	37.46	45.33	44.06	45.63	46.27
Tunable params (M)	318.31	13.18	13.46	13.43	15.79	63.0

SS and MS denote single-scale and multi-scale inference, respectively.

Key findings:

While parameter-efficient protocols do not match Full fine-tuning ( $50.07$ mIoU-MS), VPT-Deep ( $44.06$ mIoU-MS) significantly improves over Head-Only tuning ( $37.46$ mIoU-MS) and approaches Bias tuning ( $45.33$ mIoU-MS).
Combining VPT with Bias tuning (VPT + Bias) achieves $45.63$ mIoU-MS with $15.79\text{M}$ tunable parameters, rivaling a fully fine-tuned ResNet-101 ( $46.27$ mIoU-MS with $63.0\text{M}$ tunable parameters) while updating less than $5\%$ of ViT-Large parameters.

Knowl 8 — Adaptation Dynamics of Visual Prompt Tuning on Self-Supervised Vision Transformers

empirical result

When transferring ViT-B/16 backbones pre-trained with self-supervised objectives (Masked Autoencoders / MAE and MoCo v3) to VTAB-1k, adaptation behavior differs fundamentally from supervised pre-training:

	MAE			MoCo v3
Method	Natural	Specialized	Structured	Natural	Specialized	Structured
Full	59.29	79.68	53.82	71.95	84.72	51.98
Linear	18.87 (0)	53.72 (0)	23.70 (0)	67.46 (4)	81.08 (0)	30.33 (0)
Partial-1	58.44 (5)	78.28 (1)	47.64 (1)	72.31 (5)	84.58 (2)	47.89 (1)
Bias	54.55 (1)	75.68 (1)	47.70 (0)	72.89 (3)	81.14 (0)	53.43 (4)
Adapter	54.90 (3)	75.19 (1)	38.98 (0)	74.19 (4)	82.66 (1)	47.69 (2)
VPT-Shallow	39.96 (1)	69.65 (0)	27.50 (0)	67.34 (3)	82.26 (0)	37.55 (0)
VPT-Deep	36.02 (0)	60.61 (1)	26.57 (0)	70.27 (4)	83.04 (0)	42.38 (0)

Key findings:

On self-supervised backbones, VPT consistently outperforms Linear probing across all groups for both MAE and MoCo v3.
Unlike supervised ViT where VPT-Deep achieves state-of-the-art transfer performance, on MAE, Partial-1 ( $58.44\%$ Natural, $78.28\%$ Specialized, $47.64\%$ Structured) and Adapter tuning substantially outperform VPT-Deep ( $36.02\%, 60.61\%, 26.57\%$ ), with VPT-Shallow outperforming VPT-Deep.
On MoCo v3, Bias and Adapter tuning outperform VPT-Deep on Natural and Structured groups, demonstrating that representations learned via masked autoencoding and contrastive learning interact differently with input prompt conditioning compared to supervised features.

Knowl 9 — Empirical Computational Cost and Test-Time VPT-Prefix Acceleration
model/method
Although Multi-Head Self-Attention (MSA) exhibits quadratic theoretical computational complexity with respect to token sequence length $L = 1 + m + p$ (where $m$ is patch count and $p$ is prompt count), empirical benchmarking on an NVIDIA A100 GPU (batch size 64) demonstrates sub-quadratic scaling due to hardware parallelism and cache layouts:
1. Empirical Latency and GPU Memory: For ViT-B/16 ( $m=196$ ), setting $p=1$ incurs inference latency ( $69.4\text{ ms/img}$ ) and peak memory ( $0.88\text{ GB}$ ) nearly identical to Full fine-tuning ( $69.7\text{ ms/img}, 0.87\text{ GB}$ ). Doubling the effective sequence length to $p=200$ increases inference latency to $140.8\text{ ms/img}$ and memory to $1.85\text{ GB}$ (an $\approx 2\times$ increase rather than the theoretical $4\times$ ).
2. VPT-Prefix Test-Time Optimization: At test time, prompt tokens can be directly prepended to the key ( $K$ ) and value ( $V$ ) projection matrices within each self-attention module rather than appending prompt tokens to the input query sequence. Because queries remain restricted to the image patches and $[ ext{CLS}]$ token, VPT-Prefix produces mathematically identical output representations to VPT while substantially reducing inference latency and peak GPU memory usage, particularly for large prompt counts $p$ .

Knowl 10 — Pixel-Space Visual Prompting for Convolutional Neural Networks

model/method

Visual Prompt Tuning adapts to ConvNets (which lack input token sequences and location-invariant attention) by applying learnable pixel padding along the spatial boundary of the input image.

For an image of size $H \times W$ , the image borders along height and width are padded by $p$ learnable pixel values, where $p \in \{1, 3, 5, 7, 9, 11\}$ . Only the border pixels and the linear classification head are optimized during fine-tuning.

Evaluation on the 19 tasks of VTAB-1k:

	ConvNeXt-Base (87.6M)			ResNet-50 (23.5M)
Method	Natural	Specialized	Structured	Natural	Specialized	Structured
Full	77.97	83.71	60.41	59.72	76.66	54.08
Linear	74.48 (5)	81.50 (0)	34.76 (1)	63.75 (6)	77.60 (3)	30.96 (0)
Partial-1	73.76 (4)	81.64 (0)	39.55 (0)	64.34 (6)	78.64 (2)	45.78 (1)
MLP-3	73.78 (5)	81.36 (1)	35.68 (1)	61.79 (6)	70.77 (1)	33.97 (0)
Bias	69.07 (2)	72.81 (0)	25.29 (0)	63.51 (6)	77.22 (2)	33.39 (0)
VPT (Ours)	78.48 (6)	83.00 (1)	44.64 (1)	66.25 (6)	77.32 (2)	37.52 (0)

On modern, high-capacity ConvNets (ConvNeXt-Base pre-trained on ImageNet-21k), pixel-space visual prompting outperforms all head- and backbone-oriented parameter-efficient methods across all three VTAB groups, and exceeds Full fine-tuning on 8 of 19 tasks. On smaller ConvNets (ResNet-50 pre-trained on ImageNet-1k), VPT improves over Linear probing and Bias tuning on Natural and Structured but is outperformed by Partial-1.

Coverage note — All core contributions—including the mathematical formulation of VPT-Shallow and VPT-Deep, full downstream benchmarking on ViT across 24 datasets, architecture and model scaling analyses, training data size dynamics, ablation studies on prompt locations and sharing, dense prediction experiments on ADE20K, self-supervised ViT evaluations, latency profiling, and ConvNet adaptations—are covered. Minor dataset image samples and per-task t-SNE scatter plots were omitted.

References

1.Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016) 4
2.Bahng, H., Jahanian, A., Sankaranarayanan, S., Isola, P.: Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint arXiv:2203.17274 (2022) 3
3.Bao, H., Dong, L., Piao, S., Wei, F.: BEit: BERT pre-training of image transformers. In: ICLR (2022) 3
4.Beattie, C., Leibo, J.Z., Teplyashin, D., Ward, T., Wainwright, M., K¨uttler, H., Lefrancq, A., Green, S., Vald´es, V., Sadik, A., et al.: Deepmind lab. arXiv preprint arXiv:1612.03801 (2016) 16
5.Ben Zaken, E., Goldberg, Y., Ravfogel, S.: BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 1–9. Association for Computational Linguistics, Dublin, Ireland (May 2022). https://doi.org/10.18653/v1/2022.acl-short.1, https://aclanthology.org/2022.acl-short.1 3, 6
6.Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021) 1
7.Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) NeurIPS. vol. 33, pp. 1877–1901. Curran Associates, Inc. (2020) 1, 3
8.Cai, H., Gan, C., Zhu, L., Han, S.: Tinytl: Reduce memory, not parameters for efficient on-device learning. NeurIPS 33, 11285–11297 (2020) 2, 3, 6
9.Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV. pp. 213–229. Springer (2020) 3
10.Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV. pp. 801–818 (2018) 13
11.Chen*, X., Xie*, S., He, K.: An empirical study of training self-supervised vision transformers. In: ICCV (2021) 2, 3, 13, 15, 16
12.Cheng, G., Han, J., Lu, X.: Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE (2017) 16
13.Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , Vedaldi, A.: Describing textures in the wild. In: CVPR (2014) 16
14.Conder, J., Jefferson, J., Jawed, K., Nejati, A., Sagar, M., et al.: Efficient transfer learning for visual tasks via continuous optimization of prompts. In: International Conference on Image Analysis and Processing. pp. 297–309. Springer (2022) 3
15.Contributors, M.: MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation (2020) 18
16.Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009) 6, 16
17.Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019) 1, 3
18.Doersch, C., Gupta, A., Zisserman, A.: Crosstransformers: spatially-aware few-shot transfer. NeurIPS 33, 21981–21993 (2020) 3
19.Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2020) 1, 3, 4, 6, 16, 23, 25
20.Elsayed, G.F., Goodfellow, I., Sohl-Dickstein, J.: Adversarial reprogramming of neural networks. In: ICLR (2019) 14, 26
21.Feichtenhofer, C., Fan, H., Li, Y., He, K.: Masked autoencoders as spatiotemporal learners. arXiv preprint arXiv:2205.09113 (2022) 3
22.Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., Huang, G.: Domain adaptation via prompt learning. arXiv preprint arXiv:2202.06687 (2022) 3
23.Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Fei-Fei, L.: Fine-grained car detection for visual census estimation. In: AAAI (2017) 6, 16
24.Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. International Journal of Robotics Research (2013) 16
25.Girdhar, R., Carreira, J., Doersch, C., Zisserman, A.: Video action transformer network. In: CVPR. pp. 244–253 (2019) 3
26.Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS (2010) 15
27.Goyal, P., Doll´ar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017) 15
28.Guo, D., Rush, A., Kim, Y.: Parameter-efficient transfer learning with diff pruning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4884–4896. Association for Computational Linguistics, Online (Aug 2021) 3
29.He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient transfer learning. In: ICLR (2022) 3
30.He, K., Chen, X., Xie, S., Li, Y., Doll´ar, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR. pp. 16000–16009 (2022) 3, 6, 13, 15, 16, 26
31.He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016) 1, 5, 14, 16
32.Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7), 2217–2226 (2019) 11
33.Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2019) 16
34.Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: ICML. pp. 2790–2799. PMLR (2019) 3, 6, 15
35.Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021) 3
36.Jia, M., Wu, Z., Reiter, A., Cardie, C., Belongie, S., Lim, S.N.: Exploring visual engagement signals for representation learning. In: ICCV (2021) 2
37.Jiang, Z., Xu, F.F., Araki, J., Neubig, G.: How can we know what language models know? Transactions of the Association for Computational Linguistics 8, 423–438 (2020) 3
38.Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: CVPR (2017) 11, 16
39.Ju, C., Han, T., Zheng, K., Zhang, Y., Xie, W.: Prompting visual-language models for efficient video understanding. arXiv preprint arXiv:2112.04478 (2021) 3
40.Kaggle, EyePacs: Kaggle diabetic retinopathy detection (July 2015) 16
41.Khosla, A., Jayadevaprakash, N., Yao, B., Fei-Fei, L.: Novel dataset for fine-grained image categorization. In: First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO (June 2011) 6, 16
42.Krizhevsky, A.: One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997 (2014) 15
43.Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009) 16
44.LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: CVPR (2004) 16
45.Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 3045–3059. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021) 2, 3, 7, 19, 21, 26
46.Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7871–7880 (2020) 1
47.Li, F.F., Fergus, R., Perona, P.: One-shot learning of object categories. IEEE TPAMI (2006) 16
48.Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4582–4597. Association for Computational Linguistics, Online (Aug 2021) 2, 3, 25
49.Li, Y., Xie, S., Chen, X., Dollar, P., He, K., Girshick, R.: Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429 (2021) 3
50.Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021) 2, 3
51.Liu, X., Ji, K., Fu, Y., Du, Z., Yang, Z., Tang, J.: P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 (2021) 2, 3, 26
52.Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021) 3, 5, 6, 9, 16
53.Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. CVPR (2022) 14, 16
54.Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 15
55.Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008) 11
56.Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., Van Der Maaten, L.: Exploring the limits of weakly supervised pretraining. In: ECCV (2018) 2, 15
57.Matthey, L., Higgins, I., Hassabis, D., Lerchner, A.: dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/ (2017) 16
58.Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 (2011) 11, 16
59.Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722–729. IEEE (2008) 6, 16
60.Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV. pp. 69–84. Springer (2016) 6
61.Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.V.: Cats and dogs. In: CVPR (2012) 16
62.Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in PyTorch. In: NeurIPS Autodiff Workshop (2017) 15
63.Pfeiffer, J., Kamath, A., R¨uckl´e, A., Cho, K., Gurevych, I.: Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247 (2020) 3, 6, 15
64.Pfeiffer, J., R¨uckl´e, A., Poth, C., Kamath, A., Vuli´c, I., Ruder, S., Cho, K., Gurevych, I.: Adapterhub: A framework for adapting transformers. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations. pp. 46–54. Association for Computational Linguistics, Online (2020) 3, 6, 15
65.Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021) 1, 3
66.Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21(140), 1–67 (2020) 1, 3
67.Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters. NeurIPS 30 (2017) 3
68.Rebuffi, S.A., Bilen, H., Vedaldi, A.: Efficient parametrization of multi-domain deep neural networks. In: CVPR. pp. 8119–8127 (2018) 2
69.Sandler, M., Zhmoginov, A., Vladymyrov, M., Jackson, A.: Fine-tuning image transformers using learnable memory. In: CVPR. pp. 12155–12164 (2022) 3
70.Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020) 3
71.Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic segmentation. In: CVPR. pp. 7262–7272 (2021) 3
72.Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., Belongie, S.: Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In: CVPR. pp. 595–604 (2015) 6, 16
73.Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. NeurIPS 30 (2017) 3
74.Veeling, B.S., Linmans, J., Winkens, J., Cohen, T., Welling, M.: Rotation equivariant cnns for digital pathology. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2018) 16
75.Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011) 6, 16
76.Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: Superglue: A stickier benchmark for general-purpose language understanding systems. NeurIPS 32 (2019) 3
77.Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.: GLUE: A multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. pp. 353–355. Association for Computational Linguistics, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/W18-5446, https://aclanthology.org/W18-5446 3
78.Wang, H., Zhu, Y., Adam, H., Yuille, A., Chen, L.C.: Max-deeplab: End-to-end panoptic segmentation with mask transformers. In: CVPR. pp. 5463–5474 (2021) 3
79.Wang, R., Chen, D., Wu, Z., Chen, Y., Dai, X., Liu, M., Jiang, Y.G., Zhou, L., Yuan, L.: Bevt: Bert pretraining of video transformers. In: CVPR. pp. 14733–14743 (2022) 3
80.Wang, Z., Zhang, Z., Lee, C.Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., Pfister, T.: Learning to prompt for continual learning. In: CVPR. pp. 139–149 (2022) 3
81.Welch, B.L.: The generalization of ‘student’s’problem when several different population varlances are involved. Biometrika 34(1-2), 28–35 (1947) 22
82.Wilcoxon, F.: Individual comparisons by ranking methods. In: Breakthroughs in statistics, pp. 196–202. Springer (1992) 22
83.Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: CVPR (2010) 16
84.Yao, Y., Zhang, A., Zhang, Z., Liu, Z., Chua, T.S., Sun, M.: Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797 (2021) 3
85.Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? NeurIPS 27 (2014) 6
86.Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A.S., Neumann, M., Dosovitskiy, A., et al.: A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 (2019) 6, 7, 16
87.Zhang, J.O., Sax, A., Zamir, A., Guibas, L., Malik, J.: Side-tuning: a baseline for network adaptation via additive side networks. In: ECCV. pp. 698–714. Springer (2020) 2, 3, 6
88.Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV. pp. 649–666. Springer (2016) 6
89.Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: CVPR. pp. 6881–6890 (2021) 3, 12, 13, 18
90.Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. IJCV 127(3), 302–321 (2019) 12, 13, 18
91.Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134 (2021) 3
92.Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., He, Q.: A comprehensive survey on transfer learning. Proceedings of the IEEE 109(1), 43–76 (2020) 3

Citation

MLA

Jia, M., et al. “Visual Prompt Tuning”. arXiv, 2022, http://arxiv.org/abs/2203.12119v2.

APA

Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022). Visual Prompt Tuning. arXiv. http://arxiv.org/abs/2203.12119v2

Chicago

Jia, M., L. Tang, B.-C. Chen, et al. 2022. “Visual Prompt Tuning”. arXiv. http://arxiv.org/abs/2203.12119v2.

Harvard

Jia, M. et al. (2022) “Visual Prompt Tuning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2203.12119v2.

Vancouver

1. Jia M, Tang L, Chen B-C, Cardie C, Belongie S, Hariharan B, Lim S-N (2022) Visual Prompt Tuning. arXiv

BibTeX

@article{jia2022visual,
  title = {Visual Prompt Tuning},
  author = {Jia, Menglin and Tang, Luming and Chen, Bor-Chun and Cardie, Claire and Belongie, Serge and Hariharan, Bharath and Lim, Ser-Nam},
  year = {2022},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2203.12119v2},
  eprint = {2203.12119}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Visual Prompt Tuning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Visual Prompt Tuning Formulation for Vision Transformers

Knowl 2 — Benchmark Evaluation of Visual Prompt Tuning on ViT-B/16

Knowl 3 — Scaling of Visual Prompt Tuning Across Backbone Architectures and Model Sizes

Knowl 4 — Scaling Dynamics of Visual Prompt Tuning Across Training Data Sizes

Knowl 5 — Design Choices for Prompt Location, Insertion Space, and Layer Depth

Knowl 6 — Prompt Parameter Sharing, Initialization, and Interaction with Bias Tuning

Knowl 7 — Visual Prompt Tuning for Dense Prediction: Semantic Segmentation with SETR

Knowl 8 — Adaptation Dynamics of Visual Prompt Tuning on Self-Supervised Vision Transformers

Knowl 9 — Empirical Computational Cost and Test-Time VPT-Prefix Acceleration

Knowl 10 — Pixel-Space Visual Prompting for Convolutional Neural Networks

References

Citation

BibTeX

Source Code

Access the Paper