Cross-Modal Fine-Tuning: Align then Refine

Junhong ShenLiam LiLucio M. DeryCorey StatenMikhail KhodakGraham NeubigAmeet Talwalkar

article2023ICML72 citations

Proposes ORCA, a cross-modal fine-tuning framework that matches target-data distributions to pretraining modalities before model adaptation, achieving state-of-the-art results across 12 distinct modalities without requiring domain-specific pretrained architectures.

Listen

Large pretrained machine learning models have achieved strong results in fields with massive datasets, such as computer vision and natural language processing. However, domains with limited data—including the physical sciences, genomics, and specialized tabular applications—often struggle to benefit because training high-performing models from scratch requires extensive data, substantial computing power, and deep domain expertise. Existing attempts to reuse pretrained models across different data types have largely relied on rigid, ad-hoc techniques that do not generalize well.

The article introduces and evaluates ORCA, a general framework designed to adapt standard pretrained language and vision transformers to diverse, out-of-modality target tasks. The framework addresses the cross-modal transfer problem through an "align-then-refine" workflow: it first standardizes data shapes, trains an initial embedding network to align the target data distribution with the model's original pretraining data, and finally fine-tunes all model parameters on the target task.

To evaluate this approach, the authors tested ORCA across three major benchmarks spanning more than 60 datasets and 12 distinct data types, including partial differential equations, genomics, electrocardiograms, and tabular records. The evaluation compared ORCA against hand-designed models, automated machine learning (AutoML) systems, specialized cross-modal baselines, and general-purpose architectures, primarily utilizing pretrained RoBERTa and Swin Transformer backbones.

The findings show that cross-modal adaptation using data alignment consistently outperforms existing alternatives. On the multi-domain NAS-Bench-360 benchmark, ORCA achieved the lowest prediction error on 7 out of 10 tasks and placed in the top three across all 10, outperforming hand-designed and AutoML models. Crucially, the initial data-alignment stage proved essential: directly fine-tuning pretrained models without alignment resulted in worse performance and higher variance. On specialized benchmarks, ORCA matched or exceeded expert domain architectures on scientific differential equations and established AutoML systems on tabular classification. Furthermore, the framework delivered substantial performance advantages in low-data settings—matching the performance of standard fine-tuning while requiring only one-third of the target data—while adding minimal computing overhead, as the alignment phase accounted for only about 11% of the total fine-tuning time.

These results indicate that foundational knowledge captured in vision and language models can be transferred across unrelated fields without costly architecture redesigns. This approach significantly reduces the time, technical complexity, and data-gathering expenses required to deploy advanced machine learning in specialized industries. In particular, it offers an effective path for deploying high-performing models in domains where data collection is expensive or impractical.

Organizations developing machine learning for specialized or data-scarce domains should consider adapting existing pretrained models via distribution-aligned fine-tuning rather than building models from scratch. When implementing this workflow, practitioners should perform full parameter fine-tuning rather than restricting updates to specific layers, as full fine-tuning yields noticeably superior accuracy with minimal additional computational time. Additionally, teams should select pretrained backbones whose structural representations best align with the target task based on distribution-distance metrics.

While the empirical evidence across the tested benchmarks is strong, some limitations remain. The evaluations focused on one-dimensional and two-dimensional data, leaving higher-dimensional problems and complex decision-making setups for future validation. Furthermore, selecting the optimal pretrained model currently requires empirical comparison. Practitioners should view this framework as a proven, highly competitive baseline for cross-modal tasks, while validating performance on their specific domain data.

Carnegie Mellon University Hewlett Packard Enterprise cross-modal fine-tuning crossmodal transfer task-specific fine-tuning

arXiv: 2302.05738 sjunhongshen/ORCA

Paper: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation, Junnan Li et al. (2021). It introduces the core 'align before fuse' paradigm that directly inspires and provides the conceptual blueprint for the align-then-refine cross-modal adaptation framework.
Paper: Perceiver: General Perception with Iterative Attention, Andrew Jaegle et al. (2021). It establishes foundational principles for building general-purpose perceptual architectures capable of consuming arbitrary, unaligned sensory modalities.
Paper: Cross-Modal Discrete Representation Learning, Alexander H. Liu et al. (2022). It provides foundational methods for bridging disparate modalities through shared embedding alignment before downstream task transfer.
Paper: Towards a Unified View of Parameter-Efficient Transfer Learning, Junxian He et al. (2022). It provides the unified conceptual formulation of parameter-efficient transfer learning mechanisms used when adapting pretrained backbones to downstream tasks.
Paper: CLIP-Adapter: Better Vision-Language Models with Feature Adapters, Peng Gao et al. (2021). It details how lightweight embedding and feature adapters can efficiently bridge new input domains into a frozen, pretrained foundation model.

Paper: ImageBind One Embedding Space to Bind Them All, Rohit Girdhar et al. (2023). It scales cross-modal alignment across six diverse sensory modalities into a single joint embedding space without requiring all-to-all paired training.
Paper: MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning, Yi Xin et al. (2024). It extends cross-modal alignment and parameter-efficient adaptation strategies to multi-task learning settings across downstream domains.
Paper: LLaVA-OneVision: Easy Visual Task Transfer, Bo Li et al. (2024). It advances cross-modal transfer learning by demonstrating unified alignment across diverse visual input formats including single images, multi-images, and video streams.
Paper: Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning, Bingchen Zhao et al. (2024). It investigates highly parameter-efficient adaptation mechanisms for transitioning text-pretrained foundation models to multimodal domains.

Cover for Cross-Modal Fine-Tuning: Align then Refine

Abstract

Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many other modalities due to a lack of relevant pretrained models. In this work, we propose ORCA, a general cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities. ORCA adapts to a target task via an align-then-refine workflow: given the target input, ORCA first learns an embedding network that aligns the embedded feature distribution with the pretraining modality. The pretrained model is then fine-tuned on the embedded data to exploit the knowledge shared across modalities. Through extensive experiments, we show that ORCA obtains state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities, outperforming a wide range of hand-designed, AutoML, general-purpose, and task-specific methods. We highlight the importance of data alignment via a series of ablation studies and demonstrate ORCA’s utility in data-limited regimes.

Keywords

cross-modal fine-tuning crossmodal transfer task-specific fine-tuning data alignment dimensionality alignment downstream adaptation alignment metrics embedder architecture domain adaptation transfer learning

1. Introduction
2. Related Work
3. ORCA Workflow
3.1. Architecture Design for Dimensionality Alignment
3.2. Embedder Learning for Distribution Alignment
3.3. Weight Refining for Downstream Adaptation
3.4. Evaluation of Distribution Alignment Metrics
4. Experiments
4.1. A Breadth Perspective: Can Pretrained Models Transfer Across Modalities?
KEY 1: ALIGNING FEATURE DISTRIBUTIONS
KEY 2: FINE-TUNING ALL MODEL PARAMETERS
KEY 3: ADAPTING FROM THE RIGHT MODALITY
4.2. A Depth Perspective: Cross-Modal Fine-Tuning for PDE and Tabular Tasks
PDEBENCH FOR SCIENTIFIC ML
OPENML FOR TABULAR CLASSIFICATION
4.3. Comparison with Task-Specific Cross-Modal Work
4.4. Limitation and Future Work
5. Conclusion
Acknowledgments
References
A. Appendix
A.1. Embedding Learning with Optimal Transport Dataset Distance
A.1.1. LITERATURE REVIEW
A.1.2. COMPUTATIONAL CONSIDERATIONS
A.2. ORCA Implementation
A.2.1. PRETRAINED MODELS
A.2.2. HYPERPARAMETER TUNING
A.2.3. MORE DETAILS ON EMBEDDER ARCHITECTURE DESIGN
A.2.4. EMBEDDING LEARNING WITH OTDD
A.3. Baseline Implementation
A.4. Experiments on NAS-Bench-360
A.4.1. INFORMATION ABOUT THE BENCHMARK AND EXPERIMENT PROTOCOL
A.4.2. COMPLETE RESULTS FOR TABLE 2 WITH ERROR BARS
A.4.3. COMPLETE RESULTS FOR TABLE 3 WITH ERROR BARS
A.4.4. ABLATION STUDY ON EMBEDDING LEARNING METRICS
A.4.5. ABLATION STUDY ON LAYERNORM INITIALIZATION
A.4.6. RUNTIME OF ORCA VS. FPT
A.4.7. RESULTS FOR APPLYING DIFFERENT MODEL BODIES TO DEEPSEA AND SPHERICAL
A.5. Experiments on PDEBench
A.5.1. RESULTS FOR ORCA (FIGURE 5, LEFT)
A.5.2. RESULTS FOR ZERO-SHOT SUPER-RESOLUTION (FIGURE 5, RIGHT)
A.5.3. RESULTS FOR FINE-TUNING AND TRAIN-FROM-SCRATCH BASELINES
A.6. Experiments on OpenML Tabular Datasets
A.6.1. COMPLETE RESULTS FOR TABLE 4 (TOP)
A.6.2. RESULTS FOR TRAIN-FROM-SCRATCH AND FINE-TUNING BASELINES ON OPENML-CC18
A.6.3. COMPLETE RESULTS FOR TABLE 4 (BOTTOM)
A.7. Experiments on Drug Response Prediction
A.8. Additional Experiments
A.8.1. COMPATIBILITY WITH IN-MODALITY TRANSFER
A.8.2. PROMPTING

Knowls

Knowl 1 — ORCA Align-then-Refine Cross-Modal Fine-Tuning Workflow
model/method
The ORCA framework adapts a pretrained transformer model $m^s$ (pretrained on a source domain $\mathcal{D}^s = \{x_i^s, y_i^s\}_{i=1}^{n_s}$ ) to a target domain $\mathcal{D}^t = \{x_i^t, y_i^t\}_{i=1}^{n_t}$ from a completely different modality and feature/label space ( $\mathcal{X}^t \neq \mathcal{X}^s$ , $\mathcal{Y}^t \neq \mathcal{Y}^s$ ) through a three-stage workflow:
1. Dimensionality Alignment (Architecture Construction): Decompose the target model $m^t$ into a custom embedder $f^t: \mathcal{X}^t \to \dot{\mathcal{X}}$ , a pretrained transformer body $g^s: \dot{\mathcal{X}} \to \dot{\mathcal{Y}}$ , and a custom predictor head $h^t: \dot{\mathcal{Y}} \to \mathcal{Y}^t$ , where $\dot{\mathcal{X}} = \dot{\mathcal{Y}} = \mathbb{R}^{S \times D}$ is the sequence embedding space of the pretrained transformer with maximum sequence length $S$ and hidden embedding dimension $D$ . The parameters of $g^s$ are initialized from the pretrained source model.
2. Distribution Alignment (Embedder Pre-training): Before fine-tuning the transformer body, keep $g^s$ frozen and train only the embedder $f^t$ to minimize the distributional distance (such as the Optimal Transport Dataset Distance) between the embedded target representations $(f^t(x^t), y^t)$ and pre-extracted representations from an in-modality reference/proxy source dataset $(f^s(x^s), y^s)$ , where $f^s$ is the source model's embedder.
3. Weight Refining (Downstream Fine-Tuning): With the embedder $f^t$ initialized via distribution alignment and $g^s$ initialized with pretrained weights, fine-tune all parameters of the target model $m^t = h^t \circ g^s \circ f^t$ jointly using standard gradient descent to minimize the target task loss $\mathbb{E}_{(x^t, y^t) \sim P^t}[\ell(m^t(x^t), y^t)]$ .
Knowl 2 — Embedder and Predictor Architecture Design for Cross-Modal Dimensionality Alignment
model/method
To interface arbitrary 1D and higher-dimensional target inputs with a pretrained transformer body expecting input shape $(S, D)$ (where $S$ is sequence length and $D$ is embedding dimension), ORCA defines modular embedder and predictor architectures:
- Target Embedder $f^t$ : Consists of an $n$ -dimensional convolution layer with input channels $c_{\text{in}}$ matching the target input channel dimension and output channels $c_{\text{out}} = D$ . The kernel size $k$ and stride $k$ downsample the input. For 1D tasks, $k$ is selected such that the resulting sequence length is close to a target sequence length $s^* \in \{8, 64, 512\}$ . For 2D tasks using vision transformers (e.g., Swin Transformer), target inputs are resized to the pretraining resolution (e.g., $224 \times 224$ ) and use the pretrained patchify convolution ( $k=4$ ). The non-channel spatial dimensions are flattened, padded, and transposed to form an $(S, D)$ tensor, followed by a LayerNorm layer (initialized from the pretrained model) and learnable positional embeddings.
- Target Predictor $h^t$ : Converts the transformer output $\dot{y} \in \mathbb{R}^{S \times D}$ to the target output space:
  - For classification tasks with $K$ classes: Average pooling is applied along the sequence dimension $S$ to obtain a 1D representation of length $D$ , which is passed through a linear layer mapping $\mathbb{R}^D \to \mathbb{R}^K$ .
  - For dense spatial prediction tasks: A linear layer maps the sequence output to shape $(S, k^{\text{ndim}(\mathcal{Y})} K)$ , which is reshaped and upsampled via pixel-shuffle operations to match the original spatial input dimensions.
Knowl 3 — Optimal Transport Dataset Distance Formulation for Distribution Alignment
equation
Given two datasets $\mathcal{D}_A, \mathcal{D}_B \in \mathcal{P}(\dot{\mathcal{X}} \times \mathcal{Y})$ in feature-label space, the Optimal Transport Dataset Distance (OTDD) measures their distributional difference by representing each discrete label $y \in \mathcal{Y}$ as a conditional probability distribution over in-class features $P(\dot{X} \mid Y = y)$ .

The distance on the combined space $\mathcal{Z} = \dot{\mathcal{X}} \times \mathcal{P}(\dot{\mathcal{X}})$ between points $(\dot{x}_A, y_A)$ and $(\dot{x}_B, y_B)$ is defined using the $p$ -Wasserstein metric $\mathcal{W}_p$ on feature distributions:

$d_{\mathcal{Z}}((\dot{x}_A, y_A), (\dot{x}_B, y_B)) = \left( \|\dot{x}_A - \dot{x}_B\|_2^p + \mathcal{W}_p(P(\dot{X} \mid Y=y_A), P(\dot{X} \mid Y=y_B))^p \right)^{1/p}$

The total dataset distance between datasets $\mathcal{D}_A$ and $\mathcal{D}_B$ is then the optimal transport distance between their joint representations $G(\mathcal{D}_A)$ and $G(\mathcal{D}_B)$ :

$\text{OTDD}(\mathcal{D}_A, \mathcal{D}_B) = \min_{\pi \in \Pi(G(\mathcal{D}_A), G(\mathcal{D}_B))} \int_{\mathcal{Z} \times \mathcal{Z}} d_{\mathcal{Z}}(z_A, z_B)^p \, \mathrm{d}\pi(z_A, z_B)$

For continuous labels (e.g., dense regression tasks), $K$ -means clustering is performed on the continuous labels to produce discrete pseudo-labels prior to OTDD computation.
Knowl 4 — Class-Wise Subsampling for Efficient OTDD Approximation on GPUs
algorithm
Computing exact OTDD across full datasets on GPUs is memory-intensive and computationally demanding. ORCA implements an efficient approximation using entropy-regularized Sinkhorn optimal transport combined with class-wise subsampling:
Input: Target dataset {(x_i^t, y_i^t)}_{i=1}^{n_t}, Number of target classes K^t, Source proxy dataset S = {(x_j^s, y_j^s)}_{j=1}^{n_s}, Subsample batch size b, Subsample rounds R
Output: Approximated OTDD scalar distance d
for each class i in {1, ..., K^t} do
    Compute class weight w_i = (number of target samples in class i) / (total target samples)
    Construct class data loader D_i containing samples with label i
end for
for each class i in {1, ..., K^t} do
    for each round r in {1, ..., R} do
        Subsample b target data points D_ir uniformly at random from D_i
        Compute class-wise distance d_ir = OTDD(D_ir, S) using Sinkhorn algorithm
    end for
    Compute average class distance d_i = (1 / R) * sum_{r=1}^R d_ir
end for
Compute final approximated OTDD: d = sum_{i=1}^{K^t} w_i * d_i
return d
Feature vectors are averaged along the sequence length dimension prior to computing pairwise distances, reducing optimal transport empirical complexity from worst-case $\mathcal{O}(D^3 \log D)$ to $\mathcal{O}(D^2)$ .

Knowl 5 — Cross-Modal Benchmark Performance of ORCA on NAS-Bench-360

data/table

ORCA was evaluated on the NAS-Bench-360 benchmark across 10 diverse tasks spanning vision, audio, ECG, physics, protein structure, genomics, and astrophysics. Pretrained models used were RoBERTa-base (125M parameters) for 1D tasks and Swin-base (88M parameters) for 2D tasks, with CoNLL-2003 and CIFAR-10 as source proxy datasets, respectively. The table below presents test prediction errors (lower is better; mean $\pm$ standard deviation across 5 trials):

Method	CIFAR-100	Spherical	Darcy Flow	PSICOV	Cosmic
	(0-1 err %)	(0-1 err %)	(rel. $\ell_2$ )	( $\text{MAE}_8$ )	(1-AUROC)
Hand-designed	$19.39 \pm 0.20$	$67.41 \pm 0.76$	$8.0\times 10^{-3} \pm 1.0\times 10^{-3}$	$3.35 \pm 0.14$	$0.127 \pm 0.010$
NAS-Bench-360 (Best)	$23.39 \pm 0.01$	$48.23 \pm 2.87$	$2.6\times 10^{-2} \pm 1.0\times 10^{-3}$	$2.94 \pm 0.13$	$0.229 \pm 0.040$
DASH	$24.37 \pm 0.81$	$71.28 \pm 0.68$	$7.9\times 10^{-3} \pm 2.0\times 10^{-3}$	$3.30 \pm 0.16$	$0.190 \pm 0.020$
Perceiver IO	$70.04 \pm 0.44$	$82.57 \pm 0.19$	$2.4\times 10^{-2} \pm 1.0\times 10^{-2}$	$8.06 \pm 0.06$	$0.485 \pm 0.010$
FPT	$10.11 \pm 1.18$	$76.38 \pm 4.89$	$2.1\times 10^{-2} \pm 1.3\times 10^{-3}$	$4.66 \pm 0.05$	$0.230 \pm 0.002$
ORCA	$\mathbf{6.53 \pm 0.08}$	$\mathbf{29.85 \pm 0.72}$	$\mathbf{7.28\times 10^{-3} \pm 6.8\times 10^{-5}}$	$\mathbf{1.91 \pm 0.04}$	$0.152 \pm 0.005$
Method	NinaPro	FSD50K	ECG	Satellite	DeepSEA
	(0-1 err %)	(1-mAP)	(1-F1 score)	(0-1 err %)	(1-AUROC)
Hand-designed	$8.73 \pm 0.90$	$0.62 \pm 0.004$	$0.28 \pm 0.000$	$19.80 \pm 0.00$	$0.30 \pm 0.024$
NAS-Bench-360 (Best)	$7.34 \pm 0.76$	$0.60 \pm 0.001$	$0.34 \pm 0.010$	$12.51 \pm 0.24$	$0.32 \pm 0.010$
DASH	$\mathbf{6.60 \pm 0.33}$	$0.60 \pm 0.008$	$0.32 \pm 0.007$	$12.28 \pm 0.50$	$\mathbf{0.28 \pm 0.013}$
Perceiver IO	$22.22 \pm 1.80$	$0.72 \pm 0.002$	$0.66 \pm 0.010$	$15.93 \pm 0.08$	$0.38 \pm 0.004$
FPT	$15.69 \pm 2.33$	$0.67 \pm 0.007$	$0.50 \pm 0.010$	$20.83 \pm 0.24$	$0.37 \pm 0.000$
ORCA	$7.54 \pm 0.39$	$\mathbf{0.56 \pm 0.013}$	$\mathbf{0.28 \pm 0.006}$	$\mathbf{11.59 \pm 0.18}$	$0.29 \pm 0.006$

ORCA achieves the lowest error rates on 7 of the 10 tasks, outperforming hand-designed expert models on all 10 tasks and general-purpose architectures (Perceiver IO, FPT) by large margins.

Knowl 6 — Role of Distribution Alignment in Preventing Pretrained Weight Distortion
empirical result
Ablation experiments comparing ORCA against naive fine-tuning (without distribution alignment) and training from scratch demonstrate that distribution alignment is critical for out-of-modality transfer:
1. Outperformance over Naive Fine-Tuning: On NAS-Bench-360, naive fine-tuning without Stage 2 distribution alignment underperforms ORCA across all 10 tasks (e.g., Spherical error: $55.26\%$ naive vs. $29.85\%$ ORCA; DeepSEA error: $0.51$ naive vs. $0.29$ ORCA).
2. Mitigating Negative Transfer: On ECG, Satellite, and DeepSEA, training RoBERTa from scratch outperforms naive fine-tuning (e.g., DeepSEA error: $0.39$ scratch vs. $0.51$ naive fine-tuning). This occurs because direct optimization for target loss with mismatched distributions distorts pretrained weights. Pre-aligning feature distributions via ORCA resolves this weight distortion and achieves the best overall performance ( $0.29$ ).
3. Monotonic Correlation: Tracking the OTDD between embedded target features and source reference features during embedder learning shows an empirical correlation: as the OTDD distance decreases across training epochs, the final downstream fine-tuning accuracy monotonically increases, and performance variance across runs decreases.
Knowl 7 — Full Fine-Tuning vs. Layer-Norm Fine-Tuning in Cross-Modal Adaptation
empirical result
In cross-modal fine-tuning, updating all model parameters (full fine-tuning) yields substantially superior adaptation performance compared to parameter-efficient fine-tuning that updates only layer-normalization parameters (as in Frozen Pretrained Transformers, FPT):
- Across all 10 NAS-Bench-360 tasks, full fine-tuning with ORCA achieves significantly lower errors than ORCA with only layer-norm fine-tuning (e.g., CIFAR-100 error: $6.53\%$ full vs. $7.99\%$ layer-norm; Spherical error: $29.85\%$ full vs. $42.45\%$ layer-norm; ECG error: $0.28$ full vs. $0.47$ layer-norm).
- Runtime profiling demonstrates that fine-tuning only layer norms yields less than a $2\times$ speedup compared to full fine-tuning (e.g., CIFAR-100: $8.7\text{ h}$ for layer-norm tuning vs. $10.8\text{ h}$ for full fine-tuning; Satellite: $25.56\text{ h}$ vs. $37.76\text{ h}$ ) because backpropagation through the entire network graph is still required to calculate gradients.

Knowl 8 — Cross-Modal PDE Solving on PDEBench and Zero-Shot Super-Resolution

data/table

ORCA was evaluated on 8 forward prediction partial differential equation (PDE) datasets from PDEBench using single-step prediction (mapping time step $t=0$ directly to the final state) on a single V100 GPU. Models used were RoBERTa-base (1D) and Swin-base (2D). Normalized root mean squared errors (nRMSE, lower is better) are summarized below:

Dim	Dataset	Resolution	PINN	FNO	U-Net	ORCA
1D	Advection	1024	$6.7\times 10^{-1}$	$1.1\times 10^{-2}$	$1.1$	$\mathbf{9.8\times 10^{-3}}$
1D	Burgers	1024	$3.6\times 10^{-1}$	$\mathbf{3.1\times 10^{-3}}$	$9.9\times 10^{-1}$	$1.2\times 10^{-2}$
1D	Diffusion-Reaction	1024	$6.0\times 10^{-3}$	$\mathbf{1.4\times 10^{-3}}$	$8.0\times 10^{-2}$	$3.0\times 10^{-3}$
1D	Diffusion-Sorption	1024	$1.5\times 10^{-1}$	$1.7\times 10^{-3}$	$2.2\times 10^{-1}$	$\mathbf{1.6\times 10^{-3}}$
1D	Navier-Stokes	1024	$7.2\times 10^{-1}$	$6.8\times 10^{-2}$	–	$\mathbf{6.2\times 10^{-2}}$
2D	Darcy Flow	$128\times 128$	$1.8\times 10^{-1}$	$2.2\times 10^{-1}$	–	$\mathbf{8.1\times 10^{-2}}$
2D	Shallow-Water	$128\times 128$	$8.3\times 10^{-2}$	$\mathbf{4.4\times 10^{-3}}$	$1.7\times 10^{-2}$	$6.0\times 10^{-3}$
2D	Diffusion-Reaction	$128\times 128$	$8.4\times 10^{-1}$	$\mathbf{1.2\times 10^{-1}}$	$1.6$	$8.2\times 10^{-1}$

ORCA achieves the lowest nRMSE on 4 of the 8 PDE tasks, outperforming domain-specific models (PINN and U-Net) across all evaluated benchmarks and matching Fourier Neural Operators (FNO). Furthermore, when instantiated with pointwise convolutions ( $k=1$ ) and the RoBERTa backbone, ORCA exhibits zero-shot super-resolution on 1D Advection: training on resolution 256 yields an nRMSE of $1.27\times 10^{-2}$ when evaluated directly at resolution 512 without retraining (comparable to $1.02\times 10^{-2}$ when trained directly on resolution 512).

Knowl 9 — Cross-Modal Tabular Classification via Pretrained Language Models
empirical result
ORCA adapts pretrained RoBERTa models to tabular data by preprocessing numerical features (standardized to zero mean and unit variance, concatenated into a 1D vector) and one-hot encoded categorical features, using cross-entropy loss weighted by inverse class frequencies:
1. OpenML-CC18 Benchmark (30 Datasets): Evaluated under a 0.5:0.5 train-test split using One-vs-One (OVO) AUROC, ORCA achieves an average AUROC of $0.8946 \pm 0.1206$ , winning/tying on 12/30 tasks. This matches AutoGluon ( $0.8947 \pm 0.1266$ , 12/30 wins) and outperforms TabPFN ( $0.8943 \pm 0.1249$ , 7/30 wins), XGBoost ( $0.8909 \pm 0.1224$ , 3/30 wins), CatBoost ( $0.8898$ , 1/30 wins), and LightGBM ( $0.8840$ , 1/30 wins). The average training time for ORCA is 4 minutes per tabular dataset.
2. LIFT Classification Benchmark (14 Datasets): ORCA achieves an average accuracy of $83.80\% \pm 12.81\%$ and wins on 7/14 tasks, outperforming domain-specific tabular prompting with GPT-3 (LIFT: $79.63\%$ , 2/14 wins), SVM ( $80.63\%$ , 3/14 wins), Logistic Regression ( $79.58\%$ , 2/14 wins), and XGBoost ( $78.21\%$ , 2/14 wins).
3. Drug Response Prediction: On CTRP and GDSC tabular datasets, ORCA obtains coefficients of determination ( $R^2$ ) of $0.860 \pm 0.002$ and $0.831 \pm 0.002$ , outperforming IGTD-CNN ( $0.856 \pm 0.003$ and $0.740 \pm 0.006$ ), which manually transforms tabular features into synthetic 2D images.
Knowl 10 — Guiding Pretrained Backbone Selection via Post-Alignment OTDD
empirical result
When transferring across modalities, the choice of pretrained model body can be guided by the optimized distribution alignment distance (post-alignment OTDD). In cross-architecture evaluations switching RoBERTa (1D pretraining) and Swin (2D pretraining) on DeepSEA (1D genomics) and Spherical (2D vision):
- On DeepSEA (1D), RoBERTa achieves a post-alignment OTDD of $37.40$ and test error of $0.295 \pm 0.006$ , whereas Swin achieves OTDD $64.83$ and test error $0.361 \pm 0.001$ .
- On Spherical (2D), Swin achieves a post-alignment OTDD of $11.78$ and test error of $29.85 \pm 0.072$ , whereas RoBERTa achieves OTDD $19.54$ and test error $68.28 \pm 0.017$ .
In both cases, the architecture yielding the smaller post-alignment OTDD achieves substantially better downstream fine-tuning performance, providing a principled criterion for model selection.
Knowl 11 — Sample Efficiency of ORCA in Data-Limited Regimes
empirical result
The advantage of ORCA's distribution alignment over naive fine-tuning increases as downstream training data becomes scarce. On the NAS-Bench-360 Satellite task, varying the training dataset size from $10^3$ to $10^5$ samples shows:
- The performance gap between ORCA and naive fine-tuning widens significantly in lower-data regimes (e.g., at $10^3$ samples, ORCA achieves over $15\%$ higher accuracy than naive fine-tuning).
- ORCA trained on a given subset size matches or exceeds the downstream test accuracy of naive fine-tuning trained on $3\times$ the amount of target data.

Coverage note — Exploratory experiments on visual prompt tuning (VP/VPT) and in-domain DomainNet evaluations were omitted as they represent brief secondary validations rather than core contributions.

References

1.Adhikari, B. DEEPCON: protein contact prediction using dilated convolutional neural networks with dropout. Bioinformatics, 36(2):470–477, 07 2019.
2.Aghajanyan, A., Shrivastava, A., Gupta, A., Goyal, N., Zettlemoyer, L., and Gupta, S. Better fine-tuning by reducing representational collapse. International Conference on Learning Representations, 2021.
3.Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R. C., Kalashnikov, D., Kuang, Y., Lee, K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D. M., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., and Yan, M. Do as i can, not as i say: Grounding language in robotic affordances. ArXiv, abs/2204.01691, 2022.
4.Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2022.
5.Alvarez-Melis, D. and Fusi, N. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems (NeurIPS), 2020.
6.Baevski, A., Zhou, H., rahman Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems (NeurIPS), 2020.
7.Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. 2022.
8.Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. European Conference on Computer Vision, 2020.
9.Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Largescale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 2022.
10.Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
11.Cohen, T., Geiger, M., Köhler, J., and Welling, M. Spherical cnns. In International Conference on Machine Learning, 2018.
12.Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
13.Dempster, A., Petitjean, F., and Webb, G. I. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34:1454–1495, 2020.
14.Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019, 2019.
15.Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Rajput, S., Gira, M., yong Sohn, J., Papailiopoulos, D., and Lee, K. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. ArXiv, abs/2206.06565, 2022.
16.Dolan, E. D. and Moré, J. J. Benchmarking optimization software with performance profiles. Mathematical Programming, 91:201–213, 2002.
17.Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
18.Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. Autogluon-tabular: Robust and accurate automl for structured data. ArXiv, abs/2003.06505, 2020.
19.Fang, J., Sun, Y., Zhang, Q., Li, Y., Liu, W., and Wang, X. Densely connected search space for more flexible neural architecture search. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10625–10634, 2020.
20.Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X. Fsd50k: an open dataset of human-labeled sound events. ArXiv, abs/2010.00475, 2021.
21.Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. A kernel two-sample test. Journal of Machine Learning Research, 13:723–773, 2012.
22.He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., and Neubig, G. Towards a unified view of parameter-efficient transfer learning. International Conference on Learning Representations, 2022.
23.Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. Tabpfn: A transformer that solves small tabular classification problems in a second. 2022.
24.Hong, S., Xu, Y., Khare, A., Priambada, S., Maher, K. O., Aljiffry, A., Sun, J., and Tumanov, A. Holmes: Health online model ensemble serving for deep learning models in intensive care units. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
25.Hu, R. and Singh, A. Unit: Multimodal multitask learning with a unified transformer. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1419–1429, 2021.
26.Huang, G., Liu, Z., and Weinberger, K. Q. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269, 2017.
27.Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., and Carreira, J. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, 2021.
28.Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., Henaff, O. J., Botvinick, M., Zisserman, A., Vinyals, O., and Carreira, J. Perceiver IO: A general architecture for structured inputs & outputs. In International Conference on Learning Representations, 2022.
29.Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S. J., Hariharan, B., and Lim, S. N. Visual prompt tuning. In ECCV, 2022.
30.Jiang, D., Li, W., Zhang, R., Cao, M., Luo, N., Han, Y., Zou, W., Han, K., and Li, X. A further study of unsupervised pretraining for transformer based speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6538–6542. IEEE, 2021.
31.Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Zhao, T. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
32.Josephs, D., Drake, C., Heroy, A. M., and Santerre, J. semg gesture recognition with a simple model of attention. Machine Learning for Health, pp. 126–138, 2020.
33.Jumper, J. M., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D. A., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. Highly accurate protein structure prediction with alphafold. Nature, 596:583 – 589, 2021.
34.Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
35.Kiela, D., Bhooshan, S., Firooz, H., and Testuggine, D. Supervised multimodal bitransformers for classifying images and text. ArXiv, abs/1909.02950, 2019.
36.Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, 2021.
37.Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. International Conference on Learning Representations, 2022.
38.Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., and Finn, C. Surgical fine-tuning improves adaptation to distribution shifts. ArXiv, abs/2210.11466, 2022.
39.Li, F., Zhang, H., Xu, H.-S., Liu, S., Zhang, L., Ni, L. M., and yeung Shum, H. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. ArXiv, abs/2206.02777, 2022.
40.Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Ben-Tzur, J., Hardt, M., Recht, B., and Talwalkar, A. A system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230–246, 2020a.
41.Li, S., Xie, B., Wu, J., Zhao, Y., Liu, C. H., and Ding, Z. Simultaneous semantic alignment network for heterogeneous domain adaptation. In Proceedings of the 28th ACM international conference on multimedia, pp. 3866–3874, 2020b.
42.Li, Y., Ponti, E., Vulic, I., and Korhonen, A. Emergent communication pretraining for few-shot machine translation. In COLING, 2020c.
43.Li, Z., Kovachki, N. B., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
44.Liu, C., Chen, L.-C., Schroff, F., Adam, H., Hua, W., Yuille, A. L., and Fei-Fei, L. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 82–92, 2019a.
45.Liu, H., Simonyan, K., and Yang, Y. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019b.
46.Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586, 2021a.
47.Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys (CSUR), 2022.
48.Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019c.
49.Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992–10002, 2021b.
50.Lu, K., Grover, A., Abbeel, P., and Mordatch, I. Frozen pretrained transformers as universal computation engines. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7):7628–7636, Jun. 2022.
51.Ostroumova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
52.Pan, S. J. and Yang, Q. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10): 1345–1359, 2009.
53.Pele, O. and Werman, M. Fast and robust earth mover’s distances. 2009 IEEE 12th International Conference on Computer Vision, pp. 460–467, 2009.
54.Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406–1415, 2019.
55.Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018.
56.Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
57.Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 378:686–707, 2019.
58.Real, E., Liang, C., So, D. R., and Le, Q. V. Automl-zero: Evolving machine learning algorithms from scratch. In International Conference on Machine Learning, 2020.
59.Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A. D., Heess, N. M. O., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. A generalist agent. Transactions on Machine Learning Research, 2023.
60.Reid, M., Yamada, Y., and Gu, S. S. Can wikipedia help offline reinforcement learning? ArXiv, abs/2201.12122, 2022.
61.Roberts, N. C., Khodak, M., Dao, T., Li, L., Re, C., and Talwalkar, A. Rethinking neural operations for diverse tasks. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
62.Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. ArXiv, abs/1505.04597, 2015.
63.Rothermel, D., Li, M., Rocktäschel, T., and Foerster, J. N. Don’t sweep your learning rate under the rug: A closer look at cross-modal transfer of pretrained transformers. ICML 2021 Workshop: Self-Supervised Learning for Reasoning and Perception, 2021.
64.Shen, J., Khodak, M., and Talwalkar, A. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
65.Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., and Wang, Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874–1883, 2016.
66.Takamoto, M., Praditia, T., Leiteritz, R., MacKinlay, D., Alesiani, F., Pflüger, D., and Niepert, M. Pdebench: An extensive benchmark for scientific machine learning. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022.
67.Tan, S., Peng, X., and Saenko, K. Class-imbalanced domain adaptation: An empirical odyssey. In ECCV Workshops, 2020.
68.Tu, R., Roberts, N., Khodak, M., Shen, J., Sala, F., and Talwalkar, A. NAS-bench-360: Benchmarking neural architecture search on diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022.
69.Vanschoren, J., van Rijn, J. N., Bischl, B., and Torgo, L. Openml: networked science in machine learning. SIGKDD Explor., 15:49–60, 2014.
70.Vinod, R., Chen, P.-Y., and Das, P. Reprogramming pretrained language models for protein sequence representation learning. ArXiv, abs/2301.02120, 2023.
71.Wang, M. and Deng, W. Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018.
72.Wei, Y., Hu, H., Xie, Z., Zhang, Z., Cao, Y., Bao, J., Chen, D., and Guo, B. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. ArXiv, abs/2205.14141, 2022.
73.Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019.
74.Yao, Y., Zhang, Y., Li, X., and Ye, Y. Heterogeneous domain adaptation via soft transfer network. In Proceedings of the 27th ACM international conference on multimedia, pp. 1578–1586, 2019.
75.Zhang, K. and Bloom, J. S. deepcr: Cosmic ray rejection with deep learning. The Astrophysical Journal, 889(1): 24, 2020.
76.Zhang, Z., Park, C. Y., Theesfeld, C. L., and Troyanskaya, O. G. An automated framework for efficiently designing deep convolutional neural networks in genomics. bioRxiv, 2020.
77.Zhou, J. and Troyanskaya, O. G. Predicting effects of noncoding variants with deep learning–based sequence model. Nature Methods, 12:931–934, 2015.
78.Zhu, Y., Brettin, T. S., Xia, F., Partin, A., Shukla, M., Yoo, H. S., Evrard, Y. A., Doroshow, J. H., and Stevens, R. L. Converting tabular data into images for deep learning with convolutional neural networks. Scientific Reports, 11, 2021.

Citation

MLA

Shen, J., et al. “Cross-Modal Fine-Tuning: Align Then Refine”. International Conference on Machine Learning, vol. 202, 2023, pp. 31030–56, https://proceedings.mlr.press/v202/shen23e.html.

APA

Shen, J., Li, L., Dery, L. M., Staten, C., Khodak, M., Neubig, G., & Talwalkar, A. (2023). Cross-Modal Fine-Tuning: Align then Refine. International Conference on Machine Learning, 202, 31030–31056. https://proceedings.mlr.press/v202/shen23e.html

Chicago

Shen, J., L. Li, L. M. Dery, et al. 2023. “Cross-Modal Fine-Tuning: Align Then Refine”. International Conference on Machine Learning 202: 31030–56. https://proceedings.mlr.press/v202/shen23e.html.

Harvard

Shen, J. et al. (2023) “Cross-Modal Fine-Tuning: Align then Refine”, International Conference on Machine Learning. PMLR, pp. 31030–31056. Available at: https://proceedings.mlr.press/v202/shen23e.html.

Vancouver

1. Shen J, Li L, Dery LM, Staten C, Khodak M, Neubig G, Talwalkar A (2023) Cross-Modal Fine-Tuning: Align then Refine. In: International Conference on Machine Learning. PMLR, pp 31030–31056

BibTeX

@InProceedings{pmlr-v202-shen23e,
  title = 	 {Cross-Modal Fine-Tuning: Align then Refine},
  author =       {Shen, Junhong and Li, Liam and Dery, Lucio M. and Staten, Corey and Khodak, Mikhail and Neubig, Graham and Talwalkar, Ameet},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {31030--31056},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/shen23e/shen23e.pdf},
  url = 	 {https://proceedings.mlr.press/v202/shen23e.html},
  abstract = 	 {Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many other modalities due to a lack of relevant pretrained models. In this work, we propose ORCA, a general cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities. ORCA adapts to a target task via an align-then-refine workflow: given the target input, ORCA first learns an embedding network that aligns the embedded feature distribution with the pretraining modality. The pretrained model is then fine-tuned on the embedded data to exploit the knowledge shared across modalities. Through extensive experiments, we show that ORCA obtains state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities, outperforming a wide range of hand-designed, AutoML, general-purpose, and task-specific cross-modal methods. We highlight the importance of data alignment via a series of ablation studies and exemplify ORCA’s utility in data-limited regimes.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Cross-Modal Fine-Tuning: Align then Refine

Abstract

Keywords

Table of Contents

Knowls

Knowl 1 — ORCA Align-then-Refine Cross-Modal Fine-Tuning Workflow

Knowl 2 — Embedder and Predictor Architecture Design for Cross-Modal Dimensionality Alignment

Knowl 3 — Optimal Transport Dataset Distance Formulation for Distribution Alignment

Knowl 4 — Class-Wise Subsampling for Efficient OTDD Approximation on GPUs

Knowl 5 — Cross-Modal Benchmark Performance of ORCA on NAS-Bench-360

Knowl 6 — Role of Distribution Alignment in Preventing Pretrained Weight Distortion

Knowl 7 — Full Fine-Tuning vs. Layer-Norm Fine-Tuning in Cross-Modal Adaptation

Knowl 8 — Cross-Modal PDE Solving on PDEBench and Zero-Shot Super-Resolution

Knowl 9 — Cross-Modal Tabular Classification via Pretrained Language Models

Knowl 10 — Guiding Pretrained Backbone Selection via Post-Alignment OTDD

Knowl 11 — Sample Efficiency of ORCA in Data-Limited Regimes

References

Citation

BibTeX

Source Code

Access the Paper