Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Robust Cross-Modal Representation Learning

Robust cross-modal representation learning is a machine learning paradigm aimed at embedding data from multiple distinct modalities, such as images and text, into a shared feature space while maintaining high performance despite real-world noise, corrupted pairings, and distribution shifts. This approach focuses on establishing meaningful correspondences across heterogeneous data types even when web-scale training data contains noisy alignments, missing information, or ambiguous descriptions. By employing techniques such as noise-tolerant contrastive objectives, soft alignments, and self-distillation, models developed under this framework create resilient multi-modal representations that transfer effectively to diverse downstream tasks, including zero-shot classification and cross-modal retrieval, across varied and unconstrained environments.

1 item

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Alex Andonian, Shixing Chen, Raffay Hamid

OrganizationsAmazonMassachusetts Institute of Technology

Why you should read this

Proposes a progressive self-distillation framework that replaces rigid one-to-one pairings in vision-language pretraining with dynamic soft-alignment targets, consistently outperforming CLIP across zero-shot, transfer, and retrieval benchmarks without extra computational overhead.

The learning objective of vision-language approach of CLIP [63] does not effectively account for the noisy many-to-many correspondences found in web-harvested image captioning datasets, which contributes to its compute and data inefficiency. To address this challenge, we introduce a novel training framework based on cross-modal contrastive learning that uses progressive self-distillation and soft image-text alignments to more efficiently learn robust representations from noisy data. Our model distills its own knowledge to dynamically generate soft-alignment targets for a subset of images and captions in every minibatch, which are then used to update its parameters. Extensive evaluation across 14 benchmark datasets shows that our method consistently outperforms its CLIP counterpart in multiple settings, including: (a) zero-shot classification, (b) linear probe transfer, and (c) image-text retrieval, without incurring extra computational cost. Analysis using an ImageNet-based robustness test-bed [70] reveals that our method offers better effective robustness to natural distribution shifts compared to both ImageNet-trained models and CLIP itself. Lastly, pretraining with datasets spanning two orders of magnitude in size shows that our improvements over CLIP tend to scale with number of training examples.

Added

2026-09-26