Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

SoftCLIP

SoftCLIP is a vision-language pre-training framework that enhances contrastive learning by relaxing the strict one-to-one alignment typically enforced between paired image and text data. Unlike traditional contrastive models that treat all unpaired cross-modal instances as entirely negative, SoftCLIP models flexible many-to-many relationships across modalities by generating softened training targets derived from intra-modal self-similarity. This approach enables the model to effectively handle noise, partial semantic overlap, and ambiguous associations in web-scale datasets while refining cross-modal representations through the disentanglement of negative relationships within the target distribution.

1 item

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, Xing Sun

OrganizationsShanghai Jiao Tong UniversityTencent

Why you should read this

Proposes a relaxed contrastive learning framework that uses fine-grained intra-modal self-similarity and negative-pair disentanglement as soft alignment targets, significantly improving CLIP's zero-shot classification performance on noisy web-scale image-text datasets.

During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.

Added

2026-09-26