Built independently by an author, for readers. Read the story and support ChapterPal

keyword

Language Rewrites

Language rewrites refer to alternative, paraphrased variations of existing text descriptions generated as a form of text data augmentation for machine learning models. Typically produced using large language models, these rewrites introduce diverse sentence structures, phrasing, and vocabulary while strictly preserving the core semantic meaning, key entities, and context of the original source text. By substituting or randomly sampling from these varied descriptions during training alongside the original text, machine learning frameworks—particularly multimodal and vision-language systems—can mitigate overfitting, prevent reliance on textual shortcuts, and improve generalization across diverse real-world language inputs without altering the underlying meaning of the data.

1 item

Improving CLIP Training with Language Rewrites

Improving CLIP Training with Language Rewrites

Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, Yonglong Tian

Why you should read this

Introduces LaCLIP, a simple strategy that uses large language models to generate diverse text augmentations during contrastive pre-training, substantially increasing zero-shot transfer accuracy across vision-language benchmarks without incurring extra training overhead.

Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M. Code is available at this https URL.

Added

2026-09-26