keyword
language-image pretraining
Language-image pretraining is a machine learning paradigm in which neural networks are trained on large datasets of paired images and text descriptions to learn shared representations connecting visual and linguistic information. By mapping images and their corresponding textual descriptions into a common embedding space, typically using training objectives such as contrastive learning, the model learns to associate visual patterns with natural language without relying on fixed, hand-annotated categorical labels. This multimodal alignment enables the resulting models to generalize effectively across diverse downstream tasks, such as zero-shot image classification, cross-modal retrieval, and open-vocabulary visual recognition, by interpreting visual concepts through open-ended textual prompts.
3 items
SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, Tianrui Li
Why you should read this
Proposes SegCLIP, an annotation-free open-vocabulary semantic segmentation framework that dynamically aggregates Vision Transformer patches into irregular semantic regions via learnable centers while training solely on image-text pairs with auxiliary reconstruction and superpixel-guided losses.
Recently, the contrastive language-image pre-training, e.g., CLIP, has demonstrated promising results on various downstream tasks. The pre-trained model can capture enriched visual concepts for images by learning from a large scale of text-image data. However, transferring the learned visual knowledge to open-vocabulary semantic segmentation is still under-explored. In this paper, we propose a CLIP-based model named SegCLIP for the topic of open-vocabulary segmentation in an annotation-free manner. The SegCLIP achieves segmentation based on ViT and the main idea is to gather patches with learnable centers to semantic regions through training on text-image pairs. The gathering operation can dynamically capture the semantic groups, which can be used to generate the final segmentation results. We further propose a reconstruction loss on masked patches and a superpixel-based KL loss with pseudo-labels to enhance the visual representation. Experimental results show that our model achieves comparable or superior segmentation accuracy on the PASCAL VOC 2012 (+0.3% mIoU), PASCAL Context (+2.3% mIoU), and COCO (+2.2% mIoU) compared with baselines. We release the code at https://github.com/ArrowLuo/SegCLIP.
Added
2026-09-28
Improving CLIP Training with Language Rewrites
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, Yonglong Tian
Why you should read this
Introduces LaCLIP, a simple strategy that uses large language models to generate diverse text augmentations during contrastive pre-training, substantially increasing zero-shot transfer accuracy across vision-language benchmarks without incurring extra training overhead.
Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M. Code is available at this https URL.
Added
2026-09-26
Vision-Language Models for Vision Tasks: A Survey
Jingyi Zhang, Jiaxing Huang, Sheng Jin, Shijian Lu
Why you should read this
Systematizes the development of vision-language models by categorizing foundational architectures, pre-training objectives, transfer learning strategies, and knowledge distillation methods across standard visual recognition benchmarks.
Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition. A project associated with this survey has been created at this https URL.
Added
2026-09-24