Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

vision-language contrastive learning

Vision-language contrastive learning is a multimodal machine learning technique that trains computational models to understand the relationships between visual data, such as images or video frames, and textual descriptions. In this approach, separate neural network encoders map visual and language inputs into a shared embedding space using a contrastive objective, which pulls corresponding image-text pairs closer together while pushing non-matching pairs farther apart. By aligning cross-modal semantic representations at scale without requiring labor-intensive manual labels, this pre-training method enables models to learn rich, generalized representations that transfer effectively to downstream tasks such as cross-modal retrieval, zero-shot image classification, visual question answering, and visual reasoning.

1 item

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, Yujiu Yang

OrganizationsInternational Digital Economy AcademyTencentTsinghua UniversityWaseda University

Why you should read this

Proposes a vision-language pre-training framework that models multimodal features as Gaussian distributions instead of deterministic points to capture inter- and intra-modal semantic uncertainty across downstream tasks like visual reasoning and image-text retrieval.

Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets. Such uncertainty is problematic for our interpretation, including inter- and intra-modal uncertainty. Little effort has studied the modeling of this uncertainty, particularly in pre-training on unlabeled datasets and fine-tuning in task-specific downstream datasets. In this paper, we project the representations of all modalities as probabilistic distributions via a Probability Distribution Encoder (PDE) by utilizing sequence-level interactions. Compared to the existing deterministic methods, such uncertainty modeling can convey richer multimodal semantic information and more complex relationships. Furthermore, we integrate uncertainty modeling with popular pre-training frameworks and propose suitable pre-training tasks: Distribution-based Vision-Language Contrastive learning (D-VLC), Distribution-based Masked Language Modeling (D-MLM), and Distribution-based Image-Text Matching (D-ITM). The fine-tuned models are applied to challenging downstream tasks, including image-text retrieval, visual question answering, visual reasoning, and visual entailment, and achieve state-of-the-art results.

Added

2026-09-26