Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

multimodal uncertainty

Multimodal uncertainty refers to the ambiguity and variability that occur when interpreting, aligning, and integrating information across multiple distinct data modalities, such as vision, text, and audio. It encompasses both intra-modal uncertainty, which arises from incomplete or ambiguous content within a single modality, and inter-modal uncertainty, where a single representation in one modality can plausibly correspond to multiple valid targets or meanings in another. In machine learning systems, addressing multimodal uncertainty typically involves shifting from deterministic point representations to probabilistic distributions, enabling models to capture one-to-many relationships and represent the full scope of plausible semantic connections across modalities.

1 item

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, Yujiu Yang

OrganizationsInternational Digital Economy AcademyTencentTsinghua UniversityWaseda University

Why you should read this

Proposes a vision-language pre-training framework that models multimodal features as Gaussian distributions instead of deterministic points to capture inter- and intra-modal semantic uncertainty across downstream tasks like visual reasoning and image-text retrieval.

Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets. Such uncertainty is problematic for our interpretation, including inter- and intra-modal uncertainty. Little effort has studied the modeling of this uncertainty, particularly in pre-training on unlabeled datasets and fine-tuning in task-specific downstream datasets. In this paper, we project the representations of all modalities as probabilistic distributions via a Probability Distribution Encoder (PDE) by utilizing sequence-level interactions. Compared to the existing deterministic methods, such uncertainty modeling can convey richer multimodal semantic information and more complex relationships. Furthermore, we integrate uncertainty modeling with popular pre-training frameworks and propose suitable pre-training tasks: Distribution-based Vision-Language Contrastive learning (D-VLC), Distribution-based Masked Language Modeling (D-MLM), and Distribution-based Image-Text Matching (D-ITM). The fine-tuned models are applied to challenging downstream tasks, including image-text retrieval, visual question answering, visual reasoning, and visual entailment, and achieve state-of-the-art results.

Added

2026-09-26