Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Conceptual Captions datasets

Conceptual Captions datasets are large-scale multimodal benchmarks consisting of millions of paired images and descriptive text captions designed for training and evaluating vision-and-language machine learning models. Unlike traditional caption datasets built through manual human labeling, these datasets are automatically harvested from web pages by extracting online images along with their associated alternative text descriptions. The collected pairs undergo automated filtering and text transformation pipelines that eliminate noisy or non-descriptive text, verify visual-semantic relevance, and replace specific proper names and fine-grained entities with broader conceptual categories. Widely used versions, such as Conceptual Captions 3M and Conceptual 12M, provide high-scale linguistic and visual diversity to support pre-training for tasks including image captioning, cross-modal retrieval, and open-vocabulary visual recognition.

1 item

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, Ehsan Elhamifar

OrganizationsAdobeNortheastern University

Why you should read this

Proposes a cross-modal pseudo-labeling framework that aligns caption words with visual mask features and filters label noise to segment novel object classes without mask annotations.

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pre-training alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.

Added

2026-09-26