Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

sentence-image prediction objective

The sentence-image prediction objective is a self-supervised pretraining task used in multimodal machine learning models where a system is trained to determine whether a given text sentence correctly describes or corresponds to an accompanying image. Analogous to next sentence prediction in text-based language models, this objective provides the network with joint inputs composed of visual region representations and a sentence that is either the genuine matching caption or a randomly sampled, unrelated alternative. By framing the task as a binary classification problem to predict cross-modal alignment, the objective encourages the model to capture holistic relationships between visual and textual modalities, ground linguistic elements in image regions, and build unified multimodal representations that benefit downstream tasks like visual question answering and cross-modal retrieval.

1 item

VisualBERT: A Simple and Performant Baseline for Vision and Language

VisualBERT: A Simple and Performant Baseline for Vision and Language

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang

OrganizationsAllen Institute for AIPeking UniversityUniversity of California, Los Angeles

Why you should read this

Introduces VisualBERT, a simple unified Transformer framework that aligns image regions with text tokens through self-attention, establishing a performant multimodal baseline across benchmarks such as VQA, VCR, and NLVR2.

We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

Added

2026-09-15