Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

VisualBERT pre-training

VisualBERT pre-training is the self-supervised training process used to teach the VisualBERT multimodal neural network architecture to jointly represent and understand text and imagery before being adapted to specific tasks. In this phase, the model processes paired images and descriptive captions by taking visual region features extracted from an object detector alongside text token embeddings and passing them through a shared stack of Transformer layers. Using self-attention mechanisms, the network discovers implicit alignments between words and corresponding image regions without requiring explicit word-to-box annotations. The pre-training relies on visually grounded training objectives, notably masked language modeling conditioned on image features, where the network predicts masked words using both textual and visual context, as well as sentence-image matching to verify whether a caption accurately corresponds to an image. This process equips the model with a generalized understanding of cross-modal semantics and syntactic relations, establishing a robust foundation for downstream vision-and-language tasks such as visual question answering and visual reasoning.

1 item

VisualBERT: A Simple and Performant Baseline for Vision and Language

VisualBERT: A Simple and Performant Baseline for Vision and Language

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang

OrganizationsAllen Institute for AIPeking UniversityUniversity of California, Los Angeles

Why you should read this

Introduces VisualBERT, a simple unified Transformer framework that aligns image regions with text tokens through self-attention, establishing a performant multimodal baseline across benchmarks such as VQA, VCR, and NLVR2.

We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

Added

2026-09-15