Back

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Weijie SuXizhou ZhuYue CaoBin LiLewei LuFuru WeiJifeng Dai

article2019ICLR1,855 citations

Introduces VL-BERT, a pre-trainable Transformer architecture that jointly processes visual region features and text tokens to achieve state-of-the-art performance on visual commonsense reasoning, visual question answering, and referring expression comprehension.

Listen

Artificial intelligence applications combining computer vision and natural language processing traditionally relied on task-specific models trained from separate vision and language systems. This fragmented approach frequently led to overfitting when task data was limited and created high architectural complexity across different visual-linguistic problems. The article addresses this challenge by evaluating whether a unified, pre-trained generic model can effectively align multimodal data across varied vision-and-language tasks.

The article introduces and evaluates Visual-Linguistic BERT (VL-BERT), a single-stream architecture based on bidirectional Transformer attention that ingests both textual tokens and image regions. The evaluation involved pre-training the model jointly on approximately 3.3 million image-caption pairs from the Conceptual Captions dataset alongside text-only corpora from BooksCorpus and English Wikipedia. The system was then tested across three core benchmark tasks: Visual Commonsense Reasoning (VCR), Visual Question Answering (VQA), and referring expression comprehension on the RefCOCO+ dataset.

The empirical results show that the pre-trained model establishes new performance benchmarks across target tasks. First, on the VCR benchmark, the large model configuration achieved the top single-model score, reaching a holistic accuracy of 59.7% on the test set and surpassing prior specialized architectures by significant margins. Second, pre-training yielded consistent improvements across downstream tasks, boosting final reasoning accuracy on VCR by 1.0% and question answering on VQA by 1.6% compared to training from scratch. Third, ablation analyses showed that omitting Sentence-Image Relationship Prediction—a common objective in competing models—actually improved performance, while adding text-only training corpora enhanced capability on complex, long-sentence reasoning. Finally, full end-to-end tuning of the visual feature extraction components delivered noticeable gains across all evaluated settings.

These findings indicate that complex multimodal tasks do not require fragmented, ad-hoc architectures. A single unified model can process both visual regions and linguistic tokens efficiently, reducing developmental overhead and avoiding over-customization. Organizations building multimodal AI solutions can leverage this unified pre-training approach to improve model accuracy and streamline development timelines, even when target task data is relatively scarce.

Practitioners should adopt unified transformer representations for vision-language pipelines rather than maintaining separate models. Teams should also train on paired image-text and text-only corpora simultaneously to ensure strong generalization across diverse linguistic structures. When adopting this approach, practitioners should omit sentence-image prediction objectives during pre-training to prevent negative data pairing from hampering performance.

The findings are supported by comprehensive empirical evaluations across standardized industry benchmarks; however, current pre-training has focused primarily on classification and localization tasks. Capabilities on generative tasks, such as open-ended image caption creation, require further pre-training objectives and broader architectural validation before deployment in generative settings.

MicrosoftUniversity of Science and Technology of Chinacommon-senseempirical analysisencoder-decoderVisual-Linguistic BERTvisual-linguistic representationsVisual Question Answering
arXiv: 1908.08530jackroos/VL-BERT
Cover for VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Abstract

We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{this https URL}.

Topics

common-senseempirical analysisencoder-decoderquestion answeringreasoningsyntacticsvisual informationsvideocassette recordersknowledge representation and reasoningpart of speech

Keywords

Visual-Linguistic BERTvisual-linguistic representationsVisual Question Answeringvisual commonsense reasoningConceptual Captions datasetVisual GenomeSentence-Image Relationship Predictionvisual-feature regression

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 VL-BERT
  • 3.1 Revisit BERT Model
  • 3.2 Model Architecture
  • 3.3 Pre-training VL-BERT
  • 3.4 Fine-tuning VL-BERT
  • 4 Experiment
  • 4.1 Pre-training
  • 4.2 Fine-tuning on Downstream Tasks
  • 4.2.1 Visual Commonsense Reasoning (VCR)
  • 4.2.2 Visual Question Answering (VQA)
  • 4.2.3 Referring Expression Comprehension
  • 4.3 Ablation Study
  • 5 Conclusion
  • References
  • A Appendix
  • A.1 Comparison among VL-BERT and other works
  • A.2 Detailed experiment settings
  • A.3 Visualization of attention maps in VL-BERT

Knowls

  1. Knowl 1 — Unified Single-Stream Architecture of VL-BERT

    model/method

    Visual-Linguistic BERT (VL-BERT) is a generic pre-trainable neural representation designed for visual-linguistic tasks. Unlike two-stream cross-modal architectures that process visual and linguistic inputs through separate single-modal transformers before combining them via cross-attention, VL-BERT employs a single unified multi-layer bidirectional Transformer encoder where visual and textual elements interact early and freely through unconstrained self-attention across all layers.

    The input sequence to VL-BERT is formatted as a concatenated list of special, linguistic, and visual tokens:

    1. It begins with a classification token [CLS][\text{CLS}].
    2. Followed by WordPiece linguistic tokens representing input sentence(s).
    3. A separation token [SEP][\text{SEP}] is inserted between different text sentences and between the text sequence and the visual region sequence.
    4. Followed by visual elements, where each region of interest (RoI) extracted from the image corresponds to a dedicated visual token [IMG][\text{IMG}].
    5. Terminated by an ending token [END][\text{END}].

    The model is parameterized in two capacities: VL-BERTBASE\text{VL-BERT}_{\text{BASE}} (derived from and initialized with BERTBASE\text{BERT}_{\text{BASE}}) and VL-BERTLARGE\text{VL-BERT}_{\text{LARGE}} (derived from BERTLARGE\text{BERT}_{\text{LARGE}}).

  2. Knowl 2 — Four-Component Input Feature Embedding in VL-BERT

    model/method

    For every input element ii in VL-BERT (whether a linguistic sub-word token or an image Region-of-Interest (RoI)), its input representation is constructed as the element-wise sum of four distinct embedding vectors:

    1. Token Embedding: Linguistic words are mapped to continuous vectors using a WordPiece vocabulary of 30,000 tokens. Special elements ([CLS][\text{CLS}], [SEP][\text{SEP}], [END][\text{END}]) have dedicated embeddings, while each visual element is assigned a special [IMG][\text{IMG}] token embedding.
    2. Visual Feature Embedding: Formed by passing the concatenation of visual appearance features and visual geometry features through a fully connected layer:
      • Appearance Feature: For visual RoIs, visual appearance features (2048-dimensional) are extracted prior to the output classification layer of a Fast R-CNN detector (with ResNet-101 backbone). For non-visual elements (linguistic and special tokens), the visual appearance feature is extracted by applying Fast R-CNN to an RoI spanning the entire input image.
      • Geometry Embedding: To represent the spatial position of an RoI, its normalized 4D coordinate (xLTW,yLTH,xRBW,yRBH)(\frac{x_{\text{LT}}}{W}, \frac{y_{\text{LT}}}{H}, \frac{x_{\text{RB}}}{W}, \frac{y_{\text{RB}}}{H})—where (xLT,yLT)(x_{\text{LT}}, y_{\text{LT}}) and (xRB,yRB)(x_{\text{RB}}, y_{\text{RB}}) denote the top-left and bottom-right corner coordinates, and W,HW, H are image width and height—is projected into a 2048-dimensional representation using sine and cosine functions of varying wavelengths.
    3. Segment Embedding: A learned segment embedding vector distinguishing the source modality: segment AA for the first input sentence (e.g., Question / Caption), segment BB for the second input sentence (e.g., Answer / Option), and segment CC for image RoIs.
    4. Sequence Position Embedding: A learnable positional embedding indicating token order. Because visual RoIs have no intrinsic sequential ordering, a single shared positional embedding is assigned to all visual elements in the sequence.

  3. Knowl 3 — Multi-Task Pre-Training and Joint Visual-Linguistic and Text-Only Corpus Strategy

    model/method

    VL-BERT is pre-trained jointly on both visual-linguistic data (the Conceptual Captions dataset, containing ≈3.3\approx 3.3 million image-caption pairs) and text-only corpora (BooksCorpus and English Wikipedia) with a 1:1 sample ratio per mini-batch. Joint pre-training on text-only data prevents overfitting on short captions and enhances generalization to downstream tasks containing long and syntactically complex sentences.

    On the visual-linguistic corpus, VL-BERT is optimized with two pre-training objectives:

    1. Masked Language Modeling with Visual Clues: Each word token in the input caption is randomly masked with 15% probability by replacing its token embedding with [MASK][\text{MASK}]. The Transformer is trained via cross-entropy loss over the vocabulary to predict the masked word from unmasked linguistic tokens and image visual RoI embeddings.
    2. Masked RoI Classification with Linguistic Clues: Each image RoI is randomly masked with 15% probability. The final Transformer output embedding corresponding to the masked RoI is passed to a softmax classifier to predict the object category label generated by the pre-trained object detector.

    On text-only samples ⟨Text,∅⟩\langle\text{Text}, \emptyset\rangle, the visual feature embedding defaults to a shared learnable vector across all words, and optimization proceeds solely with standard BERT Masked Language Modeling. VL-BERT deliberately omits the Sentence-Image Relationship Prediction (sentence-image alignment) objective used in other vision-language frameworks, as negative/unmatched image-caption pairs degrade the representation quality for token-level visual-linguistic alignment.

  4. Knowl 4 — Raw-Pixel Masking to Prevent Visual Clue Leakage in Masked RoI Pre-Training

    model/method

    In the Masked RoI Classification with Linguistic Clues pre-training task, standard masking applied on intermediate convolutional feature maps causes information leakage: due to the large receptive field of deep convolutional neural networks (e.g., ResNet-101), neighboring unmasked feature cells contain visual cues about the masked region.

    To prevent this leakage, VL-BERT sets the raw image pixels within the masked bounding box to zero before passing the image through the Fast R-CNN backbone. The global image RoI (used for linguistic tokens' visual context) covers the entire image and is never masked. Furthermore, the convolutional parameters of the Fast R-CNN backbone are updated end-to-end alongside the Transformer during both pre-training and downstream fine-tuning.

  5. Knowl 5 — Adaptation and Fine-Tuning of VL-BERT for Downstream Tasks

    model/method

    VL-BERT adapts to downstream visual-linguistic tasks via task-specific input/output configurations while fine-tuning all model and visual backbone parameters end-to-end:

    1. Visual Commonsense Reasoning (VCR):
      • Input format: ⟨Question,Answer,Image⟩\langle\text{Question}, \text{Answer}, \text{Image}\rangle for Question Answering (Q→AQ \to A) and ⟨Question+Answer,Rationale,Image⟩\langle\text{Question} + \text{Answer}, \text{Rationale}, \text{Image}\rangle for Answer Justification (QA→RQA \to R).
      • Output: The final hidden state of the [CLS][\text{CLS}] token is fed into a softmax classifier predicting answer correctness. During fine-tuning, a multi-task loss is used: cross-entropy over answer choices plus auxiliary Masked RoI classification with linguistic clues.
    2. Visual Question Answering (VQA):
      • Input format: ⟨Question,[MASK],Image⟩\langle\text{Question}, [\text{MASK}], \text{Image}\rangle, where the answer segment consists solely of a single [MASK][\text{MASK}] token.
      • Output: The final output feature of the [MASK][\text{MASK}] token is fed into a multi-class classifier over a predefined shared answer vocabulary (3,129 candidate answers for VQA v2.0) trained with multi-class cross-entropy.
    3. Referring Expression Comprehension (RefCOCO+):
      • Input format: ⟨Query,Image⟩\langle\text{Query}, \text{Image}\rangle.
      • Output: The final output feature corresponding to each visual RoI ([IMG][\text{IMG}] token) is passed through a binary classification head to compute matching scores with binary cross-entropy. At inference, the RoI with the highest score is selected as the referred object.

  6. Knowl 6 — Empirical Performance on Visual Commonsense Reasoning Benchmark

    data/table

    VL-BERT was evaluated on the Visual Commonsense Reasoning (VCR) dataset across Question Answering (Q→AQ \to A), Answer Justification (QA→RQA \to R), and the holistic composite reasoning metric (Q→ARQ \to AR, where a question is correct only if both the answer and the rationale are correctly predicted).

    Model Q→AQ \to A QA→RQA \to R Q→ARQ \to AR
    val test val test val test
    R2C 63.8 65.1 67.2 67.3 43.1 44.0
    ViLBERT 72.4 73.3 74.5 74.6 54.0 54.8
    VisualBERT 70.8 71.6 73.2 73.2 52.2 52.4
    B2T2 71.9 72.6 76.0 75.7 54.9 55.0
    VL-BERTBASE\text{VL-BERT}_{\text{BASE}} (w/o pre-training) 73.1 - 73.8 - 54.2 -
    VL-BERTBASE\text{VL-BERT}_{\text{BASE}} 73.8 - 74.4 - 55.2 -
    VL-BERTLARGE\text{VL-BERT}_{\text{LARGE}} 75.5 75.8 77.9 78.4 58.9 59.7

    On the VCR test set, VL-BERTLARGE\text{VL-BERT}_{\text{LARGE}} achieved 75.8% on Q→AQ \to A, 78.4% on QA→RQA \to R, and 59.7% on Q→ARQ \to AR, outperforming task-specific reasoning architectures (R2C at 44.0% Q→ARQ \to AR) as well as concurrent cross-modal models (ViLBERT at 54.8%, B2T2 at 55.0%, VisualBERT at 52.4%). Comparing VL-BERTBASE\text{VL-BERT}_{\text{BASE}} with and without pre-training demonstrates a 1.0% accuracy improvement on validation Q→ARQ \to AR.

  7. Knowl 7 — Empirical Performance on VQA and RefCOCO+ Benchmarks

    data/table

    VL-BERT was evaluated on Visual Question Answering (VQA v2.0) and Referring Expression Comprehension (RefCOCO+).

    On VQA v2.0:

    Model test-dev test-std
    BUTD 65.32 65.67
    ViLBERT 70.55 70.92
    VisualBERT 70.80 71.00
    LXMERT 72.42 72.54
    VL-BERTBASE\text{VL-BERT}_{\text{BASE}} (w/o pre-training) 69.58 -
    VL-BERTBASE\text{VL-BERT}_{\text{BASE}} 71.16 -
    VL-BERTLARGE\text{VL-BERT}_{\text{LARGE}} 71.79 72.22

    Pre-training improves VL-BERTBASE\text{VL-BERT}_{\text{BASE}} accuracy by 1.58% over the non-pretrained baseline on VQA test-dev. VL-BERTLARGE\text{VL-BERT}_{\text{LARGE}} achieves 72.22% on test-std without incorporating visual question answering data during pre-training.

    On RefCOCO+:

    Model Ground-truth Regions Detected Regions
    val testA testB val testA testB
    MAttNet 71.01 75.13 66.17 65.33 71.62 56.02
    ViLBERT - - - 72.34 78.52 62.61
    VL-BERTBASE\text{VL-BERT}_{\text{BASE}} (w/o pre-training) 74.41 77.28 67.52 66.03 71.87 56.13
    VL-BERTBASE\text{VL-BERT}_{\text{BASE}} 79.88 82.40 75.01 71.60 77.72 60.99
    VL-BERTLARGE\text{VL-BERT}_{\text{LARGE}} 80.31 83.62 75.45 72.59 78.57 62.30

    On RefCOCO+, pre-training yields a 5.47% accuracy improvement on validation for VL-BERTBASE\text{VL-BERT}_{\text{BASE}} in the ground-truth region setting (74.41% to 79.88%), outperforming the dedicated MAttNet architecture.

  8. Knowl 8 — Ablation of Pre-Training Components in VL-BERT

    data/table

    An ablation study conducted using VL-BERTBASE\text{VL-BERT}_{\text{BASE}} (evaluated with 0.5×\times downstream fine-tuning epochs) isolates the contributions of individual pre-training components:

    Settings Masked Language
    Modeling with
    Visual Clues     		Masked RoI
    Classification with
    Linguistic Clues     		Sentence-Image
    Relationship
    Prediction     		with
    Text-only
    Corpus     		Tuning
    Fast R-CNN     		VCR VQA RefCOCO+
    Q→AQ \to A val QA→RQA \to R val test-dev Detected val
    w/o pre-training 72.9 73.0 69.5 62.7
    (a) ✓ 72.9 73.1 71.0 69.1
    (b) ✓ ✓ 73.0 73.1 71.1 70.7
    (c) ✓ ✓ ✓ 72.2 72.4 70.3 69.5
    (d) ✓ ✓ ✓ 73.4 73.8 71.1 70.7
    VL-BERTBASE\text{VL-BERT}_{\text{BASE}} ✓ ✓ ✓ ✓ 73.8 73.9 71.2 71.1

    Key observations:

    1. Sentence-Image Relationship Prediction degrades performance: Adding this task (setting c vs b) reduces accuracy across all downstream tasks (Q→AQ \to A drops from 73.0 to 72.2; VQA drops from 71.1 to 70.3; RefCOCO+ drops from 70.7 to 69.5), likely because creating negative unmatched pairs hampers visual-linguistic feature alignment.
    2. Text-only corpus benefits complex text: Incorporating text-only corpora (setting d vs b) significantly boosts performance on VCR (Q→AQ \to A increases from 73.0 to 73.4; QA→RQA \to R from 73.1 to 73.8), which features long sentences (averaging 16.0 words in Q→AQ \to A and 33.7 words in QA→RQA \to R), while preserving performance on tasks with shorter sentences.
    3. End-to-end visual backbone fine-tuning: Updating Fast R-CNN parameters during training further improves results on all benchmarks.

  9. Knowl 9 — Training Configuration and Optimization Setup for VL-BERT

    experimental setup

    VL-BERT pre-training and downstream fine-tuning follow specific optimization schedules across 16 NVIDIA Tesla V100 GPUs:

    • Pre-Training:

      • Training duration: 250,000 iterations using Stochastic Gradient Descent with the Adam optimizer (β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay 10−410^{-4}).
      • Learning rate: Base learning rate 2×10−52 \times 10^{-5}, linear warmup over the first 8,000 steps, followed by linear decay.
      • Mini-batch: 256 samples per batch, comprising 128 ⟨Caption,Image⟩\langle\text{Caption}, \text{Image}\rangle pairs from Conceptual Captions and 128 sequential token sequences (maximum 64 tokens each) from BooksCorpus and English Wikipedia.
      • RoI Extraction: Faster R-CNN with ResNet-101 backbone pre-trained on Visual Genome object detection; per image, up to 100 RoIs with detection scores >0.5> 0.5 (minimum 10 RoIs per image regardless of score) are selected.

    • Downstream Fine-Tuning:

      • VCR: 20 epochs, batch size 256, SGD with base learning rate 5×10−35 \times 10^{-3}, momentum 0.90.9, weight decay 10−410^{-4}, 1,000-step linear warmup, decayed by a factor of 0.1 at epochs 14 and 18.
      • VQA: 20 epochs, batch size 256, Adam optimizer with base learning rate 1×10−41 \times 10^{-4}, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay 10−410^{-4}, linear warmup over 2,000 steps, linear decay.
      • RefCOCO+: 20 epochs, batch size 256, Adam optimizer with base learning rate 1×10−41 \times 10^{-4}, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay 10−410^{-4}, linear warmup over 500 steps, linear decay.

Coverage note — None omitted; all core contributions, architectural formulations, pre-training objectives, ablation findings, downstream adaptations, and empirical benchmarks are captured.

References

  1. 1.Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. arXiv preprint arXiv:1908.05054, 2019.
  2. 2.Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077–6086, 2018.
  3. 3.Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425–2433, 2015.
  4. 4.Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  5. 5.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
  6. 6.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  7. 7.Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp. 647–655, 2014.
  8. 8.Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. From two graphs to n questions: A vqa dataset for compositional reasoning on vision and commonsense. arXiv preprint arXiv:1908.02962, 2019.
  9. 9.Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440–1448, 2015.
  10. 10.Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587, 2014.
  11. 11.Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904–6913, 2017.
  12. 12.Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision, pp. 297–312. Springer, 2014.
  13. 13.Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. arXiv preprint arXiv:1811.08883, 2018.
  14. 14.Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3588–3597, 2018.
  15. 15.Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700–6709, 2019.
  16. 16.Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901–2910, 2017.
  17. 17.Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787–798, 2014.
  18. 18.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  19. 19.Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294–3302, 2015.
  20. 20.Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.
  21. 21.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
  22. 22.Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
  23. 23.Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training, 2019a.
  24. 24.Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019b.
  25. 25.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer, 2014.
  26. 26.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  27. 27.Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440, 2015.
  28. 28.Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019.
  29. 29.Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
  30. 30.Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014.
  31. 31.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
  32. 32.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019.
  33. 33.Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91–99, 2015.
  34. 34.Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565, 2018.
  35. 35.Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743, 2019a.
  36. 36.Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766, 2019b.
  37. 37.Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
  38. 38.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017.
  39. 39.Jesse Vig. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714, 2019. URL https://arxiv.org/abs/1906.05714.
  40. 40.Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094, 2019.
  41. 41.Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
  42. 42.Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019.
  43. 43.Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014.
  44. 44.Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1307–1315, 2018.
  45. 45.Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6720–6731, 2019.
  46. 46.Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995–5004, 2016.
  47. 47.Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015.

Citation

MLA
Su, W., et al. “VL-BERT: Pre-training of Generic Visual-Linguistic Representations”. arXiv, 2019, http://arxiv.org/abs/1908.08530v4.
APA
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019). VL-BERT: Pre-training of Generic Visual-Linguistic Representations. arXiv. http://arxiv.org/abs/1908.08530v4
Chicago
Su, W., X. Zhu, Y. Cao, et al. 2019. “VL-BERT: Pre-training of Generic Visual-Linguistic Representations”. arXiv. http://arxiv.org/abs/1908.08530v4.
Harvard
Su, W. et al. (2019) “VL-BERT: Pre-training of Generic Visual-Linguistic Representations”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1908.08530v4.
Vancouver
1. Su W, Zhu X, Cao Y, Li B, Lu L, Wei F, Dai J (2019) VL-BERT: Pre-training of Generic Visual-Linguistic Representations. arXiv

BibTeX

@article{su2019bert,
  title = {VL-BERT: Pre-training of Generic Visual-Linguistic Representations},
  author = {Su, Weijie and Zhu, Xizhou and Cao, Yue and Li, Bin and Lu, Lewei and Wei, Furu and Dai, Jifeng},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1908.08530v4},
  eprint = {1908.08530}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission