Back

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Xiujun LiXi YinChunyuan LiXiaowei HuPengchuan ZhangLei ZhangLijuan WangHoudong HuLi DongFuru Wei

article2020ECCV2,282 citations

Proposes Oscar, a vision-language pre-training framework that uses detected object tags as semantic anchor points to improve cross-modal alignment and establish new state-of-the-art results across six understanding and generation benchmarks.

Listen

Artificial intelligence applications increasingly require systems that seamlessly connect visual content with natural language, such as searching image catalogs with text, automatically generating captions, or answering visual questions. Traditional vision-language pre-training methods typically concatenate raw image region features and text embeddings, relying on complex neural networks to learn alignments from scratch. However, this approach faces severe challenges due to overlapping, ambiguous visual region detections and the lack of explicit labels linking specific visual regions to corresponding words.

The article demonstrates that introducing detected object tags as textual anchor points substantially simplifies cross-modal alignment, creating a more parameter-efficient and accurate framework for vision-language tasks.

The researchers developed Oscar (Object-Semantics Aligned Pre-training), which structures training inputs as triples containing text words, detected object tags, and image region features. Because modern object detectors reliably detect salient items that are frequently mentioned in paired descriptions, these textual tags ground ambiguous image features into a shared linguistic space. The framework uses a dual-objective training strategy: a masked token loss that reconstructs masked words or tags from visual and linguistic context, and a contrastive loss that distinguishes correct image representations from randomly altered ones. The model was pre-trained on a corpus of 6.5 million text-image pairs and subsequently evaluated across seven diverse benchmark tasks covering visual understanding and language generation.

The evaluation produced four primary findings. First, Oscar established new state-of-the-art performance across six well-established vision-language benchmarks, consistently surpassing prior systems in both understanding and generation. Second, the model demonstrated exceptional parameter efficiency; the standard base model outperformed prior large models on most tasks despite being trained on roughly 30% fewer image-text pairs than comparable benchmarks. Third, in retrieval and captioning tasks, the system achieved substantial accuracy gains, improving top-1 text retrieval accuracy on the primary benchmark by nearly 7 percentage points and significantly boosting caption quality metrics. Finally, ablation and fine-tuning analyses showed that incorporating object tags accelerated task training significantly, halving the training time required to reach baseline performance levels while also improving generalizability to novel objects.

These findings indicate that providing explicit semantic bridges between modalities is far more effective and computationally efficient than relying on brute-force attention mechanisms alone. By reducing the training volume and model size required for top-tier performance, this approach lowers compute costs, reduces training timelines, and enhances model robustness when deployed on real-world data containing unseen objects.

Organizations developing or deploying multimodal artificial intelligence systems should integrate object-level semantic grounding into their pre-training and fine-tuning pipelines. Development teams can adopt the publicly released codebase and pre-trained models to accelerate existing multimodal workflows. Future technical initiatives should focus on pairing the architecture with more precise, diverse object detectors and exploring the incorporation of structural reasoning priors to further boost performance on complex logical visual reasoning tasks.

The primary limitation of the method is its operational dependence on the quality and vocabulary coverage of the underlying object detector, as missing or mislabeled tags can restrict alignment quality. Nonetheless, because the methodology demonstrated consistent, statistically strong improvements across a diverse range of benchmarks and tag sets, confidence in the reported performance advantages remains high.

MicrosoftUniversity of Washingtonimage pairsreference imagetext featureImageOscarTasks
arXiv: 2004.06165microsoft/Oscar
Cover for Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Abstract

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.

Topics

image pairsreference imagetext feature

Keywords

ImageOscarTasksObjectObject TagsPre-trainingObject-Semantics Aligned Pre-trainingVision-Language TasksLearningVLP

Table of Contents

  • 1 Introduction
  • 2 Background
  • 3 Oscar Pre-training
  • Input
  • Pre-Training Objective
  • Pre-training Corpus
  • Implementation Details
  • 4 Adapting to V+L Tasks
  • Image-Text Retrieval
  • Image Captioning
  • Novel Object Captioning (NoCaps)
  • VQA
  • GQA
  • Natural Language Visual Reasoning for Real (NLVR2)
  • 5 Experimental Results & Analysis
  • 5.1 Performance Comparison with SoTA
  • 5.2 Qualitative Studies
  • 5.3 Ablation Analysis
  • The Effect of Object Tags
  • Attention Interaction
  • Object Tags in Pre-training
  • 6 Related Work
  • Vision-Language Pre-training
  • Object Tags
  • Multimodal Embeddings
  • 7 Conclusion
  • References
  • 0.A Fine-tuning Settings
  • Image-Text Retrieval
  • Image Captioning
  • NoCaps
  • VQA
  • GQA
  • NLVR2
  • 0.B Pre-training Corpus
  • 0.C More Results

Knowls

  1. Knowl 1 — Input Representation via Word-Tag-Image Triples

    model/method

    Vision-Language Pre-training (VLP) models traditionally represent image-text instances as pairs (w,v)(w, v) containing text word tokens ww and visual region features vv. Oscar (Object-Semantics Aligned Pre-training) modifies this input formulation by introducing detected object tags qq as anchor points, structuring every input sample as a triple:

    (w,q,v)(w, q, v)

    where:

    • w=(w1,…,wT)w = (w_1, \dots, w_T) is the sequence of word embeddings corresponding to the textual sentence.
    • q=(q1,…,qM)q = (q_1, \dots, q_M) is the sequence of word embeddings of high-precision object tags detected in the image using an object detector (Faster R-CNN).
    • v={v1,…,vK}v = \{v_1, \dots, v_K\} is the set of visual region feature vectors extracted from KK regions of interest. For each detected region kk, the raw visual feature vk′∈RPv'_k \in \mathbb{R}^P (with P=2048P = 2048) is concatenated with its spatial position coordinates zk∈RRz_k \in \mathbb{R}^R (with R∈{4,6}R \in \{4, 6\}), and mapped to the embedding dimension of BERT via a learned linear projection matrix WW:

    vk=W[vk′;zk]v_k = W [v'_k; z_k]

    This input is interpreted under two distinct structural perspectives:

    1. Modality View: Differentiates the language modality ww from the combined image modality h′=[q,v]h' = [q, v].
    2. Dictionary View: Differentiates the discrete linguistic semantic space h=[w,q]h = [w, q] (which share the vocabulary dictionary of a pre-trained language model) from the continuous visual semantic space vv.

  2. Knowl 2 — Oscar Pre-training Objectives: Masked Token Loss and Contrastive Loss

    model/method

    Oscar optimizes a joint pre-training objective derived from the dictionary and modality views of the input triple (w,q,v)(w, q, v):

    LPre-training=LMTL+LC\mathcal{L}_{\text{Pre-training}} = \mathcal{L}_{\text{MTL}} + \mathcal{L}_C

    1. Masked Token Loss (MTL) from the Dictionary View: The discrete token sequence h=[w,q]h = [w, q] combines text tokens ww and object tags qq, both embedded in the linguistic space. During pre-training, each discrete token in hh is independently masked with 15%15\% probability and replaced by [MASK]. The objective minimizes the negative log-likelihood of recovering the masked token hih_i conditioned on the unmasked discrete context h∖ih_{\setminus i} and all visual region features vv:

    LMTL=−E(v,h)∼Dlog⁡p(hi∣h∖i,v)\mathcal{L}_{\text{MTL}} = -\mathbb{E}_{(v, h) \sim \mathcal{D}} \log p(h_i \mid h_{\setminus i}, v)

    Attending to the visual features vv provides visual grounding to predict missing word tokens or tags.

    2. Contrastive Loss (LC\mathcal{L}_C) from the Modality View: To align cross-modal semantics, the image modality is grouped as h′=[q,v]h' = [q, v] and the language modality as ww. A set of polluted image representations is created by replacing the tag sequence qq with probability 0.50.5 with a random tag sequence sampled from the dataset D\mathcal{D}. A linear binary classifier f(⋅)f(\cdot) applied to the pooled [CLS] token representation predicts whether the pair is an original match (y=1y = 1) or a polluted instance (y=0y = 0):

    LC=−E(h′,w)∼Dlog⁡p(y∣f(h′,w))\mathcal{L}_C = -\mathbb{E}_{(h', w) \sim \mathcal{D}} \log p(y \mid f(h', w))

    This forces the pre-trained language model embedding space to draw representations of text close to matching object tags and regions while pushing mismatched pairs apart.

  3. Knowl 3 — Pre-training Setup and Implementation Configurations for Oscar

    experimental setup

    The pre-training corpus for Oscar contains 6.56.5 million text-tag-image triples collected from 4.14.1 million unique images across public vision-language datasets, including MS COCO, Conceptual Captions (CC), SBU Captions, Flickr30k, and GQA.

    Two model sizes are pre-trained:

    • OscarB\text{Oscar}_B (Base): Initialized with BERTbase\text{BERT}_{\text{base}} weights (hidden size H=768H = 768, 12 layers, 12 attention heads). Optimized using AdamW for at least 1.0×1061.0 \times 10^6 steps with a learning rate of 5×10−55 \times 10^{-5} and a batch size of 768.
    • OscarL\text{Oscar}_L (Large): Initialized with BERTlarge\text{BERT}_{\text{large}} weights (hidden size H=1024H = 1024, 24 layers, 16 attention heads). Optimized using AdamW for at least 9.0×1059.0 \times 10^5 steps with a learning rate of 1×10−51 \times 10^{-5} and a batch size of 512.

    For every training sample, the maximum sequence length of discrete tokens h=[w,q]h = [w, q] is set to 35, and the number of region features vv is 50. The trainable parameter set is θ={θBERT,W}\theta = \{\theta_{\text{BERT}}, W\}, where WW is the linear projection matrix mapping visual region features to the BERT hidden dimension.

  4. Knowl 4 — Task Adaptation and Fine-Tuning Strategies in Oscar

    model/method

    Oscar is fine-tuned on vision-language understanding and generation tasks through task-specific head designs and attention masking:

    • Image-Text Retrieval: Formulated as binary classification on the [CLS] embedding. Aligned pairs receive label y=1y=1, while negative pairs (constructed by randomly swapping images or captions) receive label y=0y=0. At test time, the predicted probability score is used to rank candidate pairs.
    • Image Captioning: Fine-tuned via sequence-to-sequence prediction where 15%15\% of caption tokens in (w,q,v)(w, q, v) triples are masked. A causal uni-directional self-attention mask is applied to caption tokens so they attend only to prior caption tokens, while attending bidirectionally to image regions vv and object tags qq. Inference uses autoregressive decoding with beam search (beam size 5), followed by Self-Critical Sequence Training (SCST) with CIDEr reward optimization.
    • Novel Object Captioning (NoCaps): Initialized directly from BERT without cross-modal pre-training, trained strictly on COCO captions using predicted Visual Genome and Open Images tags, and decoded at test time using Constrained Beam Search (CBS) to integrate detected novel object tags.
    • Visual Question Answering (VQA v2.0): Concatenates question tokens, object tags, and region features into one sequence. The [CLS] embedding feeds into a linear classifier predicting over 3,129 answer classes, trained via cross-entropy with soft multi-label target scores.
    • GQA: Evaluated by selecting an answer from 1,852 candidates. Fine-tuning is conducted either standardly or in a two-stage regime: 5 epochs on the unbalanced "all-split" followed by 2 epochs on the "balanced-split".
    • Natural Language Visual Reasoning (NLVR2): For a text statement and two images, two separate sequence representations are constructed ([text,image1][\text{text}, \text{image}_1] and [text,image2][\text{text}, \text{image}_2]). The two output [CLS] embeddings are concatenated and classified using a multi-layer perceptron (MLP).

  5. Knowl 5 — Benchmark Results Across Vision-Language Understanding and Generation Tasks

    data/table

    Oscar establishes new state-of-the-art results across standard vision-language benchmarks, with the base model (OscarB\text{Oscar}_B) frequently outperforming previous BERT-large-sized VLP models.

    Task Image Retrieval Text Retrieval Image Captioning VQA NLVR2
    R@1 R@5 R@10 R@1 R@5 R@10 B@4 M C S test-std test-P
    SoTAS\text{SoTA}_S 39.2 68.0 81.3 56.6 84.5 92.0 38.9 29.2 129.8 22.4 70.90 53.50
    SoTAB\text{SoTA}_B 48.4 76.7 85.9 63.3 87.0 93.1 39.5 29.3 129.3 23.2 72.54 78.87
    SoTAL\text{SoTA}_L 51.7 78.4 86.9 66.6 89.4 94.3 – – – – 73.40 79.50
    OscarB\text{Oscar}_B 54.0 80.8 88.5 70.0 91.1 95.5 40.5 29.7 137.6 22.8 73.44 78.36
    OscarL\text{Oscar}_L 57.5 82.8 89.8 73.5 92.2 96.0 41.7 30.6 140.0 24.5 73.82 80.37
    Δ\Delta +5.8 +4.4 +2.9 +6.9 +2.8 +1.7 +2.2 +1.3 +10.7 +1.3 +0.42 +0.87

    Metric definitions and evaluation protocols:

    • Retrieval results denote Recall@KK (R@1, R@5, R@10) on the COCO 5K test set.
    • Captioning metrics denote BLEU@4 (B@4), METEOR (M), CIDEr (C), and SPICE (S) under CIDEr optimization on the COCO test set.
    • SoTAS\text{SoTA}_S, SoTAB\text{SoTA}_B, and SoTAL\text{SoTA}_L represent prior state-of-the-art results from small non-VLP architectures, BERT-base-sized VLP models, and BERT-large-sized VLP models (UNITER-Large), respectively.
    • Δ\Delta represents the net gain achieved by OscarL\text{Oscar}_L over the strongest existing single-model SoTA.

  6. Knowl 6 — Task Performance Breakdown Across Downstream Benchmarks

    data/table

    Detailed task comparisons between Oscar and competing vision-language pre-training methods:

    (a) Image-Text Retrieval (COCO 5K Test Set)
    Method Text R@1 Text R@5 Image R@1 Image R@5
    UNITERB_B 63.3 87.0 48.4 76.7
    UNITERL_L 66.6 89.4 51.7 78.4
    OscarB\text{Oscar}_B 70.0 91.1 54.0 80.8
    OscarL\text{Oscar}_L 73.5 92.2 57.5 82.8
    (b) Visual Question Answering (VQA v2.0)
    Method Test-dev Test-std
    LXMERT 72.42 72.54
    UNITERB_B 72.27 72.46
    UNITERL_L 73.24 73.40
    OscarB\text{Oscar}_B 73.16 73.44
    OscarL\text{Oscar}_L 73.61 73.82
    (c) Visual Reasoning (NLVR2)
    Method Dev Test-P
    LXMERT 74.90 74.50
    UNITERB_B 77.14 77.87
    UNITERL_L 78.40 79.50
    OscarB\text{Oscar}_B 78.07 78.36
    OscarL\text{Oscar}_L 79.12 80.37
    (d) Visual Reasoning (GQA)
    Method Test-dev Test-std
    LXMERT 60.00 60.33
    OscarB\text{Oscar}_B 61.19 61.23
    OscarB∗\text{Oscar}_B^* (all-split pre-finetuning) 61.58 61.62

    On COCO Image Captioning under cross-entropy optimization alone, OscarB\text{Oscar}_B achieves BLEU@4 36.5 and CIDEr 123.7, while OscarL\text{Oscar}_L achieves BLEU@4 37.4 and CIDEr 127.8, outperforming VLP models such as Unified VLP (BLEU@4 36.5, CIDEr 117.7).

  7. Knowl 7 — Novel Object Captioning Generalization on NoCaps

    data/table

    On the NoCaps validation benchmark, which evaluates describing unseen visual objects without pre-training on NoCaps images, Oscar variants consistently surpass previous state-of-the-art models (UpDown):

    Method In-domain Near-domain Out-of-domain Overall
    CIDEr SPICE CIDEr SPICE CIDEr SPICE CIDEr SPICE
    UpDown 78.1 11.6 57.7 10.3 31.3 8.3 55.3 10.1
    UpDown + CBS 80.0 12.0 73.6 11.3 66.4 9.7 73.1 11.1
    UpDown + ELMo + CBS 79.3 12.4 73.8 11.4 71.7 9.9 74.3 11.2
    OscarB\text{Oscar}_B 79.6 12.3 66.1 11.5 45.3 9.7 63.8 11.2
    OscarB\text{Oscar}_B + CBS 80.0 12.1 80.4 12.2 75.3 10.6 79.3 11.9
    OscarB\text{Oscar}_B + SCST + CBS 83.4 12.0 81.6 12.0 77.6 10.6 81.1 11.7
    OscarL\text{Oscar}_L 79.9 12.4 68.2 11.8 45.1 9.4 65.2 11.4
    OscarL\text{Oscar}_L + CBS 78.8 12.2 78.9 12.1 77.4 10.5 78.6 11.8
    OscarL\text{Oscar}_L + SCST + CBS 85.4 11.9 84.0 11.7 80.3 10.0 83.4 11.4

    The margin of improvement is largest on near-domain and out-of-domain categories (e.g., out-of-domain CIDEr improves from 71.7 for UpDown + ELMo + CBS to 80.3 for OscarL\text{Oscar}_L + SCST + CBS), confirming that grounding object tags directly in pre-trained linguistic embeddings confers strong out-of-distribution transfer.

  8. Knowl 8 — Convergence Speed and Accuracy Impact of Object Tags

    empirical result

    Ablation experiments comparing three object tag conditions during downstream fine-tuning (VQA, image retrieval, and image captioning) on a BERTbase\text{BERT}_{\text{base}} backbone show:

    1. Baseline (No Tags): Input contains only words ww and visual features vv.
    2. Predicted Tags: Input contains tags qq detected by Faster R-CNN trained on COCO.
    3. Ground-Truth Tags: Input contains human-annotated COCO object tags, serving as an upper bound.

    Fine-tuning dynamics demonstrate that incorporating predicted object tags accelerates training convergence: for both VQA and image retrieval tasks, models utilizing tags require only half the number of optimization steps to reach the final asymptotic performance of the baseline without tags. Furthermore, models using ground-truth tags achieve strictly higher performance ceilings, demonstrating that downstream performance scales directly with object detector tag fidelity.

  9. Knowl 9 — Impact of Pre-training Object Tag Source on Downstream Performance

    empirical result

    Pre-training models for a fixed budget of 589k steps across different object tag detector datasets reveals that object vocabulary diversity is more beneficial than detection precision:

    Pre-train Setting VQA Text Retrieval Image Retrieval Captioning
    Dev R@1 R@5 R@10 R@1 R@5 R@10 B@4 CIDEr
    Baseline (No Tags) 70.93 84.4 98.1 99.5 73.1 94.5 97.9 34.5 115.6
    OscarVG\text{Oscar}^{\text{VG}} 71.70 88.4 99.1 99.8 75.7 95.2 98.3 36.4 123.4
    OscarOI\text{Oscar}^{\text{OI}} 71.15 85.9 97.9 99.5 72.9 94.3 97.6 35.3 119.5

    In the table, OscarVG\text{Oscar}^{\text{VG}} uses tags from an object detector trained on Visual Genome (VG), whereas OscarOI\text{Oscar}^{\text{OI}} uses tags from an object detector trained on Open Images (OI). While both tag-augmented variants outperform the no-tag baseline across all metrics, OscarVG\text{Oscar}^{\text{VG}} outperforms OscarOI\text{Oscar}^{\text{OI}} (e.g., text retrieval R@1 of 88.4 vs. 85.9, and captioning CIDEr of 123.4 vs. 119.5), indicating that a wider, more diverse object tag vocabulary provides more effective anchor points than higher precision on a smaller vocabulary.

  10. Knowl 10 — Attention Interaction Mechanism Across Modalities

    empirical result

    Ablating the attention mask connectivity among word tokens ww, visual regions vv, and object tags qq on COCO 1K retrieval using a BERTbase\text{BERT}_{\text{base}} backbone without pre-training produces the following results:

    Active Attention Links Text Retrieval Image Retrieval
    ww-vv ww-qq vv-qq R@1 R@5 R@1 R@5
    ✓ ✓ ✓ 77.3 95.6 65.2 91.5
    ✓ 75.4 94.8 64.2 91.4
    ✓ 32.3 57.6 25.7 60.1

    Full cross-modal attention (ww-vv, ww-qq, and vv-qq) outperforms direct text-region interaction alone (ww-vv), yielding a +1.9 gain in Text R@1 and +1.0 in Image R@1. Isolated text-tag interaction (ww-qq) without region features suffers severe degradation (Text R@1 of 32.3), demonstrating that object tags function primarily as semantic bridges and alignment anchors rather than as replacements for dense visual region features.

Coverage note — No substantial contributed material was omitted; the knowls fully capture Oscar's triple input formulation, pre-training objectives, experimental setups, fine-tuning mechanisms, benchmark results, NoCaps evaluation, and ablation analyses.

References

  1. 1.Agrawal, H., et al.: Nocaps: novel object captioning at scale. In: ICCV (2019)
  2. 2.Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018)
  3. 3.Brown, P.F., Lai, J.C., Mercer, R.L.: Aligning sentences in parallel corpora. In: Proceedings of the 29th Annual Meeting on Association for Computational Linguistics (1991)
  4. 4.Chen, W., Gan, Z., Li, L., Cheng, Y., Wang, W., Liu, J.: Meta module network for compositional visual reasoning (2019). arXiv preprint arXiv:1910.03230
  5. 5.Chen, Y.C., et al.: Uniter: learning universal image-text representations (2019). arXiv preprint arXiv:1909.11740
  6. 6.Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019)
  7. 7.Frome, A., et al.: DeViSE: a deep visual-semantic embedding model. In: NeurIPS (2013)
  8. 8.Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: elevating the role of image understanding in visual question answering. In: CVPR (2017)
  9. 9.Hao, W., Li, C., Li, X., Carin, L., Gao, J.: Towards learning a generic agent for vision-and-language navigation via pre-training. In: CVPR (2020)
  10. 10.Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: ICCV (2019)
  11. 11.Hudson, D., Manning, C.D.: Learning by abstraction: the neural state machine. In: NeurIPS (2019)
  12. 12.Hudson, D.A., Manning, C.D.: GQA: a new dataset for real-world visual reasoning and compositional question answering (2019). arXiv preprint arXiv:1902.09506
  13. 13.Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015)
  14. 14.Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models (2014). arXiv preprint arXiv:1411.2539
  15. 15.Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123(1), 32–73 (2017). https://doi.org/10.1007/s11263-016-0981-7
  16. 16.Kuznetsova, A., et al.: The open images dataset v4: unified image classification, object detection, and visual relationship detection at scale (2018). arXiv preprint arXiv:1811.00982
  17. 17.Lee, K.H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: ECCV (2018)
  18. 18.Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-VL: a universal encoder for vision and language by cross-modal pre-training (2019). arXiv preprint arXiv:1908.06066
  19. 19.Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: a simple and performant baseline for vision and language (2019). arXiv preprint arXiv:1908.03557
  20. 20.Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: ECCV (2014)
  21. 21.Lu, J., Batra, D., Parikh, D., Lee, S.: VilBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019)
  22. 22.Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-Task vision and language representation learning (2019). arXiv preprint arXiv:1912.02315
  23. 23.Maaten, L.V.D., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)
  24. 24.Norouzi, M., et al.: Zero-shot learning by convex combination of semantic embeddings (2013). arXiv preprint arXiv:1312.5650
  25. 25.Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: describing images using 1 million captioned photographs. In: NeurIPS (2011)
  26. 26.Qi, D., Su, L., Song, J., Cui, E., Bharti, T., Sacheti, A.: Imagebert: cross-modal pre-training with large-scale weak-supervised image-text data (2020). arXiv preprint arXiv:2001.07966
  27. 27.Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)
  28. 28.Ren, Z., Jin, H., Lin, Z., Fang, C., Yuille, A.: Joint image-text representation by gaussian visual-semantic embedding. In: Multimedia (2016)
  29. 29.Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: CVPR (2017)
  30. 30.Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Annual Meeting of the Association for Computational Linguistics (2018)
  31. 31.Socher, R., Fei-Fei, L.: Connecting modalities: semi-supervised segmentation and annotation of images using unaligned text corpora. In: CVPR (2010)
  32. 32.Socher, R., Ganjoo, M., Manning, C.D., Ng, A.: Zero-shot learning through cross-modal transfer. In: NeurIPS (2013)
  33. 33.Su, W., et al.: VL-BERT: pre-training of generic visual-linguistic representations (2019). arXiv preprint arXiv:1908.08530
  34. 34.Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., Artzi, Y.: A corpus for reasoning about natural language grounded in photographs (2018). arXiv preprint arXiv:1811.00491
  35. 35.Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: VideoBERT: a joint model for video and language representation learning. In: ICCV (2019)
  36. 36.Tan, H., Bansal, M.: LXMERT: learning cross-modality encoder representations from transformers. In: EMNLP (2019)
  37. 37.Vaswani, A., et al.: Attention is all you need. In: NeurIPS (2017)
  38. 38.Wu, Q., Shen, C., Liu, L., Dick, A., Van Den Hengel, A.: What value do explicit high level concepts have in vision to language problems? In: CVPR (2016)
  39. 39.You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: CVPR (2016)
  40. 40.Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguist. 2, 67–78 (2014)
  41. 41.Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J.: Unified vision-language pre-training for image captioning and VQA. In: AAAI (2020)

Citation

MLA
Li, X., et al. “Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks”. arXiv, 2020, http://arxiv.org/abs/2004.06165v5.
APA
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y., & Gao, J. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. arXiv. http://arxiv.org/abs/2004.06165v5
Chicago
Li, X., X. Yin, C. Li, et al. 2020. “Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks”. arXiv. http://arxiv.org/abs/2004.06165v5.
Harvard
Li, X. et al. (2020) “Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2004.06165v5.
Vancouver
1. Li X, Yin X, Li C, et al (2020) Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. arXiv

BibTeX

@article{li2020oscar,
  title = {Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks},
  author = {Li, Xiujun and Yin, Xi and Li, Chunyuan and Zhang, Pengchuan and Hu, Xiaowei and Zhang, Lei and Wang, Lijuan and Hu, Houdong and Dong, Li and Wei, Furu and Choi, Yejin and Gao, Jianfeng},
  year = {2020},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2004.06165v5},
  eprint = {2004.06165}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF