UNITER: UNiversal Image-TExt Representation Learning

Yen-Chun ChenLinjie LiLicheng YuAhmed El KholyFaisal AhmedZhe GanYu ChengJingjing Liu

article2020ECCV2,690 citations

Presents UNITER, a universal image-text representation learning framework that incorporates conditional masking and optimal transport-based word-region alignment during pre-training to establish state-of-the-art performance across six major vision-and-language tasks.

Listen

The article addresses the challenge of developing generalizable joint embeddings for vision-and-language tasks, where earlier models produced representations tailored to narrow problems such as visual question answering or image-text retrieval and therefore transferred poorly across benchmarks.

The work set out to evaluate whether a single Transformer-based model could learn universal image-text representations through large-scale pre-training and then support multiple downstream tasks after fine-tuning.

The authors pre-trained the model on image-text pairs drawn from four datasets totaling roughly 9.5 million examples, using four objectives: masked language modeling conditioned on full image input, masked region modeling conditioned on full text input with three variants, image-text matching, and word-region alignment implemented via optimal transport. They conducted systematic ablations on task combinations and data sources, then transferred the resulting checkpoints to six tasks spanning nine datasets.

The optimal pre-training combination of masked language modeling, image-text matching, two masked-region objectives, and word-region alignment produced the strongest results; conditional masking outperformed joint random masking of both modalities; and adding out-of-domain data further improved performance. The resulting UNITER-base model set new state-of-the-art scores on visual question answering, visual commonsense reasoning, NLVR2, visual entailment, image-text retrieval (including zero-shot), and referring expression comprehension, often by margins of 2–7 percentage points, while using fewer parameters than competing two-stream models.

These outcomes indicate that a single pre-trained encoder can replace task-specific architectures for a wide range of multimodal problems, lowering development cost and improving consistency across applications. The gains from explicit fine-grained alignment and conditional masking highlight the value of reducing modality misalignment during pre-training.

Practitioners should adopt the released checkpoints for new vision-and-language work and consider two-stage pre-training when target data differ substantially from the pre-training distribution. Additional experiments are needed to test direct pixel-level interaction and to identify still more effective pre-training objectives.

The main limitations are reliance on pre-extracted region features rather than raw pixels, modest domain shift when moving from in-domain to out-of-domain pre-training data, and the need for task-specific adaptation layers on inputs that differ from image-sentence pairs. Results are therefore most reliable on tasks that closely resemble the pre-training distribution.

Microsoft common-sense genome language model UNITER Conditional Masking Optimal Transport

arXiv: 1909.11740 ChenRocks/UNITER

Paper: VisualBERT: A Simple and Performant Baseline for Vision and Language, Liunian Harold Li et al. (2019). VisualBERT established the foundational single-stream Transformer architecture for joint vision-and-language pre-training that UNITER extends with conditional masking and optimal transport alignment.
Paper: ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks, Jiasen Lu et al. (2019). ViLBERT introduced task-agnostic pre-training proxy tasks on paired image-text data, providing the baseline multimodal Transformer paradigm that UNITER improves upon.
Paper: LXMERT: Learning Cross-Modality Encoder Representations from Transformers, Hao Tan et al. (2019). LXMERT designed multi-task pre-training objectives for cross-modality reasoning across image regions and text tokens, which directly informs UNITER's pre-training design.
Paper: Deep visual-semantic alignments for generating image descriptions, Andrej Karpathy et al. (2015). This seminal work pioneered the formulation of fine-grained alignment between image regions and text fragments that UNITER optimizes using self-attention and optimal transport.
Paper: DeViSE: A Deep Visual-Semantic Embedding Model, Andrea Frome et al. (2013). DeViSE established the foundational paradigm of mapping visual features directly into a semantic linguistic embedding space to learn shared multimodal representations.

Paper: Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks, Xiujun Li et al. (2020). Oscar builds on region-based Transformer models like UNITER by introducing detected object tags as linguistic anchor points to simplify fine-grained cross-modal alignment.
Paper: ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision, Wonjae Kim et al. (2021). ViLT overcomes UNITER's key computational bottleneck—dependence on pre-extracted region bounding boxes—by operating directly on raw image patches within a single Transformer.
Paper: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation, Junnan Li et al. (2021). ALBEF advances joint vision-language representation learning by explicitly introducing contrastive alignment before cross-modal Transformer fusion and replacing heavy object detectors.
Paper: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation, Junnan Li et al. (2022). BLIP extends universal vision-language representation learning from understanding benchmarks to generative tasks using a unified mixture of encoder-decoder architecture.
Paper: CoCa: Contrastive Captioners are Image-Text Foundation Models, Jiahui Yu et al. (2022). CoCa unifies cross-modal encoder representations with autoregressive decoding and contrastive learning into a single end-to-end foundation model.
Paper: Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond, Jinze Bai et al. (2023). Qwen-VL advances universal multimodal modeling by scaling vision-language representation learning into large language model architectures with fine-grained visual grounding.
Paper: LLaVA-OneVision: Easy Visual Task Transfer, Bo Li et al. (2024). LLaVA-OneVision generalizes universal vision-language pre-training paradigms to unified transfer across single-image, multi-image, and video modalities.

Cover for UNITER: UNiversal Image-TExt Representation Learning

Abstract

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design four pre-training tasks: Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). In addition to ITM for global image-text alignment, we also propose WRA via the use of Optimal Transport (OT) to explicitly encourage fine-grained alignment between words and image regions during pre-training. Comprehensive analysis shows that both conditional masking and OT-based WRA contribute to better pre-training. We also conduct a thorough ablation study to find an optimal combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR² (Code is available at https://github.com/ChenRocks/UNITER.).

1 Introduction
2 Related Work
3 UNiversal Image-TExt Representation
3.1 Model Overview
3.2 Pre-training Tasks
3.3 Pre-training Datasets
4 Experiments
4.1 Downstream Tasks
4.2 Evaluation on Pre-training Tasks
4.3 Results on Downstream Tasks
4.4 Visualization
5 Conclusion
References

Knowls

Knowl 1 — UNITER Architecture and Input Embedding Formulation
model/method
UNITER (UNiversal Image-TExt Representation) is a single-stream multimodal Transformer model designed to learn joint contextualized embeddings over visual regions and textual tokens.

Given an image-text pair, the input representation consists of:
1. Image Embedder: An image is represented as a set of region bounding boxes extracted via a Faster R-CNN pre-trained on Visual Genome object and attribute annotations. For each detected region $j \in \{1, \dots, K\}$ , the model extracts a visual ROI feature vector $\mathbf{r}(\mathbf{v}_j)$ and a 7-dimensional location feature vector $\mathbf{l}_j = [x_1, y_1, x_2, y_2, w, h, w \cdot h]$ (normalized bounding box top-left and bottom-right coordinates, width, height, and box area). Both vectors are projected into a shared hidden dimension via fully-connected (FC) layers, summed together, added to a visual modality segment embedding, and normalized with Layer Normalization (LN).
2. Text Embedder: An input sentence is tokenized into $T$ sub-word tokens $\mathbf{w} = \{w_1, \dots, w_T\}$ using WordPiece. Each token representation is computed by summing its token embedding, positional embedding, and textual modality segment embedding, followed by Layer Normalization.
The combined sequence of token and region embeddings is processed through a multi-layer bidirectional Transformer encoder. Two model configurations are specified:
- UNITER-base: $L = 12$ Transformer blocks, hidden size $H = 768$ , $A = 12$ attention heads, and 86M trainable parameters (excluding the rare-word embedding table).
- UNITER-large: $L = 24$ Transformer blocks, hidden size $H = 1024$ , $A = 16$ attention heads, and 303M trainable parameters.
Knowl 2 — Conditional Masking Strategy for Vision-Language Pre-training
model/method
In multimodal masked pre-training, masking tokens and image regions simultaneously at random can cause severe supervision ambiguity when a masked region directly corresponds to a co-masked text token, destroying the cross-modal context needed for recovery.

UNITER introduces conditional masking, wherein only one modality is masked at a time during pre-training while the other modality is kept completely unmasked:
- Masked Language Modeling (MLM) is conditioned on full, unmasked visual region features $\mathbf{v}$ .
- Masked Region Modeling (MRM) is conditioned on full, unmasked textual tokens $\mathbf{w}$ .
When pre-training on a mini-batch with conditional masking, masking is applied either to text or to image regions, ensuring that cross-modal alignment signals remain unambiguous during masked reconstruction.
Knowl 3 — Word-Region Alignment via Optimal Transport
equation
Word-Region Alignment (WRA) pre-trains UNITER to find fine-grained alignment between textual tokens $\mathbf{w} = \{\mathbf{w}_1, \dots, \mathbf{w}_T\}$ and visual region features $\mathbf{v} = \{\mathbf{v}_1, \dots, \mathbf{v}_K\}$ by minimizing the Optimal Transport (OT) distance between their contextualized embeddings.

Let the textual sequence and visual sequence be discrete empirical distributions $\mu = \sum_{i=1}^T a_i \delta_{\mathbf{w}_i}$ and $\nu = \sum_{j=1}^K b_j \delta_{\mathbf{v}_j}$ , where probability simplex vectors $\mathbf{a} = \{a_i\}_{i=1}^T \in \Delta^T$ and $\mathbf{b} = \{b_j\}_{j=1}^K \in \Delta^K$ are uniform ( $a_i = 1/T$ , $b_j = 1/K$ ). The WRA loss is defined as: $\mathcal{L}_{\text{WRA}}(\theta) = \mathcal{D}_{\text{ot}}(\mu, \nu) = \min_{\mathbf{T} \in \Pi(\mathbf{a}, \mathbf{b})} \sum_{i=1}^T \sum_{j=1}^K T_{ij} \cdot c(\mathbf{w}_i, \mathbf{v}_j)$ subject to: $\Pi(\mathbf{a}, \mathbf{b}) = \left\{ \mathbf{T} \in \mathbb{R}_+^{T \times K} \mid \mathbf{T}\mathbf{1}_K = \mathbf{a}, \; \mathbf{T}^\top \mathbf{1}_T = \mathbf{b} \right\}$ where $\mathbf{1}_n$ denotes an $n$ -dimensional vector of ones, $\mathbf{T}$ is the optimal transport plan, and the transport cost is given by the cosine distance: $c(\mathbf{w}_i, \mathbf{v}_j) = 1 - \frac{\mathbf{w}_i^\top \mathbf{v}_j}{\|\mathbf{w}_i\|_2 \|\mathbf{v}_j\|_2}$ The transport matrix $\mathbf{T}$ is approximated iteratively via the Inexact Proximal Point Optimal Transport (IPOT) algorithm, producing an exact sparse solution with at most $2\max(K, T) - 1$ non-zero entries.
Knowl 4 — Masked Region Modeling Objectives
model/method
Masked Region Modeling (MRM) samples image region features with 15% probability, replaces the masked region visual features with zero vectors, and trains UNITER to reconstruct the masked regions $\mathbf{v}_m$ given the remaining visual regions $\mathbf{v}_{\setminus m}$ and full textual input $\mathbf{w}$ : $\mathcal{L}_{\text{MRM}}(\theta) = \mathbb{E}_{(\mathbf{w}, \mathbf{v}) \sim \mathcal{D}} f_\theta(\mathbf{v}_m \mid \mathbf{v}_{\setminus m}, \mathbf{w})$

Three objective variants $f_\theta$ are defined for MRM:
1. Masked Region Feature Regression (MRFR): Regresses the Transformer output of each masked region $\mathbf{v}_m^{(i)}$ to its original Faster R-CNN pooled ROI feature $\mathbf{r}(\mathbf{v}_m^{(i)})$ using an FC projection layer $h_\theta$ and an $L_2$ loss: $f_\theta(\mathbf{v}_m \mid \mathbf{v}_{\setminus m}, \mathbf{w}) = \sum_{i=1}^M \left\| h_\theta(\mathbf{v}_m^{(i)}) - \mathbf{r}(\mathbf{v}_m^{(i)}) \right\|_2^2$
2. Masked Region Classification (MRC): Passes the Transformer output of $\mathbf{v}_m^{(i)}$ through an FC classifier to predict class logits $\mathbf{g}_\theta(\mathbf{v}_m^{(i)}) \in \mathbb{R}^K$ . It minimizes cross-entropy against a hard one-hot target $\mathbf{c}(\mathbf{v}_m^{(i)}) \in \mathbb{R}^K$ corresponding to the object class with the highest confidence predicted by Faster R-CNN: $f_\theta(\mathbf{v}_m \mid \mathbf{v}_{\setminus m}, \mathbf{w}) = \sum_{i=1}^M \text{CE}\left(\mathbf{c}(\mathbf{v}_m^{(i)}), \mathbf{g}_\theta(\mathbf{v}_m^{(i)})\right)$
3. Masked Region Classification with KL-Divergence (MRC-kl): Avoids the assumption that the hard detector label is ground truth by distilling the soft object probability distribution $\tilde{\mathbf{c}}(\mathbf{v}_m^{(i)})$ from the Faster R-CNN detector via Kullback-Leibler divergence: $f_\theta(\mathbf{v}_m \mid \mathbf{v}_{\setminus m}, \mathbf{w}) = \sum_{i=1}^M D_{\text{KL}}\left(\tilde{\mathbf{c}}(\mathbf{v}_m^{(i)}) \parallel \mathbf{g}_\theta(\mathbf{v}_m^{(i)})\right)$
Knowl 5 — Masked Language Modeling and Image-Text Matching Objectives
model/method
UNITER utilizes two core multimodal pre-training objectives:
1. Masked Language Modeling (MLM): With 15% probability, input word tokens are selected for masking (80% replaced by [MASK], 10% replaced by random tokens, 10% kept unchanged). UNITER predicts the masked tokens $\mathbf{w}_m$ conditioned on observed words $\mathbf{w}_{\setminus m}$ and all image regions $\mathbf{v}$ by minimizing negative log-likelihood: $\mathcal{L}_{\text{MLM}}(\theta) = -\mathbb{E}_{(\mathbf{w}, \mathbf{v}) \sim \mathcal{D}} \log P_\theta(\mathbf{w}_m \mid \mathbf{w}_{\setminus m}, \mathbf{v})$
2. Image-Text Matching (ITM): A special [CLS] token is prepended to the multimodal sequence to serve as the global cross-modal representation. An FC classification layer with a sigmoid activation predicts an alignment score $s_\theta(\mathbf{w}, \mathbf{v}) \in [0, 1]$ . Training samples positive pairs $(y=1)$ and negative pairs $(y=0)$ , generated by randomly swapping the text or image with another sample, using binary cross-entropy: $\mathcal{L}_{\text{ITM}}(\theta) = -\mathbb{E}_{(\mathbf{w}, \mathbf{v}) \sim \mathcal{D}} \left[ y \log s_\theta(\mathbf{w}, \mathbf{v}) + (1 - y) \log(1 - s_\theta(\mathbf{w}, \mathbf{v})) \right]$
During pre-training, UNITER alternates between sampled tasks per mini-batch, updating model parameters $\theta$ on one loss objective per SGD step.

Knowl 6 — Ablation Analysis of Pre-training Tasks and Masking Strategies

data/table

An ablation study evaluated on UNITER-base demonstrates the relative utility of pre-training objectives, conditional masking, and pre-training corpora. Meta-Sum denotes the unweighted sum of scores across VQA (test-dev), Flickr30K Image Retrieval (IR val average R@1/5/10), Flickr30K Text Retrieval (TR val average R@1/5/10), NLVR2 (dev accuracy), and RefCOCO+ (val accuracy with detected proposals).

Pre-training Data	Pre-training Tasks	Meta-Sum	VQA	IR (Flickr)	TR (Flickr)	NLVR $^2$	RefCOCO+
None	None	314.34	67.03	61.74	65.55	51.02	68.73
Wikipedia+BookCorpus	MLM (text only)	346.24	69.39	73.92	83.27	50.86	68.80
In-domain (COCO+VG)	MRFR	344.66	69.02	72.10	82.91	52.16	68.47
	ITM	385.29	70.04	78.93	89.91	74.08	72.33
	MLM	386.10	71.29	77.88	89.25	74.79	72.89
	MLM + ITM	393.04	71.55	81.64	91.12	75.98	72.75
	MLM + ITM + MRC	393.97	71.46	81.39	91.45	76.18	73.49
	MLM + ITM + MRFR	396.24	71.73	81.76	92.31	76.21	74.23
	MLM + ITM + MRC-kl	397.09	71.63	82.10	92.57	76.28	74.51
	MLM + ITM + MRC-kl + MRFR	399.97	71.92	83.73	92.87	76.93	74.52
	MLM + ITM + MRC-kl + MRFR + WRA	400.93	72.47	83.72	93.03	76.91	74.80
	MLM + ITM + MRC-kl + MRFR (w/o cond. mask)	396.51	71.68	82.31	92.08	76.15	74.29
Out-of-domain (SBU+CC)	MLM + ITM + MRC-kl + MRFR + WRA	396.91	71.56	84.34	92.57	75.66	72.78
In-domain + Out-of-domain	MLM + ITM + MRC-kl + MRFR + WRA	405.24	72.70	85.77	94.28	77.18	75.31

Key takeaways:

Among MRM variants, soft target distillation (MRC-kl, 397.09) outperforms hard pseudo-label classification (MRC, 393.97). Combining MRC-kl with feature regression (MRFR) provides complementary gains (399.97).
Adding Optimal Transport-based WRA yields the highest in-domain score (400.93), with prominent gains on VQA (+0.55) and RefCOCO+ (+0.28).
Removing conditional masking drops the Meta-Sum score from 399.97 to 396.51, confirming the benefit of masking one modality at a time.
Pre-training on combined in-domain and out-of-domain data achieves the best overall performance (405.24).

Knowl 7 — Two-Stage Pre-training Strategy for Visual Commonsense Reasoning

model/method

Visual Commonsense Reasoning (VCR) contains narrative-style complex text distributions that differ substantially from short image-caption pairs in standard pre-training corpora. To address this, UNITER adopts a two-stage pre-training protocol:

Stage I: Pre-training on standard general-domain image-caption corpora (COCO, Visual Genome, Conceptual Captions, and SBU Captions).
Stage II: Continued task-specific self-supervised pre-training directly on the downstream VCR training dataset using MLM, MRM, and WRA before final supervised finetuning.

Evaluation on UNITER-base on the VCR validation split demonstrates the necessity of both stages across question answering ( $Q \rightarrow A$ ), rationale reasoning ( $QA \rightarrow R$ ), and combined cognition ( $Q \rightarrow AR$ ):

Stage I Pre-training	Stage II Pre-training	$Q \rightarrow A$	$QA \rightarrow R$	$Q \rightarrow AR$
No	No	72.44	73.71	53.52
No	Yes	73.52	75.34	55.60
Yes	No	72.83	75.25	54.94
Yes	Yes	74.56	77.03	57.76

Stage II pre-training yields substantial improvements (+2.08 on $Q \rightarrow AR$ without Stage I, and +2.82 on $Q \rightarrow AR$ with Stage I), indicating that second-stage in-domain pre-training effectively adapts the universal representation to complex narrative text.

Knowl 8 — Pair-Biattention Adaptation for Dual-Image NLVR2 Reasoning
model/method
Natural Language for Visual Reasoning for Real ( $ext{NLVR}^2$ ) requires determining whether a textual description holds true for a pair of input images $(I_1, I_2)$ , presenting an architectural mismatch for models pre-trained exclusively on single-image-text pairs.

Three adaptation architectures for UNITER were evaluated on $ext{NLVR}^2$ using UNITER-base:
1. Triplet: Concurrently feeds the region features of both $I_1$ and $I_2$ together with textual tokens as a single unified sequence into the Transformer. This yields dev accuracy of 73.03% and test-P accuracy of 73.89%.
2. Pair: Independently feeds $(I_1, \text{text})$ and $(I_2, \text{text})$ through the Transformer as separate pairs to obtain individual [CLS] representations. This yields dev accuracy of 75.85% and test-P accuracy of 75.80%.
3. Pair-biattn: Passes each image-text pair independently through UNITER, and then applies a bidirectional attention layer across the top of both contextualized representations to model inter-image relationships prior to the final MLP classification layer.
The Pair-biattn setting achieves 77.18% dev accuracy and 77.85% test-P accuracy, outperforming the Triplet approach by +3.96% on test-P. This demonstrates that pre-trained pair encoders adapt best to paired-image inputs via separate stream encoding combined with a lightweight top-level cross-image attention mechanism.

Knowl 9 — Downstream Vision-and-Language Task Benchmark Results

data/table

UNITER models pre-trained on in-domain and out-of-domain datasets (MLM + ITM + MRC-kl + MRFR + WRA) achieve state-of-the-art results across six vision-and-language tasks encompassing nine benchmark datasets.

Task	Split / Metric	Prior SOTA	ViLBERT	VL-BERT (Large)	LXMERT	UNITER-base	UNITER-large
VQA	test-dev	70.63	70.55	71.79	72.42	72.70	73.82
	test-std	70.90	70.92	72.22	72.54	72.91	74.02
VCR	$Q \rightarrow A$	72.60	73.30	75.80	–	75.00	77.30
	$QA \rightarrow R$	75.70	74.60	78.40	–	77.20	80.80
	$Q \rightarrow AR$	55.00	54.80	59.70	–	58.20	62.80
NLVR $^2$	dev	54.80	–	–	74.90	77.18	79.12
	test-P	53.50	–	–	74.50	77.85	79.98
SNLI-VE	val	71.56	–	–	–	78.59	79.39
	test	71.16	–	–	–	78.28	79.38
Flickr30K IR	R@1	48.60	58.20	–	–	72.52	75.56
Flickr30K TR	R@1	67.90	–	–	–	85.90	87.30
COCO IR	R@1	38.60	–	–	–	50.33	52.93
COCO TR	R@1	50.40	–	–	–	64.40	65.68
RefCOCO	$\text{val}^d$	77.48	–	–	–	81.24	81.41
RefCOCO+	$\text{val}^d$	68.19	72.34	72.59	–	75.31	75.90
RefCOCOg	$\text{val}^d$	68.22	–	–	–	74.31	74.86

Key results:

UNITER-large establishes a new state of the art across all benchmarks, notably reaching 74.02% on VQA test-std, 62.80% on VCR $Q \rightarrow AR$ , 79.98% on $ext{NLVR}^2$ test-P, and 79.38% on SNLI-VE test.
Although single-stream architectures were previously thought to underperform two-stream models (e.g., LXMERT: 183M params, ViLBERT: 221M params), UNITER-base (86M params) surpasses them across tasks while maintaining a significantly smaller parameter footprint.

Knowl 10 — Composition and Contamination Filtering of Pre-training Datasets
experimental setup
UNITER is pre-trained using image-sentence pairs drawn from four public datasets, partitioned into in-domain and out-of-domain categories:
1. In-domain Corpora:
  - COCO Captions: 533K image-text pairs across 106K images (training); 25K pairs across 5K images (internal validation).
  - Visual Genome (VG) Dense Captions: 5.06M region description pairs across 101K images (training); 106K pairs across 2.1K images (internal validation).
2. Out-of-domain Corpora:
  - Conceptual Captions (CC): 3.0M image-text pairs across 3.0M images (training); 14K pairs across 14K images (internal validation).
  - SBU Captions: 990K image-text pairs across 990K images (training); 10K pairs across 10K images (internal validation).
3. Contamination Filtering: To prevent data leakage into downstream evaluation benchmarks, all validation and test images present in downstream tasks were removed from the COCO and VG pre-training sets. Furthermore, URL matching was performed across Flickr-sourced images to eliminate overlapping images with Flickr30K (222 images removed from in-domain data, 109 images removed from out-of-domain data).

Coverage note — Qualitative attention visualization categories (e.g., vertical, diagonal, block, and reversed block attention map patterns in Section 4.4) were omitted as they constitute qualitative interpretability analysis rather than core algorithmic, structural, or experimental contributions.

References

1.Alberti, C., Ling, J., Collins, M., Reitter, D.: Fusion of detected objects in text for visual question answering. In: EMNLP (2019)
2.Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018)
3.Antol, S., et al.: VQA: visual question answering. In: ICCV (2015)
4.Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.C., Liu, J.: Behind the scene: revealing the secrets of pre-trained vision-and-language models. arXiv preprint arXiv:2005.07310 (2020)
5.Chen, L., Gan, Z., Cheng, Y., Li, L., Carin, L., Liu, J.: Graph optimal transport for cross-domain alignment. In: ICML (2020)
6.Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019)
7.Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: ICCV (2015)
8.Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. In: EMNLP (2017)
9.Gan, Z., Chen, Y.C., Li, L., Zhu, C., Cheng, Y., Liu, J.: Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195 (2020)
10.Gao, P., et al.: Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In: CVPR (2019)
11.Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018)
12.Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
13.Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: ReferItGame: referring to objects in photographs of natural scenes. In: EMNLP (2014)
14.Kim, J.H., Jun, J., Zhang, B.T.: Bilinear attention networks. In: NeurIPS (2018)
15.Kovaleva, O., Romanov, A., Rogers, A., Rumshisky, A.: Revealing the dark secrets of BERT. In: EMNLP (2019)
16.Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis. 123, 32–73 (2017). https://doi.org/10.1007/s11263-016-0981-7
17.Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: a lite BERT for self-supervised learning of language representations. In: ICLR (2020)
18.Lee, K.-H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 212–228. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_13
19.Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-VL: a universal encoder for vision and language by cross-modal pre-training. In: AAAI (2020)
20.Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)
21.Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
22.Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
23.Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019)
24.Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: multi-task vision and language representation learning. In: CVPR (2020)
25.Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving Jigsaw puzzles. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 69–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_5
26.Ordonez, V., Kulkarni, G., Berg, T.L.: Im2Text: describing images using 1 million captioned photographs. In: NeurIPS (2011)
27.Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: CVPR (2016)
28.Peters, M.E., et al.: Deep contextualized word representations. In: NAACL (2018)
29.Peyre, G., Cuturi, M., et al.: Computational optimal transport. Found. TrendsR Mach. Learn. 11(5–6), 355–607 (2019)
30.Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models. In: ICCV (2015)
31.Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
32.Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: ACL (2018)
33.Su, W., et al.: VL-BERT: pre-training of generic visual-linguistic representations. In: ICLR (2020)
34.Suhr, A., Zhou, S., Zhang, I., Bai, H., Artzi, Y.: A corpus for reasoning about natural language grounded in photographs. In: ACL (2019)
35.Sun, C., Baradel, F., Murphy, K., Schmid, C.: Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743 (2019)
36.Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: VideoBERT: a joint model for video and language representation learning. In: ICCV (2019)
37.Tan, H., Bansal, M.: LXMERT: learning cross-modality encoder representations from transformers. In: EMNLP (2019)
38.Trinh, T.H., Luong, M.T., Le, Q.V.: Selfie: self-supervised pretraining for image embedding. arXiv preprint arXiv:1906.02940 (2019)
39.Vaswani, A., et al.: Attention is all you need. In: NeurIPS (2017)
40.Wang, L., Li, Y., Lazebnik, S.: Learning deep structure-preserving image-text embeddings. In: CVPR (2016)
41.Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
42.Xie, N., Lai, F., Doran, D., Kadav, A.: Visual entailment: a novel task for finegrained image understanding. arXiv preprint arXiv:1901.06706 (2019)
43.Xie, Y., Wang, X., Wang, R., Zha, H.: A fast proximal point method for Wasserstein distance. arXiv:1802.04307 (2018)
44.Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: NeurIPS (2019)
45.Yu, L., et al.: MAttNet: modular attention network for referring expression comprehension. In: CVPR (2018)
46.Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 69–85. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_5
47.Yu, Z., Yu, J., Cui, Y., Tao, D., Tian, Q.: Deep modular co-attention networks for visual question answering. In: CVPR (2019)
48.Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: visual commonsense reasoning. In: CVPR (2019)
49.Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
50.Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J.: Unified visionlanguage pre-training for image captioning and VQA. In: AAAI (2020)

Citation

MLA

Chen, Y.-C., et al. “UNITER: UNiversal Image-TExt Representation Learning”. Lecture Notes in Computer Science, Springer International Publishing, 2020, pp. 104–20, https://doi.org/10.1007/978-3-030-58577-8_7.

APA

Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020). UNITER: UNiversal Image-TExt Representation Learning. In Lecture Notes in Computer Science (pp. 104–120). Springer International Publishing. https://doi.org/10.1007/978-3-030-58577-8_7

Chicago

Chen, Y.-C., L. Li, L. Yu, et al. 2020. “UNITER: UNiversal Image-TExt Representation Learning”. In Lecture Notes in Computer Science. Springer International Publishing. https://doi.org/10.1007/978-3-030-58577-8_7.

Harvard

Chen, Y.-C. et al. (2020) “UNITER: UNiversal Image-TExt Representation Learning”, Lecture Notes in Computer Science. Springer International Publishing, pp. 104–120. Available at: https://doi.org/10.1007/978-3-030-58577-8_7.

Vancouver

1. Chen Y-C, Li L, Yu L, El Kholy A, Ahmed F, Gan Z, Cheng Y, Liu J (2020) UNITER: UNiversal Image-TExt Representation Learning. In: Lecture Notes in Computer Science. Springer International Publishing, pp 104–120

BibTeX

@inbook{Chen_2020, title={UNITER: UNiversal Image-TExt Representation Learning}, ISBN={9783030585778}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/978-3-030-58577-8_7}, DOI={10.1007/978-3-030-58577-8_7}, booktitle={Computer Vision – ECCV 2020}, publisher={Springer International Publishing}, author={Chen, Yen-Chun and Li, Linjie and Yu, Licheng and El Kholy, Ahmed and Ahmed, Faisal and Gan, Zhe and Cheng, Yu and Liu, Jingjing}, year={2020}, pages={104–120} }

Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

UNITER: UNiversal Image-TExt Representation Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — UNITER Architecture and Input Embedding Formulation

Knowl 2 — Conditional Masking Strategy for Vision-Language Pre-training

Knowl 3 — Word-Region Alignment via Optimal Transport

Knowl 4 — Masked Region Modeling Objectives

Knowl 5 — Masked Language Modeling and Image-Text Matching Objectives