Sigmoid Loss for Language Image Pre-Training

Xiaohua Zhai$^*$
Basil Mustafa
Alexander Kolesnikov
Lucas Beyer$^*$
Google DeepMind, Zürich, Switzerland
{xzhai, basilm, akolesnikov, lbeyer}@google.com

Abstract

We propose a simple pairwise sigmoid loss for image-text pre-training. Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. With only four TPUv4 chips, we can train a Base CLIP model at $4,\text{k}$ batch size and a Large LiT model at $20,\text{k}$ batch size, the latter achieves 84.5% ImageNet zero-shot accuracy in two days. This disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of $32,\text{k}$ being sufficient. We hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

Executive Summary: Researchers have developed large-scale image-text models such as CLIP through contrastive pre-training on web image-text pairs, but the standard softmax loss requires global normalization across the entire batch. This creates high memory demands, numerical instability, and dependence on very large batch sizes that in turn require hundreds of accelerator chips.

The work set out to test whether a simpler pairwise sigmoid loss could replace the softmax objective while maintaining or improving accuracy and greatly reducing hardware requirements. The authors trained and compared sigmoid-based models (SigLIP) and locked-image variants (SigLiT) against softmax baselines across batch sizes from 512 to one million, using public ViT image encoders and transformer text towers on the WebLI and LiT datasets. Experiments ran on modest numbers of TPUv4 chips and measured zero-shot ImageNet accuracy plus multilingual retrieval.

The sigmoid loss delivered three to five points higher zero-shot accuracy than softmax at batch sizes below 16 k and matched or slightly exceeded it at larger sizes. Training a Large SigLiT model at 20 k batch size on only four TPUv4 chips reached 84.5 percent ImageNet zero-shot accuracy in two days; a Base SigLIP model trained from scratch on 32 chips reached 73.4 percent in five days. Performance for both losses saturated near a 32 k batch size, with further increases yielding negligible gains or even degradation. The sigmoid formulation proved more robust to label noise and eliminated the need for cross-device all-gathers, cutting memory use from quadratic to per-chunk.

These results show that high-quality language-image models can be trained with far fewer accelerators and simpler code than previously assumed. Practitioners can therefore obtain competitive backbones on modest hardware budgets, lowering barriers to entry and enabling faster iteration.

Further work should explore efficient hard-negative mining within the sigmoid framework and test the loss at still larger model scales. The main uncertainties are the reliance on the noisy “all non-paired items are negatives” assumption and the limited evaluation on tasks beyond classification and retrieval; additional validation on detection, segmentation, and generation would strengthen confidence in broader applicability.

1. Introduction

Section Summary: Recent work has shifted toward training computer vision models by pairing web images with text captions instead of using large labeled datasets, with methods like CLIP demonstrating strong results through a batch-wide contrastive loss that aligns matching image-text pairs while pushing apart mismatches. This paper introduces a simpler sigmoid-based loss that avoids operations across the entire batch, making training more stable, memory-efficient, and scalable to very large batches without extra synchronization steps. Experiments adapting this loss to CLIP- and LiT-style setups show it matches or exceeds the original approach at practical batch sizes, enabling high-accuracy models to be trained in just days on a handful of chips and thereby lowering barriers to language-image pre-training.

Contrastive pre-training using weak supervision from image-text pairs found on the web is becoming the go-to method for obtaining generic computer vision backbones, slowly replacing pre-training on large labelled multi-class datasets. The high-level idea is to simultaneously learn an aligned representation space for images and texts using paired data. Seminal works CLIP [34] and ALIGN [23] established the viability of this approach at a large scale, and following their success, many large image-text datasets became available privately [55, 13, 21, 45] and publicly [37, 6, 15, 7, 38].

The standard recipe to pre-train such models leverages the image-text contrastive objective. It aligns the image and text embeddings for matching (positive) image-text pairs while making sure that unrelated (negative) image-text pairs are dissimilar in the embedding space. This is achieved via a batch-level softmax-based contrastive loss, applied twice to normalize the pairwise similarity scores across all images, then all texts. A naive implementation of the softmax is numerically unstable; it is usually stabilized by subtracting the maximum input value before applying the softmax [18], which requires another pass over the full batch.

In this paper, we propose a simpler alternative: the sigmoid loss. It does not require any operation across the full batch and hence greatly simplifies the distributed loss implementation and boosts efficiency. Additionally, it conceptually decouples the batch size from the definition of the task. We compare the proposed sigmoid loss with the standard softmax loss across multiple setups. In particular, we investigate sigmoid-based loss with two prominent approaches for image-text learning: CLIP [34] and LiT [55], which we call sigmoid language image pre-training (SigLIP) and sigmoid LiT (SigLiT), respectively. We find that the sigmoid loss performs significantly better than the softmax loss when the batch size is smaller than $16,\text{k}$. As the train batch size grows, the gap closes. Importantly, the sigmoid loss is symmetric, requires just a single pass, and a typical implementation requires less memory than the softmax loss. This enables successful training of a SigLiT model at a batch size of one million. However, we find that the performance saturates with growing batch size, both for softmax and sigmoid. The good news is that a reasonable batch size, i.e. $32,\text{k}$, is sufficient for image-text pre-training. This conclusion also holds for multilingual SigLIP training on over 100 languages.

In Table 1, we present setups for image-text pre-training that require a moderate amount of TPUv4 chips for training. SigLiT is surprisingly efficient, reaching 79.7% zero-shot accuracy on ImageNet in just a single day on four chips. SigLIP's more demanding from-scratch training reaches 73.4% zero-shot accuracy in 5 days with 32 TPUv4 chips. This compares favorably to prior works such as FLIP [29] and CLIP [34], which require approximately 5 and 10 days respectively on 256 TPUv3 cores. When fine-tuning a pre-trained vision backbone in SigLIP, denoted as 🔓 in Table 1, we found that disabling the weight decay on the pre-trained backbone leads to better results (see Figure 4 for details). We hope our work paves the way for making the nascent language-image pre-training field more accessible.

: Table 1: SigLiT and SigLIP results. Sigmoid loss is memory efficient, allows larger batch sizes (BS) that unlocks language image pre-training with a small number of chips. SigLiT model with a frozen public ❄ L/16 checkpoint [39], trained on the LiT image-text dataset [55] using four TPU-v4 chips for one day, achieves 79.7% 0-shot accuracy on ImageNet. The same setup with a g/14 checkpoint [54] leads to 84.5% accuracy, trained for two days. With a public unlocked 🔓 B/16 image checkpoint [39], trained on the WebLI dataset [13], SigLIP achieves 71.0% 0-shot accuracy using 16 TPU-v4 chips for three days. The last two rows show results with randomly initialized models.

\begin{tabular}{l c c c c c c}
\hline
& Image & Text & BS & \#TPUv4 & Days & INet-0 \\
\hline
SigLiT & \textcolor{cyan}{❄} B/8 & $\text{L}^*$ & $32\,\text{k}$ & 4 & 1 & 79.7 \\
SigLiT & \textcolor{cyan}{❄} g/14 & L & $20\,\text{k}$ & 4 & 2 & 84.5 \\
\hline
SigLIP & 🔓 B/16 & B & $16\,\text{k}$ & 16 & 3 & 71.0 \\
SigLIP & B/16 & B & $32\,\text{k}$ & 32 & 2 & 72.1 \\
SigLIP & B/16 & B & $32\,\text{k}$ & 32 & 5 & 73.4 \\
\hline
\multicolumn{7}{r}{\footnotesize $^*$ We use a variant of the L model with 12 layers.}
\end{tabular}

[^1]: $^*$equal contribution

2. Related Work

Section Summary: Prior work on contrastive language-image models has largely relied on a softmax-based loss function introduced in models like CLIP and ALIGN, which learn from large image-text datasets and transfer well to tasks such as classification and retrieval. Researchers have also explored generative approaches that train text decoders instead of or alongside contrastive objectives, as seen in models like GIT and CoCa, to improve captioning and filtering. A smaller set of efforts has focused on making this pre-training more efficient through techniques like locked backbones, token dropping, larger batch sizes, or new optimizers, though these often trade off quality or require substantial hardware.

Contrastive learning with the sigmoid loss. One prior work proposes a similar sigmoid loss for the task of unsupervised dimensionality reduction [19]; in the scope of contrastive image-text learning, the vast majority of works rely on the softmax-based InfoNCE loss as popularized by [43]. In supervised classification, the sigmoid loss has already been shown to be slightly more effective and robust than the softmax loss [3, 47].

Contrastive language-image pre-training has become popular since CLIP [34] and ALIGN [23] applied softmax contrastive learning [56, 43, 10, 24] to large-scale image-text datasets. Both models perform very well on zero-shot transfer tasks, including classification and retrieval. Follow-up works show that contrastively pre-trained models produce good representations for fine-tuning [49, 16], linear regression [23], object detection [30], semantic segmentation [31] and video tasks [53].

Generative language-image pre-training Besides softmax contrastive pre-training, various alternatives have been proposed. GIT [45], SimVLM [46], and LEMON [21] successfully pre-train models using a generative text decoder instead, while CoCa [52] adds such a decoder to the discriminative CLIP/ALIGN setup, thus combining the pros and cons of both approaches into a single very capable model. BLIP [28] further proposes CapFilt which uses the generative decoder to create better captions and the discriminative part of the model to filter pairs. Language-Image pre-training is a very active field and surveys [8] rapidly become outdated.

Efficient language-image pre-training On the other hand, few works have tried making language image pre-training more efficient. LiT [55] and FLIP [29] are notable attempts, the former requires a pre-trained and locked backbone, and the latter sacrifices quality by randomly dropping visual tokens. BASIC [33] and LAION [48] look at scaling batch-size but only go up to $16,\text{k}$ and $160,\text{k}$ respectively, by using many hundreds of chips, and for the former also mixing in a large private classification dataset [33, 51]. The recent Lion optimizer [12] claims to be able to reduce the training cost to reach similar quality.

3. Method

Section Summary: The section first outlines the standard softmax contrastive loss used in vision-language pretraining, which trains image and text encoders on batches of paired examples so that matching image-text embeddings are pulled together while all other pairs are pushed apart via normalized similarity scores computed in both directions. It then presents a simpler pairwise sigmoid loss that reframes the same goal as independent binary classification decisions for every possible image-text combination in the batch, using a learnable temperature and bias term to stabilize training from the start. Finally, the authors describe a memory-efficient “chunked” implementation of this sigmoid loss that processes small blocks of pairs across devices without materializing a full pairwise similarity matrix, enabling stable training at batch sizes exceeding one million.

In this section, we first review the widely-used softmax-based contrastive loss. We then introduce the pairwise sigmoid loss and discuss its efficient implementation.

Given a mini-batch $B = {(I_1, T_1), (I_2, T_2), \dots }$ of image-text pairs, the contrastive learning objective encourages embeddings of matching pairs $(I_i, T_i)$ to align with each other, while pushing embeddings of unmatched pairs $(I_i, T_{j \neq i})$ apart. For practical purposes, it is assumed that for all images $i$, the text associated with a different image $j$ is not related to $i$, and vice-versa. This assumption is usually noisy and imperfect.

3.1. Softmax loss for language image pre-training

When using the softmax loss to formalize this objective, an image model $f(\cdot)$ and a text model $g(\cdot)$ are trained to minimize the following objective:

$ -\frac{1}{2|B|} \sum_{i=1}^{|B|} \left( \overbrace{\log \frac{e^{t \mathbf{x}_i \cdot \mathbf{y}i}}{\sum{j=1}^{|B|} e^{t \mathbf{x}_i \cdot \mathbf{y}_j}}}^{\text{image}\to\text{text softmax}} + \overbrace{\log \frac{e^{t \mathbf{x}_i \cdot \mathbf{y}i}}{\sum{j=1}^{|B|} e^{t \mathbf{x}_j \cdot \mathbf{y}_i}}}^{\text{text}\to\text{image softmax}} \right) $

where $\mathbf{x}_i = \frac{f(I_i)}{|f(I_i)|_2}$ and $\mathbf{y}_i = \frac{g(T_i)}{|g(T_i)|_2}$. In this paper, we adopt the vision transformer architecture [17] for images and the transformer architecture [44] for texts. Note that due to the asymmetry of the softmax loss, the normalization is independently performed two times: across images and across texts [34]. The scalar $t$ is parametrized as $\exp(t')$, where $t'$ is a global freely learnable parameter.

3.2. Sigmoid loss for language image pre-training

Instead of the softmax-based contrastive loss, we propose a simpler alternative that does not require computing global normalization factors. The sigmoid-based loss processes every image-text pair independently, effectively turning the learning problem into the standard binary classification on the dataset of all pair combinations, with a positive labels for the matching pairs $(I_i, T_i)$ and negative labels for all other pairs $(I_i, T_{j \neq i})$. It is defined as follows:

$ -\frac{1}{|B|} \sum_{i=1}^{|B|} \sum_{j=1}^{|B|} \underbrace{\log \frac{1}{1 + e^{z_{ij}(-t \mathbf{x}i \cdot \mathbf{y}j + b)}}}{L{ij}} $

where $z_{ij}$ is the label for a given image and text input, which equals $1$ if they are paired and $-1$ otherwise. Note that at initialization, the heavy imbalance coming from the many negatives dominates the loss, leading to large initial optimization steps attempting to correct this bias. To alleviate this, we introduce an additional learnable bias term $b$ similar to the temperature $t$. We initialize $t'$ and $b$ to $10$ and $-10$ respectively. This makes sure the training starts roughly close to the prior and does not require massive over-correction. Algorithm 1 presents a pseudocode implementation of the proposed sigmoid loss for language image pre-training.

3.3. Efficient “chunked” implementation

Contrastive training typically utilizes data parallelism. Computing the loss when data is split across $D$ devices necessitates gathering all embeddings [55] with expensive all-gathers and, more importantly, the materialization of a memory-intensive $|B| \times |B|$ matrix of pairwise similarities.

The sigmoid loss, however, is particularly amenable to a memory efficient, fast, and numerically stable implementation that ameliorates both these issues. Denoting the per-device batch size as $b = \frac{|B|}{D}$, the loss is reformulated as:

$ -\frac{1}{|B|} \underbrace{\sum_{d_i=1}^D}{\text{A: } \forall\text{ device } d_i} \overbrace{\sum{d_j=1}^D}^{\substack{\text{B: swap negs} \ \text{across devices}}} \overbrace{\sum_{i=b d_i}^{b(d_i+1)}}^{\substack{\text{all local} \ \text{positives}}} \overbrace{\sum_{j=b d_j}^{b(d_j+1)}}^{\substack{\text{negs from} \ \text{next device}}} L_{ij} $

This is particularly simple for the sigmoid loss as each pair is an independent term in the loss. Figure 1 illustrates this method. In words, we first compute the component of the loss corresponding to the positive pairs, and $b - 1$ negative pairs. We then permute representations across devices, so each device takes negatives from its neighbouring device (next iteration of sum B). The loss is then calculated with respect to this chunk (sum C). This is done independently in each device, such that each device computes the loss with respect to its local batch $b$. Losses can then simply be summed across all devices (sum A). Individual collective permutes (for sum B) are fast (and indeed $D$ collective permutes is typically faster than two all-gathers between $D$ devices), and the memory cost at any given moment is reduced from $|B|^2$ to $b^2$ (for sum C). Usually $b$ is constant as scaling $|B|$ is achieved by increasing the number of accelerators. Due to being quadratic with respect to the batch size, the vanilla loss computation rapidly bottlenecks scaling up. This chunked approach enabled training with batch sizes over 1 million on relatively few devices.

4. Results

Section Summary: The results evaluate SigLiT and SigLIP models across batch sizes from 512 to over one million, measuring zero-shot ImageNet accuracy and multilingual image-text retrieval on the XM3600 benchmark. Sigmoid loss proves more effective than standard softmax loss at smaller batches and enables larger ones on limited hardware thanks to lower memory use, though gains plateau beyond roughly 32,000–256,000 examples per batch and very large batches can even hurt multilingual performance. Additional tests cover training length, stabilization tricks, and data noise robustness.

In this section, we evaluate the proposed SigLiT and SigLIP models across a wide range of batch sizes. We discuss what can be achieved with a small number of accelerator chips, using both SigLiT and SigLIP recipes. We also briefly discuss the impact of batch size on multilingual language image pre-training. We ablate the importance of our large-batch stabilization modification and the introduced learned bias term and present a study on the effect of positive and negative pairs ratio in the sigmoid loss. Lastly, we explore SigLIP’s data noise robustness.

To validate our models, we report zero-shot transfer results on the ImageNet dataset [14] and zero-shot retrieval results across 36 languages on the XM3600 dataset [41]. We use the ScalingViT-Adafactor optimizer [54] by default for all our experiments.

4.1. SigLiT: Scaling batch size to the limit

Following [55], we use the same precomputed embeddings for the images using a ViT-g vision model, and train a base size text tower from scratch with the same hyperparameters using the LiT image-text dataset [55].

We perform a study over a wide range of batch sizes, from 512 to 1 M, demonstrating the impact of batch size for contrastive learning. Results are presented in Figure 2 (left). When the batch size is smaller than $16,\text{k}$, sigmoid loss outperforms softmax loss by a large margin. With growing batch sizes, we observe that softmax loss quickly catches up and potentially slightly underperforms sigmoid loss with a large enough batch size. Overall, we recommend using the SigLIP recipe for large batch sizes as well, due to the simplicity, compute savings, and straightforward memory efficient implementation.

There is a consensus that contrastive learning benefits from large batch sizes, while most of the existing studies stop at $64,\text{k}$ batch size [55, 33, 10]. We successfully trained an SigLiT model at one million batch size, to explore the limit of contrastive learning. To our surprise, the performance saturates at $32,\text{k}$ batch size, further scaling up the batch size only gives a minor boost, and the model peaks at $256,\text{k}$ batch size. Our best SigLiT with a B-sized text mode achieves 84.7% zero-shot transfer accuracy on ImageNet, while the original LiT paper reports a slightly better 85.2% score with a 10 times larger g-sized text model. Figure 3 presents the impact of training duration for different batch sizes. It demonstrates that large, $262,\text{k}$ batch size significantly outperforms smaller $8,\text{k}$ batch size when trained for a sufficiently long time. Note, that for short training durations, large batch size leads to the fewer absolute number of update steps and thus needs more time to ramp up.

4.2. SigLIP: Sigmoid loss is beneficial for language-image pre-training

We pre-train SigLIP models on the WebLI dataset [13], using only English image and text pairs. We use moderately-sized models: B/16 ViT for image embeddings and B-sized transformer for text embeddings. The input images are resized to $224\times224$ resolution. The text is tokenized by a $32,\text{k}$ vocabulary sentencepiece tokenizer [27] trained on the English C4 dataset [35], and a maximum of 16 text tokens are kept. Figure 2 middle plot shows SigLIP results, With less than $32,\text{k}$ batch size, SigLIP outperforms CLIP baselines with the standard softmax loss. On the other end of the scale, the memory efficiency of the sigmoid loss enabled much larger batch sizes. For example, with four TPU-v4 chips, we could fit a batch size of 4096 with a Base SigLIP but only 2048 with a corresponding CLIP model. The two advantages together demonstrate significant benefits of the sigmoid loss for language image pre-training with fixed resources, which will be discussed in Section 4.5.

As batch size increases, the gap between the sigmoid and the softmax losses diminish. SigLIP performs best at batch size $32,\text{k}$, whereas the softmax loss required $98,\text{k}$ for optimal performance and still didn’t outperform the sigmoid based variant. Scaling further, a larger batch size like $307,\text{k}$ hurts both losses.

: Table 2: Multilingual SigLIP results with various batch sizes, pre-trained for 30 billion seen examples. We report zero-shot transfer results on ImageNet (INet-0) and averaged text to image retrieval results across 36 languages on the crossmodal 3600 dataset (XM). The full table on 36 languages can be found in Appendix.

\begin{tabular}{l c c c c c}
\hline
& $16\,\text{k}$ & $32\,\text{k}$ & $64\,\text{k}$ & $128\,\text{k}$ & $240\,\text{k}$ \\
\hline
INet-0 & 71.6 & 73.2 & 73.2 & 73.2 & 73.1 \\
XM avg & 34.8 & 34.9 & 34.4 & 33.6 & 32.7 \\
XM de & 54.7 & 54.8 & 55.4 & 54.3 & 54.7 \\
XM en & 46.5 & 46.2 & 46.5 & 46.6 & 46.6 \\
XM hi & 9.1 & 8.5 & 7.9 & 8.1 & 7.3 \\
XM ru & 50.1 & 49.9 & 49.7 & 48.6 & 49.3 \\
XM zh & 30.7 & 32.5 & 32.0 & 30.6 & 23.7 \\
\hline
\end{tabular}

4.3. mSigLIP: Multi-lingual pre-training

We further scale up the training data by keeping all the 100 languages from the WebLI dataset [13]. With multilingual data, one usually needs to use a larger international vocabulary. We first verify the impact of two tokenizers: a small multilingual vocabulary with $32,\text{k}$ tokens [35], and a large multilingual vocabulary with $250,\text{k}$ tokens [50]. We train B-sized ViT and text models for $900,\text{M}$ total examples seen, and observe slightly more than 1% improvement when using a larger vocabulary.

However, the token embeddings become huge for very large vocabulary sizes. Following the standard setup, we would need to store a $N \times W$ token embedding lookup table to train the multilingual model, where $N$ is the vocabulary size mentioned above and $W$ is the embedding dimension of the text model. To save memory, we propose to use a “bottlenecked” token embedding. We use $N \times K$ embedding matrix and additional $K \times W$ projection, where the bottleneck $K$ is much smaller than $W$.

In our experiments, we observed that using a large multilingual vocabulary with a bottleneck can be scaled up as efficiently as using a small multilingual vocabulary. Specifically, by enabling the bottleneck of size $K = 96$ for Base architecture with $W = 768$, we only see about a half percent quality drop on ImageNet zero-shot transfer, compared to using the full 250k vocabulary.

With the memory improvements, we train mSigLIP models for various batch sizes, for a total of 30 billion examples seen. Table 2 and Figure 2 (right plot) show the results. We were expecting a large batch size to improve multilingual pre-training, where the model sees more examples from the same language as hard negatives in a single mini-batch. However, we didn’t observe clear improvements with a batch size larger than $32,\text{k}$. A batch size of $32,\text{k}$ is sufficient for a multilingual setup as well. On the XM3600 cross-modal retrieval tasks, we found that going beyond $32,\text{k}$ batch size leads to worse results on average while on ImageNet zero-shot transfer it stays flat. mSigLIP sets the new state-of-the-art on XM3600 text to image retrieval task, with only a Base size model. Our best result is 34.9%, which is more than 6% higher than the previously reported result 28.5% [13] with a standard LiT model [55] using a much larger four billion ViT-e model.

4.4. SigLiT with four TPU-v4 chips

For many practitioners, the important question usually is “what can be trained with a limited amount of resources?” We explore the usage of SigLiT models in this section with only four TPU-v4 chips, as the memory efficient sigmoid loss is suitable for this application scenario.

We follow the same setup as in section 4.1. We use the publicly available ViT-Augreg-B/8 [39] model as the frozen (\textcolor{cyan}{❄}) vision tower, and precompute embeddings to accelerate the training [55]. The text model is a Large Transformer, but with a depth of only 12 layers (instead of 24). It is trained using the LION [12] optimizer with decoupled weight decay $1 \times 10^{-7}$, linearly warm-up of learning rate over 6.5k steps up to a peak of $1 \times 10^{-4}$, followed by a cosine decay to 0. We train for a total of 65 000 steps with a batch size of 32k – this leads to just under one day of training. Table 1 shows the results when training a model on four chips for one day, achieving 79.7% 0-shot ImageNet classification accuracy; very competitive in this limited resource regime. With a ViT-g/14 [54] model as the vision tower and a Large text tower, we can train at $20,\text{k}$ batch size on four chips for $107,\text{k}$ steps in under two days. This further pushes the 0-shot ImageNet classification accuracy up to 84.5%.

4.5. SigLIP with a small amount of TPU-v4 chips

It’s resource demanding to train a CLIP model from-scratch in general, with SigLIP it’s possible to fit a larger train batch size with fewer amount of chips. In this section, we explore ways to train SigLIP models efficiently with pre-trained weights. We use pre-trained weights to initialize the image model to accelerate the pre-training, which was originally discussed in [55]. We use the public and unlocked ViT-Augreg-B/16 [39] model to initialize our vision tower and fine-tune on the same WebLI English data as used for SigLIP. In all the experiments, we apply a 0.1 learning rate multiplier to the pre-trained image tower to make it suitable for fine-tuning.

Figure 4 presents unlocked 🔓 fine-tuning results alongside from-scratch randomly initialized baselines. We used 16 TPU-v4 chips and train at $16,\text{k}$ batch size for 2.4 B examples seen. We found that the fine-tuning setup doesn’t perform well out-of-the-box; this is consistent with prior works [55] where finetuning image models degraded visual representation quality. This is evidenced by ImageNet 10-shot linear classification, where in Figure 4 the fine-tuned setup is barely better than the from-scratch baseline.

We hypothesize that the default weight decay applied to the pre-trained weights reduces their effectiveness. Motivated by the fine-tuning recipe from [17, 54, 25], that uses no weight decay, we also propose disabling weight decay on the pre-trained weights for SigLIP training. Weight decay is therefore only applied to the randomly initialized weights in the text model. This simple modification significantly improved SigLIP results. Figure 4 shows that with our improved recipe, SigLIP reaches 71% 0-shot accuracy on ImageNet, using 16k batch size, trained on 16 chips for three days. We also present from-scratch results in the bottom rows of Table 1: with 32 TPUv4 chips for only two days, SigLIP achieves 72.1% 0-shot accuracy. This presents a significant training cost reduction e.g. compared to CLIP (approx. 2500 TPUv3-days for 72.6%) reported in [29].

4.6. Scaling up SigLIP

In this section, we scale up SigLIP by “overtraining” the model [42, 1] for a long schedule. We present results in Table 3 using ViT-B or ViT-L as the vision encoder, while the text encoder is kept the same size (B or L respectively). Following the recipe as described in Section 4.2, we train both models for 40 billion examples seen, but increase the number of image patches from $(224/16)^2 = 196$ to $(256/16)^2 = 256$ and the number of text tokens from 16 to 64. To get SigLIP results for different resolutions, we continue training the pre-trained model for 5 billion examples on the target resolutions (i.e. 224 or 384), using a 100x smaller learning rate without weight decay. In Table 3, we report zero-shot classification results on ImageNet [14], ObjectNet [2], ImageNetv2 [36], ImageNet ReaL [3], and zero-shot image-to-text ($\text{I}\to\text{T}$) retrieval, text-to-image ($\text{I}\to\text{T}$) retrieval results on MSCOCO [11]. Our SigLIP models outperform the previous models consistently with ViT-B and ViT-L architectures.

4.7. Stabilizing large-batch training

As we move to large batch sizes, the language image pre-training using transformers becomes increasingly more unstable, even when using a modestly-sized model (e.g. Base size). The reason for these instabilities is large spikes in the gradient norms, which translate to large-magnitude changes in the weights that may destabilize the training process, see Figure 5. We observe that reducing $\beta_2$ in Adam and AdaFactor from its default 0.999 to 0.95 (which was suggested in [20, 9]) is enough to stabilize the training. Intuitively, this allows recovering from gradient spikes quicker. We opt for setting $\beta_2 = 0.95$ for all our experiments.

4.8. Negative ratio in sigmoid loss

One question which arises when shifting the perspective from the softmax’s “pick the right class” view to the sigmoid’s “rate this pair” view, is the imbalance in positive versus negative pairs. For a batch size $|B|$, the batch contains $|B|$ positive pairs, but $|B|^2 - |B|$ negative examples. In the modest batch-size of $16,\text{k}$, there are actually $268,\text{M}$ negative examples for only $16,\text{k}$ positive ones. At the same time, because the sigmoid loss decomposes into a sum of per-example losses, we can perform controlled experiments to study the effect of the mini-batch composition and distribution of examples visited. We run experiments in the SigLiT setup at batch-size $16,\text{k}$ for $900,\text{M}$ steps and vary the composition of the batch by masking out (i.e. ignoring) enough negative examples to reach a target “positive : negative” ratio, masking in the following ways:

• Random: Randomly choose negative pairs to mask.
• Hard: Keep hardest negative pairs (highest loss).
• Easy: Keep easiest negatives pairs (lowest loss).
• Hard + matching total pairs seen: Masking examples while training for a fixed number of steps does decrease the total number of pairs seen during training. Hence in the matched pairs setting, we increase the number of training steps by the masking ratio in order to keep the number of pairs seen constant.

Figure 6 shows the effect of the various masking strategies. Randomly removing negatives to rebalance does deteriorate performance. Keeping the easiest examples does not work at all, while keeping the hardest negatives does almost maintain the quality, indicating that, as could be expected, a lot of the learning on the negative side comes from the harder examples. This is further confirmed by the slightly increased performance of training longer on the hardest examples in order to match the total pairs seen.

We also look at the value of the learned bias at the end of training as well as the average logit value for positive and negative examples across these settings, and find the result mostly follows what one would expect: as fewer negatives are present, the bias and logits become more positive overall. Interestingly, when training with more hard negative pairs, the average logits of positive pairs stays mostly flat.

This study confirms that (1) the imbalance does not seem to be a major reason for concern, while at the same time (2) coming up with an efficient way of including more negative examples can be promising but is not trivial.

: Table 3: Comparison with other methods. Our SigLIP models outperform the previous models, e.g. OpenCLIP [22] and CLIP [34], by a significant margin on both the zero-shot classification tasks and the zero-shot retrieval tasks. Compared to the recently introduced EVA-CLIP [40] model, SigLIP-L outperforms EVA-CLIP-L on all the tasks with the same number of 256 patches. SigLIP-L (256 patches) also outperforms EVA-CLIP-L (576 patches) by a large margin on the retrieval tasks.

\begin{tabular}{l c c c c c c c c}
\hline
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{Image Encoder}} & \multicolumn{4}{c}{\textbf{ImageNet-1k}} & \multicolumn{2}{c}{\textbf{COCO}} \\
\cline{2-9}
& Size & Patch \# & Validation & v2 & Real & ObjectNet & $\text{I}\to\text{T}$ & $\text{T}\to\text{I}$ \\
\hline
CLIP & B & 196 & 68.3 & 61.9 & - & 55.3 & 52.4 & 33.1 \\
OpenCLIP & B & 196 & 70.2 & 62.3 & - & 56.0 & 59.4 & 42.3 \\
EVA-CLIP & B & 196 & 74.7 & 67.0 & - & 62.3 & 58.7 & 42.2 \\
SigLIP & B & 196 & 76.3 & 69.6 & 82.8 & 70.7 & 64.4 & 47.2 \\
SigLIP & B & 256 & \textbf{76.6} & \textbf{70.0} & \textbf{83.1} & \textbf{71.3} & \textbf{65.1} & \textbf{47.4} \\
\hline
CLIP & L & 256 & 75.5 & 69.0 & - & 69.9 & 56.3 & 36.5 \\
OpenCLIP & L & 256 & 74.0 & 61.1 & - & 66.4 & 62.1 & 46.1 \\
EVA-CLIP & L & 256 & 79.8 & 75.3 & - & 72.9 & 63.7 & 47.5 \\
SigLIP & L & 256 & \textbf{80.6} & 74.2 & \textbf{85.9} & \textbf{77.9} & \textbf{69.5} & \textbf{51.1} \\
\hline
CLIP & L & 576 & 76.6 & 72.0 & - & 70.9 & 57.9 & 37.1 \\
EVA-CLIP & L & 576 & 80.4 & 73.8 & - & 78.4 & 64.1 & 47.9 \\
\hline
\end{tabular}

: Table 4: Bias ($b$) and temperature ($t'$) initialization. Results are reported using Base architecture, $8,\text{k}$ batch size, trained for 900M examples. Enabling the bias term $b$ with $-10$ initialization improves results consistently.

\begin{tabular}{c c c c c}
\hline
$b$ & $t'$ & INet-0 & Pet-0 & C100-0 \\
\hline
n/a & $\log 10$ & 62.0 & 81.8 & 59.9 \\
-10 & $\log 10$ & \textbf{63.0} & \textbf{82.4} & \textbf{61.0} \\
-10 & $\log 1$ & 61.0 & 80.0 & 60.4 \\
0 & $\log 10$ & 61.7 & 79.9 & 59.0 \\
0 & $\log 1$ & 53.7 & 73.2 & 53.8 \\
\hline
\end{tabular}

4.9. Bias term in sigmoid loss

We ablate the bias term in the loss function, using the Base architecture with an $8,\text{k}$ batch size, trained for 900M examples with the SigLIP setup. Zero-shot transfer results are reported on ImageNet [14], Oxford-iiit pet [32] and Cifar100 [26]. Table 4 presents results with and without a bias term in the sigmoid loss.

Enabling the bias term with a $-10$ initialization consistently improves performance across all tasks. This is because the bias term ensures that the training starts close to the prior, preventing dramatic over-correction in early optimization. In contrast, a randomly chosen bias term initialization, such as the 0 initialization in Table 4, fails to address the over-correction issue, leading to significantly worse results. This effect is particularly noticeable when using a small temperature $t'$ initialization. We set the bias and temperature initialization to $b = -10$ and $t' = \log 10$ (hence $t = 10$) as the default for all experiments.

4.10. Label noise robustness

Prior works demonstrated improved robustness against label noise when using the sigmoid loss for classification models [3]. This property would be particularly useful here in the face of the famously noisy nature of popular large-scale image-text datasets. In order to study this for SigLIP, we train M/16 image models alongside an M text model at batch size 16384 for 3.6 billion seen examples. We corrupt the training data using one of the following methods:

• Image: With probability $p$, replace the image with uniform random noise.
• Text: With probability $p$, replace tokenized text with a new sequence of randomly sampled tokens, up to some (sampled) sequence length.
• Batch alignment: Randomly shuffle the ordering of $p%$ of the batch.
• Image & text: Apply both with probability $p$ each.
• Image, text & batch: Alongside (4), also shuffle fraction $p$ of alignments.

Results from varying the likelihood of the corruption are shown in Figure 7. Models trained with sigmoid loss are increasingly robust to all kinds of added noise.

5. Conclusion

Section Summary: The study examined two language-image pre-training models, SigLiT and SigLIP, that rely on a sigmoid loss function instead of the usual softmax approach. This change led to stronger performance, especially at smaller training batch sizes, while also using memory more efficiently so that larger batches could be run without extra hardware. Further tests identified a moderate batch size of around 32,000 as nearly optimal and explored related factors such as noise robustness, with the authors hoping the work will support research under limited resources.

We conducted a study on two language-image pre-training instances that used the sigmoid loss: SigLiT and SigLIP. Our results demonstrate that the sigmoid loss performs better than the softmax baseline, particularly for small train batch sizes. This loss function is also more memory efficient, which allows larger train batch sizes without requiring additional resources. We performed a thorough investigation of the batch size in contrastive learning. Surprisingly, we found that a relatively modest batch size of $32,\text{k}$ yielded nearly optimal performance. Further studies have been performed to understand better the introduced bias term in the sigmoid loss, robustness to data noises and the impact of positive and negative pairs ratio in the sigmoid loss. We hope this work will facilitate language-image pre-training research with limited resources.

Acknowledgements. We thank Daniel Keysers, Ilya Tolstikhin, Olivier Bousquet and Michael Tschannen for their valuable feedback and discussions on this paper. We thank Joan Puigcerver, Josip Djolonga and Black Hechtman for discussions on efficient implementations of the chunked contrastive loss. We thank Kaiming He and Xinlei Chen for the discussion of $\beta_2$ to stabilize the training. We also thank Ross Wightman for spotting a mistake in the pseudocode in the first version of this paper. Similarly, we thank Boris Dayma for spotting a typo making $t$ vs $t'$ confusing, which we fixed in the third version of this paper. We thank the Google Deepmind team for providing a supportive research environment. We use the big_vision codebase [5, 4] for all experiments in this project.

References

Section Summary: The references section consists of a numbered bibliography listing dozens of academic papers, technical reports, and online resources on topics in machine learning, computer vision, and vision-language models. These citations cover foundational datasets such as ImageNet, methods like contrastive learning and transformer architectures, scaling studies, and related tools or code repositories. The entries follow standard academic formatting with authors, titles, venues, and page or citation markers indicating where each source is referenced in the main text.

[1] Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. In NeurIPS, 2023. 7

[2] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019. 7

[3] Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? CoRR, abs/2006.07159, 2020. 2, 7, 9

[4] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet-1k, 2022. 9

[5] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision. https://github.com/google-research/big_vision, 2022. 9

[6] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset, 2022. 1

[7] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 1

[8] Feilong Chen, Duzhen Zhang, Minglun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. VLP: A survey on vision-language pre-training. Int. J. Autom. Comput., 20(1):38–56, 2023. 2

[9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pre-training from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1691–1703. PMLR, 2020. 7

[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2, 4

[11] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015. 7

[12] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms, 2023. 2, 6

[13] Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model. CoRR, abs/2209.06794, 2022. 1, 5, 6

[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 4, 7, 9

[15] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. 1

[16] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Shuyang Gu, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Clip itself is a strong fine-tuner: Achieving 85.7% and 88.0% top-1 accuracy with vit-b and vit-l on imagenet. CoRR, abs/2212.06138, 2022. 2

[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 3, 7

[18] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org. 1

[19] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, volume 2, 2006. 2

[20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15979–15988. IEEE, 2022. 7

[21] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. CoRR, abs/2111.12233, 2021. 1, 2

[22] Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP. Zenodo, 2021. 8

[23] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 1, 2

[24] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. 2

[25] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. In ECCV, 2020. 7

[26] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Univ. of Toronto, 2009. 9

[27] Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, 2018. 5, 13

[28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888–12900. PMLR, 2022. 2

[29] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. CoRR, abs/2212.00794, 2022. 2, 7

[30] Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X, volume 13670 of Lecture Notes in Computer Science, pages 728–755. Springer, 2022. 2

[31] Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip H. S. Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning, 2022. 2

[32] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. 9

[33] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer learning. CoRR, abs/2111.10050, 2021. 2, 4

[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2, 3, 8

[35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019. 5, 13

[36] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In ICML, 2019. 7

[37] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. CoRR, abs/2210.08402, 2022. 1

[38] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. WIT: wikipedia-based image text dataset for multimodal multilingual machine learning. CoRR, abs/2103.01913, 2021. 1

[39] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? Data, augmentation, and regularization in vision transformers. CoRR, abs/2106.10270, 2021. 1, 6

[40] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: improved training techniques for CLIP at scale. CoRR, abs/2303.15389, 2023. 8

[41] Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 715–729. Association for Computational Linguistics, 2022. 4

[42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023. 7

[43] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018. 2

[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3

[45] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. CoRR, abs/2205.14100, 2022. 1, 2

[46] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. 2

[47] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. CoRR, abs/2110.00476, 2021. 2

[48] Mitchell Wortsman. Reaching 80% zero-shot accuracy with OpenCLIP: VIT-G/14 trained on LAION-2B. https://web.archive.org/web/20230127012732/https://laion.ai/blog/giant-openclip/. 2

[49] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 7949–7961. IEEE, 2022. 2

[50] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In NAACL-HLT, 2021. 5

[51] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 19141–19151. IEEE, 2022. 2

[52] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. CoRR, abs/2205.01917, 2022. 2

[53] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. CoRR, abs/2111.11432, 2021. 2

[54] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CVPR, 2022. 1, 4, 6, 7, 13

[55] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18102–18112. IEEE, 2022. 1, 2, 3, 4, 6, 7, 13

[56] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Zachary C. Lipton, Rajesh Ranganath, Mark P. Sendak, Michael W. Sjoding, and Serena Yeung, editors, Proceedings of the Machine Learning for Healthcare Conference, MLHC 2022, 5-6 August 2022, Durham, NC, USA, volume 182 of Proceedings of Machine Learning Research, pages 2–25. PMLR, 2022. 2