Unsupervised Dense Information Retrieval with Contrastive Learning

Gautier Izacard $^{\diamondsuit,\clubsuit,\heartsuit}$ [email protected] Mathilde Caron $^{\diamondsuit,\heartsuit,\spadesuit}$ [email protected] Lucas Hosseini $^{\diamondsuit}$ [email protected] Sebastian Riedel $^{\diamondsuit,\triangle}$ [email protected] Piotr Bojanowski $^{\diamondsuit}$ [email protected] Armand Joulin $^{\diamondsuit}$ [email protected] Edouard Grave $^{\diamondsuit}$ [email protected] $^{\diamondsuit}$ Meta AI Research, $^{\clubsuit}$ Ecole normale supérieure, PSL University, $^{\heartsuit}$ Inria $^{\spadesuit}$ Université Grenoble Alpes, $^{\triangle}$ University College London

Abstract

Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new applications with no training data, and are outperformed by unsupervised term-frequency methods such as BM25. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings. On the BEIR benchmark our unsupervised model outperforms BM25 on 11 out of 15 datasets for the Recall@100. When used as pre-training before fine-tuning, either on a few thousands in-domain examples or on the large MS MARCO dataset, our contrastive model leads to improvements on the BEIR benchmark. Finally, we evaluate our approach for multi-lingual retrieval, where training data is even scarcer than for English, and show that our approach leads to strong unsupervised performance. Our model also exhibits strong cross-lingual transfer when fine-tuned on supervised English data only and evaluated on low resources language such as Swahili. We show that our unsupervised models can perform cross-lingual retrieval between different scripts, such as retrieving English documents from Arabic queries, which would not be possible with term matching methods.

Executive Summary: The document introduces Contriever, a dense neural retriever trained entirely without labeled query-document pairs. It tackles a practical problem: supervised dense retrievers excel on large English benchmarks but underperform the simple unsupervised baseline BM25 when applied to new domains, languages, or low-resource settings. This gap matters because creating labeled retrieval data is expensive or impossible at scale, limiting adoption in specialized or multilingual applications.

The work set out to test whether contrastive learning alone could produce competitive unsupervised dense retrievers. The authors trained a standard BERT-base bi-encoder on unlabeled text from English Wikipedia and CCNet by turning independent random crops of the same passage into positive pairs and using a large queue of negatives (via the MoCo algorithm). They evaluated the resulting model, called Contriever, in three regimes: fully unsupervised zero-shot retrieval, few-shot adaptation, and multilingual retrieval.

On the BEIR benchmark Contriever matches or exceeds BM25 on Recall@100 for 11 of 15 datasets, closing most of the previous gap between unsupervised dense and lexical methods. When the same model is then fine-tuned on the large English MS MARCO collection, it sets a new state-of-the-art recall@100 average across BEIR and, after simple re-ranking, also leads on nDCG@10 for eight datasets. In the few-shot regime it adapts faster and more effectively than a BERT model that was first fine-tuned on MS MARCO. In the multilingual setting a single model pre-trained on 29 languages outperforms BM25 on recall for Mr. TyDi and, after English-only fine-tuning, achieves strong cross-lingual retrieval (including between different scripts) on a derived MKQA benchmark.

These results matter because they remove the need for expensive in-domain labels to obtain a strong starting point for retrieval pipelines. Practitioners can now initialize dense retrievers that already rival or surpass BM25 and obtain further gains from modest fine-tuning or from re-ranking only the top retrieved candidates. The approach also opens practical multilingual and cross-lingual retrieval in languages where labeled data do not exist.

The authors therefore recommend adopting unsupervised contrastive pre-training (specifically with random cropping and a momentum queue) as the default initialization for any new dense retriever. When labeled English data are available, follow with fine-tuning on MS MARCO; when only a few hundred in-domain examples exist, fine-tune directly on them. Further gains are likely from combining the model with hard-negative mining or distillation, but those steps are secondary.

The main limitations are that performance still trails BM25 on certain domains (recent events and very long documents) and that top-ranking quality benefits from supervised fine-tuning or re-ranking. Results are stable across the datasets tested, but readers should verify behavior on any new collection whose vocabulary or document length differs markedly from the pre-training corpora.

1. Introduction

Section Summary: Document retrieval finds relevant documents from large collections to support tasks like question answering and fact checking. Traditional word-matching methods struggle when queries and documents use different terms, and neural retrievers need large labeled datasets that are rarely available outside English. This work shows that contrastive learning on synthetic queries or document crops can train effective dense retrievers from scratch, matching BM25 in zero-shot settings and extending naturally to multilingual cases.

Document retrieval is the task of finding relevant documents in a large collection to answer specific queries. This is an important task by itself and a core component to solve many natural language processing (NLP) problems, such as open domain question answering ([1]) or fact checking ([2]). Traditionally, retrieval systems, or retrievers, leverage lexical similarities to match queries and documents, using, for instance, TF-IDF or BM25 weighting ([3]). These approaches, based on near-exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well ([4]). By contrast, approaches based on neural networks allow learning beyond lexical similarities, resulting in state-of-the-art performance on question answering benchmarks, such as MS MARCO ([5]) or NaturalQuestions ([6]).

The strong retrieval results of neural networks have been possible for domains and applications where large training datasets are available. In the case of retrieval, creating these datasets requires manually matching queries to the relevant documents in the collection. This is hardly possible when the collection contains millions or billions of elements, resulting in many scenarios where only a few in-domain examples, if any, are available. A potential solution is to train a dense retriever on a large retrieval dataset such as MS MARCO, and then apply it to new domains, a setting referred to as zero-shot. Unfortunately, in this setting, dense retrievers are often outperformed by classical methods based on term-frequency, which do not require supervision ([7]). Moreover large annotated datasets are generally not available in languages other than English. Thus, using large collections of supervised data is not suitable to train multilingual retrieval systems.

A natural alternative to transfer learning is unsupervised learning, which raises the following question: is it possible to train dense retrievers without supervision, and match the performance of BM25? Training dense retrievers without supervision can be achieved by using an auxiliary task that approximates retrieval. Given a document, one can generate a synthetic query and then train the network to retrieve the original document, among many others, given the query. The inverse Cloze task (ICT), proposed by [8] to pre-train retrievers, uses a given sentence as a query and predicts the context surrounding it. While showing promising results as pre-training ([9, 10]), this approach still lags behind BM25 when used as a zero-shot retriever. ICT is strongly related to contrastive learning ([11]), which has been widely applied in computer vision ([12, 13]). In particular, computer vision models trained with the latest contrastive learning methods led to features well suited to retrieval ([14]). We thus propose to revisit how well contrastive learning performs to train dense retrievers without supervision.

In this paper, we make the following contributions. First, we show that contrastive learning can lead to strong unsupervised retrievers: our model achieves Recall@100 results competitive with BM25 on most of the BEIR benchmark. Second, in a few-shot setting, we show that our model benefits from few training examples, and obtains better results than transferring models from large datasets such as MS MARCO. Third, when used as a pre-training method before fine-tuning on MS MARCO, our technique leads to strong performance on the BEIR benchmark. We perform ablations to motivate our design choices, and show that cropping works better than the inverse Cloze task. Finally we train a multilingual dense retriever with contrastive learning and show that it achieves state-of-the-art performance.

Code and pre-trained models are available here: https://github.com/facebookresearch/contriever.

2. Related work

Section Summary: The section reviews prior work on information retrieval, starting with traditional term-frequency techniques like TF-IDF and BM25 that represent documents as sparse vectors based on exact word matches, and noting their later extension through dense vector methods such as latent semantic analysis. It then covers neural approaches, including bi-encoder models that separately encode queries and documents for fast retrieval, cross-encoder models like BERT that jointly process them for better accuracy, and refinements such as dense passage retrievers, poly-encoders, and index compression to make large-scale use practical. Finally, it discusses self-supervised pretraining methods, including sentence-level objectives and contrastive learning with data augmentation, that have been adapted to improve retrieval without needing labeled pairs.

In this section, we briefly review relevant work in information retrieval, and application of machine learning to this problem. This is not an exhaustive review, and we refer the reader to [15], [16] and [17] for a more complete introduction to the field.

Term-frequency based information retrieval.

Historically, in information retrieval, documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary. Different weighing schemes have been proposed, to determine how important a particular term is to a document in a large dataset. One of the most used weighing scheme is known as TF-IDF, and is based on inverse document frequency, or term specificity ([18]). BM25, which is still widely used today, extends TF-IDF ([19]). A well known limitation of these approaches is that they rely on near-exact match to retrieve documents. This led to the introduction of latent semantic analysis ([20]), in which documents are represented as low dimensional dense vectors.

Neural network based information retrieval.

Following the successful application of deep learning methods to natural language processing, neural networks techniques were introduced for information retrieval. [21] proposed a deep bag-of-words model, in which representations of queries and documents are computed independently. A relevance score is then obtained by taking the dot product between representations, and the model is trained end-to-end on click data from a search engine. This method was later refined by replacing the bag-of-words model by convolutional neural networks ([22]) or recurrent neural network ([23]). A limitation of bi-encoders is that queries and documents are represented by a single vector, preventing the model to capture fine-grained interactions between terms. [24] introduced a cross-encoder model, based on the BERT model ([25]), which jointly encodes queries and documents. The application of a strong pre-trained model, as well as the cross-encoder architecture, lead to important improvement on the MS MARCO benchmark ([26]).

The methods described in the previous paragraph were applied to re-rank documents, which were retrieved with a traditional IR system such as BM25. [27] first studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking. In the context of question answering, [28] introduced a dense passage retriever (DPR) based on the bi-encoder architecture. This model is initialized with a BERT network, and trained discriminatively using pairs of queries and relevant documents, with hard negatives from BM25. [29] further extended this work by mining hard negatives with the model itself during optimization, and trained on the MS MARCO dataset. Once a collection of documents, such as Wikipedia articles, is encoded, retrieval is performed with a fast k-nearest neighbors library such as FAISS [30]. To alleviate the limitations of bi-encoders, [31] introduces the poly-encoder architecture, where documents are encoded by multiple vectors. Similarly, [32] proposes the ColBERT model, which keeps a vector representation for each term of the queries and documents. To make the retrieval tractable, the term-level function is approximated to first retrieve an initial set of candidates, which are then re-ranked with the true score. In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels ([33]), or the relevance score from a cross encoder ([34]). [35] compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross- and poly-encoders.

Dense retrievers, such as DPR, can lead to indices weighing nearly 100GB when encoding document collections such as Wikipedia. [36] shows how to compress such indices, with limited impact on performance, making them more practical to use.

Self-supervised learning for NLP.

Following the success of word2vec ([37]), many self-supervised techniques have been proposed to learn representation of text. Here, we briefly review the ones that are most related to our approach: sentence level models and contrastive techniques. [38] introduced different objective functions to learn sentence representations, including next sentence prediction and sentence order prediction. These objectives were later used in pre-trained models based on transformers, such as BERT ([25]) and AlBERT ([39]). In the context of retrieval, [8] introduced the inverse cloze task (ICT), whose purpose is to predict the context surrounding a span of text. [40] integrated a bi-encoder retriever model in a BERT pre-training scheme. The retrieved documents are used as additional context in the BERT task, and the whole system is trained end-to-end in an unsupervised way. Similarly, [41] proposed to jointly learn a retriever and a generative seq2seq model, using self-supervised training. [9] compares different pre-training tasks for retrieval, including the inverse cloze task. In the context of natural language processing, [42] proposed to apply MoCo where positive pairs of sentences are obtained using back-translation. Different works augmented the masked language modeling objective with a contrastive loss ([43, 44, 45]). SBERT ([46]) uses a Siamese network similar to contrastive learning to learn a BERT-like model that is adapted to matching sentence embeddings. Their formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose to use data augmentation to leverage large unaligned textual corpora. Concurrent to this work, [47] have also shown the potential of contrastive learning for information retrieval; building on the same observation that both tasks share a similar structure. Spider ([48]), a contemporary work, uses spans appearing multiple times in a document to create pseudo examples for contrastive learning in order to train unsupervised retrievers. Finally, [49] train a dense retriever to imitate unsupervised lexical-based methods. This improves performance on a range of tasks and achieves state-of-the-art results when combining the resulting dense retriever with Contriever, our model pre-trained with contrastive learning.

3. Method

Section Summary: The section explains how to train a dense retriever model without any labeled data by using a bi-encoder architecture based on BERT. Queries and documents are encoded independently into vectors whose dot product gives a relevance score, and the model is trained with a contrastive loss that pulls together positive pairs while pushing apart negative ones. Positive pairs are created from a single document through unsupervised text augmentations such as the Inverse Cloze Task or independent cropping, allowing the model to learn useful representations from raw text alone.

In this section, we describe how to train a dense retriever with no supervision. We review the model architecture and then describe contrastive learning — a key component of its training.

The objective of a retriever is to find relevant documents in a large collection for a given query. Thus, the retriever takes as input the set of documents and the query and outputs a relevance score for each document. A standard approach is to encode each query-document pair with a neural network ([24]). This procedure requires re-encoding every document for any new query and hence does not scale to large collections of documents. Instead, we follow standard approaches ([21, 28]) in information retrieval and use a bi-encoder architecture where documents and queries are encoded independently. The relevance score between a query and a document is given by the dot product between their representations after applying the encoder. More precisely, given a query $q$ and document $d$, we encode each of them independently using the same model, $f_\theta$, parameterized by $\theta$. The relevance score $s(q, d)$ between a query $q$ and a document $d$ is then the dot product of the resulting representations:

$ s(q, d) = \langle f_\theta(q), f_\theta(d)\rangle. $

In practice, we use a transformer network for $f_{\theta}$ to embed both queries and documents. Alternatively, two different encoders can be used to encode queries and documents respectively as in DPR ([28]). Empirically, we observed that using the same encoder, such as in [29] and [46], generally improves robustness in the context of zero-shot transfer or few-shot learning, while having no impact on other settings. Finally, the representation $f_\theta(q)$ (resp. $f_\theta(d)$) for a query (resp. document) is obtained by averaging the hidden representations of the last layer. Following previous work on dense retrieval with neural networks, we use the BERT base uncased architecture and refer the reader to [25] for more details.

3.1 Unsupervised training on unaligned documents

In this section, we describe our unsupervised training method. We briefly review the loss function traditionally used in contrastive learning and also used in ICT ([8]). We then discuss obtaining positive pairs from a single text document, a critical ingredient for this training paradigm.

3.1.1 Contrastive learning

Contrastive learning is an approach that relies on the fact that every document is, in some way, unique. This signal is the only information available in the absence of manual supervision. A contrastive loss is used to learn by discriminating between documents. This loss compares either positive (from the same document) or negative (from different documents) pairs of document representations. Formally, given a query $q$ with an associated positive document $k_+$, and a pool of negative documents $(k_i)_{i=1..K}$, the contrastive InfoNCE loss is defined as:

$ \mathcal{L}(q, k_+) = - \frac{\exp(s(q, k_+) / \tau)}{\exp(s(q, k_+) / \tau) + \sum_{i=1}^K \exp(s(q, k_i) / \tau)}, $

where $\tau$ is a temperature parameter. This loss encourages positive pairs to have high scores and negative pairs to have low scores. Another interpretation of this loss function is the following: given the query representation $q$, the goal is to recover, or retrieve, the representation $k_+$ corresponding to the positive document, among all the negatives $k_i$. In the following, we refer to the left-hand side representations in the score $s$ as queries and the right-hand side representations as keys.

3.1.2 Building positive pairs from a single document

A crucial element of contrastive learning is how to build positive pairs from a single input. In computer vision, this step relies on applying two independent data augmentations to the same image, resulting in two "views" that form a positive pair ([11, 12]). While we consider similar independent text transformation, we also explore dependent transformations designed to reduce the correlation between views.

Inverse Cloze Task

is a data augmentation that generates two mutually exclusive views of a document, introduced in the context of retrieval by [8]. The first view is obtained by randomly sampling a span of tokens from a segment of text, while the complement of the span forms the second view. Specifically, given a sequence of text $(w_1, ..., w_n)$, ICT samples a span $(w_a, ..., w_b)$, where $1 \leq a \leq b \leq n$, uses the tokens of the span as the query and the complement $(w_1, ..., w_{a-1}, w_{b+1}, ..., w_n)$ as the key. In the original implementation by [8] the span corresponds to a sentence, and is kept in the document 10% of the time to encourage lexical matching. The Inverse Cloze Task is closely related to the Cloze task which uses the span complement $(w_1, ..., w_{a-1}, w_{b+1}, ..., w_n)$ as the query.

Independent cropping

is a common independent data augmentation used for images where views are generated independently by cropping the input. In the context of text, cropping is equivalent to sampling a span of tokens. This strategy thus samples independently two spans from a document to form a positive pair. As opposed to the inverse Cloze task, in cropping both views of the example correspond to contiguous subsequence of the original data. A second difference between cropping and ICT is the fact that independent random cropping is symmetric: both the queries and documents follow the same distribution. Independent cropping also lead to overlap between the two views of the data, hence encouraging the network to learn exact matches between the query and document, in a way that is similar to lexical matching methods like BM25. In practice, we can either fix the length of the span for the query and the key, or sample them.

Additional data augmentation.

Finally, we also consider additional data augmentations such as random word deletion, replacement or masking. We use these perturbations in addition to random cropping.

3.1.3 Building large set of negative pairs

An important aspect of contrastive learning is to sample a large set of negatives. Most standard frameworks differ from each other in terms of how the negatives are handled, and we briefly describe two of them, in-batch negative sampling and MoCo, that we use in this work.

Negatives within a batch.

A first solution is to generate the negatives by using the other examples from the same batch: each example in a batch is transformed twice to generate positive pairs, and we generate negatives by using the views from the other examples in the batch. We will refer to this technique as "in-batch negatives". In that case, the gradient is back-propagated through the representations of both the queries and the keys. A downside of this approach is that it requires extremely large batch sizes to work well [12], with [50] reporting improvement in the context of information retrieval up to 8192 negatives. This method has been widely used to train information retrieval models with supervised data [51, 28] and was also considered when using ICT to pre-train retrievers by [8].

Negative pairs across batches.

An alternative approach is to store representations from previous batches in a queue and use them as negative examples in the loss ([11]). This allows for smaller batch size but slightly changes the loss by making it asymmetric between "queries" (one of the view generated from the elements of the current batch), and "keys" (the elements stored in the queue). Gradient is only backpropagated through the "queries", and the representation of the "keys" are considered as fixed. In practice, the features stored in the queue from previous batches comes form previous iterations of the network. This leads to a drop of performance when the network rapidly changes during training. Instead, [13] proposed to generate representations of keys from a second network that is updated more slowly. This approach, called MoCo, considers two networks: one for the keys, parametrized by $\theta_k$, and one of the query, parametrized by $\theta_q$. The parameters of the query network are updated with backpropagation and stochastic gradient descent, similarly to when using in-batch negatives, while the parameters of the key network, or Momentum encoder, is updated from the parameters of the query network by using a exponential moving average:

$ \theta_k \leftarrow m \theta_k + (1 - m) \theta_q, $

where $m$ is the momentum parameter that takes its value in $[0, 1]$.

4. Experiments

Section Summary: In this section the authors evaluate their Contriever model, a dense retriever trained with contrastive learning on unlabeled text from Wikipedia and CCNet using random cropping and a momentum queue for negatives. They test its unsupervised retrieval performance on two open-domain question-answering collections and on the diverse 18-dataset BEIR benchmark, reporting recall and ranking metrics while comparing it to the classical BM25 baseline and to other unsupervised and supervised dense retrievers. The experiments aim to show that a simple contrastive approach can match or exceed strong traditional methods and prior learned retrievers without any labeled training data.

In this section, we empirically evaluate our best retriever trained with contrastive learning, called Contriever (contrastive retriever), which uses MoCo with random cropping. We use a contrastive learning procedure that differs from ICT ([8]) mainly in three aspects. First, positive pairs are sampled using random cropping and tokens from each element of the pair are deleted with a probability of 10%. Second we use MoCo where negatives consists of elements from previous batches stored in a queue. This allows to scale to a large number of negatives. Third we use data from Wikipedia and CCNet ([53]) for training. Ablation studies motivating these technical choices are performed in Section 6. More technical details about our model are given in Appendix A.1.

4.1 Datasets

Contriever is trained with contrastive learning on documents sampled from a mix between Wikipedia data and CCNet data ([53]), where half the batches are sampled from each source.

First, we evaluate our model on two question answering datasets: NaturalQuestions ([6]) and TriviaQA ([54]). For both datasets, we use the open domain versions as introduced by [8], and the English Wikipedia dump from Dec. 20, 2018 as the collection of documents to retrieve from. We report the top-k retrieval accuracy, i.e. the number of questions for which at least one of the top-k passages contain the answer.

Second, we use the BEIR benchmark, introduced by [7], which contains 18 retrieval datasets, corresponding to nine tasks, such as fact checking or citation prediction, and covering different domains, such as Wikipedia or scientific publications. Most datasets from BEIR do not contain a training set, and the focus of the benchmark is zero-shot retrieval. However, most machine learning based retrievers are still trained on supervised data, such as the large scale retrieval dataset MS MARCO ([26]). Following standard practice, we report two metrics on this benchmark: nDCG@10 and Recall@100. The nDCG@10 focuses on the ranking of the top 10 retrieved documents, and is good at evaluating rankings returned to humans, for example in a search engine. On the other hand, Recall@100 is relevant to evaluate retrievers that are used in machine learning systems, such as question answering. Indeed, such models can process hundreds of documents, and ignore their ranking ([55]). While nDCG@10 is the main metric of BEIR, we are more interested in the Recall@100 to evaluate bi-encoders, as our goal is to develop retrievers that can be used in ML systems. Moreover, in many settings, retrieved documents can be re-ranked with a more powerful model such as a cross-encoder, thus improving the nDCG@10.

::: {caption="Table 1: Unsupervised recall@k on the test sets of NaturalQuestions and TriviaQA. For Inverse Cloze Task and Masked Salient Spans we report the results of [10]. The Masked Salient Spans model uses annotated named entity recognition data. For BM25 we report the results of [56]"}

:::

4.2 Baselines

First, we compare Contriever to BM25, which does not require supervision. On QA datasets, we compare to dense retrievers trained with ICT and the Masked Salient Spans from [10]. On BEIR, we consider the retriever from REALM ([40]), and RoBERTa large fine-tuned with SimCSE ([52]), as unsupervised dense retrievers. We also compare to ML-based retrievers trained on MS MARCO, classified in three categories: sparse, dense and late-interaction. For sparse methods, we compare to Splade v2 ([57]), which computes sparse representations of documents with BERT pre-trained model. For dense methods, we use DPR ([28]) and ANCE ([29]), which are bi-encoders trained on supervised data such as NaturalQuestions or MS MARCO. We also compare to TAS-B ([58]), which performs distillation from a cross-encoder to a bi-encoder, and GenQ, which creates synthetic query-document pairs with a generative model.[^1] For late-interaction, we use ColBERT ([32]), which computes pairwise scores between contextualized representations of queries and documents, as well as a cross-encoder used to re-rank documents retrieved with BM25.

[^1]: GenQ thus leads to one different model for each dataset.

::: {caption="Table 2: BEIR Benchmark. We report nDCG@10 on the test sets from the BEIR benchmark for bi-encoder methods without re-ranker. We also report the average and number of datasets where a method is the best (B̈est on") over the entire BEIR benchmark (excluding three datasets because of their licence). Bold is the best overall. MS MARCO is excluded from the average. C̈Er̈efers to cross-encoder."}

:::

4.3 Results

First, we compare the performance of fully unsupervised models, i.e., without fine-tuning on MS MARCO or other annotated data. In Table 1, we report the retrieval performance on two question answering datasets: NaturalQuestions ([6]) and TriviaQA ([54]). Here, our model is competitive with a strong BM25 baseline ([56]), for example leading to 3 points improvement for the recall@100 on NaturalQuestions. It also outperforms previously proposed dense retrievers which were trained with ICT or salient span masking. In Figure 1 we report the recall@100 performance of unsupervised models on the BEIR benchmark. Interestingly, we observe that in this setting, Contriever is competitive compared to BM25 on all datasets, but TREC-COVID and Tóuche-2020. In particular, it obtains better performance than BM25 on 11 out of 15 datasets from the benchmark for the recall@100. Contriever also outperforms previously proposed unsupervised dense retrievers, which obtains lower performance than BM25 in general. For the nDCG@10, which puts more emphasis on the very first retrieved documents, while Contriever largely closes the gap between unsupervised retrievers and BM25, it is still outperformed by BM25 as reported in Table 11. The difference is mainly due to the fact that BM25 largely outperforms Contriever on two datasets with specific features: Trec-COVID and Tóuche-2020. Trec-COVID is an information retrieval dataset related to COVID. However data used to train Contriever were collected before the COVID outbreak, thus they may not be adapted. Tóuche-2020 contains long documents, which does not seem to be very well supported by dense neural retrievers: even after supervised training, models are still lagging behind BM25. Overall, these results show the potential of contrastive learning to train fully unsupervised dense retrievers.

Next, we report nDCG@10 on the BEIR benchmark for different retrievers trained on MS MARCO in Table 2 (recall@100 can be found in Table 10 of appendix). We individually report results on each dataset as well as the average over 14 datasets of the BEIR Benchmark (excluding 3 for license reasons). We observe that when used as pre-training, contrastive learning leads to strong performance: contriever obtains the best results among dense bi-encoder methods for the nDCG@10, and is state-of-the-art for the recall@100 (improving the average recall@100 from 65.0 to 67.1). This strong recall@100 performance can be further exploited by using a cross-encoder[^2] to re-rank the retrieved documents: this leads to the state-of-the-art on 8 datasets of the BEIR benchmark for the nDCG@10, as well as on average. It should be noted that our fine-tuning procedure on MS MARCO is simpler than for other retrievers, as we use a simple strategy for negative mining and do not use distillation. Our model would probably also benefits from improvements proposed by these retrievers, but this is beyond the scope of this paper.

[^2]: We use the existing ms-marco-MiniLM-L-6-v2 cross-encoder model to perform the re-ranking.

::: {caption="Table 3: Few-shot retrieval. Test nDCG@10 after training on a small in-domain training set. We compare BERT and our model, with and without an intermediate fine-tuning step on MS MARCO. Note that our unsupervised pre-training alone outperforms BERT with intermediate MS MARCO fine-tuning."}

:::

Finally, we illustrate the benefit of our retriever compared to BM25 in a few-shot setting, where we have access to a small number of in-domain retrieval examples. This setting is common in practice, and lexical based methods, like BM25, cannot leverage the small training sets to adapt its weights. In Table 3, we report nDCG@10 on three datasets from BEIR associated with the smallest training sets, ranging from 729 to 5, 500 queries. We observe that on these small datasets, our pre-training leads to better results than BERT pre-training, even when BERT is fine-tuned on MS MARCO as an intermediate step. Our pre-trained model also outperforms BM25, showing the advantage of dense retriever over lexical methods in the few-shot setting. More details are given in Appendix A.3.

5. Multilingual retrieval

Section Summary: The section explains how unsupervised dense retrievers offer a practical way to handle multilingual search when large labeled datasets exist mainly in English. A model called mContriever is pre-trained across 29 languages and can then be fine-tuned on English data like MS MARCO to deliver strong results on benchmarks such as Mr. TyDi, while also supporting cross-lingual retrieval of English documents from queries in other languages. This approach beats standard lexical methods like BM25, which struggle with differing scripts or vocabulary, and improves further when the same English fine-tuning is applied to models without contrastive pre-training.

In this section, we illustrate another advantage of learning unsupervised dense retrievers, when performing multi-lingual retrieval. While large labeled datasets are available in English, allowing to train strong dense retrievers (as shown in previous sections), this is unfortunately not the case for lower resources languages. Here, we show how unsupervised training is a promising direction. First, we show that our approach leads to strong performance, either in a full unsupervised setting, or by fine-tuning a multi-lingual model on English data such as MS MARCO. Second, we demonstrate that our model can also perform cross-lingual retrieval, by retrieving English documents from other languages queries. Unsupervised retrievers based on lexical matching, such as BM25, would obtain poor performance, especially for pairs of languages with different scripts such as English and Arabic.

5.1 Multilingual pre-training

Our multilingual model, called mContriever, is jointly pre-trained on 29 languages. The multilingual pre-training largely follows the method discussed in previous sections, but it differs by few particularities related to the pre-training data and the hyperparameters used. The model is initialized with the multilingual version of BERT, mBERT, trained on 104 languages. For the pre-training data, we consider the languages contained in CCNet ([53]) that are also part of our evaluation datasets. This results in a training set containing the CCNet data for 29 languages detailed in Table 12. During pre-training, examples are uniformly sampled over languages, i.e. the probability that a training sample comes from a specific language is the same for all languages. We observed that increasing the number of languages considered for pre-training generally deteriorates performance as reported in Appendix B.3 similarly to what has been observed for multilingual masked language models ([59]). We pre-trained our multilingual mContriever with a queue size of 32768. This generally improves stability, and is able to compensate for the additional instabilities observed in the multilingual setting. More detailed hyperparameters are given in Appendix B.1.

5.2 Fine-tuning

Large labeled datasets for information retrieval are generally available only in English. It is therefore natural to investigate whether large monolingual datasets can be leveraged for multilingual retrieval. We consider fine-tuning our pre-trained mContriever model on MS MARCO. This generally improves performance in all languages. The model trained on MS MARCO can be further fine-tuned on Mr. TyDi achieving state-of-the-art performance on this dataset. Further details regarding fine-tuning are given in Appendix B.2.

::: {caption="Table 4: Multilingual retrieval on Mr. TyDi. We report MRR@100 and Recall@100 on the test sets of Mr. TyDi. mContriever fine-tuned on MS MARCO is compared against its counterparts without contrastive pre-training using a similar fine-tuning recipe, referred to as mBERT + MS MARCO, as well as a model initialized with XLM-R referred to as XLM-R + MS MARCO. We also report the results after fine-tuning on Mr. TyDi for the model trained on MS MARCO."}

:::

5.3 Evaluation

We evaluate the performance of our pre-trained model with and without fine-tuning on English data on two different benchmarks. First, we consider Mr. TyDi ([60]), a multilingual information retrieval benchmark derived from TyDi QA ([61]). Given a question, the goal is to find relevant documents in a pool of Wikipedia documents in the same language.

In Mr. TyDi the pool of documents is restricted to the language of the query. Being able to retrieve relevant documents from another language is desirable to leverage large source of information that may no be available in all languages. In order to evaluate cross-lingual retrieval performance we derive an evaluation setting from MKQA ([62]). Given a question in a specific language, we perform retrieval in English Wikipedia, and evaluate if the English answer is in the retrieved documents. The MKQA dataset makes this possible by providing the same questions and answers in 26 languages. We remove unanswerable questions, questions accepting a binary yes/no answer and questions with long answers from the original MKQA dataset. This results in an evaluation set of 6619 queries. It should be noted that methods based on term matching such as BM25 are intrinsically limited in this cross-lingual retrieval setting because similar terms in different languages may not match.

5.4 Baselines

On Mr. TyDi ([60]) we report results from the original paper. This includes a BM25 baseline, a model fine-tuned on NaturalQuestions using the DPR pipeline, and an hybrid model combining the two.

On our cross-lingual evaluation benchmark derived from MKQA, we consider the retriever of the CORA question answering pipeline ([63]), trained on a combination of datasets containing the English NaturalQuestions and the cross-lingual XOR-TyDi QA, with data augmentation based on translation.

Additionally, to isolate the effect of contrastive pre-training, we also compare mContriever fine-tuned on MS MARCO to its counterparts without contrastive pre-training, initialized from mBERT. This model is referred as mBERT + MSMARCO in tables. We also report results obtained by fine-tuning XLM-R ([59]) on MS-MARCO. For both models we use the same hyper-parameters used to fine-tune mContriever on MS MARCO except for the temperature, additional details are reported in Appendix B.2.

5.5 Results

We report results on Mr. TyDi in Table 4. The effectiveness of the multilingual pre-training appears clearly as the pre-trained model fine-tuned on MS MARCO achieve significantly better performance than its counterparts without pre-training when fine-tuned using the same pipeline. Interestingly fine-tuning on these English-only datasets improves performance on all languages. Moreover our unsupervised mContriever outperforms BM25 for the Recall@100, and after fine-tuning on MS MARCO it achieves state-of-the-art performance for this metric. Performance can be further improved by fine-tuning on the train set of Mr. TyDi. This appears to be particularly important for the MRR@100 metric which put emphasis on the quality of the first documents retrieved. For this metric our unsupervised model is still lagging behind BM25.

Results on the cross-lingual retrieval benchmark derived from MKQA are reported in Table 5, with per language details for the Recall@100 and Recall@20 reported in Table 13 and Table 14 of appendix. Interestingly using only supervised training data in English, our mContriever fine-tuned on MS MARCO outperforms the CORA retriever. Also, similarly to the results reported on Mr. TyDi, adding multilingual contrastive pre-training before fine-tuning on MS MARCO improves performance over its counterpart without pre-training. On MKQA, evaluation is performed by lowercasing both queries and documents, we observed that this improves performance. This does not impact the CORA retriever which is based on an uncased version of mBERT.

::: {caption="Table 5: Cross-lingual retrieval on MKQA. We report the average on all languages included in MKQA for the Recall@100 and the Recall@20, and report the Recall@100 for a subset of languages. Complete results are reported in Table 13 and Table 14 of appendix."}

:::

6. Ablation studies

Section Summary: The ablation studies examine how various design decisions affect the retriever's performance, including the choice of contrastive learning approach, the number of negative examples, methods for creating positive training pairs from documents, and the sources of pre-training data such as Wikipedia or web-crawled text. Experiments show that momentum-based contrastive learning scales better than simple in-batch negatives, that larger numbers of negatives tend to help especially without task-specific fine-tuning, and that random cropping of text passages works better than prior strategies like the inverse cloze task. Using diverse pre-training data and their specialized contrastive pre-training yields gains that exceed what fine-tuning alone achieves on a standard BERT model.

In this section, we investigate the influence of different design choices on our method. In these ablations, all the models are pre-trained on English Wikipedia for $200$ k gradient steps, with a batch size of 2, 048 (on 32 GPUs). Each fine-tuning on MS MARCO takes $20$ k gradient steps with a batch size of $512$ (on 8 GPUs), using AdamW and no hard negatives.

: Table 6: MoCo vs. in-batch negatives. In this table, we report nDCG@10 on the BEIR benchmark for in-batch negatives and MoCo, without fine-tuning on the MS MARCO dataset.

	NFCorpus	NQ	FiQA	ArguAna	Quora	DBPedia	SciDocs	FEVER	AVG
MoCo	26.2	13.1	13.7	33.0	69.5	20.0	11.9	57.6	30.1
In-batch negatives	24.2	21.6	13.0	33.7	74.9	17.9	13.6	56.1	31.9

MoCo vs. in-batch negatives.

First, we compare the two contrastive pre-training methods: MoCo and in-batch negatives. As in in-batch negatives, the number of negative examples is equal to the batch size, we train models with a batch size of 4, 096 and restrict the queue in MoCo to the same number of elements. This experiment measures the effect of using of momentum encoder for the keys instead of the same network as for the queries. Using a momentum also prevents from backpropagating the gradient through the keys. We report results, without fine-tuning on MS MARCO in Table 6. We observe that the difference of performance between the two methods is small, especially after fine-tuning on MS MARCO. We thus propose to use MoCo as our contrastive learning framework, since it scales to a larger number of negative examples without the need to increase the batch size.

Number of negative examples.

Next, we study the influence of the number of negatives used in the contrastive loss, by varying the queue size of the MoCo algorithm. We consider values ranging from 2, 048 to 131, 072, and report results in Figure 2. We see that on average over the BEIR benchmark, increasing the number of negatives leads to better retrieval performance, especially in the unsupervised setting. However, we note that this effect is not equally strong for all datasets.

Data augmentations.

Third, we compare different ways to generate pairs of positive examples from a single document or chunk of text. In particular, we compare random cropping, which leads to pairs with overlap, and the inverse cloze task, which was previously considered to pre-train retrievers. Interestingly, as shown in Table 7, the random cropping strategy outperforms the inverse cloze task in our setting. We believe that random cropping, leading to the identical distributions of keys and queries, leads to more stable training with MoCo compared to ICT. This might explains part of the difference of performance between the two methods. We also investigate whether additional data perturbations, such as random word deletion or replacement, are beneficial for retrieval.

Training data.

Finally, we study the impact of the pre-training data on the performance of our retriever, by training on Wikipedia, CCNet or a mix of both sources of data. We report results in Table 8, and observe that there is no clear winner between the two data sources. Unsurprisingly, training on the more diverse CCNet data leads to strong improvements on datasets from different domains than Wikipedia, such as FiQA or Quora. On the other hand, on a dataset like FEVER, training on Wikipedia leads to better results. To get the best of both worlds, we consider two strategies to mix the two data sources. In the "50/50%" strategy, examples are sampled uniformly across domain, meaning that half the batches are from Wikipedia and the other half from CCNet. In the "uniform" strategy, examples are sampled uniformly over the union of the dataset. Since CCNet is significantly larger than Wikipedia, this means that most of the batches are from CCNet.

Impact of fine-tuning on MS MARCO.

To isolate the impact of pre-training from the impact of fine-tuning on MS MARCO, we apply the same fine-tuning to the BERT base uncased model. We report results in Table 9, and observe that when applied to BERT, our fine-tuning leads to results that are lower than the state-of-the-art. Hence, we believe that most of the improvement compared to the state-of-the-art retrievers can be attributed to our contrastive pre-training strategy.

: Table 7: Impact of data augmentions. We report nDCG@10 without fine-tuning on MS MARCO.

	NFCorpus	NQ	ArguAna	Quora	DBPedia	SciDocs	FEVER	Overall
ICT	23.2	19.4	31.6	27.6	21.3	10.6	55.6	25.9
Crop	27.6	17.7	35.6	75.4	21.0	13.3	64.5	32.2
Crop + delete	26.8	20.8	35.8	77.3	21.5	14.0	67.9	33.8
Crop + replace	27.7	18.7	36.2	75.6	22.0	13.0	66.8	32.9

: Table 8: Training data. We report nDCG@10 without fine-tuning on MS MARCO.

	NFCorpus	NQ	FiQA	ArguAna	Quora	DBPedia	SciDocs	FEVER	Overall
Wiki	27.6	17.7	16.3	35.6	75.4	21.0	13.3	64.5	33.0
CCNet	29.5	25.8	26.2	35.2	80.6	20.5	14.9	60.9	34.9
Uniform	31.0	19.4	25.1	37.8	80.4	21.5	14.7	59.8	33.9
50/50%	31.5	18.6	23.3	36.2	79.1	22.1	13.7	64.1	34.7

: Table 9: Fine-tuning. We report nDCG@10 after fine-tuning BERT and our model on MS MARCO.

	NFCorpus	NQ	FiQA	ArguAna	Quora	DBPedia	SciDocs	FEVER	Overall
BERT	28.2	44.6	25.9	35.0	84.0	34.4	13.0	69.8	42.0
Contriever	33.2	50.2	28.8	46.0	85.4	38.8	16.0	77.7	46.5

7. Discussion

Section Summary: This work explores how far unsupervised contrastive pre-training can push the performance of dense text retrievers by using the MoCo framework to incorporate large numbers of negative examples. The resulting models already match the effectiveness of the classic BM25 baseline, and they reach new state-of-the-art results on the BEIR benchmark once fine-tuned on MS MARCO and paired with a cross-encoder re-ranker. Extensive ablation studies further show that simply cropping random text segments provides a surprisingly strong training signal, rivaling more elaborate methods such as the inverse Cloze task.

In this work, we propose to explore the limits of contrastive pre-training to learn dense text retrievers. We use the MoCo technique, which allows to train models with a large number of negative examples. We make several interesting observations: first, we show that neural networks trained without supervision using contrastive learning exhibit good retrieval performance, which are competitive with BM25 (albeit not state-of-the-art). These results can be further improved by fine-tuning on the supervised MS MARCO dataset, leading to strong results, in particular for recall@100. Based on that observation, we use a cross-encoder to re-rank documents retrieved with our model, leading to new state-of-the-art on the competitive BEIR benchmark. We also performed extensive ablation experiments, and observed that independent random cropping seems to be a strong alternative to the inverse Cloze task for training retrievers.

Appendix

Section Summary: The appendix details the training procedures for the Contriever model, covering contrastive pre-training on mixed web and Wikipedia text using momentum contrast with data augmentations, followed by fine-tuning on MS MARCO with in-batch and mined hard negatives. It also outlines few-shot training protocols for specific evaluation datasets. The section further includes benchmark tables reporting recall and ranking metrics for Contriever against competing retrieval methods across multiple test collections in both supervised and unsupervised scenarios.

A. Technical details for Contriever

A.1 Contrastive pre-training

For the model with fine-tuning on MS MARCO, we use the MoCo algorithm [13] with a queue of size 131, 072, a momentum value of 0.9995 and a temperature of 0.05. We use the random cropping data augmentation, with documents of 256 tokens and span sizes sampled between 5% and 50% of the document length. Documents are simply random piece of text sampled from a mix between Wikipedia and CCNet data ([53]), where half the batches are sampled from each source. We also apply token deletion with a probability of 10%. We optimize the model with the AdamW ([64]) optimizer, with learning rate of $5 \cdot 10^{-5}$, batch size of 2, 048 and 500, 000 steps. We initialize the network with the publicly available BERT base uncased model.

A.2 Fine-tuning on MS MARCO

For the fine-tuning on MS MARCO we do not use the MoCo algorithm and simply use in-batch negatives. We use the ASAM optimizer ([65]), with a learning rate of $10^{-5}$ and a batch size of 1024 with a temperature of $0.05$, also used during pre-training. We train an initial model with random negative examples for 20000 steps, mine hard negatives with this first model, and re-train a second model with those. Each query is associated with a gold document and a negative document, which is a random document in the first phase and a hard negative 10% of the time in the second phase. For each query, all documents from the current batch aside of the gold document are used as negatives.

A.3 Few-shot training

For the few-shot evaluation presented in Table 3, we train for 500 epochs on each dataset with a batch size of 256 with in-batch random negatives. We evaluate performance performance on the development set every 100 gradient updates and perform early stopping based on this metric. For SciFact, we hold out randomly 10% of the training data and use them as development set, leading to a train set containing 729 samples.

\begin{tabular}{@l ccccccc >{\columncolor{bluegrey}}c} 
  \toprule
  & BM25 & DPR & ANCE & TAS-B & Gen-Q & ColBERT & Splade v2 & Ours \\
\midrule
MS MARCO & 65.8 & 55.2 & 85.2 & 88.4 & 88.4 & 86.5 & - & \textbf{89.1} \\
Trec-COVID & \textbf{49.8} & 21.2 & 45.7 & 38.7 & 45.6 & 46.4 & 12.3 & 40.7 \\
NFCorpus & 25.0 & 20.8 & 23.2 & 28.0 & 28.0 & 25.4 & 27.7 & \textbf{30.0} \\
NQ & 76.0 & 88.0 & 83.6 & 90.3 & 86.2 & 91.2 & \textbf{93.0} & 92.5 \\
HotpotQA & 74.0 & 59.1 & 57.8 & 72.8 & 67.3 & 74.8 & \textbf{82.0} & 77.7 \\
FiQA & 53.9 & 34.2 & 58.1 & 59.3 & 61.8 & 60.3 & 62.1 & \textbf{65.6} \\
ArguAna & 94.2 & 75.1 & 93.7 & 94.2 & \bf 97.8 & 91.4 & 97.2 & 97.7 \\
Touche-2020 & \textbf{53.8} & 30.1 & 45.8 & 43.1 & 45.1 & 43.9 & 35.4 & 29.4 \\
CQADupStack & 60.6 & 40.3 & 57.9 & 62.2 & 65.4 & 62.4 & - & \textbf{66.3} \\
Quora & 97.3 & 47.0 & 98.7 & 98.6 & 98.8 & 98.9 & 98.7 & \textbf{99.3} \\
DBPedia & 39.8 & 34.9 & 31.9 & 49.9 & 43.3 & 46.1 & \textbf{57.5} & 54.1 \\
Scidocs & 35.6 & 21.9 & 26.9 & 33.5 & 33.2 & 34.4 & 36.4 & \textbf{37.8} \\
Fever & 93.1 & 84.0 & 90.0 & 93.7 & 92.8 & 93.4 & \textbf{95.1} & 94.9 \\
Climate-fever & 43.6 & 39.0 & 44.5 & 53.4 & 45.0 & 44.4 & 52.4 & \textbf{57.4} \\
Scifact & 90.8 & 72.7 & 81.6 & 89.1 & 89.3 & 87.8 & 92.0 & \textbf{94.7} \\
\midrule
Avg. w/o CQA & 63.6 & 48.3 & 60.1 & 65.0 & 64.2 & 64.5 & 64.8 & 67.1 \\
Avg. & 63.4 & 47.7 & 60.0 & 64.8 & 64.2 & 64.3 & - & 67.0 \\
Best on & 2 & 0 & 0 & 0 & 1 & 0 & 4 & 7 \\
\bottomrule
\end{tabular}

\begin{tabular}{@l c c c c >{\columncolor{bluegrey}}c}
  \toprule
  Model ($\rightarrow$) & BM25 & BERT & SimCSE & REALM & Contriever \\
  \midrule
  Dataset ($\downarrow$) & \multicolumn{5}{c}{Recall@100}\\ 
\midrule
  MS MARCO & 65.8 & 3.5 & 33.6 & 52.6 & \bf 67.2 \\
  Trec-COVID & \bf 49.8 & 10.6 & 26.8 & 8.1 & 17.2 \\
  NFCorpus & 25.0 & 6.7 & 18.2 & 23.0 & \bf 29.4 \\
  NQ & 76.0 & 14.3 & 42.9 & 58.1 & \bf 77.1 \\
  HotpotQA & \bf 74.0 & 15.8 & 42.7 & 56.1 & 70.4 \\
  FiQA-2018 & 53.9 & 6.9 & 41.0 & 28.0 & \bf 56.2 \\
  ArguAna & 94.2 & 59.1 & \bf 95.2 & 73.1 & 90.1 \\
  T\'ouche-2020 & \bf 53.8 & 3.0 & 18.6 & 11.5 & 22.5 \\
  CQADupStack & 60.6 & 11.0 & 48.9 & 35.5 & \bf 61.4 \\
  Quora & 97.3 & 74.6 & 97.9 & 92.7 & \bf 98.7 \\
  DBPedia & 39.8 & 7.1 & 21.5 & 33.0 & \bf 45.3 \\
  SCIDOCS & 35.6 & 11.3 & 23.0 & 23.1 & \bf 36.0 \\
  Fever & 93.1 & 13.6 & 50.8 & 82.6 & \bf 93.6 \\
  Climate-fever & 43.6 & 12.8 & \bf 44.8 & 42.3 & 44.1 \\
  SciFact & 90.8 & 35.2 & 75.3 & 83.8 & \bf 92.6 \\
  \midrule
  Avg. & 63.6 & 19.0 & 45.4 & 46.9 & 60.1 \\
  Best on & 3 & 0 & 2 & 0 & 10 \\
  \midrule
  & \multicolumn{5}{c}{NDCG@10}\\
  \midrule
  MS MARCO & \bf 22.8 & 0.6 & 8.8 & 15.2 & 20.6\\
  Trec-COVID & \bf 65.6 & 16.6 & 38.6 & 20.1 & 27.4 \\
  NFCorpus & \bf 32.5 & 2.5 & 14.0 & 24.1 & 31.7 \\
  NQ & \bf 32.9 & 2.7 & 12.6 & 15.2 & 25.4 \\
  HotpotQA & \bf 60.3 & 4.9 & 23.3 & 40.5 & 48.1 \\
  FiQA-2018 & 23.6 & 1.4 & 14.8 & 9.7 & \bf 24.5 \\
  ArguAna & 31.5 & 23.1 & \bf 45.6 & 22.8 & 37.9 \\
  T\'ouche-2020 & \bf 36.7 & 3.4 & 11.6 & 7.3 & 19.3 \\
  CQADupStack & \bf 29.9 & 2.5 & 20.2 & 13.5 & 28.4 \\
  Quora & 78.9 & 3.9 & 81.5 & 71.6 & \bf 83.5 \\
  DBPedia & \bf 31.3 & 3.9 & 13.7 & 22.7 & 29.2 \\
  SCIDOCS & \bf 15.8 & 2.7 & 7.4 & 9.0 & 14.9 \\
  FEVER & \bf 75.3 & 4.9 & 20.1 & 42.9 & 68.2 \\
  Climate-fever & \bf 21.3 & 4.1 & 17.6 & 14.3 & 15.5 \\
  SciFact & \bf 66.5 & 9.8 & 38.5 & 47.1 & 64.9 \\ \midrule
  Avg. & 41.7 & 8.7 & 24.6 & 25.1 & 36.0 \\
  Best on & 12 & 0 & 1 & 0 & 2 \\
  \bottomrule
  \end{tabular}

B. Multilingual retrieval with mContriever

B.1 Hyperparameters for multilingual contrastive pre-training

The pre-trained mContriever model is pre-trained for 500, 000 steps with a queue of size 32768, and temperature of 0.05 and a momentum value of 0.999. We optimize the model with the AdamW ([64]) optimizer, with learning rate of $5 \cdot 10^{-5}$. The learning rate follows a linear warmup for 20, 000 steps followed by linear decay until the end of training. Languages used for pre-training are detailed in Table 12.

::: caption="Table 12: List of languages used for multilingual retrieval."

:::

B.2 Hyperparameters for multilingual fine-tuning

We fine-tune mContriever using in-batch negatives, AdamW optimizer ([64]), a learning rate of $10^{-5}$, and a batch size of 1024 samples with a temperature $\tau$ of $0.05$. On MS MARCO and Mr. TyDi the model is trained for 20k gradient steps. We notice overfitting on NaturalQuestions, and thus reduced the training to 1k gradient steps. We use a warmup of 1000 gradient steps with linear decay afterwards in all cases. Hard negatives are mined on Mr. TyDi with the model trained on MS MARCO. We did not observe significant improvements using hard negatives on MS MARCO and NaturalQuestions.

For the fine-tuning on MS MARCO of the models initiliazed from mBERT (resp. XLM-R) without contrastive pre-training, we use a temperature $\tau$ of $1$ (resp. $5$). We tried temperatures in ${10, 5, 2, 1, 0.1, 0.05}$ and chose the one leading to the best performance. We observed a decrease in performance for lower temperatures. We fine-tuned mContriever with $\tau = 0.05$ following the temperature used during pre-training. We followed the temperature $\tau = 0.05$ used for the training of Contriever, and did not test other temperatures for the contrastive pre-training of the multilingual model, mContriever.

B.3 Curse of multilinguality

We tried to pre-train models on different sets of languages. We generally observed performance deterioration when scaling to more languages similarly to what has been observed for general multilingual masked language models [59]. In Table 15 we report results on Mr. TyDi with a model pre-trained on the 11 languages of Mr. TyDi versus the model used in the rest of the paper which has been pre-trained on 29 languages including the 11 languages of Mr. TyDi as detailed in Table 12. We also report performance of these models after training on MS MARCO, eventually followed by further fine-tuning on Mr. TyDi. It appears that the performance of the unsupervised model and the performance after fine-tuning on MS MARCO are better for the model pre-trained only on 11 languages. The difference is mitigated after fine-tuning on Mr. TyDi.

::: caption="Table 13: Recall@100 on MKQA for cross-lingual retrieval in the setting described in Section 5.3."

:::

::: caption="Table 14: Recall@20 on MKQA for cross-lingual retrieval in the setting described in Section 5.3."

:::

::: {caption="Table 15: Performance dilution for multilingual retrievers. We report MRR@100 and R@100 on the test set of Mr. TyDi after pre-training on two different sets of languages, one containing the 11 languages of Mr. TyDi which is included in the set of 29 languages used to train mContriever used in the rest of the paper. We also report results of these models after fine-tuning on MS MARCO, potentially followed by a final fine-tuning stage on Mr. TyDi."}

:::

References

Section Summary: This section lists academic citations for research papers and books primarily focused on information retrieval, question answering systems, and related machine learning techniques. The references span foundational methods like BM25 and latent semantic analysis through modern neural approaches involving BERT, dense retrieval, and contrastive learning for tasks such as passage ranking and fact verification. They draw from conferences, journals, and preprints dating mostly from the 1990s to the early 2020s.

[1] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In Proc. ACL, 2017a.

[2] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355, 2018.

[3] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc, 2009.

[4] Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. Bridging the lexical chasm: statistical approaches to answer-finding. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 192–199, 2000.

[5] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS, 2016.

[6] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.

[7] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021.

[8] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proc. ACL, 2019.

[9] Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932, 2020.

[10] Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering, 2021.

[11] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733–3742, 2018.

[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020.

[13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738, 2020.

[14] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294, 2021.

[15] Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan. Introduction to information retrieval. Cambridge university press, 2008.

[16] Bhaskar Mitra, Nick Craswell, et al. An introduction to neural information retrieval. Foundations and Trends® in Information Retrieval, 13(1):1–126, 2018.

[17] Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained transformers for text ranking: Bert and beyond. arXiv preprint arXiv:2010.06467, 2020.

[18] Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972.

[19] Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. Okapi at TREC-3. NIST Special Publication Sp, 1995.

[20] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391–407, 1990.

[21] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333–2338, 2013.

[22] Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd international conference on world wide web, pp. 373–374, 2014.

[23] Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(4):694–707, 2016.

[24] Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085, 2019.

[25] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019.

[26] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016.

[27] Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. End-to-end retrieval in continuous space. arXiv preprint arXiv:1811.08008, 2018.

[28] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020.

[29] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020.

[30] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019.

[31] Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969, 2019.

[32] Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert, 2020.

[33] Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering, 2020a. URL https://arxiv.org/abs/2012.04584.

[34] Sohee Yang and Minjoon Seo. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010.10999, 2020.

[35] Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. arXiv preprint arXiv:2005.00181, 2020.

[36] Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. A memory efficient baseline for open domain question answering. arXiv preprint arXiv:2012.15156, 2020.

[37] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013.

[38] Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557, 2017.

[39] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.

[40] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020.

[41] Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. arXiv preprint arXiv:2006.15020, 2020.

[42] Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766, 2020.

[43] John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659, 2020.

[44] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466, 2020.

[45] Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Coco-lm: Correcting and contrasting text sequences for language model pretraining. arXiv preprint arXiv:2102.08473, 2021.

[46] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.

[47] Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540, 2021.

[48] Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. Learning to retrieve passages without supervision, 2021. URL https://arxiv.org/abs/2112.07708.

[49] Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one?, 2021. URL https://arxiv.org/abs/2110.06918.

[50] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5835–5847, Online, June 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.naacl-main.466. URL https://aclanthology.org/2021.naacl-main.466.

[51] Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. On sampling strategies for neural network-based collaborative filtering. arXiv preprint arXiv:1706.07881, 2017b.

[52] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021.

[53] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, 2020.

[54] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. ACL, 2017.

[55] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020b.

[56] Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. A replication study of dense passage retriever, 2021.

[57] Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086, 2021.

[58] Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. arXiv preprint arXiv:2104.06967, 2021.

[59] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116, 2019. URL http://arxiv.org/abs/1911.02116.

[60] Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi-lingual benchmark for dense retrieval. CoRR, abs/2108.08787, 2021. URL https://arxiv.org/abs/2108.08787.

[61] Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi QA: A benchmark for information-seeking question answering in typologically diverse languages. CoRR, abs/2003.05002, 2020. URL https://arxiv.org/abs/2003.05002.

[62] Shayne Longpre, Yi Lu, and Joachim Daiber. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. CoRR, abs/2007.15207, 2020. URL https://arxiv.org/abs/2007.15207.

[63] Akari Asai, Xinyan Yu, Jungo Kasai, and Hannaneh Hajishirzi. One question answering model for many languages with cross-lingual dense passage retrieval. CoRR, abs/2107.11976, 2021. URL https://arxiv.org/abs/2107.11976.

[64] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

[65] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiv:2102.11600, 2021.