REPLUG: Retrieval-Augmented Black-Box Language Models

Weijia Shi$^{1}$ $^{*}$
University of Washington

Sewon Min$^{1}$
University of Washington

Michihiro Yasunaga$^{2}$
Stanford University

Minjoon Seo$^{3}$
KAIST

Rich James$^{4}$
Meta AI

Mike Lewis$^{4}$
Meta AI

Luke Zettlemoyer$^{1}$ $^{4}$
University of Washington
Meta AI

Wen-tau Yih$^{4}$
Meta AI

$^{1}$ University of Washington $^{2}$ Stanford University $^{3}$ KAIST $^{4}$ Meta AI
$^{*}$ Work done while the first author was interning at Meta AI.
Correspondence to: Weijia Shi [email protected].

Abstract

We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing retrieval and language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%.

Executive Summary: REPLUG is a retrieval-augmented framework that improves large black-box language models such as GPT-3 (175B) and Codex without any access to their internal parameters or the ability to fine-tune them. The work addresses a practical bottleneck: today’s strongest models are available only through APIs, yet they still hallucinate and miss long-tail knowledge. Prior retrieval methods required white-box access to train cross-attention or index internal states, which is impossible under API constraints.

The project set out to demonstrate that a simple “retrieve-and-plug” approach—pairing any frozen LM with an external retriever—can deliver measurable gains on language modeling and downstream tasks. The authors also tested whether the LM itself can supervise retriever training, producing an improved variant called REPLUG LSR.

REPLUG retrieves the top-10 documents from a Wikipedia or Pile-derived corpus with a dense retriever, feeds each document-plus-context pair to the LM in parallel, and ensembles the output probabilities. REPLUG LSR further tunes the retriever by minimizing the KL divergence between retrieval likelihoods and the LM’s own perplexity scores on the ground-truth continuation. Experiments covered language modeling on the Pile, 5-shot MMLU, and few-shot open-domain QA (Natural Questions and TriviaQA) using GPT-2, GPT-3, OPT, BLOOM, and Codex families up to 175B parameters.

The central results are consistent and material. Adding REPLUG lowers GPT-3 175B bits-per-byte on the Pile by 3.8 %; REPLUG LSR widens the gain to 6.3 %. On MMLU, REPLUG raises Codex accuracy by 4.5 % and LSR by 5.1 %, reaching parity with Flan-PaLM (540B) despite using a three-times-smaller base model. On open-domain QA, LSR lifts Codex by 12 % on Natural Questions and 5 % on TriviaQA in the 16-shot setting, surpassing prior few-shot retrieval-augmented systems. Gains appear across model sizes and families and are largest on rare entities, confirming that retrieval supplies genuinely new information rather than mere ensembling effects.

These improvements matter for organizations that rely on API-based LLMs: modest accuracy lifts and reduced hallucination risk are obtained without model retraining or extra infrastructure beyond a tunable retriever. The framework also shows that retrieval remains beneficial even at the current frontier scale, contrary to the assumption that parametric knowledge alone suffices.

For immediate deployment, teams should integrate a Contriever-style dense retriever with their existing API calls, using the parallel ensemble method and top-10 documents. Where labeled continuations are available, periodic LM-supervised retriever updates (REPLUG LSR) are worth the modest extra cost. Longer-term priorities include improving interpretability—specifically, surfacing whether the model is relying on retrieved versus parametric knowledge—and testing the approach on multimodal or code-generation workloads. The main limitations are the current lack of such interpretability signals and the restriction to English text corpora in the reported experiments; results should be validated on additional domains and languages before broad rollout.

1. Introduction

Section Summary: Large language models like GPT-3 often hallucinate or miss rare facts because they store knowledge only in their fixed parameters and cannot be easily updated or inspected when accessed through APIs. RePlug addresses this by treating the model as a black box and adding a separate, plug-in retriever that finds relevant documents from an external collection and prepends them to the input, with an optional training method that improves the retriever using the model’s own output scores. Experiments show this approach boosts accuracy on language modeling, question answering, and benchmarks such as MMLU for models up to 175 billion parameters without any changes to the underlying language model.

Large language models (LLMs) such as GPT-3 [1] and Codex [2], have demonstrated impressive performance on a wide range of language tasks. These models are typically trained on very large datasets and store a substantial amount of world or domain knowledge implicitly in their parameters. However, they are also prone to hallucination and cannot represent the full long tail of knowledge from the training corpus. Retrieval-augmented language models [3, 4, 5, 6], in contrast, can retrieve knowledge from an external datastore when needed, potentially reducing hallucination and increasing coverage. Previous approaches of retrieval-augmented language models require access to the internal LM representations (e.g., to train the model [4, 5] or to index the datastore [3]), and are thus difficult to be applied to very large LMs. In addition, many best-in-class LLMs can only be accessed through APIs. Internal representations of such models are not exposed and fine-tuning is not supported.

In this work, we introduce $\textsc{RePlug}$ (Retrieve and Plug), a new retrieval-augmented LM framework where the language model is viewed as a black box and the retrieval component is added as a tuneable plug-and-play module. Given an input context, $\textsc{RePlug}$ first retrieves relevant documents from an external corpus using an off-the-shelf retrieval model. The retrieved documents are prepended to the input context and fed into the black-box LM to make the final prediction. Because the LM context length limits the number of documents that can be prepended, we also introduce a new ensemble scheme that encodes the retrieved documents in parallel with the same black-box LM, allowing us to easily trade compute for accuracy. As shown in Figure 1, $\textsc{RePlug}$ is extremely flexible and can be used with any existing black-box LM and retrieval model.

We also introduce $\textsc{RePlug LSR}$ ($\textsc{RePlug}$ with LM-Supervised Retrieval), a training scheme that can further improve the initial retrieval model in $\textsc{RePlug}$ with supervision signals from a black-box language model. The key idea is to adapt the retriever to the LM, which is in contrast to prior work [4] that adapts language models to the retriever. We use a training objective which prefers retrieving documents that improve language model perplexity, while treating the LM as a frozen, black-box scoring function.

Our experiments show that $\textsc{RePlug}$ can improve the performance of diverse black-box LMs on both language modeling and downstream tasks, including MMLU [7] and open-domain QA [8, 9]. For instance, $\textsc{RePlug}$ can improve Codex (175B) performance on MMLU by 4.5%, achieving comparable results to the 540B, instruction-finetuned Flan-PaLM. Furthermore, tuning the retriever with our training scheme (i.e., $\textsc{RePlug LSR}$) leads to additional improvements, including up to 6.3% increase in GPT-3 175B language modeling. To the best of our knowledge, our work is the first to show the benefits of retrieval to large LMs (>100B model parameters), for both reducing LM perplexity and and improving in-context learning performance. We summarize our contributions as follows:

• We introduce $\textsc{RePlug}$ (§ 3), the first retrieval-augmented language modeling framework for enhancing large black-box language models with retrieval.
• We propose a training scheme (§ 4) to further adapt an off-the-shelf retrieval model to the LM, using the language modeling scores as supervision signals, resulting in improved retrieval quality.
• Evaluations on language modeling (§ 6), open-domain QA and MMLU demonstrate that $\textsc{RePlug}$ can improve the performance of various language models such as GPT, OPT and BLOOM, including very large models with up to 175B parameters.

2. Background and Related Work

Section Summary: Large language models such as GPT-3 are typically available only through black-box APIs, so users cannot inspect or change their internal parameters, and conventional fine-tuning becomes impractical at their scale. Earlier retrieval-augmented systems improve performance by pulling in external documents, yet most either require retraining the model or access to its hidden states, limiting their use with these inaccessible models. The present work therefore examines ways to add retrieved documents at inference time through input reformulation and probability ensembling, while also adapting the retriever itself, and applies the approach to language modeling and understanding tasks beyond question answering.

Black-box Language Models

Large language models (i.e., >100B), such as GPT-3 [1], Codex [2], and Yuan 1.0 [10], are not open-sourced due to commercial considerations and are only available as black-box APIs, through which users can send queries and receive responses. On the other hand, even open sourced language models such as OPT-175B [11] and BLOOM-176B [12] require significant computational resources to run and finetune locally. For example, finetuning BLOOM-176B requires 72 A100 GPUs (80GB memory, \15k each [13]), making them inaccessible to researchers and developers with limited resources. Traditionally, retrieval-augmented model frameworks [3, 4, 14, 5, 15] have focused on the white-box setting, where language models are fine-tuned to incorporate retrieved documents. However, the increasing scale and black-box nature of large language models makes this approach infeasible. To address the challenges posed by large language models, we investigate retrieval-augmentation in the black-box setting, where users only have access to the model predictions and cannot access or modify its parameters.

Retrieval-augmented Models

Augmenting language models with relevant information retrieved from various knowledge stores has shown to be effective in improving performance on various NLP tasks, including language modeling [16, 4, 3] and open-domain question answering [17, 5, 18]. Specifically, using the input as query, (1) a retriever first retrieves a set of documents (i.e., sequences of tokens) from a corpus and then (2) a language model incorporates the retrieved documents as additional information to make a final prediction. This style of retrieval can be added to both encoder-decoder [14, 5] and decoder-only models [3, 4, 19, 20]. For example, Atlas [5] finetunes an encoder-decoder model jointly with the retriever by modeling documents as latent variables, while RETRO [4] changes the decoder-only architecture to incorporate retrieved texts and pretrains the language model from scratch. Both methods require updating the model parameters through gradient descent, which cannot be applied to black-box LMs. Another line of retrieval-augmented LMs such as kNN-LM [3, 21] retrieves a set of tokens and interpolates between the LM's next token distribution and kNN distributions computed from the retrieved tokens at inference. Although kNN-LM does not require additional training, it requires access to internal LM representations to compute the kNN distribution, which are not always available for large LMs such as GPT-3. In this work, we investigate ways to improve large black-box language models with retrieval. While concurrent work [22, 23, 24, 25] has demonstrated that using a frozen retriever can improve GPT-3 performance on open-domain question answering, we approach the problem in a more general setting, including language modeling and understanding tasks. We also propose an ensemble method to incorporate more documents and a training scheme to further adapt the retriever to large LMs.

3. $\textsc{RePlug}$

Section Summary: RePlug is a method for improving a language model by adding an external retrieval step that pulls in useful documents from a large collection without changing the model itself. For a given input, it finds the top few relevant documents using a dense retriever based on embedding similarity, then feeds each document-plus-input pair to the model separately. The model’s predictions are finally combined as a weighted average of the individual probabilities, giving the language model access to extra information without exceeding its context limit.

We introduce $\textsc{RePlug}$ (Retrieve and Plug), a new retrieval-augmented LM paradigm where the language model is treated as black box and the retrieval component is added as a potentially tuneable module.

As shown in Figure 2, given an input context, $\textsc{RePlug}$ first retrieves a small set of relevant documents from an external corpus using a retriever (§ 3.1). Then we pass the concatenation of each retrieved document with the input context through the LM in parallel, and ensemble the predicted probabilities (§ 3.2).

3.1 Document Retrieval

Given an input context $x$, the retriever aims to retrieve a small set of documents from a corpus $\mathcal{D}={d_1...d_m}$ that are relevant to $x$. Following prior work [26, 27, 28], we use a dense retriever based on the dual encoder architecture, where an encoder is used to encode both the input context $x$ and the document $d$. Specifically, the encoder maps each document $d \in D$ to an embedding $\textbf{E}(d)$ by taking the mean pooling of the last hidden representation over the tokens in $d$. At query time, the same encoder is applied to the input context $x$ to obtain a query embedding $\textbf{E}(x)$. The similarity between the query embedding and the document embedding is computed by their cosine similarity:

$ s(d, x) = \cos (\textbf{E}(d), \textbf{E}(x))\tag{1} $

The top- $k$ documents that have the highest similarity scores when compared with the input $x$ are retrieved in this step. For efficient retrieval, we precompute the embedding of each document $d \in D$ and construct FAISS index [29] over these embeddings.

3.2 Input Reformulation

The retrieved top- $k$ documents provide rich information about the original input context $x$ and can potentially help the LM to make a better prediction. One simple way to incorporate the retrieved documents as part of the input to the LM is to prepend $x$ with all $k$ documents. However, this simple scheme is fundamentally restricted by the number of documents (i.e., $k$) we can include, given the language model's context window size. To address this limitation, we adopt an ensemble strategy described as follows. Assume $\mathcal{D}' \subset \mathcal{D}$ consists of $k$ most relevant documents to $x$, according to the scoring function in Equation 1. We prepend each document $d \in \mathcal{D}'$ to $x$, pass this concatenation to the LM separately, and then ensemble output probabilities from all $k$ passes. Formally, given the input context $x$ and its top- $k$ relevant documents $\mathcal{D}'$, the output probability of the next token $y$ is computed as a weighted average ensemble:

$ p(y \mid x, \mathcal{D}') = \sum_{d \in \mathcal{D}'} p(y \mid d \circ x) \cdot \lambda(d, x), $

where $\circ$ denotes the concatenation of two sequences and the weight $\lambda(d, x)$ is based on the similarity score between the document $d$ and the input context $x$:

$ \lambda(d, x) = \frac{e^{s(d, x)}}{\sum_{d \in \mathcal{D}'}e^{s(d, x)}} $

Although our ensemble method requires running the LM $k$ times, the cross attention is performed between each retrieved document and the input context. Therefore, compared with the method of prepending all the retrieved documents, our ensemble methods do not incur additional computational cost overhead.

4. $\textsc{RePlug LSR}$: Training the Dense Retriever

Section Summary: RePlug LSR is a method that improves a dense retriever by letting the language model itself indicate which documents are most useful, rather than relying on static retrieval models. It works by computing how likely each retrieved document is under the current retriever and then scoring the same documents according to how much they reduce the language model’s perplexity on the correct output; the retriever is then trained to align these two distributions by minimizing their KL divergence. Because the retriever changes during training, the document index is periodically rebuilt with fresh embeddings so that retrieval remains consistent.

Instead of relying only on existing neural dense retrieval models [30, 31, 32], we further propose $\textsc{RePlug LSR}$ ($\textsc{RePlug}$ with LM-Supervised Retrieval), which adapts the retriever in $\textsc{RePlug}$ by using the LM itself to provide supervision about which documents should be retrieved.

Inspired by [33], our approach can be seen as adjusting the probabilities of the retrieved documents to match the probabilities of the output sequence perplexities of the language model. In other words, we would like the retriever to find documents that result in lower perplexity scores. As shown in Figure 3, our training algorithm consists of the four steps: (1) retrieving documents and computing the retrieval likelihood (§ 4.1), (2) scoring the retrieved documents by the language model (§ 4.2), (3) updating the retrieval model parameters by minimizing the KL divergence between the retrieval likelihood and the LM's score distribution (§ 4.3), and (4) asynchronous update of the datastore index (§ 4.4).

4.1 Computing Retrieval Likelihood

We retrieve $k$ documents $\mathcal{D}' \subset \mathcal{D}$ with the highest similarity scores from a corpus $\mathcal{D}$ given an input context $x$, as described in § 3.1. We then compute the retrieval likelihood of each retrieved document $d$:

$ P_{R}(d \mid x) = \frac{e^{s(d, x)/\gamma}}{\sum_{d \in \mathcal{D}'}e^{s(d, x)/\gamma}} $

where $\gamma$ is a hyperparameter that controls the temerature of the softmax. Ideally, the retrieval likelihood is computed by marginalizing over all the documents in the corpus $\mathcal{D}$, which is intractable in practice. Therefore, we approximate the retrieval likelihood by only marginalizing over the retrieved documents $\mathcal{D}'$.

4.2 Computing LM likelihood

We use the LM as a scoring function to measure how much each document could improve the LM perplexity. Specifically, we first compute $P_{LM}(y \mid d, x)$, the LM probability of the ground truth output $y$ given the input context $x$ and a document $d$. The higher the probability, the better the document $d_i$ is at improving the LM's perplexity. We then compute the LM likelihood of each document $d$ as follows:

$ Q(d \mid x, y) = \frac{e^{P_{LM}(y \mid d, x)/\beta}}{\sum_{d \in \mathcal{D}'}e^{P_{LM}(y \mid d, x)/\beta}} $

where $\beta$ is another hyperparameter.

4.3 Loss Function

Given the input context $x$ and the corresponding ground truth continuation $y$, we compute the retrieval likelihood and the language model likelihood. The dense retriever is trained by minimizing the KL divergence between these two distributions:

$ \mathcal{L} = \frac{1}{|\mathcal{B}|} \sum_{x \in \mathcal{B}}{ KL \Big(P_{R}(d \mid x) \parallel Q_{\text{LM}}(d \mid x, y) \Big) }, $

where $\mathcal{B}$ is a set of input contexts. When minimizing the loss, we can only update the retrieval model parameters. The LM parameters are fixed due to our black-box assumption.

4.4 Asynchronous Update of the Datastore Index

Because the parameters in the retriever are updated during the training process, the previously computed document embeddings are no longer up to date. Therefore, following [34], we recompute the document embeddings and rebuild the efficient search index using the new embeddings every $T$ training steps. Then we use the new document embeddings and index for retrieval, and repeat the training procedure.

5. Training Setup

Section Summary: The training setup begins by selecting the Contriever retrieval model for the basic RePlug approach, while RePlug LSR additionally uses GPT-3 Curie to guide updates to the retriever through likelihood comparisons. Training draws on 800,000 sequences from the Pile dataset, each split into a context prefix and a continuation target, paired with a separate collection of 36 million non-overlapping documents that serve as the retrieval pool. Efficiency comes from pre-building a FAISS index of document embeddings, retrieving the top 20 matches per query, and then optimizing the retriever with Adam for 25,000 steps while periodically refreshing the index.

In this section, we describe the details of our training procedure. We first describe the model setting in $\textsc{RePlug}$ (§ 5.1) and then describe the procedure for training the retriever in $\textsc{RePlug LSR}$ (§ 5.2).

5.1 $\textsc{RePlug}$

In theory, any type of retriever, either dense [35, 28] or sparse [36], could be used for $\textsc{RePlug}$. Following prior work [5], we use the Contriever [31] as the retrieval model for $\textsc{RePlug}$, as it has demonstrated strong performance.

5.2 $\textsc{RePlug LSR}$

For $\textsc{RePlug LSR}$, we initialize the retriever with the Contriever model [31]. We use GPT-3 Curie [37] as the supervision LM to compute the LM likelihood.

Training data

We use 800K sequences of 256 tokens each, sampled from the Pile training data [38], as our training queries. Each query is split into two parts: the first 128 tokens are used as the input context $x$, and the last 128 tokens are used as the ground truth continuation $y$. For the external corpus $D$, we sample 36M documents of 128 tokens from the Pile training data. To avoid trivial retrieval, we ensure that the external corpus documents do not overlap with the documents from which the training queries are sampled.

Training details

To make the training process more efficient, we pre-compute the document embeddings of the external corpus $D$ and create a FAISS index [29] for fast similarity search. Given a query $x$, we retrieve the top 20 documents from the FAISS index and compute the retrieval likelihood and the LM likelihood with a temperature of 0.1. We train the retriever using the Adam optimizer [39] with a learning rate of 2e-5, a batch size of 64, and a warmup ratio of 0.1. We re-compute the document embeddings every 3k steps and fine-tune the retriever for a total of 25k steps.

\begin{tabular}{llcc|cc|cc}
\toprule
Model & & # Parameters & Original & + \textsc{RePlug} & Gain \% & + \textsc{RePlug LSR} & Gain \% \\
\midrule
GPT-2 & Small & 117M & 1.33 & 1.26 & 5.3 & 1.21 & 9.0 \\
& Medium & 345M & 1.20 & 1.14 & 5.0 & 1.11 & 7.5 \\
& Large & 774M & 1.19 & 1.15 & 3.4 & 1.09 & 8.4 \\
& XL & 1.5B & 1.16 & 1.09 & 6.0 & 1.07 & 7.8 \\
\midrule
GPT-3 & Ada & 350M & 1.05 & 0.98 & 6.7 & 0.96 & 8.6 \\
(black-box) & Babbage & 1.3B & 0.95 & 0.90 & 5.3 & 0.88 & 7.4 \\
& Curie & 6.7B & 0.88 & 0.85 & 3.4 & 0.82 & 6.8 \\
& Davinci & 175B & 0.80 & 0.77 & 3.8 & 0.75 & 6.3 \\
\bottomrule
\end{tabular}

6. Experiments

Section Summary: RePlug was tested on language modeling using The Pile benchmark as well as on the MMLU knowledge test and open-domain question answering datasets like Natural Questions and TriviaQA. In each case, adding the retrieval approach to existing black-box models such as GPT-3, Codex, and GPT-2 variants produced consistent gains, with the learned retriever variant (RePlug LSR) yielding larger improvements than the basic version. The gains held across model sizes, allowed the systems to surpass some much larger models, and demonstrated that retrieval can be added effectively without retraining the underlying language model.

We perform evaluations on both language modeling (§ 6.1) and downstream tasks such as MMLU (§ 6.2) and open-domain QA (§ 6.3). In all settings, $\textsc{RePlug}$ improve the performance of various black-box language models, showing the effectiveness and generality of our approach.

6.1 Language Modeling

Datasets

The Pile [38] is a language modeling benchmark that consists of text sources from diverse domains such as web pages, code and academic papers. Following prior work, we report bits per UTF-8 encoded byte (BPB) as the metric on each subset domain.

Baselines

We consider GPT-3 and GPT-2 family language model as the baselines. The four models from GPT-3 (Davinci, Curie, Baddage and Ada) are black-box models that are only accessible through API

Our model

We add $\textsc{RePlug}$ and $\textsc{RePlug LSR}$ to the baselines. We randomly subsampled Pile training data (367M documents of 128 tokens) and use them as the retrieval corpus for all models. As the Pile dataset has made efforts to deduplicate documents across train, validation and test splits [38], we did not do additional filtering. For both $\textsc{RePlug}$ and $\textsc{RePlug LSR}$, we use a length of 128-token context to do retrieval and adopt the ensemble method (Section 3.2) to incorporate top 10 retrieved documents during inference.

Results

Table 1 reports the results of the original baselines, baselines augmented with the $\textsc{RePlug}$, and baselines augmented with the $\textsc{RePlug LSR}$. We observe that both $\textsc{RePlug}$ and $\textsc{RePlug LSR}$ significantly outperform the baselines. This demonstrates that simply adding a retrieval module to a frozen language model (i.e., the black-box setting) is effective at improving the performance of different sized language models on language modeling tasks. Furthermore, $\textsc{RePlug LSR}$ consistently performs better than $\textsc{RePlug}$ by a large margin. Specifically, $\textsc{RePlug LSR}$ results in 7.7% improvement over baselines compared to 4.7% improvement of $\textsc{RePlug}$ averaged over the 8 models. This indicates that further adapting the retriever to the target LM is beneficial.

6.2 MMLU

\begin{tabular}{llcccc|c}
\toprule
Model & # Parameters & Humanities & Social. & STEM & Other & All \\
\midrule
Codex & 175B & 74.2 & 76.9 & 57.8 & 70.1 & 68.3 \\
PaLM & 540B & 77.0 & 81.0 & 55.6 & 69.6 & 69.3 \\
Flan-PaLM & 540B & - & - & - & - & 72.2 \\
\midrule
Atlas & 11B & 46.1 & 54.6 & 38.8 & 52.8 & 47.9 \\
\midrule
Codex + \textsc{RePlug} & 175B & 76.0 & 79.7 & 58.8 & 72.1 & 71.4 \\
Codex + \textsc{RePlug LSR} & 175B & 76.5 & 79.9 & 58.9 & 73.2 & 71.8 \\
\bottomrule
\end{tabular}

Datasets

Massive Multi-task Language Understanding (MMLU [7]) is a multiple choice QA dataset that covers exam questions from 57 tasks including mathematics, computer science, law, US history and etc. The 57 tasks are grouped into 4 categories: humanities, STEM, social sciences and other. Following [40], we evaluate $\textsc{RePlug}$ in the 5-shot in-context learning setting.

Baselines

We consider two groups of strong previous models as baselines for comparisons. The first group of baselines is the state-of-the-art LLMs including Codex^1 [41], PaLM [42], and Flan-PaLM [43]. According to [43], these three models rank top-3 in the leaderboard of MMLU. The second group of baselines consists of retrieval-augmented language models. We only include Atlas [5] in this group, as no other retrieval-augmented LMs have been evaluated on the MMLU dataset. Atlas trains both the retriever and the language model, which we consider a white-box retrieval LM setting.

Our model

We add $\textsc{RePlug}$ and $\textsc{RePlug LSR}$ only to Codex because other models such as PaLM and Flan-PaLM are not accessible to the public. We use the test question as the query to retrieve 10 relevant documents from Wikipedia (2018, December) and prepend each retrieved document to the test question, resulting in 10 separate inputs. These inputs are then separately fed into the language models, and the output probabilities are ensemble together.

Results

Table 2 presents the results from the baselines, $\textsc{RePlug}$, and $\textsc{RePlug LSR}$ on the MMLU dataset. We observe that both the $\textsc{RePlug}$ and $\textsc{RePlug LSR}$ improve the original Codex model by 4.5% and 5.1%, respectively. In addition, $\textsc{RePlug LSR}$ largely outperforms the previous retrieval-augmented language model, Atlas, demonstrating the effectiveness of our black-box retrieval language model setting. Although our models slightly underperform Flan-PaLM, this is still a strong result because Flan-PaLM has three times more parameters. We would expect that the $\textsc{RePlug LSR}$ could further improve Flan-PaLM, if we had access to the model.

Another interesting observation is that the $\textsc{RePlug LSR}$ outperforms the original model by 1.9% even in the STEM category. This suggests that retrieval may improve a language model's problem-solving abilities.

6.3 Open Domain QA

Lastly, we conduct evaluation on two open-domain QA datasets: Natural Questions (NQ) [8] and TriviaQA [9].

\begin{tabular}{lcccc}
\toprule
  & \multicolumn{2}{c}{NQ} & \multicolumn{2}{c}{TQA} \\
\cmidrule(lr){2-3}
\cmidrule(lr){4-5}
Model & Few-shot & Full & Few-shot & Full \\
\midrule
Chinchilla & 35.5 & - & 64.6 & - \\
PaLM & 39.6 & - & - & - \\
Codex & 40.6 & - & 73.6 & - \\
\midrule
RETRO$^\dag$ & - & 45.5 & - & - \\
R2-D2$^\dag$ & - & 55.9 & - & 69.9 \\
Atlas$^\dag$ & 42.4 & \textbf{60.4} & 74.5 & \textbf{79.8} \\
Codex + Contriever$_{cc}$\tablefootnote{[44] augment Codex with concatenation of 10 documents retrieved by contriever.} & 44.2 & - & 76.0 & - \\
\midrule
Codex + \textsc{RePlug} & 44.7 & - & 76.8 & - \\
Codex + \textsc{RePlug LSR} & \textbf{45.5} & - & \textbf{77.3} & - \\
\bottomrule
\end{tabular}

Datasets

NQ and TriviaQA are two open-domain QA datasets consisting of questions, answers collected from Wikipedia and the Web. Following prior work [45, 44], we report results for the filtered set of TriviaQA. For evaluation, we consider the few-shot setting where the model is only given a few training examples and full data where the model is given all the training examples.

Baselines

We compare our model with several state-of-the-art baselines, both in a few-shot setting and with full training data. The first group of models consists of powerful large language models, including Chinchilla [46], PaLM [42], and Codex. These models are all evaluated using in-context learning under the few-shot setting, with Chinchilla and PaLM evaluated using 64 shots, and Codex using 16 shots. The second group of models for comparison includes retrieval-augmented language models such as RETRO [47], R2-D2 [48], and Atlas [5]. All of these retrieval-augmented models are finetuned on the training data, either in a few-shot setting or with full training data. Specifically, Atlas is finetuned on 64 examples in the few-shot setting.

Our model

We add $\textsc{RePlug}$ and $\textsc{RePlug LSR}$ to Codex with Wikipedia (2018, December) as the retrieval corpus to evaluate the model in a 16-shot in context learning. Similar to the setting in language modeling and MMLU, we incorporate top-10 retrieved documents using our proposed ensemble method.

Results

As shown in Table 3, $\textsc{RePlug LSR}$ significantly improves the performance of the original Codex by 12.0% on NQ and 5.0% on TQA. It outperforms the previous best model, Atlas, which was fine-tuned with 64 training examples, achieving a new state-of-the-art in the few-shot setting. However, this result still lags behind the performance of retrieval-augmented language models fine-tuned on the full training data. This is likely due to the presence of near-duplicate test questions in the training set (e.g., [49] found that 32.5% of test questions overlap with the training sets in NQ).

7. Analysis

Section Summary: The analysis of RePlug shows that its gains come from retrieving and combining genuinely relevant documents rather than from the mere effect of averaging multiple model runs, since using random documents actually harms performance. The approach delivers steady improvements across many different language model families and sizes, from small GPT-2 variants to large OPT and BLOOM models. It is especially helpful for passages that mention rare entities, because the retrieved documents supply background information the original model lacks.

7.1 $\textsc{RePlug}$ performance gain does not simply come from the ensembling effect

The core of our method design is the use of an ensemble method that combines output probabilities of different passes, in which each retrieved document is prepended separately to the input and fed into a language model. To study whether the gains come solely from the ensemble method, we compare our method to ensembling random documents. For this, we randomly sample several documents, concatenated each random document with the input, and ensemble the outputs of different runs (referred to as "random"). As shown in Figure 4, we evaluated the performance of GPT-3 Curie on Pile when augmented with random documents, documents retrieved by $\textsc{RePlug}$, and documents retrieved by $\textsc{RePlug LSR}$. We observed that ensembling random documents leads to worse performance, indicating that the performance gains of $\textsc{RePlug}$ do not solely come from the ensembling effect. Instead, ensembling the relevant documents is crucial for the success of $\textsc{RePlug}$. Additionally, as more documents were ensembled, the performance of $\textsc{RePlug}$ and $\textsc{RePlug LSR}$ improved monotonically. However, a small number of documents (e.g., 10) was sufficient to achieve large performance gains.

7.2 $\textsc{RePlug}$ is applicable to diverse language models

Here we further study whether $\textsc{RePlug}$ could enhance diverse language model families that have been pre-trained using different data and methods. Specifically, we focus on three groups of language models with varying sizes: GPT-2 (117M, 345M, 774M, 1.5B parameters) [1], OPT (125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B) [50] and BLOOM (560M, 1.1B, 1.7B, 3B and 7B) [12]. We evaluate each model on Wikitext-103 [51] test data and report its perplexity. For comparison, we augment each model with $\textsc{RePlug}$ that adopts the ensemble method to incorporate top 10 retrieved documents. Following prior work [3], we use Wikitext-103 training data as the retrieval corpus.

Figure 5 shows the performance of different-sized language models with and without $\textsc{RePlug}$. We observe that the performance gain brought by $\textsc{RePlug}$ stays consistent with model size. For example, OPT with 125M parameters achieves 6.9% perplexity improvement, while OPT with 66B parameters achieves 5.6% perplexity improvement. Additionally, $\textsc{RePlug}$ improves the perplexity of all the model families. This indicates that $\textsc{RePlug}$ is applicable to diverse language models with different sizes.

7.3 Qualitative Analysis: rare entities benefit from retrieval

To understand why the $\textsc{RePlug}$ improves language modeling performance, we conducted manual analysis of examples in which the $\textsc{RePlug}$ results in a decrease in perplexity. We find that $\textsc{RePlug}$ is more helpful when texts contain rare entities. Figure 4 shows a test context and its continuation from the Wikitext-103 test set. For $\textsc{RePlug}$, we use the test context as a query to retrieve a relevant document from Wikitext-103 training data. We then compute the perplexity of the continuation using the original GPT-2 1.5B and its $\textsc{RePlug}$ enhanced version. After incorporating the retrieved document, the perplexity of the continuation improves by 11%. Among all tokens in the continuation, we found that $\textsc{RePlug}$ is most helpful for the rare entity name "Li Bai". This is likely because the original LM does not have sufficient information about this rare entity name. However, by incorporating the retrieved document, $\textsc{RePlug}$ was able to match the name with the relevant information in the retrieved document, resulting in better performance.

8. Conclusion

Section Summary: RePlug is a new method that improves any existing language model by adding a separately adjustable retrieval system, all without changing the original model itself. Experiments show this boosts results on both general language tasks and specific applications, even for today's most advanced models. One remaining drawback is that it is hard to tell when the system is using outside information versus its own stored knowledge, pointing to a need for more transparent designs in future work.

We introduce $\textsc{RePlug}$, a retrieval-augmented language modeling paradigm that treats the language model as a black box and augments it with a tuneable retrieval model. Our evaluation shows that $\textsc{RePlug}$ can be integrated with any existing language model to improve their performance on language modeling or downstream tasks. This work opens up new possibilities for integrating retrieval into large-scale black-box language models and demonstrates even the state-of-the-art large-scale LMs could benefit from retrieval. However, $\textsc{RePlug}$ lacks interpretability as it is unclear when the model relies on retrieved knowledge or parametric knowledge. Future research could focus on developing more interpretable retrieval-augmented language models.

Appendix

\begin{tabularx}{\ccccccccc}{m{17cm}}
\toprule
\textbf{Knowledge}: Arctic Ocean. Although over half of Europe's original forests disappeared through the centuries of deforestation, Europe still has over one quarter of its land area as forest, such as the broadleaf and mixed forests, taiga of Scandinavia and Russia, mixed rainforests of the Caucasus and the Cork oak forests in the western Mediterranean. During recent times, deforestation has been slowed and many trees have been planted. However, in many cases monoculture plantations of conifers have replaced the original mixed natural forest, because these grow quicker. The plantations now cover vast areas of land, but offer poorer habitats for many European \\ 
\textbf{Question}: As of 2015, since 1990 forests have \rule{1cm}{0.15mm}in Europe and have \rule{1cm}{0.15mm}in Africa and the Americas. \\
A. ``increased, increased''
B. ``increased, decreased''
C. ``decreased, increased''
D. ``decreased, decreased'' \\
\textbf{Answer}: B \\
\\
\textbf{Knowledge}: Over the past decades, the political outlook of Americans has become more progressive, with those below the age of thirty being considerably more liberal than the overall population. According to recent polls, 56\% of those age 18 to 29 favor gay marriage, 68\% state environmental protection to be as important as job creation, 52\% ``think immigrants \'strengthen the country with their hard work and talents, \''' 62\% favor a ``tax financed, government-administrated universal health care'' program and 74\% ``say \'people\'s will\' should have more influence on U.S. laws than the Bible, compared to 37\%, 49\%, 38\%, 47\% and 58\% among the \\ 
\textbf{Question}: As of 2019, about what percentage of Americans agree that the state is run for the benefit of all the people? \\
A. 31\%
B. 46\%
C. 61\%
D. 76\% \\
\textbf{Answer}: B \\
... \\
\textbf{Knowledge}: last week at a United Nations climate meeting in Germany, China and India should easily exceed the targets they set for themselves in the 2015 Paris Agreement... India is now expected to obtain 40 percent of its electricity from non-fossil fuel sources by 2022, eight years ahead of schedule.'' Solar power in Japan has been expanding since the late 1990s. By the end of 2017, cumulative installed PV capacity reached over 50 GW with nearly 8 GW installed in the year 2017. The country is a leading manufacturer of solar panels and is in the top 4 ranking for countries \\
\textbf{Question}: Which of the following countries generated the most total energy from solar sources in 2019? \\
A. China
B. United States
C. Germany
D. Japan
 \\
\bottomrule
\end{tabularx}

\begin{tabularx}{\ccccccccc}{m{17cm}}
\toprule
\textbf{Knowledge}: received 122, 000 buys (excluding WWE Network views), down from the previous year\'s 199, 000 buys. The event is named after the Money In The Bank ladder match, in which multiple wrestlers use ladders to retrieve a briefcase hanging above the ring. The winner is guaranteed a match for the WWE World Heavyweight Championship at a time of their choosing within the next year. On the June 2 episode of ``Raw'', Alberto Del Rio qualified for the match by defeating Dolph Ziggler. The following week, following Daniel Bryan being stripped of his WWE World Championship due to injury, Stephanie McMahon changed the \\ 
\textbf{Question}: Who won the mens money in the bank match? \\
\textbf{Answer}: Braun Strowman \\
\\
\textbf{Knowledge}: in 3D on March 17, 2017. The first official presentation of the film took place at Disney\'s three-day D23 Expo in August 2015. The world premiere of ``Beauty and the Beast'' took place at Spencer House in London, England on February 23, 2017; and the film later premiered at the El Capitan Theatre in Hollywood, California, on March 2, 2017. The stream was broadcast onto YouTube. A sing along version of the film released in over 1, 200 US theaters nationwide on April 7, 2017. The United Kingdom received the same version on April 21, 2017. The film was re-released in \\ 
\textbf{Question}: When does beaty and the beast take place \\\
\textbf{Answer}: Rococo-era \\
... \\
\textbf{Knowledge}: Love Yourself ``Love Yourself'' is a song recorded by Canadian singer Justin Bieber for his fourth studio album ``Purpose'' (2015). The song was released first as a promotional single on November 8, 2015, and later was released as the album\'s third single. It was written by Ed Sheeran, Benny Blanco and Bieber, and produced by Blanco. An acoustic pop song, ``Love Yourself'' features an electric guitar and a brief flurry of trumpets as its main instrumentation. During the song, Bieber uses a husky tone in the lower registers. Lyrically, the song is a kiss-off to a narcissistic ex-lover who did \\
\textbf{Question}: love yourself by justin bieber is about who \\
 \\
\bottomrule
\end{tabularx}

References

Section Summary: This section compiles a numbered list of academic citations, primarily from recent machine learning conferences and preprint servers. The entries cover foundational work on large language models, retrieval-augmented techniques, code evaluation, and standard benchmarks for question answering and language understanding. They also include references to model training methods, quantization approaches, and related empirical studies.

[1] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[2] Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021a. URL https://arxiv.org/abs/2107.03374.

[3] Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HklBjCEKvH.

[4] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, pp. 2206–2240. PMLR, 2022.

[5] Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022b.

[6] Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2211.12561, 2022.

[7] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ.

[8] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi:10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026.

[9] Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:10.18653/v1/P17-1147. URL https://aclanthology.org/P17-1147.

[10] Wu, S., Zhao, X., Yu, T., Zhang, R., Shen, C., Liu, H., Li, F., Zhu, H., Luo, J., Xu, L., et al. Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning. arXiv preprint arXiv:2110.04725, 2021.

[11] Zhang, S., Diab, M., and Zettlemoyer, L. Democratizing access to large-scale language models with opt-175b. Meta AI, 2022a.

[12] Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.

[13] Younes Belkda, T. D. A gentle introduction to 8-bit matrix multiplication, 2022. URL https://huggingface.co/blog/hf-bitsandbytes-integration.

[14] Yu, W. Retrieval-augmented generation across heterogeneous knowledge. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pp. 52–58, Hybrid: Seattle, Washington + Online, July 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.naacl-srw.7. URL https://aclanthology.org/2022.naacl-srw.7.

[15] Goyal, A., Friesen, A., Banino, A., Weber, T., Ke, N. R., Badia, A. P., Guez, A., Mirza, M., Humphreys, P. C., Konyushova, K., et al. Retrieval-augmented reinforcement learning. In International Conference on Machine Learning, pp. 7740–7765. PMLR, 2022.

[16] Min, S., Shi, W., Lewis, M., Chen, X., Yih, W.-t., Hajishirzi, H., and Zettlemoyer, L. Nonparametric masked language modeling. arXiv preprint arXiv:2212.01349, 2022.

[17] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2020. URL https://arxiv.org/abs/2005.11401.

[18] Hu, Y., Hua, H., Yang, Z., Shi, W., Smith, N. A., and Luo, J. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699, 2022.

[19] Shi, W., Michael, J., Gururangan, S., and Zettlemoyer, L. Nearest neighbor zero-shot inference. 2022.

[20] Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2671, 2022.

[21] Zhong, Z., Lei, T., and Chen, D. Training language models with memory augmentation. In Empirical Methods in Natural Language Processing (EMNLP), 2022.

[22] Mallen, A., Asai, A., Zhong, V., Das, R., Hajishirzi, H., and Khashabi, D. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022.

[23] Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., and Wang, L. Prompting gpt-3 to be reliable. In Proc. of ICLR, 2023. URL https://openreview.net/forum?id=98p5x51L5af.

[24] Yu, W., Iter, D., Wang, S., Xu, Y., Ju, M., Sanyal, S., Zhu, C., Zeng, M., and Jiang, M. Generate rather than retrieve: Large language models are strong context generators. 2023.

[25] Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., and Zaharia, M. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024, 2022.

[26] Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, W. X., Dong, D., Wu, H., and Wang, H. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5835–5847, Online, June 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.naacl-main.466. URL https://aclanthology.org/2021.naacl-main.466.

[27] Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In Proc. of EACL, 2021b. URL https://arxiv.org/abs/2007.01282.

[28] Ni, J., Qu, C., Lu, J., Dai, Z., Ábrego, G. H., Ma, J., Zhao, V. Y., Luan, Y., Hall, K. B., Chang, M., and Yang, Y. Large dual encoders are generalizable retrievers, 2021. URL https://arxiv.org/abs/2112.07899.

[29] Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547, 2019.

[30] Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781, Online, November 2020a. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.550. URL https://aclanthology.org/2020.emnlp-main.550.

[31] Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022a. URL https://openreview.net/forum?id=jKN1pXi7b0.

[32] Su, H., Kasai, J., Wang, Y., Hu, Y., Ostendorf, M., Yih, W.-t., Smith, N. A., Zettlemoyer, L., Yu, T., et al. One embedder, any task: Instruction-finetuned text embeddings. arXiv preprint arXiv:2212.09741, 2022.

[33] Sachan, D. S., Lewis, M., Yogatama, D., Zettlemoyer, L., Pineau, J., and Zaheer, M. Questions are all you need to train a dense passage retriever. arXiv preprint arXiv:2206.10658, 2022.

[34] Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pp. 3929–3938. PMLR, 2020.

[35] Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781, 2020b.

[36] Robertson, S., Zaragoza, H., et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009.

[37] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Proc. of NeurIPS, 2020b. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[38] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.

[39] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.

[40] Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022a.

[41] Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021b. URL https://arxiv.org/abs/2107.03374.

[42] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.

[43] Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022b.

[44] Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., and Wang, L. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022.

[45] Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874–880, Online, April 2021a. Association for Computational Linguistics. doi:10.18653/v1/2021.eacl-main.74. URL https://aclanthology.org/2021.eacl-main.74.

[46] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.

[47] Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. v. d., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. arXiv preprint arXiv:2112.04426, 2021.

[48] Fajcik, M., Docekal, M., Ondrej, K., and Smrz, P. R2-D2: A modular baseline for open-domain question answering. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 854–870, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.findings-emnlp.73. URL https://aclanthology.org/2021.findings-emnlp.73.

[49] Lewis, P., Stenetorp, P., and Riedel, S. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1000–1008, 2021.

[50] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022b.

[51] Stephen, M., Caiming, X., James, B., and Socher, R. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.