Active Retrieval Augmented Generation

Zhengbao Jiang $^{1}$ $^{}$ Frank F. Xu $^{1}$ $^{}$ Luyu Gao $^{1}$ $^{}$ Zhiqing Sun $^{1}$ $^{}$ Qian Liu $^{2}$
Jane Dwivedi-Yu $^{3}$ Yiming Yang $^{1}$ Jamie Callan $^{1}$ Graham Neubig $^{1}$
$^{1}$ Language Technologies Institute, Carnegie Mellon University
$^{2}$ Sea AI Lab $^{3}$ FAIR, Meta
{zhengbaj, fangzhex, luyug, zhiqings, gneubig}@cs.cmu.edu

$^{*}$ Lead contributors.
$^{1}$ Code and datasets are available at https://github.com/zbjyb/FLARE.

Abstract

Despite the remarkable ability of large language models (LMs) to comprehend and generate language, they have a tendency to hallucinate and create factually inaccurate output. Augmenting LMs by retrieving information from external knowledge resources is one promising solution. Most existing retrieval augmented LMs employ a retrieve-and-generate setup that only retrieves information once based on the input. This is limiting, however, in more general scenarios involving generation of long texts, where continually gathering information throughout generation is essential. In this work, we provide a generalized view of active retrieval augmented generation, methods that actively decide when and what to retrieve across the course of the generation. We propose Forward-Looking Active REtrieval augmented generation (FLARE), a generic method which iteratively uses a prediction of the upcoming sentence to anticipate future content, which is then utilized as a query to retrieve relevant documents to regenerate the sentence if it contains low-confidence tokens. We test FLARE along with baselines comprehensively over 4 long-form knowledge-intensive generation tasks/datasets. FLARE achieves superior or competitive performance on all tasks, demonstrating the effectiveness of our method.$^{1}$

Executive Summary: Large language models (LMs) such as GPT-3.5 generate fluent text yet frequently produce factually incorrect or invented content. For short, focused questions this problem can be reduced by retrieving external documents once at the start. For longer outputs—detailed answers, multi-step reasoning, or aspect-based summaries—the information needed is rarely known in advance, so one-time retrieval often leaves gaps or inserts irrelevant material.

The paper therefore set out to test whether an LM can decide for itself when and what additional information to fetch during generation, without any extra training. The authors introduced a single generic method, Forward-Looking Active REtrieval augmented generation (FLARE), that works with existing models at inference time. At each step the system first drafts the next sentence without retrieval. If any token in that draft has low probability (indicating uncertainty), it uses the draft itself—or a masked or question-rewritten version of it—as a search query, fetches fresh documents, and regenerates the sentence. Retrieval therefore occurs only when the model signals it is needed and the query reflects what the model intends to say next.

The method was evaluated on four long-form tasks (2WikiMultihopQA, StrategyQA, ASQA, and WikiAsp) using the same GPT-3.5 model and off-the-shelf retrievers. Across all tasks FLARE matched or exceeded every baseline. The largest gain appeared on the two-hop QA task, where exact-match accuracy rose from 47.8 % (strongest baseline) to 51.0 %. Forward-looking queries outperformed both “use the previous sentence” and “retrieve every fixed window” strategies; active triggering based on confidence proved more efficient than retrieving at every sentence. Masking low-confidence tokens or turning them into explicit questions further improved retrieval quality.

These results indicate that LMs already possess useful internal signals for when they lack knowledge, and that planning one step ahead is sufficient to produce targeted retrievals. Consequently, long-form generation can be made more factual and coherent with modest added cost and no model changes. Stakeholders who rely on LMs for reports, explanations, or decision support can therefore obtain higher reliability without waiting for larger models or costly fine-tuning.

The authors recommend deploying FLARE (or similar active-retrieval loops) wherever outputs exceed a few sentences, starting with a modest set of tasks that reward factuality. They also flag two practical limits: the approach adds repeated API calls and latency, and it yielded little benefit on short dialogue or highly open-ended tasks where grounding criteria remain unclear. A next step would therefore be to integrate retrieval into the underlying architecture so that repeated forward passes are no longer required, while collecting more data on domains where active retrieval should or should not be triggered.

1. Introduction

Section Summary: Generative language models often produce plausible but incorrect information because they lack up-to-date or detailed facts, even though they have absorbed some knowledge during training. Standard retrieval methods fetch outside documents only once at the beginning based on the user’s initial query, which works for short answers but falls short for longer, more complex tasks such as multi-step reasoning or detailed summaries where the needed information only becomes clear as writing unfolds. The paper therefore introduces FLARE, a simple method that lets the model decide on its own when it is uncertain and then uses a provisional next sentence to fetch the most relevant facts before continuing, improving performance across several long-form generation benchmarks without any extra training.

Generative language models (LMs) [1, 2, 3, 4, 5, 6, 7] have become a foundational component in natural language processing (NLP) systems with their remarkable abilities. Although LMs have memorized some world knowledge during training [8, 9, 10], they still tend to hallucinate and create imaginary content [11, 12]. Augmenting LMs with retrieval components that look up relevant information from external knowledge resources is a promising direction to address hallucination [13, 14].

Retrieval augmented LMs commonly use a retrieve-and-generate setup where they retrieve documents based on the user's input, and then generate a complete answer conditioning on the retrieved documents [15, 16, 17, 18, 19, 20, 21, 14, 22, 23, 24, 25]. These single-time retrieval augmented LMs outperform purely parametric LMs, particularly for short-form knowledge-intensive generation tasks such as factoid question answering (QA) [26, 27], where the information needs are clear in the user's input, and it is sufficient to retrieve relevant knowledge once solely based on the input.

Increasingly powerful large LMs have also demonstrated abilities in more complex tasks that involve generating long-form output, such as long-form QA [28, 29], open-domain summarization [30, 31, 32], and (chain-of-thought; CoT) reasoning [33, 34, 35, 36]. In contrast to short-form generation, long-form generation presents complex information needs that are not always evident from the input alone. Similar to how humans gradually gather information as we create content such as papers, essays, or books, long-form generation with LMs would require gathering multiple pieces of knowledge throughout the generation process. For example, to generate a summary about a particular topic, the initial retrieval based on the topic name (e.g., Joe Biden) may not cover all aspects and details. It is crucial to retrieve extra information as needed during generation, such as when generating a certain aspect (e.g., Joe Biden's education history) or a specific detail (e.g., the date of Joe Biden's presidential campaign announcement).

Several attempts have been made to retrieve multiple times throughout generation. These attempts include methods that passively use the past context to retrieve additional information at a fixed interval [13, 37, 38, 39] which might not accurately reflect what LMs intend to generate in the future or retrieve at inappropriate points. Some works in multihop QA decompose the full question into sub-questions, each of which is used to retrieve extra information [40, 41, 42, 43].

We ask the following question: can we create a simple and generic retrieval augmented LM that actively decides when and what to retrieve throughout the generation process, and are applicable to a variety of long-form generation tasks? We provide a generalized view of active retrieval augmented generation. Our hypothesis regarding when to retrieve is that LMs should retrieve information only when they lack the required knowledge to avoid unnecessary or inappropriate retrieval that occurs in passive retrieval augmented LMs [13, 37, 38, 39]. Given the observation that large LMs tend to be well-calibrated and low probability/confidence often indicates a lack of knowledge [44], we adopt an active retrieval strategy that only retrieves when LMs generate low-probability tokens. When deciding what to retrieve, it is important to consider what LMs intend to generate in the future, as the goal of active retrieval is to benefit future generations. Therefore, we propose anticipating the future by generating a temporary next sentence, using it as a query to retrieve relevant documents, and then regenerating the next sentence conditioning on the retrieved documents. Combining the two aspects, we propose Forward-Looking Active REtrieval augmented generation (FLARE), as illustrated in Figure 1. FLARE iteratively generates a temporary next sentence, use it as the query to retrieve relevant documents if it contains low-probability tokens and regenerate the next sentence until reaches the end.

FLARE is applicable to any existing LMs at inference time without additional training. Considering the impressive performance achieved by GPT-3.5 [2] on a variety of tasks, we examine the effectiveness of our methods on text-davinci-003. We evaluate FLARE on 4 diverse tasks/datasets involving generating long outputs, including multihop QA (2WikiMultihopQA), commonsense reasoning (StrategyQA), long-form QA (ASQA), and open-domain summarization (WikiAsp) [34, 35, 29, 31]. Over all tasks, FLARE achieves superior or competitive performance compared to single-time and multi-time retrieval baselines, demonstrating the effectiveness and generalizability of our method.

2. Retrieval Augmented Generation

Section Summary: The section defines retrieval-augmented generation, in which a language model retrieves relevant documents from a corpus and conditions its output on them to improve factual accuracy. The standard single-time approach performs one retrieval using the original user query and then generates the full response at once. In contrast, the proposed active framework interleaves retrieval and generation by deciding at each step what to query next based on the input and prior output, retrieving fresh documents only when needed and discarding older ones to stay within length limits.

We formally define single-time retrieval augmented generation and propose the framework of active retrieval augmented generation.

2.1 Notations and Definitions

Given a user input $\bm{x}$ and a document corpus $\mathcal{D}={\bm{d}i}{i=1}^{|\mathcal{D}|}$ (such as all Wikipedia articles), the goal of retrieval augmented LMs is to generate the answer $\bm{y}=[\bm{s}_1, \bm{s}_2, ..., \bm{s}_m]=[w_1, w_2, ..., w_n]$ containing $m$ sentences or $n$ tokens leveraging information retrieved from the corpus.

In retrieval augmented LM, the LM typically pairs with a retriever that can retrieve a list of documents $\mathcal{D}{\bm{q}}=\text{ret}(\bm{q})$ for a query $\bm{q}$; the LM conditions on both the user input $\bm{x}$ and retrieved documents $\mathcal{D}{\bm{q}}$ to generate the answer. Since we focus on examining various methods of determining when and what to retrieve, we follow existing methods [38, 39] to prepend the retrieved documents before the user input to aid future generation for both baselines and our method for fair comparisons: $\bm{y}=\text{LM}([\mathcal{D}_{\bm{q}}, \bm{x}])$, where $[\cdot, \cdot]$ is concatenation following the specified order.

2.2 Single-time Retrieval Augmented Generation

The most common choice is to directly use the user input as the query for retrieval and generate the complete answer at once $\bm{y}=\text{LM}([\mathcal{D}_{\bm{x}}, \bm{x}])$.

2.3 Active Retrieval Augmented Generation

To aid long-form generation with retrieval, we propose active retrieval augmented generation. It is a generic framework that actively decides when and what to retrieve through the generation process, resulting in the interleaving of retrieval and generation. Formally, at step $t (t \ge 1)$, the retrieval query $\bm{q}t$ is formulated based on both the user input $\bm{x}$ and previously generated output $\bm{y}{<t}=[\bm{y}0, ..., \bm{y}{t-1}]$:

$ \bm{q}t=\text{qry}(\bm{x}, \bm{y}{<t}), $

where $\text{qry}(\cdot)$ is the query formulation function. At the beginning ($t=1$), the previous generation is empty ($\bm{y}_{<1}=\emptyset$), and the user input is used as the initial query ($\bm{q}1=\bm{x}$). Given retrieved documents $\mathcal{D}{\bm{q}_t}$, LMs continually generate the answer until the next retrieval is triggered or reaches the end:

$ \bm{y}t=\text{LM}([\mathcal{D}{\bm{q}t}, \bm{x}, \bm{y}{<t}]), $

where $\bm{y}t$ represents the generated tokens at the current step $t$, and the input to LMs is the concatenation of the retrieved documents $\mathcal{D}{\bm{q}t}$, the user input $\bm{x}$, and the previous generation $\bm{y}{<t}$. We discard previously retrieved documents $\cup_{t'<t}\mathcal{D}{\bm{q}{t'}}$ and only use the retrieved documents from the current step to condition the next generation to prevent reaching the input length limit of LMs.

3. FLARE: Forward-Looking Active REtrieval Augmented Generation

Section Summary: FLARE is an approach that lets language models fetch outside information only when they need it during generation, instead of retrieving constantly or at fixed points. The system looks ahead by first drafting the next sentence on its own, then checks how confident the model is in that draft; low-confidence tokens trigger a search using the tentative sentence (or key parts of it) as the query so the model can pull useful documents and rewrite before continuing. Two practical versions are described, one that prompts the model to insert explicit search commands and another that directly treats uncertain generated text as the retrieval query.

Our intuition is that (1) LMs should only retrieve information when they do not have the necessary knowledge to avoid unnecessary or inappropriate retrieval, and (2) the retrieval queries should reflect the intents of future generations. We propose two forward-looking active retrieval augmented generation (FLARE) methods to implement the active retrieval augmented generation framework. The first method prompts the LM to generate retrieval queries when necessary while generating the answer using retrieval-encouraging instructions, denoted as FLARE $\text{instruct}$. The second method directly uses the LM's generation as search queries, denoted as FLARE $\text{direct}$, which iteratively generates the next sentence to gain insight into the future topic, and if uncertain tokens are present, retrieves relevant documents to regenerate the next sentence.

3.1 FLARE with Retrieval Instructions

Inspired by Toolformer [45], a straightforward way of expressing information needs for retrieval is to generate "[Search(query)]" when additional information is needed [45], e.g., "The colors on the flag of Ghana have the following meanings. Red is for [Search(Ghana flag red meaning)] the blood of martyrs, ..." When working with GPT-3.5 models that offer only API access, we elicit such behavior by few-shot prompting [1].

Specifically, for a downstream task, we place the search-related instruction and exemplars at the beginning as skill 1, followed by the instruction and exemplars of the downstream task as skill 2. Given a test case, we ask LMs to combine skills 1 and 2 to generate search queries while performing the task. The structure of the prompt is shown in Prompt 1, and full details can be found in Prompt 4.

Skill 1. An instruction to guide LMs to generate search queries.

Several search-related exemplars.

Skill 2. An instruction to guide LMs to perform a specific downstream task (e.g., multihop QA).

Several task-related exemplars.

An instruction to guide LMs to combine skills 1 and 2 for the test case.

The input of the test case.

As shown in Figure 2, when the LM generates "[Search(query)]" (shown in gray italic), we stop the generation and use the query terms to retrieve relevant documents, which are prepended before the user input to aid future generation until the next search query is generated or reaches the end. Additional implementation details are included in Appendix A.

3.2 Direct FLARE

Since we cannot fine-tune black-box LMs, we found queries generated by FLARE $_\text{instruct}$ through retrieval instructions might not be reliable. Therefore, we propose a more direct way of forward-looking active retrieval that uses the next sentence to decide when and what to retrieve.

3.2.1 Confidence-based Active Retrieval

As shown in Figure 1, at step $t$, we first generate a temporary next sentence $\hat{\bm{s}}t=\text{LM}([\bm{x}, \bm{y}{<t}])$ without conditioning on retrieved documents. Then we decide whether to trigger retrieval and formulate queries based on $\hat{\bm{s}}_t$. If the LM is confident about $\hat{\bm{s}}_t$, we accept it without retrieving additional information; if not, we use $\hat{\bm{s}}_t$ to formulate search queries $\bm{q}_t$ to retrieve relevant documents, and then regenerate the next sentence $\bm{s}_t$. The reason we utilize sentences as the basis of our iteration is due to their significance as semantic units that are neither too short nor too lengthy like phrases and paragraphs. However, our approach can also utilize phrases or paragraphs as the basis.

Since LMs tend to be well-calibrated that low probability/confidence often indicates a lack of knowledge [46, 44, 47], we actively trigger retrieval if any token of $\hat{\bm{s}}_t$ has a probability lower than a threshold $\theta \in [0, 1]$. $\theta=0$ means retrieval is never triggered, while $\theta=1$ triggers retrieval every sentence.

$ \bm{y}_t = \begin{cases} \hat{\bm{s}}_t \quad\quad \text{if all tokens of } \hat{\bm{s}}_t \text{ have probs} \ge \theta \ \bm{s}t=\text{LM}([\mathcal{D}{\bm{q}t}, \bm{x}, \bm{y}{<t}]) \quad\quad \text{otherwise} \end{cases} $

where the query $\bm{q}_t$ is formulated based on $\hat{\bm{s}}_t$.

3.2.2 Confidence-based Query Formulation

One way to perform retrieval is to directly use the next sentence $\hat{\bm{s}}_t$ as the query $\bm{q}_t$. This shares a similar spirit with methods that use generated hypothetical titles or paragraphs from LMs as retrieval queries or evidences [48, 49, 50, 51]. We generalize such techniques to long-form generation where active information access is essential.

We found retrieving with the next sentence achieves significantly better results than with the previous context, as shown later in Section 6.2. However, it has a risk of perpetuating errors contained in it. For example, if the LM produces the sentence "Joe Biden attended the University of Pennsylvania" instead of the correct fact that he attended the University of Delaware, using this erroneous sentence as a query might retrieve misleading information. We propose two simple methods to overcome this issue as illustrated in Figure 3.

Masked sentences as implicit queries.

The first method masks out low-confidence tokens in $\hat{\bm{s}}_t$ with probabilities below a threshold $\beta \in [0, 1]$, where a higher $\beta$ results in more aggressive masking. This removes potential distractions from the sentence to improve retrieval accuracy.

Generated questions as explicit queries.

Another method is to generate explicit questions that target the low-confident span in $\hat{\bm{s}}_t$. For example, if the LM is uncertain about "the University of Pennsylvania", a question like "Which university did Joe Biden attend?" can help retrieve relevant information. Self-ask [40] achieved this by manually inserting follow-up questions into downstream task exemplars as shown later in Prompt 3, which requires task-specific annotation efforts. Instead, we developed a universal approach that generates questions for low-confidence spans without additional annotation. Specifically, We first extract all spans from $\hat{\bm{s}}t$ with probabilities below $\beta$. For each extracted span $\bm{z}$, we prompt gpt-3.5-turbo to generate a question $\bm{q}{t, \bm{z}}$ that can be answered with the span:

User input $\bm{x}$.

Generated output so far $\bm{y}_{\le t}$.

Given the above passage, ask a question to which the answer is the term/entity/phrase " $\bm{z}$ ".

We retrieve using each generated question and interleave the returned documents into a single ranking list to aid future generations. In summary, queries $\bm{q}_t$ are formulated based on $\hat{\bm{s}}_t$ as follows:

$ \bm{q}_t = \begin{cases} \emptyset \quad\quad \text{if all tokens of } \hat{\bm{s}}_t \text{ have probs} \ge \theta \ \text{mask}(\hat{\bm{s}}_t) \text{ or } \text{qgen}(\hat{\bm{s}}_t) \quad\quad \text{otherwise} \end{cases} $

3.3 Implementation Details

Base LM

We validate our method on one of the most advanced GPT-3.5 LMs text-davinci-003 by iteratively querying their API.[^1]

[^1]: https://api.openai.com/v1/completions April 23.

Document corpus and retrievers.

Since we focus on the integration of retrieval and generation, we use off-the-shelf retrievers that take queries as inputs and return a list of relevant documents. For datasets that mainly rely on knowledge from Wikipedia, we use the Wikipedia dump from [52] and employ BM25 [53] as the retriever. For datasets that rely on knowledge from the open web, we use the Bing search engine as our retriever.^2

Retrieved document formatting.

Multiple retrieved documents are linearized according to their ranking and then added to the beginning of the user input using Prompt 2.

Other implementation details such as sentence tokenization and efficiency are included Appendix A.

4. Multi-time Retrieval Baselines

Section Summary: The section explains how prior work on retrieval-augmented language models that fetch information multiple times during generation can be grouped into three categories that differ only in retrieval timing and query content. The first two categories retrieve at fixed intervals—either every fixed number of tokens or every sentence—by using the immediately preceding text as the search query, while the third category relies on manually designed prompts that make the model emit explicit sub-questions to trigger retrieval. The authors note that all three approaches suffer from using past text as a potentially poor signal for future needs, inefficient fixed schedules, and limited applicability across tasks.

Existing passive multi-time retrieval augmented LMs can also be formulated using our framework (Section 2.3). In this section, we formally introduce three baseline categories based on when and what to retrieve. These baselines are not exact reproductions of the corresponding paper because many design choices differ which makes direct comparisons impossible. We implemented them using the same settings, with the only variation being when and what to retrieve.

Previous-window

approaches trigger retrieval every $l$ tokens, where $l$ represents the window size. Generated tokens from the previous window are used as the query:

$ \begin{aligned} \bm{q}t &= \bm{y}{t-1} \quad (t \ge 2), \ \bm{y}t &= [w{(t-1)l+1}, ..., w_{tl}]. \end{aligned} $

Some existing methods in this category are RETRO [37], IC-RALM [38], which retrieve every few tokens, and KNN-LM [13], which retrieves every token.[^3] We follow [38] to use a window size of $l=16$.

[^3]: Since KNN-LM uses the contextualized representation corresponding to the current decoding position to retrieve relevant information which encodes all previous tokens. Strictly speaking, $\bm{q}t$ should be $\bm{y}{<t}$.

Previous-sentence

approaches trigger retrieval every sentence and use the previous sentence as the query, and IRCoT [39] belongs to this category:

$ \begin{aligned} \bm{q}t &= \bm{y}{t-1} \quad (t \ge 2), \ \bm{y}_t &= \bm{s}_t. \end{aligned} $

Question decomposition

approaches manually annotated task-specific exemplars to guide LMs to generate decomposed sub-questions while producing outputs. For example, self-ask [40], a method in this category, manually inserts sub-questions in exemplars using Prompt 3. For the test case, retrieval is triggered dynamically whenever the model generates a sub-question.

The aforementioned approaches can retrieve additional information while generating. However, they have notable drawbacks: (1) Using previously generated tokens as queries might not reflect what LMs intend to generate in the future. (2) Retrieving information at a fixed interval can be inefficient because it might occur at inappropriate points. (3) Question decomposition approaches require task-specific prompt engineering, which restricts their generalizability in new tasks.

5. Experimental Setup

Section Summary: The experiments test FLARE on four knowledge-intensive tasks—multi-hop question answering, commonsense reasoning, long-form question answering, and open-domain summarization—using few-shot prompting of large language models. For each task the authors subsample up to 500 examples from standard benchmarks such as 2WikiMultihopQA, StrategyQA, ASQA, and WikiAsp, selecting hyperparameters on development data and reporting the primary metrics used in prior work. They focus on the direct variant of FLARE unless otherwise noted and describe how answers or summaries are extracted and scored for each dataset.

We evaluate the effectiveness of FLARE on 4 diverse knowledge-intensive tasks using few-shot in-context learning [54, 1, 55]. We follow previous works [39] to sub-sample at most 500 examples from each dataset due to the cost of running experiments. Datasets, metrics, and settings are summarized in Table 7 of Appendix B. The hyperparameters of FLARE are selected based on the development set and listed in Table 9. FLARE refers to FLARE $_\text{direct}$ if not specifically stated.

Multihop QA

The goal of multihop QA is to answer complex questions through information retrieval and reasoning. We use 2WikiMultihopQA [34] which contains 2-hop complex questions sourced from Wikipedia articles that require composition, comparison, or inference, e.g., "Why did the founder of Versus die?" We follow [56] to generate both the chain-of-thought and the final answer. Experimental setting details are included in Appendix B.

We use regular expressions to extract the final answer from the output and compare it with the reference answer using exact match (EM), and token-level F$_1$, precision, and recall.

Commonsense reasoning

Commonsense reasoning requires world and commonsense knowledge to generate answers. We use StrategyQA [35] which is a collection of crowdsourced yes/no questions, e.g., "Would a pear sink in water?" We follow [33] to generate both the chain-of-thought and the final yes/no answer. Details are included in Appendix B.

We extract the final answer and match it against the gold answer using exact match.

Long-form QA

Long-form QA aims to generate comprehensive answers to questions seeking complex information [28, 29]. We use ASQA [29] as our testbed where inputs are ambiguous questions with multiple interpretations, and outputs should cover all of them. For example, "Where do the Philadelphia Eagles play their home games?" could be asking about the city, sports complex, or stadium. We found in many cases it is challenging even for humans to identify which aspect of the question is ambiguous. Therefore, we created another setting (ASQA-hint) where we provide a brief hint to guide LMs to stay on track when generating answers. The hint for the above case is "This question is ambiguous in terms of which specific location or venue is being referred to." Experimental setting details are included in Appendix B.

We use metrics from [29], including EM, RoBERTa-based QA score (Disambig-F$_1$), ROUGE [57], and an overall score combining Disambig-F$_1$ and ROUGE (DR).

Open-domain summarization

The goal of open-domain summarization is to generate a comprehensive summary about a topic by gathering information from open web [32]. We use WikiAsp [31] which aims to generate aspect-based summaries about entities from 20 domains in Wikipedia, e.g., "Generate a summary about Echo School (Oregon) including the following aspects: academics, history." Experimental setting details are included in Appendix B.

Metrics include ROUGE, named entity-based F$_1$, and UniEval [58] which measures factual consistency.

6. Experimental Results

Section Summary: The experiments show that FLARE outperforms all tested baseline methods across four different tasks, delivering the largest gains on multihop question answering where its active retrieval of future-relevant information helps models stay on track and produce correct answers. On more open-ended tasks such as ambiguous question answering and aspect-based summarization, the improvements are smaller yet still consistent, and FLARE beats even strong alternatives that rely on prior context or manually designed sub-questions. Follow-up ablation tests confirm that retrieving based on anticipated next steps works noticeably better than using previous sentences or fixed windows of text.

We first report overall results across 4 tasks/datasets and compare the performance of FLARE with all the baselines introduced in Section 4. We then run ablation experiments to study the efficacy of various design choices of our method.

6.1 Comparison with Baselines

Overall results.

The overall performance of FLARE and baseline across all tasks/datasets are reported in Figure 4. FLARE outperforms all baseline on all tasks/datasets, indicating that FLARE is a generic method that can effectively retrieve additional information throughout the generation.

Among various tasks, multihop QA shows the most significant improvement. This is largely due to the task's clear definition and specific objective of producing the final answer through a 2-hop reasoning process, which makes it easier for LMs to generate on-topic output. In contrast, ASQA and WikiAsp are more open-ended, which increases the difficulty of both generation and evaluation. The improvement on ASQA-hint is larger than that of ASQA because identifying ambiguous aspects is challenging even for humans in many cases, and providing a generic hint helps LMs to stay on topic.

::: {caption="Table 1: FLARE and baselines on 2WikiMultihopQA. Previous-window [37, 38], previous-sentence [39], and question decomposition [40, 41] methods are reimplemented for fair comparisons."}

:::

\begin{tabular}{@l@{\smallcol}c|c@{\smallcol}c@{\smallcol}c@{\smallcol}c|c@{\smallcol}c@{\smallcol}c@{\smallcol}c|c@{\smallcol}c@{\smallcol}c@}
\toprule
\textbf{Datasets} & \textbf{StrategyQA} & \multicolumn{4}{c|}{\textbf{ASQA}} & \multicolumn{4}{c|}{\textbf{ASQA-hint}} & \multicolumn{3}{c}{\textbf{WikiAsp}} \\
\textbf{Metrics} & \textbf{EM} & \textbf{EM} & \textbf{D-F$_1$} & \textbf{R-L} & \textbf{DR} & \textbf{EM} & \textbf{D-F$_1$} & \textbf{R-L} & \textbf{DR} & \textbf{UniEval} & \textbf{E-F$_1$} & \textbf{R-L} \\
\midrule
No retrieval & 72.9 & 33.8 & 24.2 & 33.3 & 28.4 & 40.1 & 32.5 & 36.4 & 34.4 & 47.1 & 14.1 & 26.4 \\
Single-time retrieval & 68.6 & 40.0 & 27.1 & 34.0 & 30.4 & 43.2 & 34.8 & 37.4 & 36.0 & 52.4 & 17.4 & 26.9 \\
\midrule
\multicolumn{13}{c}{\emph{Multi-time retrieval}} \\
Previous-window & 71.2 & 39.9 & 27.0 & \textbf{34.3} & 30.4 & 43.7 & 35.7 & 37.5 & 36.6 & 51.8 & 18.1 & 27.3 \\
Previous-sentence & 71.0 & 39.9 & 27.9 & \textbf{34.3} & 30.9 & 44.7 & 35.9 & 37.5 & 36.7 & 52.6 & 17.8 & 27.2 \\
FLARE (ours) & \textbf{77.3} & \textbf{41.3} & \textbf{28.2} & \textbf{34.3} & \textbf{31.1} & \textbf{46.2} & \textbf{36.7} & \textbf{37.7} & \textbf{37.2} & \textbf{53.4} & \textbf{18.9} & \textbf{27.6} \\
\bottomrule
\end{tabular}

Thorough comparisons with baselines.

The performance of all baselines on 2WikiMultihopQA are reported in Table 1. FLARE outperforms all baselines by a large margin, which confirms that forward-looking active retrieval is highly effective. Most multi-time retrieval augmented approaches outperform single-time retrieval but with different margins. The improvement of retrieving using the previous sentence is relatively small which we hypothesize is mainly because the previous sentence often describes entities or relations different from those in the next sentence in 2WikiMultihopQA. While the previous-window approach might use the first half of a sentence to retrieve information potentially helpful for generating the second half. Among all baselines, the question decomposition approach [40] achieves the best performance. which is not surprising since the in-context exemplars manually annotated with decomposed sub-questions (Prompt 3) guide LMs to generate sub-questions that align with the topic/intent of future generations. FLARE outperforms this baseline, indicating that manual exemplar annotation is not necessary for effective future-aware retrieval. The gap between FLARE $_\text{instruct}$ and question decomposition is large, indicating that teaching LMs to generate search queries using task-generic retrieval instructions and exemplars is challenging.

We report all metrics for the other datasets in Table 2. FLARE outperforms baselines with respect to all metrics. Retrieval using the previous window underperforms single-time retrieval on ASQA, which we hypothesize is because the previous window does not accurately reflect future intent. Since we focus on evaluating factuality, metrics with an emphasis on factual content (such as EM, Disambig-F$_1$, UniEval) are more reliable than metrics computed over all tokens (ROUGE-L).

6.2 Ablation Study

\begin{tabular}{@l@{\smallcol}c@{\smallcol}c@{\smallcol}c@{\smallcol}c|c@{\smallcol}c@{\smallcol}c@{\smallcol}c}
\toprule
& \multicolumn{4}{c|}{\textbf{2WikiMultihopQA}} & \multicolumn{4}{c}{\textbf{ASQA-hint}} \\
& \textbf{EM} & \textbf{F$_1$} & \textbf{Prec.} & \textbf{Rec.} & \textbf{EM} & \textbf{D-F$_1$} & \textbf{R-L} & \textbf{DR} \\
\midrule
Previous & 39.0 & 49.2 & 48.9 & 51.8 & 42.5 & 34.1 & 36.9 & 35.5 \\
Next & 48.8 & 57.6 & 57.1 & 60.5 & 45.9 & 35.7 & 37.5 & 36.6 \\
\bottomrule
\end{tabular}

: Table 4: Previous-window approaches using different numbers of tokens as queries.

#Tokens	EM	$F_1$	Prec.	Rec.
16	43.2	52.3	51.7	54.5
32	43.6	52.4	52.0	55.0
48	40.0	49.3	49.0	52.0
All	39.0	48.5	48.2	51.1

Importance of forward-looking retrieval.

We first validate that forward-looking retrieval is more effective than past-context-based retrieval. We run ablation experiments on 2WikiMultihopQA and ASQA-hint comparing retrieval using the previous versus the next sentence. Specifically, both methods retrieve every sentence and directly use the complete previous/next sentence as queries. As shown in Table 3, using the next sentence to retrieve is clearly better than using the previous sentence, confirming our hypothesis.

We also run previous-window approaches using different numbers of past tokens as queries. As shown in Table 4, using too many tokens (

gt;32$) in the past hurts the performance, further confirming our hypothesis that previous context might not be relevant to intent of future generations.

Importance of active retrieval.

Next, we investigate how active retrieval threshold $\theta$ affects performance. To alter our method from not retrieving to retrieving every sentence, we adjust the confidence threshold $\theta$ that determines when to trigger retrieval from 0 to 1. We then calculate the proportion of steps/sentences where retrieval is activated, and present the performance based on it. As shown in Figure 5, on 2WikiMultihopQA, the performance plateaus when the retrieval percentage exceeds 60%, indicating that retrieval when LMs are confident is not necessary. On StrategyQA, the performance drops when the retrieval percentage exceeds 50%, indicating that unnecessary retrieval can introduce noise and impede the original generation process. We found triggering retrieval for 40%-80% of sentences usually leads to a good performance across tasks/datasets.

\begin{tabular}{lcccc}
\toprule
\textbf{$\beta$} & \textbf{EM} & \textbf{F$_1$} & \textbf{Prec.} & \textbf{Rec.} \\
\midrule
0.0 & 0.488 & 0.576 & 0.571 & 0.605 \\
0.2 & 0.498 & 0.588 & 0.582 & 0.616 \\
0.4 & 0.510 & 0.597 & 0.591 & 0.627 \\
0.6 & 0.506 & 0.593 & 0.586 & 0.622 \\
\bottomrule
\end{tabular}

\begin{tabular}{l@{\smallcol}c@{\smallcol}c@{\smallcol}c@{\smallcol}c|c@{\smallcol}c@{\smallcol}c}
\toprule
& \multicolumn{4}{c|}{\textbf{ASQA-hint}} & \multicolumn{3}{c}{\textbf{WikiAsp}} \\
& \textbf{EM} & \textbf{D-F$_1$} & \textbf{R-L} & \textbf{DR} & \textbf{UniEval} & \textbf{E-F$_1$} & \textbf{R-L} \\
\midrule
Implicit & 45.7 & 36.9 & 37.7 & 37.3 & 53.4 & 18.8 & 27.7 \\
Explicit & 46.2 & 36.7 & 37.7 & 37.2 & 53.4 & 18.9 & 27.6 \\
\bottomrule
\end{tabular}

Effectiveness of different query formulation methods

We study implicit query formation by masking and explicit query formulation through question generation. In Table 5, we compare the performance of FLARE with different masking thresholds $\beta$. Retrieving directly with the complete sentence ($\beta=0$) is worse than masking tokens with low probabilities, confirming our hypothesis that low-confidence erroneous tokens can distract retrievers. We compare implicit and explicit query formulation methods in Table 6. Performances of both methods are similar, indicating that both methods can effectively reflect information needs.

7. Related Work

Section Summary: This section reviews prior research on retrieval-augmented language models and situates FLARE within that landscape by pointing back to earlier sections for details on single-step and multi-step retrieval methods. It notes that earlier studies have explored iterative retrieval and refinement for text and code tasks, as well as adaptive retrieval triggered by question popularity or model uncertainty, yet FLARE stands out by applying these ideas to long-form generation that requires ongoing information gathering. The authors also compare their approach to browser-enhanced models such as WebGPT, which train language models to issue multiple web queries via reinforcement learning or supervision, while observing that FLARE’s text-retrieval foundation could be paired with such browsing tools if needed.

We refer to Section 2.2 and Section 4 for extensively discussion on single-time and multi-time retrieval augmented LMs, which is the most relevant area to this paper.

Iterative and adaptive retrieval

Iterative retrieval and refinement has been studied in both text and code generation tasks [59, 60, 61, 62]. FLARE differs from these methods in the granularity of generation and retrieval strategies. Adaptive retrieval has been studied in single-time retrieval scenarios based on either question popularity or generation probabilities [63, 64], while we focus on long-form generation requiring active information access.

Browser-enhanced LMs

WebGPT [22] and WebCPM [65] train LMs to interact with browser to enhance factuality using reinforcement learning or supervised training where multiple queries can be triggered before generation. FLARE is built on text-based retrievers but can be combined with a browser to potentially improve retrieval quality.

8. Conclusion

Section Summary: The researchers introduce a method called active retrieval-augmented generation that helps language models produce longer texts by deciding on its own when extra outside information is needed and fetching it mid-process. Their version checks the upcoming sentence for uncertain spots, pulls relevant details if necessary, and rewrites the sentence before moving on. Tests on four tasks show the approach improves results, and the authors point to future work on smarter retrieval rules and more efficient model designs.

To aid long-form generation with retrieval augmentation, we propose an active retrieval augmented generation framework that decides when and what to retrieve during generation. We implement this framework with forward-looking active retrieval that iteratively uses the upcoming sentence to retrieve relevant information if it contains low-confidence tokens and regenerates the next sentence. Experimental results on 4 tasks/datasets demonstrate the effectiveness of our methods. Future directions include better strategies for active retrieval and developing efficient LM architectures for active information integration.

9. Limitations

Section Summary: FLARE showed little benefit on the Wizard of Wikipedia and ELI5 datasets. The first involves short dialogue turns that rarely need multiple retrievals, while the second poses long-form questions whose grounding and evaluation challenges limit gains from retrieval. On the practical side, repeatedly switching between generation and retrieval also raises computational overhead, though custom model designs that keep retrieved passages separate from the main input could ease this cost.

We also conduct experiments on Wizard of Wikipedia [66] and ELI5 [28], and found that FLARE did not provide significant gains. Wizard of Wikipedia is a knowledge-intensive dialogue generation dataset where the output is relatively short ($\sim$ 20 tokens on average) so retrieving multiple disparate pieces of information might not be necessary. ELI5 [28] is a long-form QA dataset requiring in-depth answers to open-ended questions. Due to issues mentioned in [67] such as difficulties of grounding generation in retrieval and evaluation, both single-time retrieval and FLARE did not provide significant gains over not using retrieval. From an engineering perspective, interleaving generation and retrieval with a naive implementation increases both overheads and the cost of generation. LMs need to be activated multiple times (once for each retrieval) and a caching-free implementation also requires recomputing the previous activation each time after retrieval. This issue can be potentially alleviated with special architectural designs that encode the retrieved documents $\mathcal{D}_{\bm{q}t}$ and the input/generation ($\bm{x}$ / $\bm{y}{<t}$) independently.

Acknowledgements

Section Summary: The research received partial funding from a grant by the Singapore Defence Science and Technology Agency as well as the IBM PhD Fellowship. The authors also thanked a number of colleagues for their helpful discussions and support with carrying out the experiments.

This work was supported in part by a grant from the Singapore Defence Science and Technology Agency and the IBM PhD Fellowship. We thank Chunting Zhou, Amanda Bertsch, Uri Alon, Hiroaki Hayashi, Harsh Trivedi, Patrick Lewis, Timo Schick, Kaixin Ma, Shuyan Zhou, and Songwei Ge for their insightful discussions and help with the experiments.

Appendix

Section Summary: The appendix describes the technical implementation of the FLARE model, including adjustments to language model outputs to control how often and when search queries are generated, the use of an initial user query to start retrieval, and sentence-level tokenization via NLTK during generation. It also details the experimental setups for multiple tasks, such as multihop and commonsense QA as well as long-form question answering and open-domain summarization, covering the number of examples, retrieval corpora and methods like BM25 or Bing, and prompt structures with in-context exemplars. Efficiency notes compare FLARE’s selective retrieval frequency against other systems that fetch information more often.

A. FLARE Implementation Details

FLARE $_\text{instruct}$ implementation details

We found that LMs can effectively combine retrieval and downstream task-related skills and generate meaningful search queries while performing the task. However, there are two issues: (1) LMs tend to generate fewer search queries than necessary. (2) Generating excessive search queries can disrupt answer generation and adversely affect performance. We address these issues using two methods respectively. First, we increase the logit of the token "[" by 2.0 to improve the chances of LMs generating "[Search(query)]". Second, whenever LMs generate a search query, we use it to retrieve relevant information, promptly remove it from the generation, and generate the next few tokens while forbidding "[" by adding a large negative value to the logit of "[".

The initial query of FLARE.

FLARE starts with the user input $\bm{x}$ as the initial query to retrieve documents to generate the first sentence $\hat{\bm{s}}1=\text{LM}([\mathcal{D}{\bm{x}}, \bm{x}])$ to bootstrap the iterative generation process. For the following steps, the temporary forward-looking sentence is generated without retrieved documents.

Sentence tokenization.

For each step $t$, we generate 64 tokens which are longer than most sentences, and use NLTK sentence tokenizer^4 to extract the first sentence and discard the rest.

Efficiency

As shown in Section 6.2, on average retrieval is triggered for $30% \sim 60%$ of sentences depending on downstream tasks. In comparision, KNN-LM [13] retrieves every token, RETRO or IC-RALM [37, 38] retrievers every 4 $\sim$ 32 tokens, and IRCoT [39] retrieves every sentence. Compared to single-time retrieval, however, interleaving retrieval and generation with a naive implementation indeed increases overheads, which we discuss in the limitation section (Section 9).

B. Datasets and Settings

Datasets, metrics, and experimental settings are summarized in Table 7.

Multihop QA

For "Why did the founder of Versus die?", the output we aim to generate is "The founder of Versus was Gianni Versace. Gianni Versace was shot and killed on the steps of his Miami Beach mansion on July 15, 1997. So the answer is shot." We use 8 exemplars from [39] listed in Prompt 5 for in-context learning, BM25 as the retriever, and Wikipedia articles as the retrieval corpus. Similar to the observation in [39], we found incorporating retrieval results for exemplars improves the performance, we use the input $\bm{x}$ of each exemplar to retrieve several documents and then add them using the format in Prompt 2. We found increasing the number of retrieval documents often increases performance. Therefore, we use the maximum number of documents that can fit within the input length limit of text-davinci-003, which is 2 for 2WikiMultihopQA.

Commonsense Reasoning

For "Would a pear sink in water?", the output we aim to generate is "The density of a pear is about 0.6g/cm$^3$, which is less than water. Objects less dense than water float. Thus, a pear would float. So the final answer is no." We use 6 exemplars from [33] listed in Prompt 6, BM25 on the Wikipedia corpus, and 3 retrieved documents to run experiments.

Long-form QA

For "Where do the Philadelphia Eagles play their home games?", the output we aim to generate is "We need to consider the different possible locations or venues that could be considered the home field of the Philadelphia Eagles. These include the city, the sports complex, or the stadium. Therefore, this question has 3 interpretations and the answers are: (1) The city is Philadelphia. (2) The sports complex is the South Philadelphia Sports Complex. (3) The stadium is the Lincoln Financial Field stadium." For both the original setting (ASQA) and the setting with hints (ASQA-hint), we manually annotate 8 exemplars (Prompt 7 Prompt 9), use BM25 on the Wikipedia corpus, and 3 retrieved documents to run experiments.

Open-domain Summarization

The original WikiAsp dataset is designed for multi-document summarization and provides a list of references to systems. We converted it into the open-domain setting by removing the associated references and instead gathering information from the open web. For "Generate a summary about Echo School (Oregon) including the following aspects: academics, history.", the output we aim to generate is "# Academics. In 2008, 91% of the school's seniors received their high school diploma... # History. The class of 2008 was the 100th class in the school's history." where # is used to indicate aspects. We manually annotate 4 exemplars (Prompt 11), and use the Bing search engine to retrieve 5 documents from the open web. To avoid leaking, we exclude several Wikipedia-related domains listed in Table 8 from Bing's search results.

\begin{tabular}{lc@{\tinycol}c@{\tinycol}c@{\tinycol}c}
\toprule
\textbf{Settings} & \textbf{2WikiMultihopQA} & \textbf{StrategyQA} & \textbf{ASQA} & \textbf{WikiAsp} \\
 & [34] & [35] & [29] & [31] \\
\midrule
\multicolumn{5}{c}{\emph{Dataset statistics}} \\
Task & multihop QA & commonsense QA & long-form QA & open-domain summarization \\
#Examples & 500 & 229 & 500 & 500 \\
\midrule
\multicolumn{5}{c}{\emph{Evaluation settings}} \\
Metrics & EM, F$_1$, Prec., Rec. & EM & EM, Disambig-F$_1$, ROUGE, DR & UniEval, entity-F$_1$, ROUGE \\
\midrule
\multicolumn{5}{c}{\emph{Retrieval settings}} \\
Corpus & Wikipedia & Wikipedia & Wikipedia & open web \\
Retriever & BM25 & BM25 & BM25 & Bing \\
Top-k & 2 & 3 & 3 & 5 \\
\midrule
\multicolumn{5}{c}{\emph{Prompt format}} \\
#Exemplars & 8 & 6 & 8 & 4 \\
Ret. for exemplars & \ding{51} & \ding{55} & \ding{55} & \ding{55} \\
\bottomrule
\end{tabular}

\begin{tabular}{l}
\toprule
wikipedia.org, wikiwand.com, wiki2.org, wikimedia.org \\
\bottomrule
\end{tabular}

C. Hyperparameters

Hyperparameters of FLARE on different datasets are listed in Table 9.

\begin{tabular}{lcccc}
\toprule
\textbf{Dataset} & \textbf{$\theta$} & \textbf{$\beta$} & \textbf{Query formulation} & \textbf{Combine single- \& multi-time retrieval} \\
\midrule
2WikiMultihopQA & 0.8 & 0.4 & implicit & \ding{55} \\
StrategyQA & 0.4 & 0.4 & implicit & \ding{55} \\
ASQA \& ASQA-hint & 0.8 & 0.4 & explicit & \ding{51} \\
WikiAsp & 0.8 & 0.4 & explicit & \ding{51} \\
\bottomrule
\end{tabular}

D. Prompts and Few-shot exemplars

The prompt used to linearize multiple documents is shown in Prompt 2. The prompt used in self-ask [40] is shown in Prompt 3. Prompts and exemplars of different tasks/datasets are shown in Prompt 4, Prompt 5, Prompt 6, Prompt 7, Prompt 9, Prompt 11, respectively.

Search results:

$[$ 1 $]$ Document 1

$[$ 2 $]$ Document 2

... The user input $\bm{x}$

Question: Who lived longer, Theodor Haecker or Harry Vaughan Watkins?

Are follow up questions needed here: Yes.

Follow up: How old was Theodor Haecker when he died?

Intermediate answer: Theodor Haecker was 65 years old when he died.

Follow up: How old was Harry Vaughan Watkins when he died?

Intermediate answer: Harry Vaughan Watkins was 69 years old when he died.

So the final answer is: Harry Vaughan Watkins.

References

Section Summary: This references section compiles a list of influential academic papers on large language models, highlighting foundational systems such as GPT-3, GPT-4, PaLM, Llama, and OPT. It covers topics like model training with human feedback, scaling laws, and how these systems store and retrieve knowledge. Additional entries address challenges like factual accuracy and hallucinations, along with methods such as retrieval augmentation to improve outputs.

[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

[2] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155.

[3] OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

[4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311.

[5] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01068.

[6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.

[7] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303.18223.

[8] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases?In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2463–2473. Association for Computational Linguistics.

[9] Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model?In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 5418–5426. Association for Computational Linguistics.

[10] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know. Trans. Assoc. Comput. Linguistics, 8:423–438.

[11] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.

[12] Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, Online. Association for Computational Linguistics.

[13] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

[14] Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. CoRR, abs/2208.03299.

[15] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1870–1879. Association for Computational Linguistics.

[16] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: retrieval-augmented language model pre-training. CoRR, abs/2002.08909.

[17] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

[18] Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 874–880. Association for Computational Linguistics.

[19] Devendra Singh Sachan, Siva Reddy, William L. Hamilton, Chris Dyer, and Dani Yogatama. 2021. End-to-end training of multi-document reader and retriever for open-domain question answering. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 25968–25981.

[20] Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christopher D. Manning, and Kyoung-Gu Woo. 2021. You only need one model for open-domain question answering. CoRR, abs/2112.07381.

[21] Zhengbao Jiang, Luyu Gao, Jun Araki, Haibo Ding, Zhiruo Wang, Jamie Callan, and Graham Neubig. 2022. Retrieval as attention: End-to-end learning of retrieval and reading within a single transformer. CoRR, abs/2212.02027.

[22] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332.

[23] Hongjing Qian, Yutao Zhu, Zhicheng Dou, Haoqi Gu, Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Webbrain: Learning to generate factually correct articles for queries by grounding on large web corpus. CoRR, abs/2304.04358.

[24] Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. 2022. Internet-augmented language models through few-shot prompting for open-domain question answering. CoRR, abs/2203.05115.

[25] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. REPLUG: retrieval-augmented black-box language models. CoRR, abs/2301.12652.

[26] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452–466.

[27] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601–1611. Association for Computational Linguistics.

[28] Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: long form question answering. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3558–3567. Association for Computational Linguistics.

[29] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. ASQA: factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 8273–8288. Association for Computational Linguistics.

[30] Nachshon Cohen, Oren Kalinsky, Yftah Ziser, and Alessandro Moschitti. 2021. Wikisum: Coherent summarization dataset for efficient human-evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 212–219. Association for Computational Linguistics.

[31] Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021. Wikiasp: A dataset for multi-domain aspect-based summarization. Trans. Assoc. Comput. Linguistics, 9:211–225.

[32] John M. Giorgi, Luca Soldaini, Bo Wang, Gary D. Bader, Kyle Lo, Lucy Lu Wang, and Arman Cohan. 2022. Exploring the challenges of open domain multi-document summarization. CoRR, abs/2212.10526.

[33] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.

[34] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6609–6625. International Committee on Computational Linguistics.

[35] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361.

[36] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. CoRR, abs/2009.03300.

[37] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR.

[38] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083.

[39] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. CoRR, abs/2212.10509.

[40] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350.

[41] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. CoRR, abs/2210.03629.

[42] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. CoRR, abs/2210.02406.

[43] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP. CoRR, abs/2212.14024.

[44] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. CoRR, abs/2207.05221.

[45] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.

[46] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know Whenlanguage models know? on the calibration of language models for question answering. Trans. Assoc. Comput. Linguistics, 9:962–977.

[47] Neeraj Varshney, Man Luo, and Chitta Baral. 2022. Can open-domain QA reader utilize external knowledge efficiently like humans?CoRR, abs/2211.12707.

[48] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without relevance labels. CoRR, abs/2212.10496.

[49] Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. CoRR, abs/2210.01296.

[50] Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. CoRR, abs/2209.10063.

[51] Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-augmented retrieval for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4089–4100. Association for Computational Linguistics.

[52] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769–6781. Association for Computational Linguistics.

[53] Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

[54] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).

[55] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9):195:1–195:35.

[56] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. CoRR, abs/2203.11171.

[57] Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

[58] Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi-dimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2023–2038. Association for Computational Linguistics.

[59] Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. CoRR, abs/2302.12813.

[60] Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. CoRR, abs/2303.12570.

[61] Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, and Fei Sha. 2022. Generate-and-retrieve: Use your predictions to improve retrieval for semantic parsing. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 4946–4951. International Committee on Computational Linguistics.

[62] Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. 2023. Improving language models via plug-and-play retrieval feedback. CoRR, abs/2305.14002.

[63] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. CoRR, abs/2212.10511.

[64] Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jingyuan Wang, Jian-Yun Nie, and Ji-Rong Wen. 2023. The web can be your oyster for improving large language models. CoRR, abs/2305.10998.

[65] Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023. Webcpm: Interactive web search for chinese long-form question answering. CoRR, abs/2305.06849.

[66] Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.

[67] Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In North American Association for Computational Linguistics.