RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Harrison Lee
Google DeepMind

Samrat Phatale
Google DeepMind

Hassan Mansoor
Google DeepMind

Thomas Mesnard
Google DeepMind

Johan Ferret
Google DeepMind

Kellie Lu
Google

Colton Bishop
Google DeepMind

Ethan Hall
Google

Victor Carbune
Google DeepMind

Abhinav Rastogi
Google DeepMind

Sushant Prakash
Google

Correspond to: Harrison Lee [email protected]

Keywords: Reinforcement Learning, Large Language Models, Alignment with Human Feedback, Alignment from AI Feedback, Self-improvement

Abstract

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in [1], offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.

Executive Summary: Aligning large language models with human preferences is essential for making them helpful, accurate, and safe. Reinforcement Learning from Human Feedback has served as the standard approach for this alignment, but gathering large volumes of high-quality human preference annotations is slow, expensive, and difficult to scale. This operational bottleneck limits how quickly and cost-effectively advanced models can be aligned and updated.

This article set out to determine whether Reinforcement Learning from AI Feedback—where an off-the-shelf language model generates the preference ratings—can match or exceed the performance of human feedback across multiple text generation tasks. The authors evaluated this across three specific domains: text summarization, helpful dialogue generation, and harmless dialogue generation.

The authors conducted comprehensive experiments using the PaLM 2 model family to compare models aligned with AI feedback against those aligned with human feedback and standard baseline models. The study benchmarked these approaches using thousands of examples across public datasets, including Reddit summaries and conversational datasets. AI preference labelers were evaluated on their alignment with human preferences, and final models were assessed using human evaluators through pairwise win rates and safety classifications, with post-hoc adjustments applied to control for response length biases.

The primary finding is that AI-aligned models achieve performance on par with or superior to human-aligned models across all tested tasks. For text summarization and helpful dialogue, human evaluators preferred both AI- and human-aligned models over standard baselines at nearly identical rates (approximately 71% to 73% for summarization and 63% to 64% preference for helpful dialogue), with direct head-to-head comparisons resulting in an even split. On safety, the AI-aligned model outperformed the human-aligned model, generating harmless responses 88% of the time compared to 76% for human feedback and 64% for the baseline. The research also revealed that models can achieve "self-improvement," demonstrating performance gains even when the AI labeler is the same size or identical checkpoint as the model being trained. Furthermore, a proposed direct scoring method—which queries an AI model directly during training rather than training an intermediate scoring model—outperformed canonical AI feedback setups, achieving a 74% win rate over the baseline on summarization.

These findings indicate that AI feedback provides a viable, highly scalable alternative to human annotation that can reduce preference labeling costs by an estimated factor of ten or more while drastically reducing project timelines. However, utilizing AI feedback transfers any underlying biases from the labeling model into the trained policy. This creates governance considerations: while AI feedback makes model alignment accessible and rapid, organizations deploying models in high-stakes environments such as law, medicine, or hiring must exercise caution and retain human expert oversight.

Organizations developing language models should consider adopting AI feedback pipelines to accelerate alignment workflows and lower development costs. Teams can implement direct scoring mechanisms to streamline infrastructure and avoid training intermediate reward models. When designing prompts for AI labelers, practitioners should incorporate chain-of-thought reasoning to maximize alignment with human judgments, while adjusting candidate order to mitigate position bias.

Confidence in these findings is supported by consistent results across tasks, statistically significant human evaluation trials, and controls for response length. Nevertheless, key limitations remain: smaller AI labelers exhibit substantial position bias, and combining human and AI feedback in a single training set did not show additive gains. Qualitative reviews also showed occasional factual hallucinations in human-aligned models and minor fluency degradations in AI-aligned models. Further work is recommended to explore more complex multi-turn interactions, credit assignment methods, and hybrid training curricula before deploying purely AI-aligned models into critical production domains.

1. Introduction

Section Summary: Reinforcement Learning from Human Feedback aligns language models with human preferences and has driven success in systems like ChatGPT, but it depends on expensive human annotations that limit scaling. This paper examines whether AI-generated feedback from large language models can serve as a practical substitute, comparing the two approaches on summarization, helpful dialogue, and harmless dialogue tasks. Results indicate that the AI-based method matches or exceeds human-feedback performance in human evaluations while enabling techniques for direct rewards from the models themselves and potential self-improvement.

Reinforcement Learning from Human Feedback (RLHF) is an effective technique for aligning language models to human preferences ([2, 3]). It is cited as one of the key drivers of success in modern conversational language models, such as ChatGPT ([4]) and Bard ([5]). A key advantage of training language models with reinforcement learning (RL) is that it enables optimization on complex, sequence-level objectives that are not easily differentiable and therefore ill-suited for traditional supervised fine-tuning (SFT).

One obstacle for employing RLHF at scale is its dependence on high-quality human preference labels. Modern large language models (LLMs) have shown a high degree of alignment with human judgment ([6, 7]), suggesting that LLM-generated preference labels may be a viable substitute for human labels. [1] was the first effort to explore Reinforcement Learning from AI Feedback (RLAIF), where RL was conducted using a reward model trained on a hybrid of human and AI preferences. In conjunction with their "Constitutional AI" self-revision technique, their final policy outperformed supervised fine-tuning for training a conversational assistant. However, it did not directly compare the efficacy of human vs. AI feedback, leaving the question of whether RLAIF can be a suitable alternative to RLHF unanswered.

In this work, we compare the effectiveness of RLAIF and RLHF (see Figure 2) on three tasks: summarization, helpful dialogue generation, and harmless dialogue generation. Our experiments show that RLAIF and RLHF are preferred by humans over a SFT baseline 71% and 73% of the time for summarization and 63% and 64% of the time for helpful dialogue generation, respectively, where the win rates for RLAIF and RLHF are not statistically significantly different. Furthermore, in a head-to-head comparison of RLAIF against RLHF, both policies are equally preferred[^2]. For harmless dialogue generation, human evaluators rated the harmlessness of each response independently. RLAIF scored a higher harmless rate than RLHF, and both outperformed the SFT baseline (88%, 76%, and 64%, respectively). These results suggest that RLAIF is a viable alternative to RLHF that does not depend on human annotation, while offering appealing scaling properties.

[^2]: The win rate for one policy over the other is not statistically significantly different from 50%

Additionally, we conduct two related studies. First, in a step towards LLM self-improvement, we demonstrate that RLAIF significantly improves upon the SFT baseline even when the AI labeler is the same size as the policy model. Second, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents reward model training by obtaining rewards directly from an off-the-shelf LLM during RL. In our experiments, d-RLAIF matches or outperforms canonical RLAIF. Furthermore, for the task of helpful dialogue generation, the initial policy and the LLM providing rewards are the same model checkpoint, demonstrating an instance of strict LLM self-improvement.

Finally, we study techniques to maximize the alignment of AI-generated preferences to human preferences. We find that soliciting chain-of-thought reasoning ([8]) consistently improves alignment, while using a detailed preamble and few-shot prompting ([9]) are only beneficial for certain tasks. We conduct scaling experiments to examine the trade-off between the size of the LLM labeler and alignment with human preferences.

The main contributions of this work are as follows:

1. We demonstrate that RLAIF achieves comparable performance to RLHF for the tasks of summarization, helpful dialogue generation and harmless dialogue generation.
2. We show that RLAIF can improve upon an SFT policy when the LLM labeler is the size same as the policy, or even the exact same checkpoint as the policy.
3. We introduce direct RLAIF (d-RLAIF), which derives the reward directly from an off-the-shelf LLM during RL and matches or outperforms canonical RLAIF.
4. We study techniques to maximize the alignment of AI-generated preferences to human preferences.

2. Methodology

Section Summary: The methodology section explains how researchers use an ordinary large language model to label which of two responses is better by feeding it carefully designed prompts, extracting probability scores for each choice, and averaging results from swapped orders to reduce bias. These AI-generated preference labels then support two reinforcement learning approaches: one that trains a separate reward model on the labels before using it to guide policy training, and a direct method that lets the original LLM score responses on the fly during learning to avoid outdated rewards. The section also briefly notes experiments with chain-of-thought reasoning in the labeling step and standard RL procedures drawn from prior work.

This section describes the techniques used to generate preferences with an LLM, the reinforcement learning setups, and evaluation metrics. RLHF preliminaries are provided in Appendix A.

2.1 Preference Labeling with LLMs

We annotate preferences with an "off-the-shelf" LLM - a model pre-trained or instruction-tuned ([10]) for general usage but not fine-tuned for a specific downstream task. Given a piece of text and two candidate responses, the LLM is asked to rate which response is preferred. The prompt is structured as follows (examples in Table 15 and Table 21):

1. Preamble - Introduction and instructions describing the task at hand
2. Few-shot exemplars (optional) - An example input context, a pair of responses, a chain-of-thought rationale (optional), and a preference label
3. Sample to annotate - An input context and a pair of responses to be labeled
4. Ending - The ending text to prompt the LLM (e.g. "Preferred Response=")

After the prompt is given to the LLM, we extract the log-probabilities of generating the tokens "1" and "2" and compute the softmax to obtain a preference distribution.

There are numerous alternatives to obtain preference labels from LLMs, such as extracting the preference from a free-form generated response (e.g. "The first response is better"), or representing the preference distribution as a one-hot encoding. However, we choose our method because it is straightforward to implement and conveys more information than a one-hot encoding through its distributed representation of preferences.

We experiment with two styles of preambles: "Base", which essentially asks which response is better, and "Detailed", which resembles detailed rating instructions typically given to human annotators (see Table 16 for preambles used in the summarization task). We also experiment with in-context learning ([9]), using high-quality exemplars hand-selected to cover a range of topics.

2.1.1 Addressing Position Bias

The order in which candidates are shown to an LLM can bias which candidate it prefers ([11, 12]). We find evidence of position bias, which is especially prevalent in smaller LLM labelers (see Appendix B).

To mitigate the effect of position bias, two inferences are made for every pair of candidates, where the order in which candidates are presented to the LLM is reversed for the second inference. The results from both inferences are then averaged to obtain the final preference distribution.

2.1.2 Eliciting Chain-of-thought Reasoning

We experiment with eliciting chain-of-thought (CoT) reasoning ([8]) from our AI labelers through a two-step inference procedure. First, we replace the Ending of the standard prompt with a sentence asking for thoughts and explanation (e.g. "Consider the coherence, accuracy, coverage, and overall quality of each summary and explain which one is better. Rationale:") and decode a response from the LLM. Then, we concatenate the original prompt, the response, and the standard Ending string together, and follow the scoring procedure in Section 2.1 to obtain a preference distribution. Figure 3 illustrates this process.

For zero-shot prompts, the LLM is not given an example of what reasoning should look like. In few-shot prompts, we provide examples of CoT reasoning for the model to follow. See Table 17 and Table 18 for examples.

2.2 Reinforcement Learning

from AI Feedback

2.2.1 Canonical RLAIF

We describe our adaptation of the canonical RLAIF setup below. Unless otherwise mentioned, RLAIF is carried out using this method.

A reward model (RM) is trained on the LLM-generated preference labels following the methodology in Appendix A.2. Since our approach produces soft labels (e.g. $[0.6, 0.4]$), we train the RM with a cross-entropy loss on the softmax of the scores generated by the RM. The softmax converts the RM scores into a probability distribution. We note that training a RM on a dataset of AI labels can be viewed as a form of model distillation.

Finally, we conduct reinforcement learning to train the RLAIF policy model, using the RM to assign rewards to model responses, as described in Appendix A.3.

2.2.2 Direct-RLAIF (d-RLAIF)

One issue with RLAIF is that the reward model may become "stale" as the policy is trained. In the typical setup, the RM is trained on generations sampled from the initial policy. As the policy is trained, the generated trajectories become increasingly out-of-distribution from the dataset the RM was trained on, leading to suboptimal performance ([13]). One solution is to conduct iterative RLAIF, where a new RM is periodically trained on the latest policy, though this is a time consuming process.

We introduce direct-RLAIF (d-RLAIF) - a simple alternative to canonical RLAIF that directly uses LLM feedback as the reward signal in RL. D-RLAIF addresses the RM staleness issue, as the off-the-shelf LLM directly scores generated responses during RL without undergoing training. Additionally, d-RLAIF eliminates the need for the time consuming process of AI preference labeling and RM training. Figure 4 depicts this process.

In d-RLAIF, the LLM is prompted to rate the quality of a generation between 1 and 10. Similar to Section 2.1, the prompt instructs the LLM on how to rate a generation. Then, the likelihood of each score token between 1 and 10 is computed, the likelihoods are normalized to a probability distribution, a weighted score is calculated as $s(y | x) = \sum_{i = 1}^{10} i P(i | y, x)$, and finally the score is again normalized to the range $[-1, 1]$. Additional details on the prompting technique can be found in the Appendix D.

RL is then conducted in a similar manner to canonical RLAIF, where the direct score is used as reward instead of a RM score.

2.3 Evaluation

We evaluate our results with three metrics - AI Labeler Alignment, Win Rate, and Harmless Rate.

AI Labeler Alignment measures the accuracy of AI-labeled preferences with respect to human preferences. For a single example, a soft AI-labeled preference is first converted to a binary representation (e.g. $[0.6, 0.4] \rightarrow [1, 0]$). Then, a score of 1 is assigned if the label agrees with the human preference and 0 otherwise. The alignment accuracy $z_{acc}$ can be expressed as follows:

$ z_{acc} = \frac{1}{D}\sum_{i=1}^{D} \mathbb{1} [\operatorname*{\arg\max}j P^{AI}{i, j} = p^H_i], $

where $D$ is the size of the preference dataset, $P^{AI} \in \mathbb{R}^{D \times 2}$ is the matrix of soft AI preferences, and $p^{H} \in \mathbb{R}^{D}$ is the corresponding vector of human preferences, containing elements $0$ or $1$ to denote whether the first or second response is preferred, respectively.

Win Rate evaluates the end-to-end quality of two policies by measuring how often one policy is preferred by human annotators over another. Given an input and two generations, human annotators select their preferred generation. The percentage of instances where policy $A$ is preferred over policy $B$ is referred to as the "win rate of A vs. B". A 50% win rate indicates that $A$ and $B$ are equally preferred.

Harmless Rate measures the percentage of responses that are considered harmless by human evaluators. We evaluate the harmless dialogue generation task with this metric instead of Win Rate, because we find that many responses are equally safe, making it difficult to assign relative rankings.

3. Experimental Details

Section Summary: The experiments rely on several public datasets of human preferences over summaries or AI assistant responses, which were downsampled to a few thousand examples each for faster iteration. Researchers used the PaLM 2 family of models to generate AI preference labels, trained small reward models on those labels or on the original human ratings, and then applied a REINFORCE-style reinforcement learning procedure to improve policy models starting from supervised fine-tuning checkpoints. Final quality was assessed through human raters who compared or judged responses on held-out test examples.

3.1 Datasets

We use the following datasets for our experiments:

• Reddit TL;DR ([2]) - posts from Reddit^1 accompanied by summaries of the posts.
• OpenAI's Human Preferences ([2]) - a dataset created from a subset of Reddit TL;DR. Each example comprises a post, two candidate summaries, and a rating from a human annotator indicating which summary is preferred.
• Anthropic Helpful and Harmless Human Preferences ([13]) - conversations between a human and an AI assistant, where each conversation has two possible AI assistant responses - one preferred and the other non-preferred, according to a human annotator. Preferences are based on which response is more informative and honest for the helpful task, and which response is safer for the harmless task.

More dataset details can be found in Appendix C.

We also explored the Stanford Human Preferences dataset ([14]), but we found that both RLHF and RLAIF policies did not show meaningful improvements over the SFT baseline after correcting for length biases as described in Appendix J.

3.2 LLM Labeling

To enable fast experiment iteration when evaluating AI labeling techniques, we randomly downsampled the training split of each preference dataset. For summarization, an additional filter was applied to only include examples where human annotators preferred one summary over the other with high confidence[^3]. After downsampling and filtering, there remained 3-4k examples for each task[^4]. AI labeler alignment was calculated on these downsampled datasets.

[^3]: This follows the evaluation procedure in [2]. Examples with confidence scores of 1, 2, 8, and 9 were considered to be "high-confidence"

[^4]: We sampled 15%, 10%, and 10% of the training splits for summarization, helpful dialogue generation, and harmless dialogue generation, respectively.

We use the PaLM 2 ([15]) family of models for labeling preferences. All versions were instruction-tuned but not previously trained with RL. Unless otherwise specified, AI labels were generated using PaLM 2 Large (L) with the best-performing prompt for each task in Section 4.4. For more details on LLM labeling, see Appendix D.

3.3 Model Training

All SFT models are initialized from PaLM 2 Extra-Small (XS). For summarization, the SFT model is produced by fine-tuning PaLM 2 XS on the Reddit TL;DR dataset. For all other tasks, an instruction-tuned variant of PaLM 2 is used in lieu of task-specific fine-tuning.

All RMs are trained from PaLM 2 XS checkpoints. Each RM is fine-tuned on the full training split of a preference dataset, where the label is the AI preference for AI feedback RMs and the original human preference label for human feedback RMs. RM accuracies can be found in Appendix G.

In the RL phase, the policy is trained with a modified version of REINFORCE ([16]) adapted to the language modeling domain (see Appendix E). While many recent works use Proximal Policy Optimization (PPO) ([17]), we use REINFORCE with a baseline given that it is simpler yet still effective for the problem at hand. Both policy and value models are initialized from the SFT model. For summarization, the policy is rolled out on the training split of the Reddit TL;DR dataset. In other words, the initial state for each trajectory is the original Reddit post. For the helpful and harmless tasks, the initial states are drawn from the training splits of the preference datasets. For summarization, simple post-processing is applied to responses generated by RL-trained policies as described in Appendix H.

For additional details on model training, see Appendix F.

3.4 Human Evaluation

To measure win rates, evaluators were presented with an input context and multiple responses generated from different policies (e.g. RLAIF, RLHF, and SFT). They were then asked to rank responses in order of quality without ties, as seen in Figure 5. Input contexts were drawn from the test splits of each dataset, which were not used for training or any other evaluation[^5]. Rankings were subsequently used to compute win rates for pairs of policies. For harmless dialogue generation, evaluators were asked to independently rate each response as harmless or harmful.

[^5]: For summarization, we used the test split of Reddit TL;DR. For helpful and harmless dialogue generation, we used test splits from the preference datasets, detailed in Appendix C.

For more details on human evaluation, see Appendix I.

4. Results

Section Summary: Results show that RLAIF matches or exceeds RLHF across summarization and dialogue tasks, with human raters preferring both methods over a standard supervised baseline and RLAIF achieving an even higher harmlessness score. The gains hold when the AI labeler is the same size as the model being trained, and they become larger still when preferences are applied directly rather than distilled through a reward model, producing a clear case of self-improvement. Experiments with prompting also indicate that chain-of-thought reasoning improves the reliability of the AI feedback used for training.

::: {caption="Table 1: Left side: Win rates for pairs of policies on the summarization and the helpful dialogue tasks. Right side: Harmless rates across policies for the harmless dialogue task. All numbers are based on human evaluation."}

:::

4.1 RLAIF vs. RLHF

RLAIF achieves performance gains on par with or better than RLHF on all three tasks (see Figure 1 and Table 1). Specifically, RLAIF and RLHF are preferred by human evaluators over the baseline SFT policy 71% and 73% of the time for summarization[^6] and 63% and 64% for helpful dialogue generation, respectively. The difference in win rates between RLAIF vs. SFT and RLHF vs. SFT are not statistically significant. When directly comparing RLAIF against RLHF, they are equally preferred - i.e. the win rate is not statistically significantly different from 50%. For harmless dialogue generation, RLAIF achieves a harmless rate of 88%, outperforming both RLHF and SFT, which score 76% and 64%, respectively[^7].

[^6]: RLAIF and RLHF are also preferred over the human reference summaries in Reddit TL;DR 79% and 80% of the time, respectively.

[^7]: RLAIF achieves a statistically significant improvement over RLHF and SFT, according to two-sided paired t-tests.

Figure 6 contains an example of SFT, RLAIF, and RLHF summaries. To better understand how RLAIF compares to RLHF, we qualitatively compare responses generated by both policies for summarization in Section 5.

Similar to [2], we observe that RLAIF and RLHF policies tend to generate longer responses than the SFT policy, which may bias human evaluation. We conduct post-hoc analysis to control for length and find that both RLAIF and RLHF policies still outperform the SFT policy. See Appendix J for details.

One natural question that arises is whether there is value in combining human and AI feedback. We experimented with combining both types of feedback but did not see an improvement beyond using human feedback alone. However, we believe that there are several alternative training setups that could demonstrate value in combining both forms of feedback. See Appendix K for details.

These results suggest that RLAIF is a viable alternative to RLHF that does not depend on human annotation. In addition to expediting the time to collect labels, another benefit of AI labeling is cost reduction. We estimate the cost of LLM labeling to be over 10x cheaper than human annotation. See Appendix L for detailed analysis.

4.2 Towards Self-Improvement

In Section 4.1, the LLM used to label preferences (PaLM 2 L) is much larger than the policy being trained (PaLM 2 XS). Going one step further, we explore whether RLAIF can yield improvements when the AI labeler is the same size as the policy. On the task of summarization, we conduct RLAIF where PaLM 2 XS is used as the AI labeler instead of PaLM 2 L. The rest of the setup mimics the experiment in Section 4.1. We refer to this setup as "same-size RLAIF".

Same-size RLAIF still improves greatly over the SFT baseline, with human annotators preferring same-size RLAIF 68% of the time over SFT (see Table 1). For reference, RLAIF using an AI labeler larger than the policy is preferred 71% over SFT[^8]. This result demonstrates that RLAIF can yield improvements even when the AI labeler is the same size as the policy LLM.

[^8]: The difference between win rates between "same-size RLAIF vs. SFT" and the original "RLAIF vs. SFT" is not statistically significant. For a two-sample t-test, p-value = 0.07. At alpha = 0.05, this difference is not statistically significant.

We note that this experiment is not a strict example of "self-improvement" ([18]), as the AI labeler is the instruction-tuned PaLM 2 XS, whereas the initial policy is PaLM 2 XS fine-tuned on Reddit TL;DR summarization. However, we demonstrate a case of strict self-improvement on the helpfulness task in the following section.

4.3 D-RLAIF

In Section 4.1 and Section 4.2, AI feedback was distilled into a RM. On the summarization and helpfulness tasks, we experiment with d-RLAIF (see Section 2.2.2). We use the smaller instruction-tuned PaLM 2 XS as our AI labeler to reduce compute costs.

For summarization, human annotators prefer d-RLAIF over SFT 74% of the time (see Table 1). To understand the impact of directly utilizing LLM feedback versus distilling feedback to a RM, we compare this result to the same-size RLAIF policy from Section 4.2, which only differs in terms of the reward function. D-RLAIF outperforms same-size RLAIF, which achieves a statistically significantly lower win rate of 68%. Furthermore, when shown responses side-by-side, annotators prefer d-RLAIF over same-size RLAIF 60% of the time[^9]. We hypothesize that this improvement is a result of directly querying the AI labeler for preferences rather than first distilling its preferences into a RM, as well as circumventing the "staleness" issue described in Section 2.2.2.

[^9]: This is statistically significantly different from 50% according to the binomial test.

For helpful dialogue generation, we find that d-RLAIF achieves a a win rate of 66% over the SFT baseline. Since the LLM providing feedback and the starting policy are exactly the same model checkpoint, this constitutes a strict example of LLM self-improvement.

4.4 Prompting Techniques

::: {caption="Table 2: We observe that eliciting chain-of-thought reasoning tends to improve AI labeler alignment, while few-shot prompting and detailed preambles have mixed effects across tasks. Above, 'Help.' and 'Harm.' refer to helpfulness and to harmlessness, respectively."}

:::

We experiment with three types of prompting variations - preamble specificity, chain-of-thought reasoning, and in-context learning (see Table 2). The best prompts outperform the base prompts ("Base 0-shot") by +1.9%, +1.3%, and +1.7% for summarization, helpfulness, and harmlessness, respectively.

Detailed preambles improve alignment for summarization, while yielding mixed results for helpful and harmless dialogue generation. We hypothesize that summarization benefits more from a detailed preamble due to the greater complexity of the task. Rating helpfulness and harmlessness are easier to grasp, and therefore may benefit less from detailed instructions.

Chain-of-thought reasoning generally improves alignment. For summarization, the improvement is consistent. For helpful and harmless dialogue generation, CoT only improves alignment when paired with the "Base" preamble.

Surprisingly, we observe that in-context learning only improves alignment for harmless dialogue generation[^10]. For summarization and helpfulness, alignment monotonically decreases as the number of exemplars increases. To verify that this was not a result of poorly chosen exemplars, we conducted 10 trials for "Base 1-shot" on summarization, where a different exemplar was randomly selected for each trial. The maximum AI labeler alignment from all trials was 76.1%, which still does not surpass "Base 0-shot" in terms of AI labeler alignment. One hypothesis is that the summarization and helpful dialogue generation tasks may already be sufficiently well-understood by the off-the-shelf AI labeler, rendering the exemplars unhelpful or distracting. It is worth noting that in-context learning is still an important research area that is not fully understood ([19, 20]).

[^10]: We verified that all inputs used in these experiments fit within our AI labeler's context length.

For summarization, we also compare against human inter-annotator agreement to get a sense of how well our LLM labeler performs in absolute terms. [2] estimated that agreement rate for the OpenAI human preference dataset was 73-77%, suggesting that the off-the-shelf LLM achieving 78% alignment performs well in absolute terms.

We also conduct experiments with self-consistency ([21]), where multiple chain-of-thought rationales are sampled with temperature $T > 0$. The preference distributions generated by the LLM are averaged together to arrive at the final preference label. However, we find that self-consistency strictly degrades AI labeler alignment (see Appendix M).

We hypothesize that higher AI labeler alignment leads to improvements in RLAIF policies. To this end, we conduct an experiment on the end-to-end sensitivity to AI labeler alignment. Two RLAIF policies are trained that only differ in the alignment scores of AI labels. Results show that the policy trained with more aligned AI labels achieves a significantly higher win rate, which aligns with our expectations. However, this study only compares two policies, and rigorous experimentation is required to draw definitive conclusions. See Appendix N for details.

4.5 Size of LLM Labeler

: Table 3: AI labeler alignment increases as the size of the LLM labeler increases.

Large model sizes are not widely accessible and can be expensive to run. On the task of summarization, we vary the LLM size for labeling preferences and observe a strong positive relationship between size and alignment (see Table 3). Alignment decreases by 4% when substituting PaLM 2 L with PaLM 2 S, and decreases another 11% when using PaLM 2 XS - a trend consistent with scaling behaviors observed in other work ([22]). One contributing factor to this trend, apart from a decrease in general model capability, may be that smaller LLMs display greater position bias (see Appendix B).

From another angle, these results also suggest that scaling up AI labeler size may produce even higher quality preference labels. Since the AI labeler is only used to generate preference examples once and is not called during RL for canonical RLAIF, using an even larger AI labeler is not necessarily prohibitively expensive.

5. Qualitative Observations

Section Summary: When comparing summaries produced by RLAIF and RLHF, the two approaches often generated similar results. However, RLHF occasionally included plausible-sounding details that were not supported by the source text, while RLAIF summaries were sometimes less fluent, featuring run-on sentences or repeated phrases that did not match the original meaning. A small human evaluation of 70 examples found no statistically significant differences in accuracy, coverage, or coherence, suggesting that larger studies are needed to confirm these patterns.

To gain a deeper understanding of how RLAIF compares to RLHF, we visually inspected responses generated by both policies for the summarization task. In many cases, the two policies produced similar summaries, which is reflected in their similar win rates. However, we identified a few patterns where they sometimes diverged.

First, we observe that in some cases, RLHF hallucinated when RLAIF did not. The hallucinations in RLHF summaries sounded plausible but were inconsistent with the original text. For instance, in Example #1 of Table 23, the RLHF summary states that the author is 20 years old, but this is neither mentioned nor implied by the source text.

Second, we observed that RLAIF sometimes produced less fluent summaries than RLHF. For instance, in Table 24, all three RLAIF summaries contain run-on sentences. We also observed cases where RLAIF responses repeated phrases that failed to convey the intention of the original text. For example, several summaries concluded with "How do I get over this?", despite the fact that the original text did not convey this question implicitly or explicitly.

We conducted a small-scale evaluation on 70 examples, where human annotators were asked to blindly rank RLHF and RLAIF summaries in terms of accuracy, coverage, and coherence. However, the difference in scores was not statistically significant. More systematic analysis is required to identify if these patterns exist at scale, which we leave to future work.

6. Related Work

Section Summary: Large language models have achieved strong results on many language tasks, and researchers have increasingly used reinforcement learning to refine their outputs. Early approaches relied on automatic scoring metrics as rewards, but these often failed to match human judgments of quality, leading to methods that instead train reward models directly from human preference comparisons. Recent work has also explored using the models themselves to generate training data or rewards, including hybrid setups that combine model and human feedback, which motivates investigating whether models can fully replace human labelers for preference collection.

LLMs have shown impressive performance on a wide range of NLP tasks ([9, 23, 24, 15, 25]). For several of these tasks, RL has emerged as an effective optimization technique. While initial applications of RL on tasks such as translation ([26, 27]) and summarization ([28, 29]) used automatic evaluation metrics as rewards, such simplified formulations of rewards did not fully align with human notions of quality.

Reinforcement learning (RL) from human feedback ([30]) has been used as a technique to directly align LLMs with human preferences ([31]) through training a reward model on pairwise comparisons of natural language responses. It has been successfully applied for summarization ([2]), instruction following ([3, 32]), dialogue ([6, 5, 33, 13]) and question answering ([34]). To mitigate some stability and efficiency challenges of conducting RL, DPO ([35]) reformulates the training objective to rely on a classification loss, while RaFT ([36]) employs the reward model for conducting rejection-sampling fine-tuning.

LLMs have also been extensively used for data generation ([37, 38]), augmentation ([39]) and in self-training setups ([21, 40]). [1] introduced the idea of RLAIF, which used LLM and human labeled preferences together to jointly optimize for the two objectives of helpfulness and harmlessness. Recent works have also explored related techniques for generating rewards from LLMs ([41, 42, 43]). These works demonstrate that LLMs can generate useful signals for RL fine-tuning, which inspired this work's investigation into whether LLMs can serve as a viable alternative to humans in collecting preference labels for RL.

7. Conclusion

Section Summary: This section concludes that RLAIF delivers results comparable to RLHF across several text generation tasks, with humans rating the outputs similarly in direct comparisons. It further shows that the approach enables self-improvement in large language models, since it remains effective even when the same model serves as both the policy and the labeler, and that directly prompting the model for rewards outperforms first training a separate reward model. The authors also note open questions around further adaptations, such as model-based reinforcement learning, that are left for future work.

We show that RLAIF achieves comparable improvements to RLHF on three text generation tasks. In head-to-head comparisons, RLAIF and RLHF are preferred at similar rates by humans. Furthermore, we demonstrate evidence of LLM self-improvement by showing that RLAIF is effective even when the LLM labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Additionally, we also direct-RLAIF, which directly prompts the LLM labeler to provide rewards during RL, outperforming the canonical RLAIF setup that first distills LLM preferences into a separate RM. Finally, we study the impact of various AI labeling techniques on alignment to human preferences.

While this work highlights the potential of RLAIF, there remain many fascinating open questions, such as how RLAIF can be adapted to a model-based RL setting where both human and assistant are modeled by LLMs, or how AI feedback can be leveraged for granular credit assignment. We leave these questions to future work.

Acknowledgements

Section Summary: The authors thank several collaborators for technical assistance such as optimizing language model setups, refining prompts, highlighting position bias concerns, and providing overall research support. They also acknowledge a group of reviewers for their detailed feedback, along with contributors who helped create a training approach that uses large language models to generate reward scores. Finally, they recognize the team responsible for designing and building the reinforcement learning infrastructure used throughout the project.

We would like to thank many people who have helped make this work complete. We thank Chen Zhu for optimizing our LLM inference setup, Le Hou for suggesting prompt improvements and experimenting with self-consistency, Léonard Hussenot for bringing the problem of position bias in LLMs to our attention, and Bradley Green, Ewa Dominowska, and Blaise Aguera y Arcas for supporting this research.

We thank everyone who thoroughly reviewed our work and provided valuable feedback: Hakim Sidahmed, Meiqi Guo, Michal Valko, Nevan Wichers, Sian Gooding, and Yuan Cao.

We thank Mo Azar, Daniel Guo, Andrea Michi, Nicolas Perez-Nieves, and Marco Selvi for their contribution to developing a RLAIF training setup that directly prompts an LLM to obtain reward scores.

Finally, we thank the individuals who designed and built the RL training infrastructure used in this paper: Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Alexis Jacq, Sabela Ramos, Piotr Stanczyk, Sertan Girgin, Danila Sinopalnikov, Amélie Héliou, Nikola Momchev, and Olivier Bachem.

Impact Statement

Section Summary: This paper aims to make it easier and cheaper for developers to align AI models with human values by using feedback generated by other AI systems instead of relying on human labelers. However, this approach risks transferring and amplifying biases from existing AI models into new ones, which could cause harm in sensitive areas like medicine, law, and hiring, where expert human oversight remains essential. It also raises concerns that lowering the barriers to model alignment could enable misuse, such as creating convincing misinformation or abusive content, underscoring the need for strict controls on access to powerful AI systems.

This paper seeks to better understand and improve the utility of AI models in a scalable fashion. Methods presented in this paper make model alignment more accessible to developers, as generating preferences from LLMs is more affordable and faster than human labeling. However, the use of AI Feedback presents two ethical considerations.

Utilizing AI-generated feedback as a source for model alignment has the potential risk of transferring biases from off-the-shelf LLMs to generated preferences. This in turn may result in RL-trained policies that further amplify biases, thereby inadvertently misaligning models and potentially causing harm. Extreme caution must be exercised, especially when deploying these models in high-stakes domains such as medicine, law, and employment, where models have the potential to significantly impact human lives in adverse ways. In such domains, we believe that human experts trained to carefully assign preferences according to strict policies should be considered the gold standard.

Another ethical consideration is that reducing the barriers to aligning LLMs also carries the risk of facilitating their misuse for malicious purposes. For instance, RLAIF could be employed to train models to generate convincing misinformation or produce hateful and abusive content. The best mitigation to this risk is to carefully govern the access and usage of powerful LLMs (e.g. limiting "white-box" access), to prevent bad actors from abusing them.

Appendix

Section Summary: The appendix first reviews the standard RLHF process, covering supervised fine-tuning of a language model, training a reward model on human preference pairs via a ranking loss, and then using reinforcement learning to optimize a policy while adding a KL penalty to avoid reward hacking and unnatural outputs. It next examines position bias in LLM-based preference labeling, showing through swap experiments that smaller models are far more likely to favor responses based on order rather than content. Finally, it describes the specific datasets employed, including filtered Reddit posts for summarization and Anthropic’s helpful/harmless dialogue pairs, along with basic labeling setup details.

A. RLHF Preliminaries

We review the RLHF pipeline introduced in [2, 3], which consists of 3 phases: supervised fine-tuning, reward model training, and reinforcement learning.

A.1 Supervised Fine-tuning

A pre-trained LLM is fine-tuned on a high quality labeled dataset for a downstream task using token-level supervision to produce a supervised fine-tuned (SFT) model $\pi^{SFT}$.

A.2 Reward Modeling

Given an input $x$, we sample a pair of responses $(y_1, y_2) \sim \pi$ from one or more models, where oftentimes $\pi$ is the SFT model. The input and responses are sent to human annotators to rate which response is better according to some criteria. These annotations form a dataset of triplets $\mathcal{D} = {(x, y_w, y_l)}$, where $y_w$ and $y_l$ are the preferred and non-preferred responses, respectively. A reward model (RM) $r_{\phi}$ is trained by minimizing the following loss:

$ \mathcal{L}r (\phi) = \mathop{-\mathbb{E}}{(x, y_w, y_l) \sim \mathcal{D} } \Big[\log \sigma \big( r_{\phi} (x, y_w) - r_{\phi} (x, y_l) \big) \Big], $

where $\sigma$ is the sigmoid function.

A.3 Reinforcement Learning

A policy $\pi_{\theta}^{RL}$ is initialized from the SFT model weights and then optimized with reinforcement learning to maximize the reward given by the RM, which serves as a proxy for human preferences. Optionally, a Kullback-Leibler (KL) divergence term $D_{KL}$ is added to the objective to penalize $\pi_{\theta}^{RL}$ for deviating from the original SFT policy $\pi^{SFT}$, controlled by the hyperparameter $\beta$ ([44, 45]). The KL loss helps prevent $\pi_{\theta}^{RL}$ from drifting into a region where it generates language that is highly rewarded by the RM yet consists of low-quality or unnatural language - a phenomenon known as "reward hacking" ([46, 47]). The optimization objective is described by the equation below:

$ \begin{aligned} J(\theta) = \mathop{ \mathbb{E}}{y \sim \pi\theta(\cdot | x)} &\Big[ (1 - \beta) r_\phi(y | x) \ &- \beta D_{KL} \big(\pi^{RL}_\theta(y|x) , || , \pi^{SFT} (y|x)\big) \Big], \end{aligned} $

where $\beta$ is a hyperparameter between 0 and 1.

B. Position Bias in LLM Labelers

: Table 4: Position bias is more prevalent in smaller model sizes, measured by the percentage of examples where the LLM prefers the same position even after swapping the order of candidates ("% Same Position Preferred"). Analysis is conducted using the "Detailed + CoT 0-shot" prompt on the summarization task.

Our analysis on the summarization task suggests that the LLMs used for preference labeling are biased by the order in which candidates are shown. For each example in our AI labeling evaluation set, we query the LLM preferences for the pair of candidates, swap the order in which candidates are presented, and then query the LLM preferences again.

We consider an LLM to be more biased if it prefers the same position on both the original and reversed inferences. For example, let candidates A and B be in positions 1 and 2 for the first inference and in positions 2 and 1 for the second inference. If the LLM prefers the same position on both inferences, we consider the LLM to be position-biased. We measure position bias by computing "% Same Position Preferred" - the percentage of inference pairs where this occurs. A higher metric value indicates a more biased LLM.

We find that PaLM 2 L, S, and XS prefer the same position 18%, 21%, and 56% of the time, respectively, suggesting that position bias is inversely correlated with model size (see Table 4). One hypothesis is that larger models are more capable and therefore more faithfully judge preferences based on the content of the candidates rather than their positions, which are supposed to be immaterial.

We also observe that for PaLM 2 L, of the 18% of cases where it prefers the same position on both inferences, 94% of the time it prefers the first candidate shown. On the other hand, PaLM 2 S and XS show affinity for the second candidate shown when the same position is preferred on both inferences, preferring it 91% and 99% of the time, respectively. These biases are statistically significant under a two-sided binomial test at $\alpha = 0.05$.

C. Dataset Details

For summarization, we use the filtered Reddit TL;DR dataset ([2]), containing posts from Reddit^11 that have been filtered to ensure high quality. The dataset contains 123k posts, where $\sim$5% is held out as a validation set.

Additionally, we use OpenAI's human preference dataset created from the filtered Reddit TL;DR dataset. For a given post, two candidate summaries were generated - often from different policies, and human labelers were asked to rate which summary they preferred. The total dataset comprises 92k pairwise comparisons.

For helpful and harmless dialogue generation, we use Anthropic's Helpful and Harmless preference datasets[^12] ([13]). Each example consists of a conversation history between a human and an AI assistant accompanied by a preferred and non-preferred response from the AI assistant. Preference is based on which response is more helpful and honest for the helpful task, and which response is safer and less harmful for the harmless task. Each dataset comprises over 40k training examples and 2k test examples. We further split each original test set into a validation and a test set by randomly assigning two-thirds of examples to validation and one-third to test.

[^12]: We use the helpful-base and harmless-base datasets from https://huggingface.co/datasets/Anthropic/hh-rlhf.

D. LLM Labeling Details

For LLM labeling, we set a maximum input context length of 4096 tokens. For chain-of-thought generation, we set a maximum decoding length of 512 tokens and sample with temperature $T = 0.0$ (i.e. greedy decoding). For self-consistency experiments in Appendix M, we use temperatures varying from $T = 0.3$ to $T = 1.0$ with top-K sampling ([48]), where $K = 40$.

In Section 4.3, we use the AI labeler to directly compute a score that we leverage as the reward for RL. For summarization, we use the following prompt "You are an expert summary rater. Given a TEXT (completed with a SUBREDDIT and a TITLE) and a SUMMARY, your role is to provide a SCORE from 1 to 10 that rates the quality of the SUMMARY given the TEXT, with 1 being awful and 10 being a perfect SUMMARY.", followed by the input Reddit post, then the summary to score preceded by "SUMMARY: ", and a final "SCORE: ". For helpful task generation, we use the prompt "You are an expert rater of helpful and honest Assistant responses. Your role is to provide a SCORE from 1 to 10 that rates the helpfulness and honesty of the RESPONSE for a given CONTEXT. Where SCORE of 1 refers to useless and dishonest RESPONSE and a SCORE of 10 refers to a perfectly helpful and honest RESPONSE.", followed by the conversation history and a response.

PaLM 2 models are publicly available through Google Cloud's Vertex AI^13, though exact reproducibility is not guaranteed as the models accessible through Google Cloud are subject to change.

E. REINFORCE for Language Models

Consider a deterministic, finite-horizon MDP $M = (\mathcal{X},\mathcal{A},R,P,\gamma)$ ([49]). At each step $t$, given the current state $X_t \in \mathcal{X}$ and the next action $A_t \in \mathcal{A}$, the model receives a reward $R_t = R(X_t,A_t)$ and transitions to the next state $X_{t+1}= P(X_t,A_t)$.

In the context of language models, $X_t$ is the concatenation of the input text and all text generated by the policy until time $t$. Action $A_t$ is the token from the considered vocabulary decoded at time $t$ by the stochastic policy $\pi_\theta(\cdot|X_t)$, where $\theta$ represents the policy parameters. The reward $R_t$ is given by the RM, which is only evaluated when the language model response has been fully generated; all rewards prior to the final token are set to $0$, while the reward corresponding to the final token is set to $R_{T}$.

The cumulative sum of rewards received when following the policy $\pi_\theta$ from time-step $t$ is called the return. Generally, it is defined as $Z_t= \sum_{s=t}^{T} \gamma^{s-t} R_s$. However, since only the terminal reward is non-zero and we set $\gamma=1$, the return can be simplified to $Z_t=R_{T}$.

Given a trajectory $(X_t,A_t,R_t){t=0}^{T}$ generated under $\pi\theta$, the policy gradient loss from REINFORCE is then defined as follows:

$ \mathcal{L}\text{PG}(\theta)=-\sum{t} \log \pi_\theta (A_t|X_t) \overline{\left(Z_t - V^{\pi}_\psi(X_t)\right)}, $

where the bar notation denotes that no gradient is passed through the advantage term during backpropagation.

The baseline value function $V^\pi_\psi(x)$ estimates the return-to-go $Z_t$ when following the policy $\pi_\theta$, and it is parameterized by $\psi$ ([16, 50]). It is trained with the following loss:

$ \mathcal{L}V(\psi)=\sum_t (Z_t-V^\pi\psi(X_t))^2. $

Our full optimization objective is written in Appendix A.3. We incorporate the KL divergence in the policy gradient loss described above, as commonly seen in other work ([51]).

F. Model Training Details

SFT models for the summarization task are trained on the Reddit TL;DR dataset with a batch size of 128 and for one epoch. We use the Adafactor ([52]) optimizer with a learning rate of $10^{-5}$, and the maximum input and output lengths are 1024 and 128 tokens, respectively. For helpful and harmless dialogue generation tasks, an instruction-tuned version of PaLM 2 XS serves as the SFT model.

RMs for all tasks are trained until the training loss and accuracy curves plateau, which happens in 2-3 epochs. We use the Adafactor optimizer with a learning rate of $10^{-5}$. Batch size is 128 for summarization RMs and 32 for RMs of other tasks. We train all our RMs with maximum input length of 1152 tokens to account for 1024 context tokens and 128 response tokens. We report the accuracies of the RMs in Appendix G.

For summarization, the AI feedback RM is initialized from the SFT model (i.e. PaLM 2 XS fine-tuned on Reddit TL;DR), and the human feedback RM is initialized from PaLM 2 XS. We experimented with initializing the human feedback RM from the SFT model but found that it resulted in lower accuracy on the held out set of human preferences (see Table 6). For helpful and harmless dialogue generation tasks, we initialize both the human and AI feedback RMs from the instruction-tuned version of PaLM 2 XS.

For reinforcement learning, we use the SFT model for each task as the initial policy. We sample from our language model policies for all tasks with a temperature of $T = 0.9$ to encourage exploration. We train with a batch size of 128 and learning rate of $10^{-5}$ for 8 epochs. We set $\beta = 0.05$ for the KL divergence loss.

To select the final checkpoint for each RL policy, we first selected 4 candidate checkpoints from RL training that scored high rewards on validation prompts. We then prompted an off-the-shelf LLM to judge the win rate of the RL checkpoint's responses vs. the SFT policy's responses. We also conducted manual inspection of a dozen examples. We picked the checkpoint with the best combination of win rate and quality as judged by manual inspection as our final RL policy.

G. Reward Model Accuracy

\begin{tabular}{|c|c|c|}
\hline
\makecell{Task} & \makecell{Human \\ Feedback} & \makecell{AI \\ Feedback} \\
\hline
Summarization & 79.3\% & 74.2\% \\
\hline
\makecell{Helpful Dialogue} & 76.0\% & 67.8\% \\
\hline
\makecell{Harmless Dialogue} & 72.1\% & 69.7\% \\
\hline
\end{tabular}

\begin{tabular}{|c|c|c|}
\hline
\makecell{Initialization} & \makecell{Human \\ Feedback} & \makecell{AI \\ Feedback} \\
\hline
PaLM 2 XS & \textbf{79.3\%} & 73.0\% \\
\hline
SFT & 78.7\% & \textbf{74.2\%} \\
\hline
\end{tabular}

\begin{tabular}{|c|c|}
\hline
\makecell{RM Variant} & \makecell{AI \\ Feedback}\\
\hline
Trained on ``Base 0-shot'' labels & 77.9\% \\
\hline
Trained on ``Detailed CoT 0-shot'' labels from PaLM 2 XS & 66.4\% \\
\hline
\end{tabular}

Pairwise Accuracy measures how accurate a trained reward model is with respect to a holdout set of human preferences. Given an input context and pair of candidate responses, the value is 1 if the RM scores the preferred candidate higher than the non-preferred candidate, according to the human label. Otherwise the value is 0. This quantity is averaged over multiple examples to obtain the pairwise accuracy.

We report RM accuracy on a holdout set of human preferences for all tasks in Table 5. For summarization, we also report RM accuracy when initializing on different checkpoints in Table 6. In Table 7, we report accuracy for RM variants used in the end-to-end sensitivity experiment in Appendix N and the same-size RLAIF experiment in Section 4.2.

We observe that RMs trained on human feedback outperform those trained on AI feedback, both of which are measured against a holdout set of human preferences. This pattern seems natural, given that the human preferences are trained on data drawn from the same distribution as the validation dataset. However, it is interesting to note that despite the gap in accuracy between AI and human preference RMs, RLAIF achieves comparable results to RLHF on two tasks and surpasses it on another. Additionally, we note that the summarization RMs trained on "Base 0-shot" and "Detailed + CoT 0-shot" (i.e. the default prompting technique) achieve accuracies of 77.9% and 74.2%, respectively, which is the inverse order of their final performance after RL (see Appendix N). These gaps in RM accuracy suggest that RM accuracy, while correlated with RM usefulness, may not accurately reflect a RM's effectiveness in RLHF and RLAIF. Ultimately, we believe that the usefulness of RMs is assessed through conducting RL and evaluating the final policies through human evaluation.

H. Post-RL Response Formatting

For summarization, we observed that summaries generated by RLHF and RLAIF policies often included superfluous symbols like periods or spaces at the end of the response - possibly due to "reward hacking". Given that these extra tokens do not have any meaningful content, we programmatically removed certain symbols at the end of summaries. This ensured that human evaluators could focus on the content without being distracted by the formatting of the response.

I. Human Evaluation Details

To conduct human evaluation, in total we generated $\sim$2k unique rating instances. Each instance comprised a single context and multiple distinct model responses (e.g. responses from SFT, RLAIF, and RLHF policies), resulting in a total of $\sim$6k unique (context, response) pairs subjected to human evaluation. Additionally, each instance was assessed by three independent raters, resulting in $\sim$18k (context, response, rating) tuples.

We measure the inter-annotator agreement with Kendall's Coefficient of Concordance W ([53]) - a non-parametric statistic for assessing the agreement among multiple raters ranking multiple items. The values of Kendall's W range from 0 to 1, where 0 indicates perfect disagreement and 1 indicates perfect agreement. We conducted multiple human evaluation sessions, and the W statistic ranged from 0.6-0.7, indicating a reasonable level of agreement.

J. Controlling for Response Length

Response length often can influence human evaluators' perception of quality ([2]), and our various policies generate responses that differ in length. For example, in the summarization task, the summaries produced by RLAIF, RLHF, and SFT policies sent to human evaluation had an average character-length of 164, 161, and 132, respectively. We conduct post-hoc analysis to estimate the win rates after controlling for length.

We take an approach similar to [2] and calculate the "length-adjusted win rate of policy A vs. policy B". Given policy A, we train a logistic regression model where the input is the ratio of the policy A's response length to policy B's summary length (in characters), and the target is a binary label indicating whether policy A's response was preferred over policy B's response. After fitting the model, we estimate a length-controlled win rate by asking the logistic regressor to predict the win rate given a length ratio of 1.0, which represents the scenario where both the responses are of equal length.

After controlling for length for the summarization task, our length-adjusted win rates for RLAIF and RLHF vs. SFT are 59% and 61%, respectively (see Table 8). Both RL policies continue to outperform the SFT policy by a similar margin, supporting our initial statement that RLAIF is comparable to RLHF.

We reach similar conclusions for the helpful dialogue generation task (Table 9), same-size RLAIF and direct RLAIF (d-RLAIF) experiments (Table 11), the end-to-end sensitivity to AI labeler alignment experiment (Table 12), and combining human and AI feedback (Table 13).

For the harmless dialogue generation task, we use a different setup. Since human evaluators rated each response independently as harmful or harmless, we compute the harmless rate instead of the win rate. We use the average generation length from the SFT policy as the reference point for all other policies (Table 10).

We note that this post-hoc method of controlling for length is imperfect, as it assumes the logistic regression model accurately learns the relationship between summary length and human preference. A more principled approach is to encourage all policies generate summaries of similar length through an auxiliary training loss.

\begin{tabular}{|c|c|c|}
\hline
Models & \makecell{Length \\ uncorrected} & \makecell{Length \\ corrected} \\\hline
RLAIF vs SFT & 71\% & 59\% \\\hline
RLHF vs SFT & 73\% & 61\% \\\hline
RLAIF vs RLHF & 50\% & 47\% \\\hline
\end{tabular}

\begin{tabular}{|c|c|c|}
\hline
Models & \makecell{Length \\ uncorrected} & \makecell{Length \\ corrected} \\\hline
RLAIF vs SFT & 63\% & 61\% \\\hline
RLHF vs SFT & 64\% & 61\% \\\hline
RLAIF vs RLHF & 52\% & 50\% \\\hline
\end{tabular}

\begin{tabular}{|c|c|c|}
\hline
Models & \makecell{Length \\ uncorrected} & \makecell{Length \\ corrected} \\\hline
SFT & 64\% & 64\% \\ \hline
RLHF & 76\% & 78\% \\ \hline
RLAIF & 88\% & 91\% \\ \hline
\end{tabular}

\begin{tabular}{|c|c|c|}
\hline
Models & \makecell{Length \\ uncorrected} & \makecell{Length \\ corrected} \\\hline
\makecell{Same-size RLAIF \\ vs SFT} & 68\% & 59\% \\\hline
\makecell{d-RLAIF \\ vs SFT} & 74\% & 65\% \\\hline
\makecell{d-RLAIF vs \\ Same-size RLAIF} & 60\% & 56\% \\\hline
\end{tabular}

\begin{tabular}{|c|c|c|}
\hline
Models & \makecell{Length \\ uncorrected} & \makecell{Length \\ corrected} \\\hline
\makecell{Base RLAIF \\ vs SFT} & 63\% & 59\% \\\hline
\makecell{Detailed RLAIF \\ vs SFT} & 67\% & 63\% \\\hline
\makecell{Base RLAIF vs \\ Detailed RLAIF} & 41\% & 45\% \\\hline
\end{tabular}

\begin{tabular}{|c|c|c|}
\hline
Models & \makecell{Length \\ uncorrected} & \makecell{Length \\ corrected} \\\hline
\makecell{RLHF + RLAIF \\ vs SFT} & 71\% & 61\% \\\hline
\makecell{RLHF \\ vs SFT} & 74\% & 67\% \\\hline
\makecell{RLHF + RLAIF \\ vs RLHF} & 48\% & 46\% \\\hline
\end{tabular}

K. Combining Human and AI Feedback

We investigate the effectiveness of combining human feedback and AI feedback on the task of summarization. We refer to this approach as RLHF + RLAIF and compare it against RLHF.

First, given contexts randomly drawn from the Reddit TL;DR dataset, responses are generated by RLHF and SFT policies with temperature $T = 1.0$. The instruction-tuned PaLM 2 L is then called to generate AI preferences on the OpenAI human preference dataset. Finally, a new RM is trained on both the dataset with human labels and the dataset with AI labels.

We observe that RLHF + RLAIF does not improve beyond RLHF alone. RLHF + RLAIF and RLHF achieve win rates of 71% and 74% over SFT, respectively. The difference in win rates is not statistically significant. When compared head-to-head, raters prefer both policies equally.

While this experiment did not show positive results from combining RLAIF and RLHF, there are many alternative setups which could prove successful. One such setup could involve first conducting RLAIF, then collecting generations and human preferences using the RLAIF policy as the initialization point for RLHF. In this curriculum learning approach, RLAIF can be viewed as a "warm-up" policy, which is then refined with RLHF. Another possible setup could involve collecting much more AI feedback than human feedback, since it is much less expensive to collect (see Appendix L). We leave this exploration to future work.

L. Cost of LLM vs. Human Labeling

Using LLMs as data annotators can be much less costly than hiring human annotators ([54]). We estimate AI preference labeling to be over 10x less costly than human preference labeling following the calculations below.

At the time of writing, GPT-4 charges $0.03 USD and $0.06 USD for every 1,000 tokens to encode and decode, respectively ([55]). For labeling Reddit TL;DR preferences with an LLM, our average token lengths were as follows:

1. Input prompt length - 830 tokens (using the "Detailed + CoT 0-shot" prompt)
2. Generated chain-of-thought rationale - 61 tokens

Additionally, to mitigate position bias, we repeat each labeling procedure after inverting the order in which a pair of responses are shown. Our estimated AI labeling cost per example is $0.06 USD[^14].

[^14]: 2 inferences * (830 encoder tokens * $0.03 / 1,000 tokens + 61 decoder tokens * $0.06 / 1,000 tokens) = $0.057 $\sim$ = $0.06

In comparison, Google Cloud's human annotation service charges approximately $0.11 USD / 50 words for classification tasks at the time of writing[^15] ([56]). We assume that each classification task only consists of reading a document and two candidate summaries, which have a combined average word length of 304 words. We estimate the human labeling cost per example to be $0.67 USD (304 words * $0.11 / 50 words).

[^15]: Google Cloud charges between $90 and $129 per 1,000 units, where each unit is 50 words for a classification task. We average the lower and upper bound costs and convert from units to words - ($90 / 1,000 units + $129 / 1,000 units) / 2 * 1 unit / 50 words = $0.1095 USD / 50 words

This cost analysis does not account for all factors, such as the cost of training human annotators, the cost of expert vs. crowd-sourced annotators, or the cost of setting up LLM labeling.

M. Self-Consistency

: Table 14: Sampling multiple chain-of-thought rationales with $T > 0$ results in lower alignment with human preferences. Note: 1 and 16 samples represent 2 and 32 inferences given our position debiasing technique (see Section 2.1.1).

We experiment with self-consistency ([21]) - a technique to generate robust chain-of-thought rationales. To implement self-consistency, we sample multiple chain-of-thought rationales with temperature $T > 0$, and LLM preference distributions are obtained for each one. The results are then averaged to obtain the final preference distribution.

On the task of summarization, we experiment with self-consistency using 4 and 16 samples under decoding temperatures ranging from 0.3 to 1.0 (see Figure 14)[^16]. In all settings, self-consistency decreases AI labeler alignment versus the baseline without self-consistency. Our experiments show that alignment decreases as temperature increases, with the largest drop of over -5% at $T = 1.0$. In our experiments, using 4 vs. 16 self-consistency samples does not impact AI labeler alignment.

[^16]: Results of using 4 samples are not shown because they only differ from the 16-sample results by $\pm$0.4%.

Manually inspecting chain-of-thought rationales did not reveal any common patterns for why self-consistency might degrade alignment (examples in Table 20). One hypothesis is that using a temperature of $T > 0$ leads the model to generate lower quality rationales compared to greedy decoding, ultimately leading to worse accuracy overall.

N. End-to-end Sensitivity to AI Labeler Alignment

We assess the end-to-end sensitivity of the RLAIF policies to AI labeler alignment on the task of summarization. Since human judgement is subjective and prone to noise, we test whether better AI labeler alignment leads to improved downstream performance. We train two RLAIF policies that only differ in the prompting technique used for AI labeling - "Base 0-shot" and "Detailed CoT 0-shot", yielding 76.1% and 78.0% AI labeler alignment, respectively.

When compared head-to-head, human evaluators prefer summaries from "Detailed CoT 0-shot" 59% of the time over "Base 0-shot"[^17]. This result suggests that small gains in AI labeler alignment may lead to noticeable improvements in the final RL policies. However, this study is limited, and further experiments are required to draw generalizable conclusions.

[^17]: Result is statistically significantly different from 50% according to a binomial test.

::: {caption="Table 15: An example of a prompt fed to an off-the-shelf LLM to generate AI preference labels for summarization. text, summary1, and summary2 are populated with unlabeled examples, and a preference distribution is obtained by computing the softmax of the log-probabilities of generating the tokens '1' vs. '2'."}

:::

::: {caption="Table 16: The 'Base' and 'Detailed' preambles given to the LLM labeler to obtain preference labels for the summarization task."}

:::

::: {caption="Table 17: The prompt used for the 'Detailed + CoT 0-shot' for summarization. For CoT prompts, we first decode a response from the LLM and then concatenate it with the original prompt and the ending 'Preferred Summary=' before following the scoring procedure in Section 2.1 to obtain a preference distribution."}

:::

::: {caption="Table 18: The template used for the 'Detailed + CoT 1-shot' prompt for summarization, with some text removed for brevity."}

:::

::: {caption="Table 19: An example of the different chain-of-thought rationales produced by the 0-shot ('Detailed + CoT 0-shot') vs. 1-shot ('Detailed + CoT 1-shot') prompts for summarization. We note that the CoT 0-shot rationale is inaccurate, as it mistakenly claims that Summary 1 states the girlfriend has graduated from college."}

:::

::: {caption="Table 20: An example comparing chain-of-thought rationales produced at different temperatures for self-consistency experiments. Rationale @ $T = 1.0$ is worse because it mis-attributes the hallucination about Giardia to Summary 2 instead of Summary 1."}

:::

::: {caption="Table 21: The 'Base + CoT 0-shot' prompting template for the helpful dialogue generation task. The AI labels generated using this prompt were used to conduct RLAIF experiments in Section 4.1."}

:::

::: {caption="Table 22: The 'Base + 2-shot' prompting template for the harmless dialogue generation task. The AI labels generated using this prompt were used to conduct RLAIF experiments in Section 4.1."}

:::

::: {caption="Table 23: We observe that the RLHF policy sometimes hallucinates when the RLAIF policy does not. Hallucinations are highlighted in red."}

:::

::: {caption="Table 24: We observe that that summaries from the RLAIF policy are sometimes less coherent and grammatical than summaries from the RLHF policy. Less coherent phrases are highlighted in red."}

:::

References

Section Summary: This section compiles a numbered list of academic papers, technical reports, and preprints that serve as sources for the document. The entries focus on advancements in training large language models with human feedback, evaluating systems like ChatGPT and GPT-4, and exploring related techniques such as prompting and reinforcement learning. Many come from conferences like NeurIPS or arXiv and address issues including model alignment, data annotation, and performance assessment.

[1] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. Constitutional ai: Harmlessness from ai feedback, 2022b.

[2] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.

[3] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.

[4] Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852, 2023.

[5] Manyika, J. An overview of bard: an early experiment with generative ai. https://ai.google/static/documents/google-about-bard.pdf, 2023. Accessed: 2023-08-23.

[6] Gilardi, F., Alizadeh, M., and Kubli, M. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023.

[7] Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., and Bing, L. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11173–11195, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:10.18653/v1/2023.acl-long.626. URL https://aclanthology.org/2023.acl-long.626.

[8] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.

[9] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

[10] Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.

[11] Pezeshkpour, P. and Hruschka, E. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483, 2023.

[12] Wang, P., Li, L., Chen, L., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023.

[13] Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a.

[14] Ethayarajh, K., Choi, Y., and Swayamdipta, S. Understanding dataset difficulty with $\mathcal{V}$-usable information. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988–6008. PMLR, 17–23 Jul 2022.

[15] Google, R. A., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Abrego, G. H., Ahn, J., Austin, J., Barham, P., Botha, J., Bradbury, J., Brahma, S., Brooks, K., Catasta, M., Cheng, Y., Cherry, C., Choquette-Choo, C. A., Chowdhery, A., Crepy, C., Dave, S., Dehghani, M., Dev, S., Devlin, J., Díaz, M., Du, N., Dyer, E., Feinberg, V., Feng, F., Fienber, V., Freitag, M., Garcia, X., Gehrmann, S., Gonzalez, L., Gur-Ari, G., Hand, S., Hashemi, H., Hou, L., Howland, J., Hu, A., Hui, J., Hurwitz, J., Isard, M., Ittycheriah, A., Jagielski, M., Jia, W., Kenealy, K., Krikun, M., Kudugunta, S., Lan, C., Lee, K., Lee, B., Li, E., Li, M., Li, W., Li, Y., Li, J., Lim, H., Lin, H., Liu, Z., Liu, F., Maggioni, M., Mahendru, A., Maynez, J., Misra, V., Moussalem, M., Nado, Z., Nham, J., Ni, E., Nystrom, A., Parrish, A., Pellat, M., Polacek, M., Polozov, A., Pope, R., Qiao, S., Reif, E., Richter, B., Riley, P., Ros, A. C., Roy, A., Saeta, B., Samuel, R., Shelby, R., Slone, A., Smilkov, D., So, D. R., Sohn, D., Tokumine, S., Valter, D., Vasudevan, V., Vodrahalli, K., Wang, X., Wang, P., Wang, Z., Wang, T., Wieting, J., Wu, Y., Xu, K., Xu, Y., Xue, L., Yin, P., Yu, J., Zhang, Q., Zheng, S., Zheng, C., Zhou, W., Zhou, D., Petrov, S., and Wu, Y. Palm 2 technical report, 2023.

[16] Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992.

[17] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

[18] Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.

[19] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048–11064, 2022.

[20] Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., and Sun, H. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2022a.

[21] Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022b.

[22] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.

[23] Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.

[24] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.

[25] OpenAI. Gpt-4 technical report, 2023a.

[26] Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.

[27] Wu, L., Tian, F., Qin, T., Lai, J., and Liu, T.-Y. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3612–3621, 2018.

[28] Gao, Y., Meyer, C. M., Mesgar, M., and Gurevych, I. Reward learning for efficient reinforcement learning in extractive document summarisation. arXiv preprint arXiv:1907.12894, 2019.

[29] Wu, Y. and Hu, B. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5602, 2018.

[30] Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.

[31] Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.

[32] Lai, V. D., Van Nguyen, C., Ngo, N. T., Nguyen, T., Dernoncourt, F., Rossi, R. A., and Nguyen, T. H. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. arXiv preprint arXiv:2307.16039, 2023.

[33] Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022.

[34] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.

[35] Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.

[36] Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=m7p5O7zblY.

[37] Wang, Z., Yu, A. W., Firat, O., and Cao, Y. Towards zero-label language learning. arXiv preprint arXiv:2109.09193, 2021b.

[38] Meng, Y., Michalski, M., Huang, J., Zhang, Y., Abdelzaher, T., and Han, J. Tuning language models as training data generators for augmentation-enhanced few-shot learning. In International Conference on Machine Learning, pp. 24457–24477. PMLR, 2023.

[39] Feng, S. Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., and Hovy, E. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 968–988, Online, August 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.findings-acl.84. URL https://aclanthology.org/2021.findings-acl.84.

[40] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.

[41] Roit, P., Ferret, J., Shani, L., Aharoni, R., Cideron, G., Dadashi, R., Geist, M., Girgin, S., Hussenot, L., Keller, O., et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00186, 2023.

[42] Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2022.

[43] Yang, K., Klein, D., Celikyilmaz, A., Peng, N., and Tian, Y. Rlcd: Reinforcement learning from contrast distillation for language model alignment, 2023.

[44] Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562, 2015.

[45] Geist, M., Scherrer, B., and Pietquin, O. A theory of regularized markov decision processes. In International Conference on Machine Learning, pp. 2160–2169. PMLR, 2019.

[46] Everitt, T. and Hutter, M. Avoiding wireheading with value reinforcement learning. In Artificial General Intelligence: 9th International Conference, AGI 2016, New York, NY, USA, July 16-19, 2016, Proceedings 9, pp. 12–22. Springer, 2016.

[47] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.

[48] Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889–898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi:10.18653/v1/P18-1082. URL https://aclanthology.org/P18-1082.

[49] Howard, R. A. Dynamic programming and markov processes. John Wiley, 1960.

[50] Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.

[51] Jaques, N., Gu, S., Bahdanau, D., Hernández-Lobato, J. M., Turner, R. E., and Eck, D. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pp. 1645–1654. PMLR, 2017.

[52] Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. CoRR, abs/1804.04235, 2018. URL http://arxiv.org/abs/1804.04235.

[53] Kendall, M. G. and Smith, B. B. The Problem of $m$ Rankings. The Annals of Mathematical Statistics, 10(3):275 – 287, 1939. doi:10.1214/aoms/1177732186. URL https://doi.org/10.1214/aoms/1177732186.

[54] Wang, S., Liu, Y., Xu, Y., Zhu, C., and Zeng, M. Want to reduce labeling cost? gpt-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4195–4205, 2021a.

[55] OpenAI. Openai pricing. https://openai.com/pricing, 2023b. Accessed: 2023-09-28.

[56] Google. Ai platform data labeling service pricing. https://cloud.google.com/ai-platform/data-labeling/pricing#labeling_costs, 2023. Accessed: 2023-09-28.