From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models

Tarun Raheja $^{,1}$ Nilay Pochhi $^{,1}$
$^{1}$ Independent Researchers
[email protected], [email protected]
$^{*}$ Equal contribution

Abstract

Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives---Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others---has left practitioners without clear guidance on method selection. This survey provides a theoretical unification of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: (I) Preference Model (what likelihood model underlies the objective), (II) Regularization Mechanism (how deviation from reference policies is controlled), and (III) Data Distribution (online vs.\ offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes---length hacking, mode collapse, likelihood displacement---arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner's decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline.

Keywords: Preference learning, RLHF, direct preference optimization, Bradley-Terry model, language model alignment

Executive Summary: Large language models require alignment with human preferences for safe deployment, yet the shift from Reinforcement Learning from Human Feedback to simpler direct methods has created confusion over which approach to choose. The article addresses this by unifying the growing set of preference learning techniques under a single theoretical lens.

The article set out to show that methods such as DPO, IPO, KTO, SimPO, and ORPO are not unrelated inventions but different choices along three clear axes: the preference model that turns human judgments into an objective, the regularization that limits deviation from a reference policy, and the data distribution that governs online versus offline learning. It draws on formal theorems, coverage arguments, and a synthesis of findings from more than fifty papers.

The analysis establishes that offline contrastive methods need broad data coverage while online reinforcement learning needs far less; that length hacking, mode collapse, and likelihood displacement each trace to predictable combinations of design choices; and that SimPO frequently outperforms DPO because length normalization and an explicit margin correct biases that DPO leaves unaddressed. Empirical comparisons confirm these patterns across instruction-following and safety benchmarks.

These results matter because practitioners can now match method to constraints rather than relying on trial and error. When preference data are diverse, SimPO offers the best balance of simplicity and performance; when data coverage is limited or exploration beyond the data is required, PPO remains preferable despite higher cost. The framework also shows that data quality improvements often outweigh algorithmic tweaks.

The article recommends SimPO as the default for most settings with moderate compute, PPO when coverage is incomplete, and hybrid online-offline approaches when both stability and exploration matter. It notes that richer feedback beyond pairwise comparisons and methods for multi-objective or shifting preferences remain open areas. Limitations include reliance on idealized coverage assumptions and the fact that real-world preference distributions are rarely stationary, so results should be validated on domain-specific data before large-scale deployment.

1. Introduction

Section Summary: Reinforcement learning from human feedback, or RLHF, became the standard method for aligning large language models with human values after early successes with systems like InstructGPT. It typically involves supervised fine-tuning on demonstrations, training a reward model on preference data, and then using reinforcement learning to optimize the model, but this approach often proves unstable and prone to exploiting flaws in the reward signal. In response, researchers developed numerous simpler alternatives such as DPO and SimPO, yet the variety left practitioners without clear guidance on when to use each one; this survey addresses the gap by organizing the methods into a shared theoretical framework based on choices about preference modeling, regularization, and data assumptions.

The Orthodoxy.

Since Christiano et al. [1] introduced learning from human preferences and Ouyang et al. [2] demonstrated its transformative potential with InstructGPT, Reinforcement Learning from Human Feedback (RLHF) has become the canonical approach for aligning large language models with human values. The standard RLHF pipeline consists of three stages: (1) supervised fine-tuning (SFT) on high-quality demonstrations, (2) reward model training on pairwise human preferences, and (3) policy optimization via Proximal Policy Optimization (PPO) [3] to maximize the learned reward while staying close to the SFT policy. This paradigm underlies the alignment of GPT-4 [4], Claude, and other frontier models.

The Challenge.

Yet RLHF's dominance belies significant practical difficulties. PPO is notoriously unstable, requiring careful hyperparameter tuning and multiple model copies [5, 6]. The reward model introduces a "proxy" that can be exploited, leading to Goodhart's law violations where optimizing the proxy degrades true performance [7]. These challenges motivated a wave of alternatives: Direct Preference Optimization (DPO) [8] eliminates the reward model entirely; Identity Preference Optimization (IPO) [9] addresses DPO's overfitting to deterministic preferences; Kahneman-Tversky Optimization (KTO) [10] requires only binary feedback; Simple Preference Optimization (SimPO) [11] removes the reference model; and Odds Ratio Preference Optimization (ORPO) [12] unifies SFT with preference learning.

The resulting landscape is bewildering. When should one use DPO versus PPO? Why does SimPO outperform DPO despite apparent simplicity? Why do all methods suffer from length hacking? Practitioners face an embarrassment of riches without theoretical guidance.

The Resolution.

This survey provides that guidance through a unified theoretical framework. We show that the zoo of preference learning methods reduces to principled choices along three orthogonal axes:

• Pillar I: Preference Model — What probabilistic model relates rewards to preferences? (Bradley-Terry, Plackett-Luce, prospect-theoretic, game-theoretic)
• Pillar II: Regularization — How is policy deviation from a reference controlled? (Explicit KL, implicit KL, reference-free, f-divergences)
• Pillar III: Data Distribution — What assumptions on data coverage enable learning? (Online, offline, hybrid)

This taxonomy is not merely organizational—it is predictive. We show that failure modes arise from specific combinations: length hacking from reward model bias interacting with insufficient regularization; mode collapse from overly strong KL penalties; likelihood displacement from offline data limitations.

Contributions.

This survey makes four contributions:

1. A unified theoretical framework showing DPO, IPO, KTO, SimPO, ORPO, and GRPO as special cases of a general $\Psi$PO objective (Section 3).
2. Formal theorems establishing the coverage separation between online/offline methods, conditions for preference collapse, and overoptimization scaling laws (Section 5 and Section 6).
3. A systematic taxonomy of failure modes connecting theoretical properties to empirical pathologies (Section 7).
4. A practitioner's decision guide providing actionable recommendations based on constraints (Section 9).

2. Preliminaries: Choice Theory Meets Language Models

Section Summary: This section links classical ideas from choice theory to the mathematics of aligning large language models. It defines policies as probability distributions over responses to prompts and reward functions that score those pairs, then introduces the Bradley-Terry model, which converts reward differences into preference probabilities via a sigmoid while assuming preferences are scale-free and transitive. The section also presents the standard RLHF training objective, which maximizes expected reward subject to a KL penalty that keeps the policy close to a reference model, and states the resulting closed-form optimal policy that underpins later direct preference methods.

We establish the mathematical foundations connecting classical choice theory to modern LLM alignment.

2.1 Notation and Setup

Let $\mathcal{X}$ denote the space of prompts and $\mathcal{Y}$ the space of responses. A policy $\pi: \mathcal{X} \to \Delta(\mathcal{Y})$ maps prompts to distributions over responses. We write $\pi(y|x)$ for the probability of response $y$ given prompt $x$. A reward function $r: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ assigns scalar values to prompt-response pairs.

2.2 The Bradley-Terry Model

The Bradley-Terry model [13] provides the foundational preference model for RLHF.

########## {caption="Definition 1: Bradley-Terry Preference Model"}

Given a reward function $r$, the probability that response $y_1$ is preferred to $y_2$ given prompt $x$ is:

$ p^*(y_1 \succ y_2 | x) = \sigma(r(x, y_1) - r(x, y_2)) $

where $\sigma(z) = 1/(1 + e^{-z})$ is the sigmoid function.

The Bradley-Terry model makes two key assumptions: (1) preferences depend only on reward differences, making the reward scale-free; (2) preferences are transitive—if $y_1 \succ y_2$ and $y_2 \succ y_3$, then $y_1 \succ y_3$ with high probability. Both assumptions are violated in practice, motivating extensions we discuss in Section 4.

2.3 The Canonical RLHF Objective

The standard RLHF objective balances reward maximization against deviation from a reference policy $\pi_{\mathrm{ref}}$:

########## {caption="Definition 2: KL-Regularized Reward Maximization"}

The RLHF objective is:

$ \max_\pi \mathbb{E}{x \sim \mathcal{D}, y \sim \pi(\cdot|x)} \left[ r(x, y) - \beta \mathrm{KL}(\pi(\cdot|x) | \pi{\mathrm{ref}}(\cdot|x)) \right] $

where $\beta > 0$ controls regularization strength and $\mathcal{D}$ is the prompt distribution.

The KL penalty serves multiple purposes: preventing reward hacking by keeping $\pi$ near the capable $\pi_{\mathrm{ref}}$; maintaining response diversity; and ensuring the optimization problem is well-posed.

########## {caption="Theorem 3: Optimal Policy Form [8, 14]"}

The optimal policy for Equation 2 has the closed form:

$ \pi^*(y|x) = \frac{1}{Z(x)} \pi_{\mathrm{ref}}(y|x) \exp\left(\frac{r(x,y)}{\beta}\right) $

where $Z(x) = \sum_y \pi_{\mathrm{ref}}(y|x) \exp(r(x,y)/\beta)$ is the partition function.

This result is the foundation of direct preference optimization methods: by inverting Equation 3, rewards can be expressed in terms of policies, eliminating the need for explicit reward modeling.

3. A Unified Framework: $\Psi$PO

Section Summary: The ΨPO framework unifies a range of preference-learning methods by expressing them as different choices of a single convex function applied to the gap between a model’s log-probability ratios on preferred and dispreferred responses. It corrects two common shortcuts in standard RLHF: the conversion of pairwise preferences into pointwise rewards and the need to train a separate reward model that must generalize to new outputs. By varying the convex function, the same template recovers DPO, IPO, and other algorithms, revealing that they mainly differ in how strongly they penalize small or negative margins while keeping training reward-free.

We present the $\Psi$PO framework of Azar et al. [9], which unifies preference learning methods and reveals their structural relationships.

3.1 The Two Approximations in RLHF

Standard RLHF makes two approximations that introduce potential errors:

1. Pointwise Reward Approximation: Pairwise preferences are converted to pointwise rewards via the Bradley-Terry model, discarding relational information.
2. Reward Model Generalization: A reward model trained on finite data must generalize to out-of-distribution responses sampled by the policy.

DPO addresses the second approximation by eliminating the reward model but retains the first. The $\Psi$PO framework addresses both.

3.2 The General $\Psi$PO Objective

########## {caption="Definition 4: $\Psi$PO Objective [9]"}

For a convex function $\Psi: \mathbb{R} \to \mathbb{R}$, the $\Psi$PO objective is:

$ \mathcal{L}{\Psi}(\pi) = \mathbb{E}{(x, y_w, y_l) \sim \mathcal{D}} \left[ \Psi\left( \beta \log \frac{\pi(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta \log \frac{\pi(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} \right) \right] $

where $y_w$ denotes the preferred ("winning") response and $y_l$ the dispreferred ("losing") response.

Different choices of $\Psi$ recover existing methods:

########## {caption="Theorem 5: Instantiations of $\Psi$PO"}

• $\Psi(z) = -\log\sigma(z)$: DPO [8]
• $\Psi(z) = (z - 1)^2$: IPO [9]
• $\Psi(z) = \max(0, 1-z)$: SLiC-HF [15]

This unification reveals that the methods differ primarily in how they handle the margin between preferred and dispreferred log-ratios.

3.3 DPO: The Reparameterization Trick

DPO's key insight is that the reward can be expressed purely in terms of policies:

########## {caption="Theorem 6: DPO Reparameterization [8]"}

Under the optimal policy form Equation (3), the reward satisfies:

$ r(x, y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\mathrm{ref}}(y|x)} + \beta \log Z(x) $

Substituting into the Bradley-Terry model yields the DPO loss:

$ \mathcal{L}{\mathrm{DPO}}(\pi\theta) = -\mathbb{E} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} \right) \right] $

The partition function $Z(x)$ cancels in the preference probability, enabling reward-free optimization.

3.4 Beyond DPO: IPO, KTO, and Reference-Free Methods

IPO.

Identity Preference Optimization [9] addresses DPO's tendency to overfit when preferences are deterministic. Instead of the log-sigmoid, IPO uses a squared loss:

$ \mathcal{L}{\mathrm{IPO}}(\pi\theta) = \mathbb{E}\left[ \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} - 1 \right)^2 \right] $

The target margin of 1 (in log-odds) provides regularization absent in DPO.

KTO.

Kahneman-Tversky Optimization [10] draws on prospect theory, requiring only binary "good"/"bad" labels rather than pairwise comparisons:

$ \mathcal{L}{\mathrm{KTO}}(\pi\theta) = \mathbb{E}{y \sim \mathcal{D}^+}\left[ w(y) \cdot \ell+(y) \right] + \mathbb{E}{y \sim \mathcal{D}^-}\left[ w(y) \cdot \ell-(y) \right] $

where $\mathcal{D}^+, \mathcal{D}^-$ denote good and bad responses, and the losses are asymmetric reflecting loss aversion.

SimPO.

Simple Preference Optimization [11] eliminates the reference model entirely by using length-normalized log-probabilities as implicit rewards:

$ \mathcal{L}{\mathrm{SimPO}}(\pi\theta) = -\mathbb{E}\left[ \log \sigma\left( \frac{\beta}{| y_w|} \log \pi_\theta(y_w|x) - \frac{\beta}{| y_l|} \log \pi_\theta(y_l|x) - \gamma \right) \right] $

where $\gamma > 0$ is a target margin. The length normalization addresses verbosity bias.

ORPO.

Odds Ratio Preference Optimization [12] combines SFT and preference learning in a single stage:

$ \mathcal{L}{\mathrm{ORPO}} = \mathcal{L}{\mathrm{SFT}}(y_w) + \lambda \cdot \mathcal{L}_{\mathrm{OR}}(y_w, y_l) $

where $\mathcal{L}_{\mathrm{OR}}$ contrasts odds ratios rather than log-probabilities.

Table 1 summarizes the relationships between methods.

\begin{tabular}{@lcccccc@}
\toprule
\textbf{Method} & \textbf{Pref. Model} & \textbf{Regularization} & \textbf{Reference?} & \textbf{Data} & \textbf{Loss Form} & \textbf{Key Property} \\
\midrule
PPO/RLHF & Bradley-Terry & Explicit KL & Yes & Online & Policy gradient & Flexible, unstable \\
DPO [8] & Bradley-Terry & Implicit KL & Yes & Offline & $-\log\sigma(\cdot)$ & Simple, overfit-prone \\
IPO [9] & Squared margin & Implicit + margin & Yes & Offline & $(z-1)^2$ & Regularized margin \\
KTO [10] & Prospect theory & Implicit KL & Yes & Offline & Asymmetric & Binary feedback \\
SimPO [11] & Bradley-Terry & Target margin & No & Offline & $-\log\sigma(\cdot-\gamma)$ & Reference-free \\
ORPO [12] & Odds ratio & SFT + odds & No & Offline & Combined & Single-stage \\
GRPO [16] & Bradley-Terry & Group relative & Yes & Online & Policy gradient & Group normalization \\
\bottomrule
\end{tabular}

4. Pillar I: Preference Models and Their Limits

Section Summary: The choice of preference model fundamentally determines what can be learned from human feedback, as it dictates how rewards are inferred from observed choices. The widely used Bradley-Terry model assumes that preferences reflect consistent underlying quality scores, yet this breaks down with intransitive human judgments and differing annotator tastes, which pairwise data alone cannot reliably distinguish. Alternatives such as Plackett-Luce rankings or Nash equilibrium approaches address some of these gaps by handling multi-response orderings or non-transitive preferences without assuming a single fixed reward function.

The choice of preference model—how rewards relate to human choices—fundamentally shapes what can be learned.

4.1 Bradley-Terry: Assumptions and Violations

The Bradley-Terry model assumes preferences arise from comparing latent "quality" scores. This implies:

########## {caption="Assumption 7: Bradley-Terry Regularity"}

There exists a reward function $r^*: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ such that for all $x, y_1, y_2$:

$ p(y_1 \succ y_2 | x) = \sigma(r^*(x, y_1) - r^*(x, y_2)) $

This assumption is violated in several ways:

Intransitivity.

Human preferences exhibit cycles: $A \succ B$, $B \succ C$, but $C \succ A$. This occurs when different attributes dominate different comparisons. Munos et al. [17] formalize this via Nash Learning from Human Feedback (NLHF), treating alignment as a two-player game.

Annotator Heterogeneity.

Different annotators have different preferences. Qin et al. [18] show that binary comparisons cannot identify latent annotator types:

########## {caption="Proposition 8: Identification Failure [18]"}

With heterogeneous annotators and only pairwise preferences, the latent preference distribution is not identifiable from finite data. Rankings over three or more responses are necessary for identification.

This motivates methods like Expectation-Maximization DPO that explicitly model annotator mixtures.

4.2 Beyond Bradley-Terry

Plackett-Luce.

For ranking multiple responses, the Plackett-Luce model generalizes Bradley-Terry:

$ p(\sigma | x, y_1, \ldots, y_k) = \prod_{i=1}^{k} \frac{\exp(r(x, y_{\sigma(i)}))}{\sum_{j=i}^{k} \exp(r(x, y_{\sigma(j)}))} $

where $\sigma$ is a permutation (ranking). RRHF [19] and listwise methods exploit this structure.

Nash Learning.

When preferences are non-transitive, the goal shifts from reward maximization to computing a Nash equilibrium:

########## {caption="Definition 9: Nash Learning from Human Feedback [17]"}

Find policy $\pi^*$ such that for all $\pi$:

$ \mathbb{E}{y \sim \pi^*, y' \sim \pi}[p(y \succ y')] \geq \mathbb{E}{y \sim \pi, y' \sim \pi^*}[p(y \succ y')] $

EGPO [20] achieves last-iterate convergence to this equilibrium using extragradient methods.

5. Pillar II: The Role of Regularization

Section Summary: Regularization limits how far a learned policy can stray from a reference model, which helps keep preference optimization stable and prevents erratic outputs. In methods like PPO the constraint appears directly in the reward, while in DPO it is only enforced at the final solution, allowing the model to overfit more readily during training. Without this control the policy collapses to a single high-reward response and ignores minority preferences, prompting work on alternative divergence measures and reference-free approaches.

Regularization—controlling how far the learned policy deviates from a reference—is crucial for stable preference learning.

5.1 Explicit vs. Implicit KL

In PPO-based RLHF, the KL penalty appears explicitly in the reward:

$ \tilde{r}(x, y) = r(x, y) - \beta \log \frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)} $

In DPO, the KL constraint is implicit: the reparameterization (Theorem 6) assumes the policy takes the optimal form, which automatically satisfies a KL constraint. However, this implicit regularization has different properties.

########## {caption="Proposition 10: DPO's Implicit Regularization [9]"}

DPO's objective can be written as:

$ \mathcal{L}{\mathrm{DPO}} = -\mathbb{E}\left[ \log \sigma\left( r\theta(y_w) - r_\theta(y_l) \right) \right] $

where $r_\theta(y) = \beta \log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x))$ is the implicit reward. The KL constraint is enforced only at the optimal solution, not during optimization.

This distinction explains why DPO can overfit more easily than PPO: the regularization doesn't actively constrain intermediate iterates.

5.2 What Happens Without Regularization

Removing or weakening KL regularization leads to degenerate solutions:

########## {caption="Theorem 11: Preference Collapse [21]"}

Without KL regularization, maximizing expected reward under the Bradley-Terry model leads to:

$ \pi^*(y|x) = \begin{cases} 1 & \text{if } y = \arg\max_{y'} r(x, y') \ 0 & \text{otherwise} \end{cases} $

This deterministic policy ignores minority preferences, achieving high reward but low diversity.

Xiao et al. [21] term this preference collapse and show it disproportionately affects underrepresented groups in preference data.

5.3 Reference Model Dependence

DPO-family methods depend critically on the reference model $\pi_{\mathrm{ref}}$:

########## {caption="Proposition 12: Reference Sensitivity [22]"}

The implicit reward in DPO is:

$ r_\theta(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)} $

This reward is undefined for $y$ where $\pi_{\mathrm{ref}}(y|x) = 0$, and poorly calibrated where $\pi_{\mathrm{ref}}(y|x)$ is small.

This motivates reference-free methods (SimPO, ORPO) and multi-reference approaches (MRPO [23]) that average over multiple reference models.

5.4 Alternative Regularizers

The f-divergence framework generalizes KL regularization:

########## {caption="Definition 13: f-Divergence Regularization [24]"}

For convex $f$ with $f(1) = 0$:

$ D_f(\pi | \pi_{\mathrm{ref}}) = \mathbb{E}{y \sim \pi{\mathrm{ref}}}\left[ f\left( \frac{\pi(y|x)}{\pi_{\mathrm{ref}}(y|x)} \right) \right] $

Wang et al. [24] show that different f-divergences (reverse KL, Jensen-Shannon, $\alpha$-divergence) yield different trade-offs between mode-seeking and mode-covering behavior.

6. Pillar III: Online vs. Offline Learning

Section Summary: The core design choice in preference optimization is whether to rely on offline data collected in advance or online data generated by the model itself during training. Offline approaches need training examples that broadly cover all plausible responses a model might produce, while online methods can succeed with narrower data because they explore and collect new examples on their own. This difference explains why hybrids that start with offline data and then refine online often achieve better results with less total data.

Perhaps the most consequential design choice is whether to use online (on-policy) or offline (off-policy) data.

6.1 Coverage Conditions

The key theoretical concept is coverage: how well the training data distribution covers the space of responses the policy might generate.

########## {caption="Definition 14: Global Coverage"}

A preference dataset $\mathcal{D}$ satisfies $C$-global coverage if for all policies $\pi$ and prompts $x$:

$ \mathbb{E}_{y \sim \pi(\cdot|x)}\left[ \frac{\pi(y|x)}{\mu(y|x)} \right] \leq C $

where $\mu$ is the data collection distribution.

########## {caption="Definition 15: Partial Coverage"}

A dataset satisfies partial coverage if the coverage condition holds only for the optimal policy $\pi^*$, not all policies.

6.2 The Coverage Separation

Song et al. [25] establish a fundamental separation:

########## {caption="Theorem 16: Coverage Separation [25]"}

1. Offline contrastive methods (DPO, IPO) require global coverage for convergence to the optimal policy.
2. Online RL methods (PPO) require only partial coverage.

Proof Sketch: Offline methods optimize over a fixed dataset and cannot explore. If the dataset lacks coverage of good responses, these responses have zero gradient signal. Online methods generate their own data, enabling exploration of high-reward regions even if initially uncovered.

This theorem explains empirical findings that PPO outperforms DPO when preference data is not diverse [26, 27].

6.3 Hybrid Approaches

The coverage separation motivates hybrid methods combining offline initialization with online refinement:

HyPO.

Hybrid Preference Optimization [25] uses offline data for the contrastive loss and online data for KL regularization:

$ \mathcal{L}{\mathrm{HyPO}} = \mathcal{L}{\mathrm{DPO}}^{\mathrm{offline}} + \lambda \cdot \mathrm{KL}(\pi | \pi_{\mathrm{ref}})^{\mathrm{online}} $

Iterative DPO.

Xiong et al. [28] alternate between generating new preference data with the current policy and running DPO, converting offline DPO into an online algorithm.

########## {caption="Theorem 17: Hybrid Sample Complexity [29]"}

Hybrid methods achieve sample complexity:

$ O\left( \frac{1}{\epsilon^2} \cdot \min(C_{\mathrm{global}}, C_{\mathrm{partial}} + n_{\mathrm{online}}) \right) $

interpolating between offline and online rates.

7. Failure Modes and Pathologies

Section Summary: This section examines several recurring problems that arise when training language models with preference-based methods such as RLHF and DPO. The first is reward overoptimization, in which models exploit inaccuracies in learned reward signals so that proxy scores keep rising while actual performance eventually falls. Additional issues include a tendency for models to produce unnecessarily long outputs because both human raters and reward models favor length, and an imbalance in DPO’s gradient updates that lowers the likelihood of dispreferred responses more sharply than it raises preferred ones, sometimes harming overall capabilities.

We systematically analyze failure modes, connecting them to the theoretical framework.

7.1 Reward Overoptimization

Gao et al. [7] establish scaling laws for reward model exploitation:

########## {caption="Theorem 18: Overoptimization Scaling [7]"}

Let $R_{\mathrm{gold}}$ denote true reward and $R_{\mathrm{proxy}}$ the learned reward model. Under RL optimization:

$ R_{\mathrm{gold}} = d_0 + d_1 \sqrt{\mathrm{KL}} - d_2 \cdot \mathrm{KL} $

Under best-of-$n$ sampling:

$ R_{\mathrm{gold}} = d_0 + d_1 \sqrt{d_3 \log n} - d_2 \cdot d_3 \log n $

where coefficients scale predictably with reward model size.

Critically, Rafailov et al. [30] show that DPO exhibits identical overoptimization patterns despite lacking an explicit reward model, occurring even before completing one epoch of training.

7.2 Length Hacking

Models optimized with RLHF or DPO systematically increase response length [31]:

########## {caption="Proposition 19: Length Bias [31]"}

Human annotators and reward models exhibit length bias: longer responses are preferred even controlling for quality. Under DPO:

$ \nabla_\theta \mathcal{L}{\mathrm{DPO}} \propto \nabla\theta \log \pi_\theta(y_w|x) - \nabla_\theta \log \pi_\theta(y_l|x) $

If $| y_w| > | y_l|$ systematically, the gradient encourages verbosity.

SimPO's length normalization Equation (9) directly addresses this by using per-token log-probabilities.

7.3 Likelihood Displacement

DPO's gradient structure creates an asymmetry:

########## {caption="Proposition 20: Gradient Asymmetry [32]"}

The DPO gradient satisfies:

$ \left| \frac{\partial \mathcal{L}}{\partial \log \pi(y_l|x)} \right| > \left| \frac{\partial \mathcal{L}}{\partial \log \pi(y_w|x)} \right| $

when $\pi(y_w|x) > \pi(y_l|x)$. DPO decreases dispreferred likelihood faster than it increases preferred likelihood.

This "3D property" (Drastic drop, Degradation, Dispersion) [33] explains why DPO can harm capabilities on responses similar to dispreferred examples.

7.4 Failure Mode Summary

Table 2 connects failure modes to their theoretical causes.

: Table 2: Failure Modes and Their Causes. Each pathology traces to specific design choices in preference learning.

8. Empirical Landscape

Section Summary: Empirical results across standard benchmarks show that methods like SimPO often achieve stronger performance on instruction-following and reasoning tasks than DPO or PPO, while requiring less training compute. Comparisons reveal that PPO can edge out DPO with careful tuning, yet DPO matches it when given high-quality data, and SimPO benefits from built-in fixes for length bias and regularization. Overall, gains from better preference data consistently outweigh those from switching algorithms.

Table 3 summarizes empirical comparisons across major benchmarks.

\begin{tabular}{@llcccccc@}
\toprule
\textbf{Method} & \textbf{Base Model} & \textbf{AlpacaEval 2 (LC)} & \textbf{MT-Bench} & \textbf{Arena-Hard} & \textbf{GSM8K} & \textbf{Training Cost} \\
\midrule
SFT (baseline) & Llama-3-8B & 15.2\% & 7.1 & 18.3\% & 72.1\% & 1$\times$ \\
\midrule
PPO [2] & Llama-3-8B & 28.4\% & 7.8 & 31.2\% & 74.3\% & 4-8$\times$ \\
DPO [8] & Llama-3-8B & 25.1\% & 7.6 & 27.8\% & 73.2\% & 1.5$\times$ \\
IPO [9] & Llama-3-8B & 24.8\% & 7.5 & 26.9\% & 73.0\% & 1.5$\times$ \\
KTO [10] & Llama-3-8B & 23.6\% & 7.4 & 25.4\% & 72.8\% & 1.3$\times$ \\
SimPO [11] & Llama-3-8B & 31.5\% & 7.9 & 34.1\% & 73.9\% & 1.2$\times$ \\
ORPO [12] & Llama-3-8B & 26.3\% & 7.6 & 28.5\% & 73.1\% & 1.0$\times$ \\
\midrule
SimPO [11] & Gemma-2-9B-it & 72.4\% & 8.5 & 59.1\% & --- & 1.2$\times$ \\
WPO [34] & Gemma-2-9B-it & 76.7\% & 8.6 & 62.3\% & --- & 1.5$\times$ \\
\bottomrule
\end{tabular}

8.1 Key Empirical Findings

PPO vs. DPO.

The comparison is nuanced. Xu et al. [26] find PPO superior with careful tuning, while Ivison et al. [27] show DPO can match PPO with high-quality preference data. The coverage theorem (Theorem 16) predicts this: DPO wins when data is diverse, PPO when exploration matters.

SimPO's Success.

SimPO consistently outperforms DPO despite simpler design [11]. Our framework explains this: (1) length normalization addresses verbosity bias; (2) the target margin $\gamma$ provides explicit regularization missing in DPO; (3) reference-free design avoids reference model miscalibration.

Data Quality Dominates.

Ivison et al. [27] find that preference data quality matters more than algorithm choice—an 8% improvement from better data vs. 2.5% from PPO over DPO. This suggests the field may be optimizing the wrong variable.

9. Practitioner's Guide

Section Summary: The practitioner's guide offers a decision table to help select among alignment methods such as PPO, DPO, and SimPO depending on practical constraints like available compute, data quality and diversity, stability needs, and whether a reference model exists. For typical users with moderate resources and decent preference data, SimPO is presented as the default choice that balances simplicity, reliability, and results. PPO is advised when data coverage is narrow and extra exploration or compute is feasible, while DPO suits cases with high-quality data and a well-matched reference model.

Table 4 provides actionable recommendations.

: Table 4: Practitioner's Decision Guide. ✓ = suitable; ✓✓ = strongly recommended; — = not suitable.

Default Recommendation.

For most practitioners with moderate compute and reasonable preference data: SimPO offers the best trade-off of simplicity, stability, and performance.

When to Use PPO.

Choose PPO when: (1) preference data coverage is limited; (2) you need to explore capabilities beyond the data distribution; (3) you have compute budget for 4-8$\times$ training cost.

When to Use DPO.

Choose DPO when: (1) you have high-quality, diverse preference data; (2) you need a simple baseline; (3) reference model is well-calibrated to your domain.

10. Open Problems and Future Directions

Section Summary: Several important challenges remain in advancing how AI systems learn from human preferences. Researchers still lack a clear understanding of how such learning scales with model size or data volume, and current techniques are mostly restricted to simple pairwise comparisons rather than richer forms of feedback. Additional open questions involve balancing conflicting goals like helpfulness and safety, enabling models to adapt to shifting preferences over time, and explaining why alignment can sometimes occur through prompting alone.

10.1 Scaling Laws for Preference Learning

While Gao et al. [7] establish overoptimization scaling, fundamental questions remain: How does preference learning scale with model size? With data size? Is there a "Chinchilla" for alignment?

10.2 Beyond Pairwise Comparisons

Current methods are limited to pairwise (or binary) feedback. Listwise methods using Plackett-Luce models [19] and partial rankings remain underexplored. As Proposition 8 shows, richer feedback may be necessary for identifying heterogeneous preferences.

10.3 Multi-Objective Alignment

Real alignment involves multiple objectives (helpfulness, harmlessness, honesty) that may conflict. Sequential Preference Optimization [35] and multi-objective methods remain nascent. How to aggregate conflicting preferences while respecting Pareto efficiency is an open theoretical question.

10.4 Distribution Shift and Continual Learning

Preference distributions shift over time. How can aligned models adapt without catastrophic forgetting? The interaction between preference learning and continual learning is largely unexplored.

10.5 Theoretical Foundations of In-Context Alignment

Recent work shows LLMs can be aligned through prompting alone [36]. Understanding when and why in-context alignment works could reveal fundamental principles about preference learning in neural networks.

11. Conclusion

Section Summary: Preference learning began with a single dominant approach but has since branched into many competing methods, leaving users without clear direction. A new organizing framework based on preference modeling, regularization, and data distribution turns this collection of techniques into a more coherent, theoretically grounded field, with specific insights about why certain methods succeed or fail and when online approaches become necessary. These foundations matter increasingly as language models grow more powerful and alignment with human values grows more critical.

Preference learning has evolved from a single paradigm (RLHF with PPO) to a diverse ecosystem of methods. This proliferation reflects genuine algorithmic innovation but has left practitioners without clear guidance. Our unified framework—organizing methods along the axes of preference model, regularization mechanism, and data distribution—transforms the landscape from an empirical art into a theoretically grounded discipline.

The key insights are actionable: DPO requires diverse data due to coverage requirements; SimPO's success stems from addressing length bias and reference miscalibration; failure modes like overoptimization arise predictably from specific design choices. The coverage separation theorem explains when online methods are necessary; the preference collapse theorem explains why regularization is essential.

As language models grow more capable, aligning them with human values becomes both more important and more challenging. The theoretical foundations established here—connecting classical choice theory to modern deep learning—provide the principled basis for this ongoing effort.

Acknowledgments

We thank the broader research community whose work made this survey possible. We are grateful for open-source implementations that enable reproducible research in preference learning.

References

Section Summary: This section lists dozens of academic papers on training large language models to align with human preferences and instructions. The works focus on techniques such as reinforcement learning from human feedback, reward modeling, and various preference optimization methods, drawing from major AI conferences and preprint archives between 2017 and 2025. They reflect rapid progress in making AI outputs more helpful, safe, and consistent with human judgments.

[1] P.F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. NeurIPS, 2017.

[2] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.

[3] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

[4] OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.

[5] R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing? arXiv preprint arXiv:2210.01241, 2022.

[6] R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, et al. Secrets of RLHF in large language models part I: PPO. arXiv preprint arXiv:2307.04964, 2023.

[7] L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. ICML, 2023.

[8] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C.D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.

[9] M.G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036, 2023.

[10] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. KTO: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.

[11] Y. Meng, M. Xia, and D. Chen. SimPO: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024.

[12] J. Hong, N. Lee, and J. Thorne. ORPO: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024.

[13] R.A. Bradley and M.E. Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.

[14] J. Peters, K. Mülling, and Y. Altün. Relative entropy policy search. AAAI, 2010.

[15] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P.J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.

[16] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.

[17] R. Munos, M. Valko, D. Calandriello, M.G. Azar, M. Rowland, Z. Guo, Y. Tang, M. Geist, T. Mesnard, et al. Nash learning from human feedback. ICML, 2024.

[18] Y. Qin, et al. Direct preference optimization with unobserved preference heterogeneity. arXiv preprint arXiv:2510.15716, 2024.

[19] Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.

[20] R. Zhou, M. Fazel, and S.S. Du. Extragradient preference optimization (EGPO): Beyond last-iterate convergence for Nash learning from human feedback. arXiv preprint arXiv:2503.08942, 2025.

[21] J. Xiao, Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W.J. Su. On the algorithmic bias of aligning large language models with RLHF: Preference collapse and matching regularization. Journal of the American Statistical Association, 2024.

[22] Y. Lin, S. Seto, M. ter Hoeve, K. Metcalf, B. Theobald, X. Wang, Y. Zhang, C. Huang, and T. Zhang. On the limited generalization capability of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:2409.03650, 2024.

[23] H. Le, Q. Tran, D. Nguyen, K. Do, S. Mittal, K. Ogueji, and S. Venkatesh. Multi-reference preference optimization for large language models. arXiv preprint arXiv:2405.16388, 2024.

[24] C. Wang, Y. Jiang, C. Yang, H. Liu, and Y. Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240, 2023.

[25] Y. Song, G. Swamy, A. Singh, J.A. Bagnell, and W. Sun. The importance of online data: Understanding preference fine-tuning via coverage. NeurIPS, 2024.

[26] S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, G. Wang, C. Yu, and Y. Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. arXiv preprint arXiv:2404.10719, 2024.

[27] H. Ivison, Y. Wang, J. Liu, Z. Wu, V. Pyatkin, N. Lambert, N.A. Smith, Y. Choi, and H. Hajishirzi. Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. arXiv preprint arXiv:2406.09279, 2024.

[28] W. Xiong, H. Dong, C. Ye, H. Zhong, N. Jiang, and T. Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under KL-constraint. arXiv preprint arXiv:2312.11456, 2023.

[29] A. Bose, Z. Xiong, A. Saha, S.S. Du, and M. Fazel. Hybrid preference optimization for alignment: Provably faster convergence rates. arXiv preprint arXiv:2412.10616, 2024.

[30] R. Rafailov, Y. Chittepu, R. Park, H.S. Sikchi, J. Hejna, B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024.

[31] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024.

[32] D. Feng, B. Qin, C. Huang, Z. Zhang, and W. Lei. Towards analyzing and understanding the limitations of DPO: A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024.

[33] Y. Yan, Y. Miao, J. Li, Y. Zhang, J. Xie, Z. Deng, and D. Yan. 3D-Properties: Identifying challenges in DPO and charting a path forward. arXiv preprint arXiv:2406.07327, 2024.

[34] W. Zhou, R. Agrawal, S. Zhang, S. Indurthi, S. Zhao, K. Song, S. Xu, and C. Zhu. WPO: Enhancing RLHF with weighted preference optimization. arXiv preprint arXiv:2406.11827, 2024.

[35] X. Lou, J. Zhang, J. Xie, L. Liu, D. Yan, and K. Huang. SPO: Multi-dimensional preference sequential alignment with implicit reward modeling. arXiv preprint arXiv:2405.12739, 2024.

[36] B.Y. Lin, S. Lee, R. Khanna, and X. Ren. Unlocking the potential of language models in alignment through in-context learning. arXiv preprint, 2023.