Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Generalized preference optimization

Generalized preference optimization is a unified algorithmic framework for aligning generative models, such as large language models, with human preferences directly from offline comparison datasets. Instead of relying on separate reward modeling and reinforcement learning loops, it formulates alignment through a broad family of loss functions parameterized by convex functions. This mathematical structure encompasses various direct preference learning algorithms, including direct preference optimization and identity preference optimization, as specific instances within a single theoretical foundation. By altering the chosen convex function, generalized preference optimization characterizes how different offline objectives implicitly enforce regularization relative to a reference policy, enabling systematic analysis and development of diverse offline alignment strategies.

1 item

Generalized Preference Optimization: A Unified Approach to Offline Alignment

Generalized Preference Optimization: A Unified Approach to Offline Alignment

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, Bilal Piot

OrganizationsGoogle

Why you should read this

Unifies offline alignment methods under a single convex-loss framework that explains how algorithms like DPO and IPO enforce implicit regularization and provides principled guidance for tuning their hyperparameters.

Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al. (2023), we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners. Please see https://arxiv.org/pdf/2402.05749 for the full version of the paper.

Added

2026-10-01