Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

self-play fine-tuning

Self-play fine-tuning is an iterative machine learning training method in which a language model improves its capabilities by generating its own training data and competing against earlier versions of itself. In this approach, which typically builds upon an initial supervised baseline, the current iteration of the model generates candidate responses while the training process optimizes the model to discern and favor high-quality target demonstration data over its self-generated outputs. By progressively refining its policy across multiple rounds so that its generated responses become indistinguishable from the target data distribution, the model enhances its overall performance and alignment without requiring additional human-annotated labels, external reward models, or feedback from stronger teacher systems.

1 item

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu

OrganizationsUniversity of California, Los Angeles

Why you should read this

Proposes Self-Play Fine-Tuning (SPIN), a method that allows fine-tuned language models to continuously improve by competing against their past iterations, achieving superior performance to direct preference optimization without needing additional human labels or AI-generated feedback.

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at this https URL.

Added

2026-09-25