Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Zixuan Fu$^{,1}$, Bingxiang He$^{,\dagger,\ddagger,1}$, Yuxin Zuo$^{,\dagger,1}$, Haohuan Huang$^{,1,2}$, Jinqian Zhang$^{1}$, Ruhang Xiao$^{3}$, Cheng Qian$^{4}$, Qinyu Luo$^{5}$, Huan-ang Gao$^{1}$, Yudong Wang$^{1}$, Zhiyuan Liu$^{1}$, Ning Ding$^{\ddagger,1}$, Chaojun Xiao$^{\ddagger,1}$
$^{1}$ Tsinghua University $^{2}$ University of Chinese Academy of Sciences $^{3}$ Northeastern University
$^{4}$ University of Illinois Urbana-Champaign $^{5}$ Johns Hopkins University

$^{*}$ Equal Contribution. $^{\dagger}$ Project Lead. $^{\ddagger}$ Corresponding Authors.
Code: https://github.com/Thinking-Space/One-Shot-OPD.
[email protected], {dingning,xcj}@tsinghua.edu.cn

Abstract

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches 71.5%, most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach 98.9% and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.

Executive Summary: On-policy distillation trains a student language model on its own generated responses while receiving dense token-level guidance from a teacher at every prefix the student visits. Frontier systems already rely on this method alongside supervised fine-tuning and reinforcement learning, yet prior analyses examined only algorithmic behavior and left the role of training data unexamined. The present work therefore asks how much data OPD actually needs and why a minimal supply still produces large, sustained gains.

The authors trained models on a single query, on small semantically diverse sets, and on full training corpora across mathematical reasoning, code generation, instruction following, and agentic tool use. They tracked validation accuracy, the fraction of reasoning states visited relative to full-data runs, and the rate at which the student closed its distributional gap to the teacher. Experiments used three model families, fixed rollout budgets, and controlled ablations that isolated query count from state coverage and on-policy from off-policy sampling.

A single query recovered 62–89 percent of the gain achieved by full-data OPD after 300–1000 steps and reached 71.5 percent of the states visited by the full corpus, most of them within the first 100 steps. Sixteen semantically distinct queries raised state coverage to 98.9 percent and matched full-data performance in both single-domain and multi-teacher settings. The rate at which the student absorbed remaining teacher–student disagreement declined at nearly identical speed whether training used one query or seventeen thousand; even a fixed set of states required hundreds of steps to exhaust. Content-light templates and off-domain chat logs induced useful states and produced comparable gains, showing that explicit task content is not required.

These results indicate that OPD supplies far more supervision than its current update rule can absorb. Consequently, data selection should prioritize the breadth of states a prompt elicits rather than the number or difficulty of problems collected. The same logic implies that further scaling will be limited by step efficiency unless the absorption rate itself is improved.

Future work should therefore develop cheap estimators of state coverage for query selection, test trust-region or token-weighting schemes that reuse batches without degrading the signal, and extend the regime to larger students, longer contexts, and additional teachers. State coverage remains a semantic proxy rather than an exhaustive accounting, and the precise cause of the slowing absorption rate is still open; readers should treat the quantitative coverage figures as relative rather than absolute. The core pattern, however, appears consistently across domains, model families, and controlled ablations.

1. Introduction

Section Summary: On-policy distillation trains a student model by letting it generate its own responses while receiving the teacher’s full next-token guidance at every step along those responses. The paper shows that even a single training example produces large, long-lasting gains because one query already covers most of the reasoning states the model would encounter with a full dataset. The remaining limit on progress comes from the algorithm itself, which absorbs the available teacher signal more and more slowly as training continues.

On-policy distillation (OPD) is becoming increasingly common in frontier LLM post-training. Qwen3 ([1]), MiMo ([2]), GLM-5 ([3]), DeepSeek-V4 ([4]), and Kimi K3 ([5]) all use OPD alongside supervised fine-tuning (SFT) and reinforcement learning (RL). What makes OPD distinctive is its combination of on-policy state visitation and dense distillation supervision ([6, 7]). The student samples its own rollouts, while the teacher provides the full next-token distribution at every visited prefix, yielding dense, token-level supervision rather than the single outcome-level reward typically used in reinforcement learning with verifiable rewards (RLVR).

A growing line of work has systematically investigated OPD's training dynamics and mechanisms, largely from an algorithmic perspective, to explain how and why the method works ([8, 9, 10, 11]). Yet no prior work has studied how training data shapes OPD, or how the data and the algorithm interact. Closing this gap is essential to a complete understanding of OPD. In RLVR, [12] introduce one-shot RLVR as an extreme controlled experiment to isolate the role of training data. We bring the same experimental lens to OPD by training on a single query, a setting we call one-shot OPD. To our surprise, one-shot OPD produces a learning curve strikingly similar to that of one-shot RLVR as shown in: despite repeatedly training on just one query, the model continues to improve for hundreds of steps and ultimately achieves a substantial gain. We therefore devote this work to answering a single question:

Why can OPD, when trained on just one example, keep learning for so long and improve by so much?

This question exposes two aspects of OPD efficiency that standard training entangles: keep learning for so long concerns how fast the algorithm absorbs supervision, whereas improve by so much concerns why a small amount of data supply sufficient supervision. By reducing the data supply to its minimum, one-shot OPD disentangles the two, and the phenomenon itself proves robust: the gain holds across four task domains and three model families, and persists even on queries the student never solves (Section 3). Our answer is that OPD is data-overfed but algorithm-starved. From the data perspective, a query acts on the student through the states its rollouts reach, each state being a prefix $s=(x, y_{<i})$ at which the teacher supplies token-level supervision, so a query set is worth the part of that space it covers. We quantify this by state coverage: we group the states full-data OPD visits into semantic clusters and report the fraction a setting's rollouts reach. A single query already supplies enough states on its own, covering $71.5%$ of what full-data OPD visits, and 16 semantically diverse queries cover $98.9%$ and match full-data training (Section 4). This is why so little data improves the student by so much. From the algorithm perspective, what falls as training proceeds is the absorption rate, the proportion of the remaining teacher–student gap that one update closes, and it declines in much the same way on one query as on all 17k (Section 5), so the pace of a run is a property of OPD rather than of the training set. In other words, the supervision a single query supplies remains largely unexploited, rate-limited by how quickly an on-policy student can absorb it.

The same mechanism informs data design beyond the one-shot setting. In multi-teacher OPD (MOPD) ([2, 4]), where a single student is trained across several domains and each query is routed to its domain teacher, we find that 16 semantically diverse queries per domain suffice to match full-data training (Section 6). Taking the data side to its extreme, we further show that even a training input that states no problem can remain effective: off-domain WildChat prompts and content-free templates drive OPD nearly as effectively as the full training set (Section 7.1). Therefore, an input is useful largely because it starts the student reasoning. Furthermore, we conduct a controlled comparison with one-shot RLVR on the same query to make the algorithmic contrast concrete: the outcome reward is exhausted once the query is solved on nearly every rollout, whereas OPD's token-level signal persists throughout training, and OPD's validation gain over $1000$ steps is more than twice that of RLVR (Section 7.2).

In short, OPD is supplied with more supervision than its algorithm can absorb. What data curation has to settle is therefore no longer how many problems to collect, but which states an input induces. We hope this state-level view of the training data can guide future work on selecting queries by the states they induce, and on raising the rate at which a student absorbs them.

2. Preliminaries

Section Summary: The preliminaries introduce notation for training inputs, token sequences, and states in language models, along with two models: a trainable student policy and a fixed teacher. On-policy distillation trains the student by sampling its own outputs and minimizing the difference between its next-token predictions and the teacher's at every point along those sequences, either via a single sampled token or a weighted top-k set; this yields a local, dense training signal rather than a single end-of-sequence reward. The section also defines several dynamic metrics, such as the fraction of the initial performance gap to the teacher that is recovered over time and measures of how closely the two models agree on their highest-probability tokens.

2.1 Notation

Let $x\sim\mathcal{D}$ denote a training input and $y=(y_1, \ldots, y_L)$ a response, with $y_{<i}=(y_1, \ldots, y_{i-1})$ the prefix up to token position $i$. We consider two LLMs: a student $\pi_{\theta}$ and a teacher $\pi_T$, each defining a next-token distribution over a shared vocabulary. A trajectory is a pair $(x, y)$ with $y$ sampled autoregressively from a policy given $x$. We use $i$ to index token positions and $t$ to index optimization steps, writing $\theta_t$ for the student parameters at step $t$. A state is the autoregressive context $s_i=(x, y_{<i})$, the object on which the OPD objective is defined.

2.2 On-Policy Distillation

On-policy distillation (OPD) samples trajectories from the student and aligns the student to the teacher on the prefixes the student actually visits ([7]). OPD can also be viewed as a special case of dense KL-constrained reinforcement learning, where the teacher distribution induces a token-level reward and the KL regularizer has a fixed relative weight ([13]).

In the full-distribution form, OPD minimizes a per-token KL divergence on visited states:

$ \mathcal{L}_{\mathrm{OPD}}(\theta)

\mathbb{E}{x\sim\mathcal{D}, , y\sim\pi\theta} \left[\sum_{i=1}^{L} \operatorname{KL}!\left(\pi_\theta(\cdot\mid s_i) , \Vert, \pi_T(\cdot\mid s_i) \right) \right].\tag{1} $

In practice, we estimate it with a per-token advantage:

$ A_i^{\mathrm{OPD}} = \log \pi_T(y_i \mid s) - \log \pi_\theta(y_i \mid s),\tag{2} $

where $s = (x, y_{<i})$. Two properties follow from this formulation: (i) the signal is local in that it depends only on the prediction problem at state $s$, not on the trajectory's outcome; and (ii) it is dense in that the teacher supplies a distributional signal at every visited state instead of a terminal scalar reward.

Equation 2 evaluates the correction only at the token the student emitted. A second estimator of the same divergence keeps the student's $k$ most likely tokens at each visited state and weights them by the student's probability:

$ A_i^{\text{top-}k}

\sum_{v \in \mathcal{V}i} \tilde{\pi}\theta(v \mid s) \left[\log \pi_T(v \mid s) - \log \pi_\theta(v \mid s) \right],\tag{3} $

where $\mathcal{V}i = \operatorname{TopK}(\pi\theta(\cdot \mid s), k)$ and $\tilde{\pi}\theta$ is $\pi\theta$ renormalized over that set. It truncates the divergence to $k$ terms rather than estimating it from one sampled token, trading the distribution's tail for lower variance. Section 3.1 states which form each run uses. The alignment metrics below are diagnostics computed from the two policies' logged distributions, so they are available under either form.

2.3 Dynamic Metrics

Gap recovery.

Because evaluation criteria and initial teacher–student performance gaps vary across domains and model pairs, raw score improvements are not directly comparable. We therefore report improvement as a ratio to the initial teacher–student gap. Let $M_0$, $M_T$, and $M_t$ denote the evaluation scores of the initial student, the teacher, and the student at optimization step $t$, respectively. The gap recovery ratio at step $t$ is $\frac{M_t-M_0}{M_T-M_0}\times100%.$ A value of $0%$, $100%$, or above $100%$ indicates no improvement toward the teacher, matching the teacher, or surpassing the teacher, respectively.

Full-data recovery.

When the reference is full-data OPD rather than the teacher, we normalize by the gain of full-data OPD instead. Let $M_F$ denote the full-data OPD score at the same optimization step. The full-data recovery ratio is $\frac{M_t-M_0}{M_F-M_0}\times100%.$ A value approaching $100%$ means that a reduced query set nearly matches the gain of full-data OPD while a value above $100%$ means it surpasses it.

Top- $k$ token overlap ratio.

This metric measures agreement between the two policies' high-probability token sets. Let $S_t^\theta(s)=\operatorname{TopK}(\pi_{\theta_t}(\cdot\mid s), k)$ and $S^T(s)=\operatorname{TopK}(\pi_T(\cdot\mid s), k)$. The overlap at optimization step $t$ is

$ O_t^{(k)}

\frac{1}{|\mathcal{T}t|} \sum{i\in\mathcal{T}_t} \frac{|S_t^\theta(s_i)\cap S^T(s_i)|}{k},\tag{4} $

where $\mathcal{T}_t$ indexes the non-padding response positions of the rollouts in the batch collected at step $t$ ([8]).

Overlap-token advantage.

Inside that agreement, we average the teacher–student log-probability difference over the shared tokens $S_t^\theta(s_i)\cap S^T(s_i)$, weighting each shared token by the probability the student assigns it. The result reports how far apart the two policies remain on the tokens they already rank highly. Because of that weighting, and because it averages over top- $k$ entries rather than over sampled tokens, it sits on a much smaller scale than the teacher–student distance of Section 5.1, and the levels of the two are not comparable.

3. The One-Shot Phenomenon

Section Summary: The section examines the one-shot OPD phenomenon, testing how much of the performance boost from full-dataset training can be achieved with just a single query. Experiments across mathematical reasoning, code generation, instruction following, and agentic tool use show that one query recovers most of the teacher-student gap for multiple model families, with gains that remain stable regardless of query difficulty, response length, or sampling settings. Even queries the student model consistently fails to solve prove as effective as those it always succeeds on.

This section presents the experimental setup and results of One-Shot OPD. We investigate how much of full-data OPD's gain a single query recovers, and how robust that gain is.

**One query is enough to induce robust OPD gains.**
  A single query recovers most of the teacher–student gap across task domains and model families. The gain is insensitive to query difficulty, response length, and sampling temperature, and a query the student never solves is as effective as one it always solves.

3.1 Experimental Setup

We establish one-shot OPD across math, code generation, instruction following, and agentic tool use for controlled analysis to test generality.

Models.

\begin{tabularx}{\ccccccccc}{@lXX@}
\toprule
\textbf{Setting} & \textbf{Student} & \textbf{Teacher} \\
\midrule

\multirow{3}{*}{Mathematical reasoning}
 {} & DeepSeek-R1-Distill-Qwen-1.5B\newline
{\footnotesize (R1-Distill-1.5B)}
 {} & JustRL-DeepSeek-1.5B\newline
{\footnotesize (JustRL-1.5B)} \\

 {} & Llama-3.2-3B-Instruct\newline
{\footnotesize (Llama-3B-It)}
 {} & GT-Llama-3.2-3B-Instruct-MATH\newline
{\footnotesize (GT-Llama-3B-Math)} \\

 {} & OLMo-3-7B-Instruct-DPO\newline
{\footnotesize (OLMo-7B-It-DPO)}
 {} & OLMo-3-7B-Instruct\newline
{\footnotesize (OLMo-7B-It)} \\

\midrule

Code generation
 {} & DeepSeek-R1-Distill-Qwen-1.5B\newline
{\footnotesize (R1-Distill-1.5B)}
 {} & Nemotron-Research-Reasoning-Qwen-1.5B-v2-RLVE\newline
{\footnotesize (Nemotron-1.5B)} \\

Instruction following
 {} & DeepSeek-R1-Distill-Qwen-1.5B\newline
{\footnotesize (R1-Distill-1.5B)}
 {} & UltraData-IF-1.5B \\

Agentic tool use
 {} & Qwen2.5-Coder-1.5B-Instruct\newline
{\footnotesize (Qwen-Coder-1.5B)}
 {} & Hammer2.1-1.5b\newline
{\footnotesize (Hammer-1.5B)} \\

\bottomrule
\end{tabularx}

For each domain, we pair a student with a post-trained teacher from the same family. Table 1 lists the full identifiers and the short names used throughout. Math, code generation, and instruction following share the student R1-Distill-1.5B ([14]), whose teachers are, respectively, JustRL-1.5B ([15]), Nemotron-1.5B ([16]), and an instruction-following variant we post-train from the same student (Appendix A.1). For agentic tool use, the student is Qwen-Coder-1.5B ([17]) and the teacher is Hammer-1.5B ([18]). To test whether the phenomenon is specific to Qwen-based pairs, we add two mathematical-reasoning pairs from other families: Llama-3B-It ([19]) with GT-Llama-3B-Math ([20]), and OLMo-7B-It-DPO with OLMo-7B-It ([21]).

Datasets.

The domain-specific training sets comprise DAPO-Math-17K ([22]) for mathematical reasoning, Open-R1 Codeforces ([23]) for code generation, a sampled subset of UltraData-SFT-2605 ([24]) for instruction following, and xLAM-function-calling-60K ([25]) for agentic tool use. For One-shot OPD in mathematics, we select three queries spanning easy, medium, and hard initial difficulty, scored by the student's pass rate over 8 rollouts before training ($8/8$, $4/8$, and $0/8$, respectively). In each of the other domains, the one-shot query is sampled randomly from the corresponding training set. Appendix A.1 details the dataset preprocessing and the selected one-shot queries.

Training.

We implement OPD in veRL ([26]). The mathematical-reasoning runs of this section optimize the top- $k$ advantage of Equation 3 with $k=16$ ; the code, instruction-following, and agentic runs optimize the sampled-token advantage of Equation 2. Every update uses a batch of $64$ rollouts. We use AdamW with a learning rate of $10^{-6}$, a rollout temperature of $1.0$, and a gradient clip norm of $1.0$; remaining hyperparameters are listed in Table 3.

Evaluation.

For math, we evaluate on MATH-500 ([27]), AMC 2023 ([28]), and AIME 2025 ([29]), sampling 16 responses per problem and reporting avg@16 accuracy. For code generation, LiveCodeBench v6 (LCB v6) ([30]) uses 3 sampled solutions per problem and reports avg@3 with the official execution-based evaluator. For instruction following, Multi-IF ([31]) evaluates three-turn conversations and reports the final-turn score averaged over its eight languages. For agentic tool use, BFCL v3 ([32]) reports avg@8 over the evaluated subsets. And the response caps are $31{,}744$ tokens for mathematics, $65{,}536$ for LCB v6, $16{,}384$ per turn for Multi-IF, and $4{,}096$ for BFCL v3. Unless stated otherwise, mathematics figures report validation accuracy macro-averaged over MATH-500, AMC 2023, and AIME 2025, and a dashed line marks the teacher.

3.2 Experimental Results

One-shot OPD recovers most of full-data OPD's gain in mathematics.

Figure 2 shows that training on a single query improves accuracy on MATH-500, AIME 2025, and AMC 2023, approaching the full-data OPD on all three benchmarks. Averaged over them, one-shot OPD reaches $68.5$ against $69.8$ for full-data OPD, recovering $69%$ of the teacher–student gap and $87%$ of full-data OPD's gain at step 300. Beyond step $300$ both curves stay within a band of about $3$ points, and the recovered fraction ranges from $62%$ to $89%$ through step $1000$, where one-shot OPD reaches $68.4$ against $72.1$ and recovers $72%$ of full-data OPD's gain (). The training dynamics follow the same alignment process under both settings: the top-16 overlap ratio climbs to the full-data level, the overlap-token advantage approaches zero, and the absolute entropy gap nearly closes. Thus, even when all rollouts originate from a single query, OPD improves accuracy while progressively aligning the student distribution with the teacher on the visited states.

The one-shot effect is robust across families and task domains.

Figure 3 shows that one-shot OPD improves mathematical reasoning across all three student–teacher families. At the final checkpoint, the averaged scores increase from $77.1$, $28.2$, and $70.8$ for the respective student baselines to $85.5$, $40.2$, and $82.4$ for R1-Distill-1.5B, Llama-3B-It, and OLMo-7B-It-DPO. Figure 4 further shows that the effect extends beyond mathematical reasoning: on code generation, instruction following, and agentic tool use, one-shot OPD recovers $73%$, $66%$, and $64%$ of the corresponding teacher–student gaps, respectively. Together, these results show that the one-shot effect is robust across both model families and task domains.

The one-shot effect is robust to query and rollout properties.

We next examine whether one-shot OPD remains effective under varying query difficulty, response-length budgets, and rollout sampling temperature. As shown in Figure 5, one-shot OPD works across easy, medium, and hard queries, even though the fraction of correct rollouts evolves very differently in the three settings (Appendix A.1): the easy query is solved on nearly every step, the medium query becomes largely solvable during training, and the hard query is never solved. Tightening the response-length cap and lowering the rollout temperature both preserve the one-shot gain. Together, these results show that one-shot OPD is robust to query difficulty, response-length budget, and rollout sampling temperature.

4. Data Perspective: Abundant States

Section Summary: One-shot OPD already captures most of full-data OPD’s gains because a single query, when rolled out repeatedly, reaches roughly 71 percent of the distinct reasoning states that the full dataset would visit, with the bulk of that coverage occurring in the first hundred steps. Each rollout generates many unique token-by-token prefixes that serve as training examples, so query count alone understates how much supervision is actually supplied. Adding a modest number of semantically different queries further expands state coverage, and sixteen such queries are enough to match the performance of training on the entire dataset.

Section 3 showed that One-Shot OPD recovers much of full-data OPD's gain robustly. This section asks why training on a single query produces such a large gain, and whether the same explanation also covers the cases where n-shot OPD can match full-data OPD. [^2]

[^2]: The runs analyzed here and in Section 5 optimize the sampled-token advantage of Equation 2, rather than the top- $k$ advantage used by the mathematics runs of Section 3, so their absolute accuracies differ slightly from those reported there. Every comparison below is between runs that share this form.

- **One query already reaches a large part of the full-data state space.** What OPD consumes is primarily states rather than queries. In practice, repeated rollouts from a single query can reach $71.5\%$ state coverage, with most of it within the first $100$ steps.
- **Diverse but few queries reach sufficient states and rival full-data performance.** Adding semantically distinct queries raises both state coverage and validation accuracy, and 16 queries already match full-data OPD. The value of an extra example lies in whether it covers states missed by earlier ones.

4.1 State Coverage of One-Shot OPD

Setup.

OPD trains on states rather than queries. A query $x$ and a sampled response $y$ produce one state $s=(x, y_{<i})$ at every token position $i$, each paired with a target distribution from the teacher. With $64$ rollouts per update, even a single query yields tens of thousands of supervised states, so query count can substantially understate the amount of supervision available to OPD.

A single query can provide enough supervision to recover most of full-data OPD's gain if its rollouts cover a broad part of the state space visited by full-data OPD.

To test this hypothesis, we measure the breadth rather than the raw number of visited states, since every generated token creates a distinct prefix. We call this measure state coverage: we represent states in a shared representation space, partition that space into clusters, and report the fraction reached by a setting's rollouts.

• Representation. We represent each state by its teacher signature $h_T(s)$, the teacher's final-layer hidden vector at the state's last token.
• Reference space. We pool states visited by full-data OPD over its entire run on DAPO-Math-17K, sampling $8$ uniformly spaced positions from each rollout. A held-out portion of these rollouts takes no part in defining the clusters and is measured as a setting of its own, full data (held-out).[^1]
• Clusters. After PCA, $K$ -means partitions the reference signatures into $K=200$ clusters. Each state falls in the nearest one, denoted $c(s)$ .

[^1]: Three of every five collected rollouts form the pool; the other two are the held-out setting, so it is what full-data OPD reaches on a fresh sample under the same budget.

The state coverage of a set of states $S$ is then the fraction of the $K$ clusters it reaches:

$ \mathrm{Cov}(S)

\frac{1}{K} \left| \left{, c(s);:;s\in S, \right} \right|.\tag{5} $

Coverage records which clusters a setting reaches, not how often it visits them. We fit the clusters once and measure every setting over the same $300$ steps. Full data (held-out) reaches $100%$ on this budget, so the top of the scale is a level full-data OPD attains rather than a maximum the construction guarantees. Appendix B.1 gives the construction and schedule, and shows that the comparisons below are stable across choices of $K$, reference set, and state positions.

One-shot OPD reaches $71.5%$ of the state space.

Figure 6 shows that one-shot OPD reaches $71.5%$ state coverage by step $300$ . Most of this coverage appears early: the run reaches $65.9%$ by step $100$, then adds only $5.6$ percentage points over the next $200$ steps. Repeated rollouts from the same query therefore continue to discover new clusters, but at a sharply diminishing rate. The same run raises validation accuracy from $59.1$ to $66.9$, compared with $70.8$ for full-data OPD at step $300$ . These results show the same pattern on the data side: a single query already covers most of the state-space clusters reached by full-data OPD and produces a substantial validation gain. One-shot OPD is therefore small in query count, yet broad in the supervision it generates.

4.2 Diversity Expands State Coverage

The analysis above shows that one query already reaches most of the full-data state space, but it leaves the causal question open: a run that trains well might simply visit more states along the way, with the extra states doing none of the work. We therefore ablate state coverage directly, varying how many distinct states the training data reaches while holding the rest of the setup fixed.

Setup.

We raise data diversity from two directions, each with its own control.

(i) Response diversity (off-policy). We sample $64$ trajectories once from the initial student on the one-shot query and keep that pool fixed, which also removes a confound of on-policy training, where the states keep changing as the student does. Nested subsets retain $1$, $4$, $16$, or all $64$ trajectories, and the retained ones are repeated to fill each batch of $64$ . Every condition shares the query, the batch size, and the optimization budget, so the number of distinct trajectories, and hence of states, is the only quantity that changes.

(ii) Query diversity (on-policy). These runs instead change the query set, and all follow the mathematical configuration. Starting from the medium query used by one-shot OPD, we build 4- and 16-shot sets by clustering DAPO-Math-17K with BGE-M3 ([33]) and taking one representative per semantic cluster, so each query added to the ladder is semantically distinct from those already in it, and we compare these sets with full-data OPD. We report validation accuracy for every run, and state coverage for the on-policy ones, where the rollouts are those the student actually generates during training. Off-policy scores are averaged over the five evaluation checkpoints from step $300$ to $500$ . On the coverage side, the full-data bar is the held-out split that sets the $100%$ ceiling.

Data diversity is state diversity, and a few queries match full-data OPD.

Figure 7 shows off-policy validation accuracy rising monotonically with the number of distinct trajectories. Everything else is held fixed, so the only thing the extra trajectories add is states. The on-policy runs reach the same conclusion from the query side. Semantically distinct queries raise validation accuracy until 16 of them match full-data OPD, and they match it on state coverage at the same point, which rises from $71.5%$ for one query to $98.9%$ for 16. Every query on the ladder is a new semantic cluster, so it raises diversity and count together, and two controls in Appendix B.2 separate them: holding the count at 16, drawing those queries from 16 semantic clusters rather than from one raises state coverage and validation accuracy sharply, whereas the order in which the student trains on a fixed query set does not matter. What an added query is worth is therefore set by whether it reaches new states, and a few diverse queries already reach nearly all the states full-data OPD does.

5. Algorithm Perspective: Slow Alignment

Section Summary: From the algorithm side, the process of aligning the student model to the teacher slows steadily over hundreds of steps as the rate at which each update reduces their disagreement steadily declines, even though improvement never fully stops. This gradual slowdown occurs in much the same way regardless of how many training queries are used. Experiments that reuse a fixed set of states throughout training show that the long duration is not driven by a need for fresh data, since progress on alignment metrics and accuracy still unfolds over hundreds of steps.

Section 4 analyzed the one-shot gain from the data side: what a query is worth to the student is set by the states its rollouts reach. Turning to the algorithm side, this section asks why one query keeps producing gains over so many steps.

- **Alignment slows over time, and this trend is insensitive to query count.** The student continuously narrows the teacher–student gap throughout the run, but the rate of progress diminishes steadily, which is why a run requires hundreds of steps rather than tens. Increasing the number of queries does not accelerate or decelerate this process.
- **Even a fixed set of states takes hundreds of steps to learn from.** A run that trains on the same states from beginning to end still improves over hundreds of steps, so a steady supply of fresh states is not required to keep a run going that long.

5.1 Alignment Metrics

OPD aligns the student with the teacher. Two quantities describe the state of that alignment at any moment: how much teacher–student disagreement is left, and how fast it is falling. We measure both on the states the student actually visits.

• Distance. The teacher and the student disagree at each visited position, by the per-token advantage of Equation 2. We take the distance $d_t$ to be the average size of that disagreement over the positions $\mathcal{T}_t$ the student visits at step $t$ :

$ d_t

\frac{1}{|\mathcal{T}t|} \sum{i\in\mathcal{T}t} \left| \log\pi_T(y_i\mid s_i)-\log\pi{\theta_t}(y_i\mid s_i) \right|.\tag{6} $

Here $s_i=(x, y_{<i})$ is the state at token position $i$, as defined in Section 2. Averaging magnitudes keeps positive and negative per-token gaps from cancelling, so $d_t$ falls to zero only as the two policies come to agree at the positions the student visits.

• Absorption rate. The absorption rate $v_t$ is then the proportion of that distance one update absorbs,

$ v_t = \frac{d_t-d_{t+1}}{d_t}.\tag{7} $

We track both metrics from step 30, after gradient clipping no longer binds. Since each run measures distance over the states it visits, these runs have settled at different levels by then. We therefore report each run against its own value at step 30, which we refer to as the distance left $d_t/d_{30}$ .[^3]

[^3]: Late in training a single pair of consecutive steps is too noisy to read, so $v_t$ is estimated over the geometric window $[t/1.4, , 1.4t]$, which is why it stops short of step $300$ . Appendix C.1 gives the estimators and the evidence behind both conventions.

5.2 Alignment Slows Similarly Across Data

We track $d_t$ and $v_t$ for OPD trained on 1, 4, 16, and all DAPO-Math-17k queries, over the same $300$ steps and under the mathematical configuration of Section 3.1.

The student aligns ever more slowly.

The distance falls for the whole run, so the student is never stuck. What slows is the absorption rate, which falls throughout the run in the right panel of Figure 8: on the logarithmic distance axis at left a constant rate would trace a straight line, and all four curves bend flatter instead. Each update thus absorbs less of what is left than the last, which is why a run takes hundreds of steps rather than tens.

The absorption rate falls in much the same way at every training-set size.

The four runs do this alike: each removes $78%$ to $84%$ of its step- $30$ distance by step $300$, and all four slow by a similar factor during training (Appendix C.1 gives the per-run numbers and the conventions these readings are taken under). One query therefore does not make alignment slower or faster: the pace is a property of OPD rather than of the training set. The learning rate changes how many steps that takes, but not the way the distance falls (Appendix C.2).

5.3 Ablation on Fixed States

On-policy training keeps producing new states as the student changes, and one reading of the hundreds of steps is that the run lasts as long as that supply does. Holding the training states fixed tests that reading directly: if a fixed set of states is exhausted quickly, the off-policy run should finish early.

Setup.

We compare two one-shot runs that differ only in where their training states come from. The always-on-policy run draws fresh rollouts from the current student at every step. The always-off-policy run reuses the $64$ trajectories of Section 4.2, sampled once from the initial student, for every update, so its training states stay fixed while the student changes.

Holding the states fixed leaves the run just as long.

Figure 9 shows the off-policy run gains accuracy steadily for about $200$ steps and then stops, so its gain builds up over hundreds of steps rather than over the first few updates. The alignment metrics move just as slowly: the off-policy run's top- $16$ token overlap rises and its overlap-token advantage approaches zero only over hundreds of steps. Removing that supply therefore does not shorten the run. This suggests that a fixed set of states is enough on its own to keep an OPD run going for hundreds of steps, so the length of a run is not explained by how long fresh states keep arriving.

6. One-Shot Extends to Multi-Teacher OPD

Section Summary: The section examines whether the benefits of small, diverse query sets observed in single-domain distillation also apply when training one student model simultaneously on multiple domains through multi-teacher on-policy distillation. Experiments across mathematical reasoning, code generation, and instruction following show that just 16 carefully chosen queries per domain produce validation accuracy nearly identical to using the full datasets, while a single query per domain falls short. This outcome indicates that semantic diversity within each domain is enough to drive effective learning even in a combined multi-domain training run.

Section 4.2 showed that a small, semantically diverse query set can match full-data OPD within one domain. Modern post-training pipelines often use multi-teacher on-policy distillation (MOPD) ([2, 4]), where one student is trained on several domains in a single run and each query is routed to its domain teacher. This section asks whether the same query-diversity result carries over: can a small, diverse query set for each domain match full-data MOPD?

16 semantically diverse queries within each domain are sufficient to match full-data MOPD, which reaches nearly the same average validation accuracy as separate full-data OPD.

Setup.

We train one R1-Distill-1.5B student with MOPD on mathematical reasoning, code generation, and instruction following. The domain teachers from Section 3.1 are JustRL-1.5B for mathematics, Nemotron-1.5B for code, and UltraData-IF-1.5B for instruction following. We exclude agentic tool use because it uses a different student, Qwen-Coder-1.5B. The three settings are one-shot, 16-shot, and full-data MOPD. As in Section 4.2, the 16-shot setting selects one representative from each of 16 BGE-M3 ([33]) semantic clusters fitted separately within each domain. All settings run for $300$ steps with the hyperparameters in Table 8.

Full-data MOPD matches separate full-data OPD.

At step $300$, full-data MOPD raises the average validation accuracy from $43.5$ to $52.8$, recovering $79%$ of the teacher–student gap. The three corresponding full-data OPD runs reach $53.8$ . This $1.0$ -point difference shows that MOPD remains effective when all three domains are trained in a single run and establishes full-data MOPD as the reference for the query-diversity comparison below.

16 diverse queries per domain match full-data MOPD.

Increasing the query count from 1 to 16 per domain raises the average validation accuracy from $50.1$ to $52.9$ at step $300$ . Full-data MOPD reaches $52.8$, so 16-shot MOPD recovers $101%$ of full-data MOPD's gain. The result is not driven by one domain: calculated separately, the corresponding proportions are $93%$ for mathematics, $136%$ for code generation, and $109%$ for instruction following. Because 16-shot and full-data MOPD use the same rollout and optimization budgets, expanding each domain's query set beyond 16 diverse queries produces no additional average gain in this setting.

Together, these results are consistent with a state-space account for MOPD: semantically diverse queries can induce complementary states within each domain, allowing 16 queries per domain to match full-data MOPD.

7. Discussion

Section Summary: The discussion reframes prompts as tools that guide a model into specific reasoning states where a teacher can offer effective feedback, rather than as containers of task content. Experiments show that even minimal or unrelated inputs can trigger useful learning trajectories, producing gains comparable to real math problems while using far fewer tokens. In contrast to outcome-based methods like RLVR, this approach extracts denser supervision from the same starting query, enabling longer improvement because it relies on step-by-step teacher alignment instead of final-answer variation.

The mechanism above shifts attention from prompts to the regions of reasoning states they induce. An input is useful when the student's rollouts reach regions where the teacher can provide useful supervision. Stating a problem is only one way to reach them. This view raises two broader questions: how much task content must the input itself contain, and why does the same one-shot setting behave differently under OPD and RLVR? We discuss these questions through content-light and off-domain inputs, followed by a controlled comparison with one-shot RLVR.

7.1 Inputs as State Generators

The state-generator view suggests that task content may be unnecessary if an input still induces trajectories that the teacher can supervise. We stress-test this prediction with an empty user turn ending in <think>, the same template with a simple domain system prompt, and general-domain queries from WildChat ([34]) (Figure 11). The WildChat set contains $192{,}824$ predominantly English queries from general conversation, creative writing, role-playing, and rewriting. Conservative heuristics label only $0.17%$ as mathematics-related and $2.63%$ as code-related.

Content-light and off-domain inputs can still induce useful states.

All three conditions track the real-query baseline, which lifts the three-benchmark average from $59.1$ to $69.8$, and all three reach that level on a third to a half of its rollout tokens (Figure 12). Code generation gives the same picture (Appendix E.1). However, the requirement is not empty: a scaffold that closes the block at once, <think>\n</think>\n, collapses into short meta-level replies we cannot train on. These results do not imply that task content is generally dispensable. They show that content is not the only source of useful signal in OPD. An input also acts as a state generator, and even content-light or off-domain inputs can sometimes lead the student to states that support useful teacher supervision. Explicit domain content changes final performance by about one point in these experiments. This provides a complementary view of OPD data efficiency. Data quality depends not only on the problems being collected, but also on the state regions they induce and the teacher targets available there.

7.2 Relation to One-Shot RLVR

One-shot OPD and one-shot RLVR ([12]) share the same surface phenomenon that a single query supports continued improvement over hundreds of updates in OPD and over a thousand updates in RLVR. The source of learning is different. OPD receives a teacher target at every visited state, while RLVR receives a verifiable outcome for each trajectory. We compare them on the same medium query to isolate this difference (Figure 13).

Useful inputs differ across objectives.

RLVR updates from trajectory outcomes. Its training input must therefore define a task whose sampled outcomes can be verified. The outcomes must also vary enough to produce a useful advantage. OPD instead updates from the teacher–student gap at each visited state. It can receive useful signal whenever a prompt leads to trajectories that expose such gaps. The value of an OPD input is therefore not tied directly to verifiability or difficulty. Our results further show that even weak domain alignment can work in the settings we study. Coverage of the induced state regions still matters (Section 4), but obtaining such coverage is less restrictive than obtaining tasks with verifiable outcomes.

OPD extracts more signal from the same query.

Both algorithms obtain fresh rollouts from the training query at every step. Over $1000$ steps, OPD closes $72%$ of its gap to the teacher. Its gain in validation accuracy is more than twice that of RLVR. This advantage remains when the two methods are matched by rollout tokens rather than training steps (Figure 13).

The RLVR signal weakens as the model learns to solve the training query. Its groups soon become nearly unanimous, which leaves no advantage under GRPO. OPD does not depend on variation in trajectory outcomes. It can continue learning from local teacher–student gaps after the query is solved, although these gaps gradually shrink. One-shot RLVR is therefore limited by the outcome variation available from its query. One-shot OPD is instead limited by how quickly it absorbs dense teacher supervision, as shown in Section 4 and Section 5. The objectives also imply different ceilings. RLVR is not tied to a teacher's distribution, while OPD is trained to match one.

8. Related Work

Section Summary: Previous work on on-policy distillation has explored training a student model on its own outputs under teacher guidance, often using techniques like policy gradients or KL constraints, and has analyzed why these dense per-token signals can be more effective than standard reinforcement learning. Related efforts have also examined how to achieve strong reasoning with very little data, whether through curated examples in supervised fine-tuning or sparse-reward reinforcement learning, and have developed ways to generate synthetic training inputs or remove the need for human labels. This paper builds on those ideas but shifts focus to the minimal amount of data required for distillation, showing that even simple or off-domain queries can work well because the dense teacher feedback allows the student to learn efficiently from whatever starting points it encounters.

On-policy distillation.

On-policy distillation (OPD) trains the student on its own rollouts with teacher supervision ([35]). MiniLLM ([6]) optimized this under reverse KL by policy gradient, and GKD ([7]) generalized it across divergences and on- and off-policy mixtures. [13] cast OPD as dense KL-constrained RL whose per-token log-ratio acts as an implicit reward. OPD now appears across frontier post-training stacks ([1, 3, 2, 4, 36]), several of which adopt the multi-teacher form we study in Section 6, routing each query to a domain specialist. A growing line explains why OPD is effective, and does so entirely on the algorithm side. One thread studies its objective and update geometry: the dense per-token target replaces the single sequence-level scalar of outcome-reward RL, which in parameter space steers an early-locked, low-rank update subspace ([10, 37]) and can be recast as token-level policy gradient ([38]), connecting to analyses that separate the dense updates of SFT from the localized subnetworks of RL ([39, 40]). A second thread charts when OPD succeeds or fails: a thinking pattern shared between teacher and student and the limits of an over-strong teacher ([8]), which tokens are actually learnable ([41, 42]), and recurring failure modes with simple fixes ([11, 9]). These accounts take the training set as given. We instead ask how many training inputs OPD needs, and find that the binding resource is not the data but the rate at which the student absorbs it.

Data-efficient reasoning post-training.

A parallel line studies how little data elicits strong reasoning. Under sparse outcome-reward RL, [12] show that a single example keeps improving a model for over a thousand steps, the precedent we adapt to the dense-signal setting of OPD. LIMR ([43]) prunes the RLVR training set several-fold by ranking each example's learning impact. With supervised fine-tuning, LIMA ([44]), LIMO ([45]), and s1 ([46]) elicit competitive alignment or reasoning from on the order of a thousand curated examples. Within OPD, scheduling supervision over reasoning prefixes cuts training cost severalfold ([47]). Broader post-training work selects high-value subsets by influence or quality ([48, 49, 50, 51]). These works share our data-efficiency theme, but their supervision still depends on curated problems, delivered as a sparse outcome scalar in the RL case or as reference solutions in the SFT case. OPD instead supplies a dense per-token target at each visited prefix and remains effective on queries the student never solves (Section 3), where the teacher signal stays informative even on incorrect rollouts ([41]), so the limiting factor moves from whether a useful signal is available, the standing difficulty under sparse reward, to how fast an on-policy student absorbs an already-abundant one (Section 5.2).

Synthetic and unsupervised post-training data.

A growing body of work reduces post-training's dependence on curated, human-written data. One direction synthesizes the inputs: Self-Instruct ([52]) and Evol-Instruct ([53]) bootstrap instructions from a language model, and Magpie ([54]) elicits full instruction–response pairs from nothing but a chat-template prefix. A second direction removes ground-truth labels from RL ([55]), replacing verifiable rewards with self-generated signals such as model confidence or entropy ([56, 57]) and majority agreement across rollouts ([58]), or drawing reward from unlabeled corpora ([59]). A third removes external problems altogether, letting the model propose its own tasks ([60, 61]). These lines relax supervision on the reward or the query source. Our template- and WildChat-OPD results (Section 7.1) push the query side further: because the teacher supplies dense supervision at each visited prefix, the query's role reduces to placing the student in a useful reasoning state, so an off-domain chat log or even a content-free template drives OPD nearly as well as a curated problem.

9. Conclusion, Limitations, and Future Work

Section Summary: The study concludes that a single training query suffices for effective one-shot preference optimization, allowing the student model to keep improving over many steps and recover most of the gains from full datasets, because the core bottleneck is how efficiently each update absorbs the teacher-student gap rather than the sheer volume of data. Limitations include reliance on a reference-based measure of state coverage that may overlook frequency or value of visited states, plus uncertainty about what controls the absorption rate and how results scale beyond the tested domains and teachers. Future efforts could focus on choosing queries by the states they trigger, reusing batches more efficiently during training, and extending the approach to larger models, more diverse teachers, and complex settings like tool use.

Conclusion.

Motivated by one-shot RLVR ([12]), we asked whether a single query is also enough for OPD, and it is: one-shot OPD keeps the student improving for hundreds of steps and recovers most of full-data OPD's gain, across model families and task domains. The data and the algorithm sides give one answer: OPD is data-overfed but algorithm-starved. A query acts on the student through the states its rollouts reach, and one query already covers most of the state space full-data OPD visits, while 16 semantically diverse queries match full-data training in single-domain OPD and in MOPD alike. What limits a run is the absorption rate: the proportion of the remaining teacher–student gap one update absorbs keeps falling, at the same pace on one query as on all 17k, so the optimizer rather than the training set sets how long a run takes. An input need only start the student reasoning, which turns data design from collecting problems into choosing teachers.

Limitations.

State coverage is a semantic-level proxy measured against a reference space built from full-data rollouts, so it reports how much of that space a query set reaches rather than what it covers on its own, and it weights every cluster equally, however often the cluster is visited and however much teacher signal it still carries. What sets the absorption rate is still open. Our MOPD runs use three domains with one teacher each, leaving open how far the diversity result extends as teachers are added.

Future work.

Three directions follow.

• Selecting data by state coverage: the question is no longer how many queries to collect but which states they induce, so state coverage becomes a selection criterion once it can be estimated from the queries themselves, without a full-data run to measure it against.
• Making training more step-efficient, by reusing a batch for several epochs under a trust region on the per-token gap, or weighting tokens by how much teacher signal they still carry.
• Growing the setting, to MOPD with more teachers and domains, and to larger students, agentic tool use, and long contexts, where training data is most expensive to collect.

Appendix

Section Summary: The appendix supplies technical details supporting the paper's main experiments. It first describes how an instruction-following teacher model was post-trained from a base checkpoint using rule-based rewards on synthetic tasks, along with the specific math queries chosen to test easy, medium, and hard difficulty levels and the default OPD training settings such as batch sizes, learning rates, and sequence lengths. It then explains the procedure for measuring training-state coverage by collecting rollouts across optimization steps, clustering them with PCA and K-means, and verifying that the relative coverage ordering among one-shot, few-shot, and full-data regimes remains stable under changes in cluster count, reference data, and token positions.

A. Details for Section 3

A.1 Experimental Setup and Training Queries

Instruction-following setup.

Unlike the other primary-domain teachers, the instruction-following teacher is trained specifically for our experiments. Starting from DeepSeek-R1-Distill-Qwen-1.5B, we post-train the model on synthetic instruction-following tasks with programmatically checkable constraints. We use GRPO with rule-based outcome rewards that indicate whether the response satisfies the target-turn constraints. Training samples four completions per prompt at temperature $0.9$, uses a maximum prompt length of $4{,}096$ tokens and a maximum response length of $16{,}384$ tokens, and optimizes with AdamW at learning rate 1 x 10^-6, train batch size $16$, and clip ratio $0.2$ . The step-600 checkpoint is used as the instruction-following teacher, improving IFBench ([62]) from $14.33$ to $20.65$ and Multi-IF final-turn accuracy from $20.84$ to $28.58$ .

For this domain, "full-data" denotes the subset sampled from UltraData-SFT-2605 ([24]) rather than the complete dataset, and the one-shot query is drawn at random from that subset.

Mathematical queries for one-shot OPD (Table 2, Figure 14).

We sample 8 rollouts from the untrained student for each of the first 64 queries of DAPO-Math-17K and assign difficulty by the resulting pass rate, which selects t59 ($8/8$), t22 ($4/8$), and t56 ($0/8$) as the easy, medium, and hard queries. How often each is solved during training follows that assignment: the easy query stays near $1.0$ throughout, the medium one rises from $0.36$ to about $0.89$, and the hard one is never solved in $300$ steps, yet all three produce comparable benchmark gains in Figure 5.

::: {caption="Table 2: Mathematical queries used in the query-difficulty experiment, labelled by the student's pass rate before OPD training."}

:::

Default hyperparameters for OPD (Table 3).

Unless otherwise noted, all experiments use these defaults. The top- $K$ row covers both objectives: the mathematical one-shot runs of Section 3.2 use $K=16$, and all other experiments use the sampled-token objective.

\begin{tabular}{ll}
\toprule
\textbf{Item} & \textbf{Value} \\
\midrule
Rollout batch size & 64 \\
Mini batch size & 64 \\
Responses per prompt & 1 \\
KL coefficient & 0.0 \\
LogProb top- $K$ & 0(default) / 16 \\
Top- $K$ strategy & Student Top- $K$ \\
Loss aggregation & token-mean \\
Training temperature & 1.0 \\
Top- $p$ & 1.0 \\
Optimizer & AdamW ($\beta_1{=}0.9$, $\beta_2{=}0.999$, wd $=0.01$) \\
Learning rate & 1e-6 \\
Gradient clip norm & 1.0 \\
Max prompt length & 1024 (math); 4096 (code, IF, agentic) \\
Max response length & 7168 (math, code, IF); 2048 (agentic) \\
\bottomrule
\end{tabular}

B. Details for Section 4

B.1 Training-State Coverage

Construction.

Rollouts are collected by optimization step rather than at random, so that every stage of training is equally represented and the budget is comparable across settings, which run for the same number of steps: at each step from $1$ to $500$ of the full-data run we take $5$ rollouts, three building the clusters and two forming full data (held-out), for $12{,}000$ reference and $8{,}000$ held-out states. PCA reduces these to $50$ dimensions, retaining $71.9%$ of the variance, before $K$ -means. Every evaluated setting is then measured on that same schedule, drawing $2$ of the $5$ rollouts at each step and accumulating over the run, so the step- $300$ budget of $600$ rollouts matches the held-out split. We repeat the draw $30$ times; bands and error bars are one standard deviation over these repetitions, which reflects rollout subsampling rather than variation across trained models.

The ordering is stable across $K$, reference construction, and state positions (Table 4 and Table 5, Figure 15).

Coverage depends on how finely the reference states are clustered and on which states define the clusters, so we vary both. The first table repeats the analysis at $K\in{50, 100, 200, 500}$ : finer clusters lower absolute coverage, but coverage still increases from 1- to 4- to 16-shot with 16-shot close to full data. The second rebuilds the clusters from the union of $2{,}000$ rollouts per setting instead of from full-data states alone, and the ordering survives; because states from all settings help define these pooled clusters, pooled coverage is not a recovery measure, and its rows are to be read only against each other. The figure varies the positions read from each rollout, comparing the final non-padding token and fixed absolute positions $2{,}000$ and $4{,}000$ against the 8 relative positions of the main analysis. All four preserve the ordering. The fixed-position variants land close to the main construction, whereas the final token gives markedly lower coverage when there are few queries, which is what one expects of a position governed by response length and termination; we therefore read 8 relative positions, which sample throughout a response of any length.

: Table 4: Full-data-anchored training-state coverage at step $300$ under different cluster counts $K$ .

: Table 5: Training-state coverage under alternative cluster constructions, at $K=200$ .

One-shot has a heavier tail of distant states, but is not separable from full data (Table 6).

Nearest-centroid assignment places a state in some cluster however far it lies from every full-data state, so coverage alone cannot tell a setting that fills the reference space thinly from one that leaves it. We therefore also measure, for each state, the mean cosine distance to its ten nearest reference states in the unreduced $1536$ -dimensional space, and call a state off-reference when that distance exceeds the $95$ th percentile of the held-out full-data states. One-shot OPD exceeds the threshold more often than the other settings, so its distances have a heavier upper tail. But an AUROC of $0.552$ against held-out full-data states is close to chance, so one-shot states are not separable from full-data states as a whole.

: Table 6: Reference-distance diagnostics across query-set sizes. AUROC compares each setting with the held-out full-data states. The last row is the complete full-data pool, including the rollouts that built the clusters, so its exceedance rate need not equal the 5% the threshold is set at.

B.2 Controls on the Training Data

Diversity in 16-shot OPD (Figure 16).

Every query on the ladder of Section 4.2 represents a different semantic cluster, so it raises diversity and count together. Holding the count at 16 and drawing the second set from a single cluster separates them: the 16 cluster representatives reach $70.9$ validation accuracy at step $300$ against $69.9$ for the single-cluster set, which is about what 4-shot OPD reaches with a quarter as many queries, and their state coverage separates the same way and does so early, $98.9%$ against a single-cluster set that flattens near $76.8%$ within $100$ steps. Queries from one cluster keep returning the student to the part of the state space it has already seen.

Query scheduling in 2-shot OPD (Figure 17).

Holding a two-query set fixed and varying only when each query is shown, the sequential schedule trains on the hard query for the first $300$ steps of a $600$ -step budget and on the medium query for the rest, while the mixed schedule puts $32$ copies of each in every batch. At step $600$ they reach $68.8$ and $69.1$, both above the $67.4$ of the one-shot baseline they share, so ordering a fixed query set barely matters next to what the second query itself buys. These runs train longer than those of Section 4.2 and take the hard query as their baseline, so neither their horizon nor their baseline is interchangeable with Figure 7.

C. Details for Section 5

C.1 Estimating the Absorption Rate and the Measurement Window

Section 5.2 measures under three conventions. Why each one:

• $\boldsymbol{[t/1.4, , 1.4t]}$ . A difference between adjacent steps is mostly noise once $d$ moves slowly, so $v_t$ is read as the slope of $\log d$ across the endpoints of this window, using the median $d$ near each. Only centers whose window fits inside the measured range are plotted, which is why the main text reads the rate at steps $50$ and $200$ rather than at step $300$ .
• $\boldsymbol{d_t/d_{30}}$ . Each run measures its distance on the states it visits and the four sit at different levels by step $30$, so plotting each against its own value there lets the curves be compared by shape rather than by level.
• Start at step $30$ . Gradient clipping binds until then, and a clipped update is not the update the objective asked for: $v_t$ inside that window measures what one fixed-size step buys rather than what one step buys, which is why the measured rate rises there rather than falls.

Table 7 collects the readings Section 5.2 quotes from this range: the proportion of the step- $30$ distance still left at step $300$, and the factor by which the absorption rate falls between steps $50$ and $200$ .

: Table 7: Alignment dynamics over steps $30$ – $300$ for on-policy OPD, at the default learning rate.

C.2 The Learning Rate Rescales the Step Axis

We repeat on-policy one-shot OPD at half, the default, and twice the default learning rate, holding every other setting fixed, and read all three over the same $300$ steps as Section 5.2.

The learning rate rescales the step axis and leaves the slowdown untouched.

A larger learning rate reaches any given distance sooner, the three curves separated by roughly the ratio of the learning rates (Figure 18). Changing the horizontal axis from the step $t$ to $u=(lr/10^{-6}), t$ brings them onto one curve: the spread between the three runs narrows from about $2.2\times$ to about $1.1\times$ . The learning rate therefore sets only how fast a run moves along that one curve, not the shape of the curve itself. It bends flatter as it goes, just as the four query settings of Section 5.2 do against the step count. A proportionally larger learning rate therefore shortens the run proportionally, over the fourfold range we tested, without reaching whatever makes alignment slow in the first place.

D. Details for Section 6

Table 8 shows the default hyperparameters for MOPD.

\begin{tabular}{ll}
\toprule
\textbf{Item} & \textbf{Value} \\
\midrule
Number of teachers & 3 (math / code / IF) \\
LogProb top- $k$ & 0 \\
KL coefficient & 0.0 \\
Loss aggregation & seq-mean-token-mean \\
\midrule
Rollout batch size & 64 \\
Mini batch size & 64 \\
Responses per prompt & 1 \\
Training temperature & 1.0 \\
Top- $p$ & 1.0 \\
\midrule
Optimizer & AdamW ($\beta_1{=}0.9$, $\beta_2{=}0.999$, wd $=0.01$) \\
Learning rate & 1e-6 \\
Gradient clipping & 1.0 \\
\midrule
Max prompt length & 4096 \\
Max response length & 7680 \\
\bottomrule
\end{tabular}

E. Details for Section 7

E.1 Template and WildChat OPD in Code Generation

Section 7.1 reports the four input conditions in mathematics. Here we report the same four in code generation.

Setup.

The student is R1-Distill-1.5B, the teacher DeepCoder-1.5B-Preview ([63]), and the baseline full-data OPD on TACO ([64]). The base template, the system template, and the WildChat queries are exactly those of Section 7.1; the system template's one-line instruction is the competitive-programming instruction of Figure 11 b. We evaluate on LiveCodeBench v6 under avg@3.

Results.

Figure 19 shows the four conditions ending within $1.7$ points of one another, and the token accounting separating them much further than the step accounting does: over the same $500$ steps the baseline spends $277$ M rollout tokens against $18$ M for the base template, with the other two in between. Actor entropy orders the conditions exactly as it does in mathematics, with WildChat highest and the system template lowest. The one difference is that no condition falls behind here, whereas WildChat costs about a point in mathematics.

References

Section Summary: The references section compiles dozens of recent academic papers, technical reports, and datasets from 2024 to 2026, centered on advances in large language models from teams working on systems like Qwen, DeepSeek, and Llama. It emphasizes methods such as knowledge distillation, reinforcement learning for reasoning, and the building of training data for math and coding tasks, alongside various benchmarks for evaluating model capabilities. Most entries are arXiv preprints, with a smaller number drawn from major conferences like NeurIPS and ICLR.

[1] Yang et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.

[2] Xiao et al. (2026). Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780.

[3] Zeng et al. (2026). Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763.

[4] Xu et al. (2026). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv preprint arXiv:2606.19348.

[5] Team et al. (2026). Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653.

[6] Gu et al. (2024). Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations. pp. 32694–32717.

[7] Agarwal et al. (2024). On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations. pp. 21246–21263.

[8] Li et al. (2026). Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016.

[9] Fu et al. (2026). Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv preprint arXiv:2603.25562.

[10] Cai et al. (2026). Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation. arXiv preprint arXiv:2605.11739.

[11] Zhu et al. (2026). The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes. arXiv preprint arXiv:2605.11182.

[12] Wang et al. (2026). Reinforcement learning for reasoning in large language models with one training example. Advances in Neural Information Processing Systems. 38. pp. 122721–122764.

[13] Yang et al. (2026). Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. arXiv preprint arXiv:2602.12125.

[14] Guo et al. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

[15] He et al. (2025). Justrl: Scaling a 1.5 b llm with a simple rl recipe. arXiv preprint arXiv:2512.16649.

[16] Zeng et al. (2025). Rlve: Scaling up reinforcement learning for language models with adaptive verifiable environments. arXiv preprint arXiv:2511.07317.

[17] Hui et al. (2024). Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186.

[18] Lin et al. (2024). Hammer: Robust function-calling for on-device language models via function masking. arXiv preprint arXiv:2410.04587.

[19] Grattafiori et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

[20] Zhang et al. (2025). Co-reward: Self-supervised reinforcement learning for large language model reasoning via contrastive agreement. arXiv e-prints. pp. arXiv–2508.

[21] Olmo et al. (2025). Olmo 3. arXiv preprint arXiv:2512.13961.

[22] Yu et al. (2026). Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems. 38. pp. 113222–113244.

[23] Guilherme Penedo et al. (2025). CodeForces. https://huggingface.co/datasets/open-r1/codeforces.

[24] OpenBMB (2026). UltraData-SFT-2605. https://huggingface.co/datasets/openbmb/UltraData-SFT-2605.

[25] Liu et al. (2024). Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems. 37. pp. 54463–54482.

[26] Sheng et al. (2025). Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems. pp. 1279–1297.

[27] Dan Hendrycks et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS.

[28] Li et al. (2024). Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 13(9). pp. 9.

[29] Balunovic et al. (2025). Matharena: Evaluating llms on uncontaminated math competitions. Advances in Neural Information Processing Systems. 38.

[30] Jain et al. (2025). Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations. pp. 58791–58831.

[31] He et al. (2024). Multi-if: Benchmarking llms on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410.15553.

[32] Patil et al. (2025). The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning.

[33] Chen et al. (2024). Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216. 4(5).

[34] Zhao et al. (2024). Wildchat: 1m chatgpt interaction logs in the wild. In International Conference on Learning Representations. pp. 34590–34605.

[35] Song, Mingyang and Zheng, Mao (2026). A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626.

[36] Blakeman et al. (2026). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv preprint arXiv:2606.15007.

[37] Shen et al. (2026). On the Geometry of On-Policy Distillation. arXiv preprint arXiv:2606.07082.

[38] Ko et al. (2026). Scaling Reasoning Efficiently via Relaxed On-Policy Distillation. arXiv preprint arXiv:2603.11137.

[39] Mukherjee et al. (2026). Reinforcement learning finetunes small subnetworks in large language models. Advances in Neural Information Processing Systems. 38. pp. 132119–132138.

[40] Zhu et al. (2025). The path not taken: Rlvr provably learns off the principals. arXiv preprint arXiv:2511.08567.

[41] Armandpour et al. (2026). Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why. arXiv preprint arXiv:2605.10889.

[42] Wang et al. (2026). Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation. arXiv preprint arXiv:2605.26844.

[43] Li et al. (2025). Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886.

[44] Zhou et al. (2023). Lima: Less is more for alignment. Advances in Neural Information Processing Systems. 36. pp. 55006–55021.

[45] Ye et al. (2025). Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387.

[46] Muennighoff et al. (2025). s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 20286–20332.

[47] Zhang et al. (2026). Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026. pp. 25553–25569.

[48] Xia et al. (2024). Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333.

[49] Chen et al. (2024). Alpagasus: Training a better alpaca with fewer data. In International Conference on Learning Representations. pp. 34767–34797.

[50] He et al. (2025). The right time matters: Data arrangement affects zero-shot generalization in instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2025. pp. 222–243.

[51] He et al. (2025). Air: A systematic analysis of annotations, instructions, and response pairs in preference dataset. arXiv preprint arXiv:2504.03612.

[52] Wang et al. (2023). Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers). pp. 13484–13508.

[53] Xu et al. (2024). WizardLM: Empowering large pre-trained language models to follow complex instructions. In International Conference on Learning Representations. pp. 30745–30766.

[54] Xu et al. (2025). Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. In International Conference on Learning Representations. pp. 76346–76382.

[55] He et al. (2026). How Far Can Unsupervised RLVR Scale LLM Training?. In International Conference on Learning Representations. pp. 14823–14865.

[56] Zhao et al. (2026). Learning to reason without external rewards. In International Conference on Learning Representations. pp. 2548–2581.

[57] Agarwal et al. (2026). The unreasonable effectiveness of entropy minimization in llm reasoning. Advances in Neural Information Processing Systems. 38. pp. 107150–107180.

[58] Zuo et al. (2026). Ttrl: Test-time reinforcement learning. Advances in Neural Information Processing Systems. 38. pp. 131459–131483.

[59] Dong et al. (2025). Reinforcement Pre-Training. arXiv preprint arXiv:2506.08007.

[60] Zhao et al. (2026). Absolute zero: Reinforced self-play reasoning with zero data. Advances in Neural Information Processing Systems. 38. pp. 105816–105879.

[61] Zweiger et al. (2026). Self-adapting language models. Advances in Neural Information Processing Systems. 38. pp. 74084–74115.

[62] Pyatkin et al. (2026). Generalizing verifiable instruction following. Advances in Neural Information Processing Systems. 38.

[63] Luo et al. (2025). Deepcoder: A fully open-source 14b coder at o3-mini level. Notion Blog. 1.

[64] Rongao Li et al. (2023). TACO: Topics in Algorithmic COde generation dataset. arXiv preprint arXiv:2312.14852.