Weak-to-Strong On-Policy Distillation

Fangxu Yu$^{1,*}$, Zinan Lin$^{2}$, Xiaodong Liu$^{2}$, Weijia Xu$^{2}$, Michael Xu$^{2}$, Tianyi Zhou$^{3}$, Jianfeng Gao$^{2}$
$^{1}$University of Maryland, College Park, $^{2}$Microsoft Research, $^{3}$MBZUAI

$^{*}$ Work done during the internship at Microsoft Research.

Project page: https://w2s-opd.github.io

Abstract

On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, has become an effective paradigm for transferring capabilities across large language models (LLMs). Prevailing approaches assume a teacher at least as capable as the student, and either distill a larger model into a smaller one, which fails at the frontier when no larger teacher exists, or train multiple domain experts from a shared base and consolidate them into one student, which requires costly training at the student's scale. To tackle these challenges, we introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. Specifically, W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is then added to the student's own base model. The resulting proxy teacher thus couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct and wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD consistently outperforms OPD and even enables the student to surpass the domain teacher and continues to improve the student when every supervision source is weaker. Further analysis shows that different contrasts yield distinct learning signals: the post-RL and hint contrast emphasizes reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.

Executive Summary: W2S-OPD addresses a practical limit in improving frontier large language models. As models reach the performance frontier, no larger teacher exists for standard distillation, and training multiple domain experts at the student's own scale costs too much. Direct distillation from weaker models also fails: their distributions differ too much from the student, and the weaker teacher caps or even erodes the student's existing capability.

The work evaluates a new weak-to-strong on-policy distillation method. It constructs a proxy teacher by taking the logit difference between a positive weak model and a negative weak model, then adds that difference to the student's own base model. The student then distills from this proxy on its own generated trajectories using reverse KL divergence. Three cheap contrast pairs supply the signal: pre- and post-RL versions of a small model, off-the-shelf base models of different sizes, and a single small model given correct versus wrong hints. Experiments cover Qwen3 models trained on math and code data and tested on seven benchmarks.

The method consistently improves the 8B student. In the pre-/post-RL setting it raised math accuracy 11.4 % relative to ordinary on-policy distillation and placed the student above the small domain expert itself; multi-teacher merging showed similar gains. The scale and hint contrasts improved the student even though every source model was smaller and weaker. Different contrasts strengthened different reasoning steps: RL and hint contrasts emphasized planning and monitoring tokens, while scale contrast emphasized solution steps. Out-of-domain performance rose on GPQA-Diamond and did not degrade on instruction following, unlike ordinary distillation.

These results matter because they show frontier models can continue to advance by extracting directional signals from abundant, inexpensive weak models rather than waiting for stronger teachers or paying for same-scale expert training. The approach adds only modest runtime overhead and preserves, or even improves, general capabilities.

Two next steps follow directly. First, test the framework with additional contrast sources and larger capability gaps to measure how far gains continue before saturation. Second, integrate the technique into routine post-training pipelines so that every released model family can supply improvement signals to larger successors.

The evidence rests on one model family and three contrast types; results could shift with other architectures or data regimes. Performance also depends on choosing a moderate amplification strength; values that are too high distort the proxy teacher. Overall, the reported improvements are consistent across settings and benchmarks, supporting near-term use while further scaling tests proceed.

1. Introduction

Section Summary: Reinforcement learning and knowledge distillation are two main ways to improve large language models’ reasoning, but each has drawbacks: the first gives sparse feedback while the second can cause the student to copy mistakes from off-policy data. On-policy distillation tries to combine their strengths by letting a student learn from a teacher’s detailed signals on the student’s own outputs, yet it usually requires a teacher that is already stronger than the student. The paper introduces Weak-to-Strong On-Policy Distillation, which instead builds a helpful “proxy teacher” from pairs of weaker models (for example, before and after small-scale training, or models of different sizes) by isolating the useful capability difference and adding it to the student’s own distribution, allowing the large model to keep improving at modest cost.

Reinforcement learning with verifiable rewards (RLVR) ([1, 2, 3]) and knowledge distillation ([4, 5]) are two dominant paradigms for improving the reasoning ability of large language models (LLMs). RLVR scales optimization directly from verifiable outcomes, yet its sparse reward provides limited fine-grained supervision. In contrast, knowledge distillation offers dense token-level supervision from a teacher, but the learning from teacher-generated off-policy trajectories suffers from exposure bias. On-policy distillation (OPD) ([6, 7]) combines the complementary strengths of both, which supervises the student with a teacher's token-level fine-grained supervision, yet on trajectories sampled from the student's own policy. This approach delivers dense credit assignment that alleviates exposure bias. However, the premise is a teacher at least as capable as the student.

This premise breaks down in the two ways current practice makes concrete. The strong-to-weak paradigm distills a larger model into a smaller one. For instance, Qwen3 ([8]) distills a large-scale model (e.g. 235B) into smaller variants (e.g. 8B, 14B). As models approach the frontier, however, no larger teacher exists to distill from, capping the performance ceiling. A second line instead performs same-origin consolidation: rather than assuming a single superior teacher, it trains multiple domain experts in parallel from a shared base model and distills their capabilities back into one student. MOPD ([9]) instantiates this by applying per-domain RL to obtain a set of domain teachers and merging them in policy space, while DeepSeek-V4 ([10]) scales the same specialize-then-consolidate recipe to the trillion-parameter regime. Yet training multiple domain experts at the student's scale is computationally expensive, inflating the cost of OPD. We therefore move to the weak-to-strong paradigm ([11, 12]), which induces supervisory signals from weaker models that already exist or can be trained inexpensively. This enables a strong model to keep improving when no stronger teacher is available at a lower cost.

However, directly applying OPD to weak-to-strong learning is challenging. First, the weak teacher's distribution is distinct from the student's, and such a mismatch weakens the distillation signals ([13, 14]). Second, since the teacher is weaker than the student, distillation toward it bounds the student at the teacher's level and risks eroding the general capability it already possesses.

To address these challenges, we propose Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that distills the strong student from multiple weak models. Figure 2 provides an overview. Specifically, W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive model and a negative model, both substantially smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, and W2S-OPD adds this direction to the student's own base model. The proxy teacher therefore inherits the isolated direction while remaining distributionally close to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts.

We propose three ways to construct the proxy teacher from weak models: i) pre-RL and post-RL models, where the positive model is a domain expert trained by RL and the negative model is its pre-RL initialization, whose difference isolates the skill that RL instills; since the RL need only be run at the small model's scale, this transfers an expensive capability to the large student without ever running RL at the student's scale; ii) larger and smaller base models, two off-the-shelf base models of different sizes, whose difference isolates the capability that emerges from scale; this signal comes for free from models that already exist, requiring no new data, reward design, or training; and iii) a single base model conditioned on correct and wrong hints, which extracts the instance-level direction toward the solution from single model and provides token-level supervision more efficiently. We evaluate all three settings on four math reasoning and three code generation benchmarks. Our main findings are threefold:

• Surpassing the teacher it learns from. In the pre-RL / post-RL setting, W2S-OPD outperforms OPD by 11.4% and 12.0% relative on math reasoning under single- and multi-teacher distillation, and lifts the student above the domain expert itself.
• Improving from purely weaker models. In the smaller/larger and contrastive-hints settings, W2S-OPD still improves the student even though every supervision source is weaker than it.
• Different contrasts reinforce complementary reasoning patterns. The post-RL and hint contrast places more emphasis on the tokens for reasoning framework (e.g. planning and monitoring the solving process), whereas the scale contrast emphasizes the solving procedure.

More broadly, W2S-OPD reframes weak-to-strong learning as isolating and transferring a capability direction rather than imitating a weak supervisor, suggesting how frontier models can keep improving from abundant existing weak signals rather than waiting for a stronger teacher to be built.

2. Preliminary: On-Policy Distillation

Section Summary: On-policy distillation trains a student model by letting it first generate its own sequences of text and then having a fixed teacher model provide token-by-token feedback on those sequences, so the student reduces the divergence between its predictions and the teacher’s at every step. Because the sequences come from the student itself rather than a fixed dataset, the teacher corrects the student exactly where it tends to go wrong, avoiding the mismatch that arises when a student only sees teacher-generated text. This approach is commonly used to compress a larger model into a smaller one or to merge several specialized models back into one, yet it runs into limits when the student is already at the frontier of capability and no stronger teacher is available.

On-policy distillation (OPD), formalized by the Generalized Knowledge Distillation (GKD) framework ([6]), bridges reinforcement learning and imitation learning by providing dense, token-level supervision on the student's own trajectories. Unlike off-policy distillation that learns from a fixed teacher-generated dataset, OPD lets the student policy $\pi_{S}$ generate rollouts on-policy, and a powerful frozen teacher $\pi_{T}$ then scores every token of these self-generated trajectories, against which the student minimizes a per-token divergence:

$ \mathcal{L}{\mathrm{OPD}}(\pi{S}) ;=; \mathbb{E}{x \sim \mathcal{D}, ; y \sim \pi{S}(\cdot \mid x)} \bigg[\sum_{t=1}^{|y|} D\big(\pi_{S}(\cdot \mid s_t) , \big|, \pi_{T}(\cdot \mid s_t) \big) \bigg],\tag{1} $

where $s_t = (x, y_{<t})$ is the student-generated prefix and $D$ is a per-token divergence. Because states are drawn from the student's own rollouts, the teacher corrects the student precisely on the states the student actually visits, thereby avoiding the exposure bias of off-policy distillation. We instantiate $D$ as the reverse KL divergence $\mathrm{KL}(\pi_{S} , |, \pi_{T})$. In practice, we employ Top- $k$ OPD for computational efficiency; see more details in Appendix A.

Two prominent uses of OPD are strong-to-weak distillation and same-origin consolidation. The former compresses a larger, more capable teacher into a smaller student ([8]), while the latter distills several RL-trained domain experts back into a single model, with all experts initialized from the student's base so that their distributions stay close ([5, 9]). Both presuppose a teacher at least as capable as the student, which becomes ineffective when training the frontier model, since no larger teacher exists to distill from, and training qualified experts at the student's scale is computationally expensive. These limitations motivate us to instead leverage small, cheaply obtained models to supervise a stronger student.

3. Weak-to-Strong On-Policy Distillation

Section Summary: W2S-OPD improves a capable student model by learning from weaker models without letting those weaker models limit its performance or create a mismatch in output style. It works by subtracting the output logits of a weaker “negative” model from those of a stronger “positive” model to isolate a transferable capability direction, then adding that direction to the student’s own base model to create a proxy teacher that stays close to the student’s distribution. The student then generates its own responses and is trained to match this proxy teacher via a reverse KL objective, allowing it to absorb the desired skill while retaining its general abilities.

Improving a strong student with supervision from weaker models presents two central challenges: i) the distributional mismatch between the weak models and the student renders direct distillation ineffective ([13, 14]); and ii) since the supervision sources are weaker than the student, direct distillation constrains the student to a lower-capability policy and degrades the general ability it already possesses. To address both challenges, we adopt a directional perspective, as illustrated in Figure 2. Instead of imitating a weak model directly, we isolate the capability direction that separates a stronger positive model from a weaker negative model, a signal that is largely disentangled from model scale and hence transferable across scales, and re-anchor this direction onto the student's base model. The resulting proxy teacher is simultaneously equipped with the target domain capability and distributionally adjacent to the student, which stabilizes optimization and preserves the student's general ability. We detail the proxy teacher construction and distillation objective in § 3.1, the instantiations of the contrast pair in § 3.2, and its extension to multi-teacher distillation in § 3.3.

3.1 Proxy Teacher Synthesis and Distillation

Suppose the student $\pi_{S}$ is initialized from a strong base model, while the available weak models form a contrast pair: a positive model $m^{+}$ and a weaker negative model $m^{-}$. Specifically, $m^{+}$ is obtained either by applying domain-specific RL to its initialization $m^{-}$, or taken as a stronger base model than $m^{-}$ (we detail these instantiations in § 3.2). W2S-OPD constructs the teacher directly from these three frozen models, requiring only their output logits.

The key observation behind W2S-OPD is that, although both $m^{+}$ and $m^{-}$ are weaker than the strong base model, their difference still encodes a transferable capability direction. What the two weak models agree on reflects their shared, limited ability, so subtracting their logits cancels this common component and retains precisely the direction along which $m^{+}$ improves over $m^{-}$. Adding this capability direction onto the strong base model therefore yields a proxy teacher that combines it with the student's general strength while staying distributionally close to the student. We instantiate this construction following decoding-time experts ([15, 16]), where $m^{+}$ acts as the positive model whose logits are additively combined, $m^{-}$ acts as the negative model whose logits are negatively combined, and the student's base model serves as the anchor. Formally, at each position $t$, we condition the three models on the prefix $s_t$ to obtain the logit scores $z_{\mathrm{base}}$, $z^{+}$, and $z^{-}$, and synthesize the proxy teacher as:

$ \pi_{T, \alpha}(\cdot \mid s_t) ;=; \mathrm{softmax}\Big(, z_{\mathrm{base}}(s_t) , +, \alpha \big(z^{+}(s_t) - z^{-}(s_t) \big) \Big),\tag{2} $

where $\alpha \geq 0$ is an amplification coefficient controlling the strength of the injected capability direction, and thus trades off signal strength against distributional proximity. A small $\alpha$ injects little of the direction and keeps the proxy teacher close to the base model, yielding supervision too weak to drive improvement, whereas a large $\alpha$ injects a stronger signal but risks distorting the distribution and pushing the teacher away from the student. Given the constructed proxy teacher, W2S-OPD instantiates the OPD objective in Equation 1 with $\pi_{T} = \pi_{T, \alpha}$:

$ \mathcal{L}{\text{W2S-OPD}}(\pi{S}) ;=; \mathbb{E}{x \sim \mathcal{D}, ; y \sim \pi{S}(\cdot \mid x)} \bigg[\sum_{t=1}^{|y|} \mathrm{KL}\big(\pi_{S}(\cdot \mid s_t) , \big|, \pi_{T, \alpha}(\cdot \mid s_t) \big) \bigg],\tag{3} $

Taken together, W2S-OPD address both challenges raised at the beginning of this section. Anchoring the proxy teacher at the student's own base model keeps the supervision target within a distribution the student already realizes (with $\alpha$ bounding how far it moves), and since the weak models enter only through their logit difference rather than their absolute level, distillation transfers their capability direction without pulling the student down toward them, which alleviates the capability ceiling. Appendix B gives an alternative view of this objective as reward maximization under a KL trust region.

3.2 Instantiations of the Positive and Negative Models

W2S-OPD requires a contrast pair $(m^{+}, m^{-})$ to construct the proxy teacher. We propose three instantiations and summarize in Table 1:

i) Pre-RL and post-RL: the positive model $m^{+}$ is a domain expert obtained by applying RL to a small base model, and the negative model $m^{-}$ is its pre-RL initialization, whose difference isolates the domain skill acquired through RL. This is a cheaper way of reducing the cost of training domain experts at the student's own scale.

ii) Smaller and larger: the positive and negative models are two off-the-shelf base models of different sizes (e.g. Qwen3-4B and Qwen3-0.6B). Their difference isolates the capability that emerges purely from scale and requires no additional training. This signal is already available in released models and can be used for free to further improve the capability of frontier models.

iii) Correct and wrong hints: the positive and negative models are a single base model conditioned on a correct and a wrong solution hint of the identical format, respectively. Since the hint-conditioning is shared, their difference cancels the style shift it induces ([17]) and isolates the instance-level direction toward the correct solution. Needing only one small model and a reference solution, which token-level supervision efficiently.

These instantiations indicate that W2S-OPD can support various origins of the contrast, in which the pair exhibiting a separable capability gap can serve as the supervision source, since the construction of the proxy teacher requires only the output logits of the pair.

3.3 Unification of Single- and Multi- Proxy Teacher Distillation

W2S-OPD can naturally extend to multiple proxy teacher distillation scenarios. Given $K$ positive models ${m^{+}k}{k=1}^{K}$, each paired with a weaker negative model $m^{-}_k$, W2S-OPD composes their capability directions on the shared base model:

$ \pi_{T, \boldsymbol{\alpha}}(\cdot \mid s_t) ;=; \mathrm{softmax}\Big(, z_{\mathrm{base}}(s_t) , +, \sum_{k=1}^{K} \alpha_k \big(z^{+}_k(s_t) - z^{-}_k(s_t) \big) \Big),\tag{4} $

where $\alpha_k$ controls the strength of capability direction $k$. Since each subtraction isolates its own capability direction, the summation injects multiple domain skills into a single proxy teacher, and one distillation run with Equation 3 merges them into a unified student. For example, $\alpha_k$ can be set to ${0, 1}$. In this way, each query is routed to its most suitable positive and negative model pair for distillation.

::: {caption="Table 1: The three instantiations of the contrast pair in W2S-OPD. All models are from Qwen3 series."}

:::

4. Experiments

Section Summary: In the experiments, researchers tested W2S-OPD across three contrast-pair setups involving math and code tasks, pitting it against standard distillation from a single expert model on benchmarks such as AIME, HumanEval, and GPQA. The method consistently delivered stronger gains than baselines, often surpassing the original expert teachers themselves and improving an 8B student even when the paired models were weaker or merely differed by correct versus incorrect hints. Training proved faster and more stable under the contrastive approach, with results holding across single-teacher, multi-teacher, and off-the-shelf model scenarios.

We evaluate W2S-OPD under the three contrast-pair settings. § 4.2 reports the main results for each, § 4.3 examines the factors behind its effectiveness, and § 4.4 characterizes the learning signals the contrast provides.

4.1 Experimental Setups

Benchmarks. For math reasoning, we evaluate on four competition-level benchmarks: AIME24 ([18]), AIME25 ([19]), HMMT25 (Feb.), and HMMT25 (Nov.) ([20]). For code generation, we adopt HumanEval+, MBPP+ ([21]), and LiveCodeBench-V6 ([22]). To assess out-of-domain generalization, we further evaluate on GPQA-Diamond ([23]) for scientific reasoning and IFBench ([24]) for instruction following.

Baselines and Evaluation Metrics. We compare W2S-OPD against OPD, which distills the student directly from the positive model under an identical training configuration. As reference points, we additionally report the positive and negative models that form the contrast pair, as well as the student itself. In all evaluations, we set the temperature to 1.0 and top- $p$ to 1.0. On each math reasoning benchmark, we sample 32 solutions per problem, whereas on each code generation benchmark, we sample 4 solutions per problem, and report the average accuracy over all samples on each benchmark for a robust evaluation. We adopt Math-Verify to validate answer correctness for math reasoning.

Training details. We use Qwen3-8B ([8]) in non-thinking mode as the student $\pi_{S}$; a frozen copy of the same checkpoint serves as the anchor. Both W2S-OPD and OPD are trained for 100 steps under an identical configuration. See Appendix C for more implementation details.

Contrast pairs. Table 1 summarizes the contrast pair of each setting. In the pre-RL / post-RL setting, the positive model $m^{+}$ is a Qwen3-4B domain expert obtained by applying GRPO ([2]) to the Qwen3-4B base, for 500 steps on the level-6 subset of DeepMath-103K ([25]) for math reasoning and for 300 steps on Eurus-RL-Code ([26]) for code generation, and the negative model $m^{-}$ is its pre-RL initialization, i.e. the original Qwen3-4B model. In the smaller/larger setting, the positive and negative models are the off-the-shelf Qwen3-4B and Qwen3-0.6B base models, requiring no additional training. In the contrastive-hint setting, the positive and negative models share the Qwen3-4B base model, conditioned on a correct and a wrong solution hint of the identical format, respectively.

4.2 Main Results

::: {caption="Table 2: Results for Pre-RL / Post-RL contrast setting. W2S-OPD beats OPD and even surpasses the 4B-expert on math. Improv. reports the absolute gain over OPD."}

:::

::: {caption="Table 3: Results for contrastive hints setting. W2S-OPD improves the 8B student above its own base with only a single weaker and smaller model. Improv. reports the absolute gain of W2S-OPD over the student base model."}

:::

i) Distillation from Pre-RL / Post-RL. Table 2 compares W2S-OPD against OPD under single-teacher distillation, which learns from a single domain expert, and multi-teacher distillation, which merges multiple domain experts (here a math and a code expert) into one student. We have the following observations:

(a) A small post-RL expert substantially improves the large student. W2S-OPD consistently outperforms SFT and OPD across all benchmarks in the single-teacher setting, with 11.4% and 3.7% average relative improvements on math and code, respectively. Notably, on math reasoning W2S-OPD surpasses the domain teacher itself, whereas OPD remains below it. On code generation, W2S-OPD likewise narrows the gap to the teacher.

(b) W2S-OPD outperforms OPD when distilling from multiple domain teachers. We conduct experiments in the multi-teacher setting, where we aim to merge the capabilities from different domain teachers into a single student through OPD, which is obtained by applying domain-specific RL to the same base model. Specifically, we mix the math and coding training data and route each sample to the corresponding domain teacher through their domain labels. The results in Table 2 demonstrate that W2S-OPD consistently leads to better performance than OPD on all benchmarks.

(c) W2S-OPD learns faster and more stably. Figure 3 tracks benchmark performance over training and shows that W2S-OPD outperforms OPD throughout the training.

::: {caption="Table 4: Results for Smaller and Larger contrast setting. W2S-OPD improves the 8B student above its own base even though both source models are weaker than it. Improv. reports the absolute gain of W2S-OPD over the student base model."}

:::

ii) Distillation from Smaller / Larger Base Models. In this setting, we find that

Two weak base models can improve the stronger student. As shown in Table 4, when the proxy teacher is constructed from Qwen3-4B and Qwen3-0.6B, both weaker than the student, W2S-OPD still improves the student by an absolute 6.0% on math reasoning and 1.2% on code generation on average. This confirms that the inherent gap between two off-the-shelf weak models also encodes a transferable improving direction to distill from.

iii) Distillation from a Single Model with Correct / Wrong Hints. As shown in Table 3, a contrastive hint direction from a single weak model improves the student. W2S-OPD improves the student by an absolute 1.4% on math reasoning and 1.1% on code generation on average, despite the hint model being a 4B model weaker than the student. This construction distills the privileged information and shows that a meaningful learning signal can also be extracted from a difference in context without requiring two different models.

4.3 Training Analysis

Using the Pre-RL / Post-RL setting, we conduct two further analyses: the impact of the amplification coefficient $\alpha$ on distillation, and out-of-domain generalization compared with OPD.

4.3.1 Effect of the Amplification Coefficient $\alpha$

To investigate the impact of the amplification coefficient, we vary $\alpha$ with all other configurations fixed and report the results in Figure 4. When $\alpha$ is small, $\pi_{T, \alpha}$ stays close to the student's own base model, where the capability direction is barely injected, the distillation signal is weak, and the distilled student falls behind OPD. As $\alpha$ increases, more domain skill is injected into the proxy teacher, and the student's performance improves accordingly. When the $\alpha$ continues increasing, performance starts to decline on most benchmarks, likely because an overly large $\alpha$ drives the proxy teacher further away from the student's distribution, making the supervision increasingly hard for the student to learn. Consequently, a moderate $\alpha$ strikes the best balance between the strength of the injected signal and the adjacency of the teacher.

4.3.2 Generalization to Out-of-Domain Tasks

::: {caption="Table 5: Results for OOD generalization on GPQA-Diamond and IFBench. Both distillation methods are trained only on the math task, W2S-OPD transfers out of domain and improves general ability, whereas OPD can degrade it below the base. Improv. indicates the absolute gain over OPD."}

:::

A crucial concern for weak-to-strong distillation is whether learning from a small domain expert erodes the strong student's general capability. To investigate this, we evaluate the student trained in the Pre-RL / Post-RL setting on two out-of-domain benchmarks. As shown in Table 5, the distilled reasoning skill transfers well beyond the training domain. W2S-OPD lifts the student from 38.9 to 56.5 on GPQA-Diamond, outperforming OPD by an absolute 2.1 %. The two models diverge on IFBench, where OPD degrades the student below its initialization, consistent with the student absorbing the small expert's limitations, yet W2S-OPD instead improves compared to the base model and outperforms OPD by 1.1%. These results indicate that W2S-OPD transfers domain skill without sacrificing and even enhancing the student's general ability.

4.4 What Kinds of Tokens Are Most Strengthened?

Since all settings improve the student, we raise the question of what each capability direction actually provides. We quantitatively analyze this question by recording each token's offset between the positive and negative models on a shared solution generated by the student model. At every position $t$ along this trace, the capability direction in logit space is $\Delta z_t = z^{+}(s_t) - z^{-}(s_t)$. To measure how strongly it reinforces the realized token $x_t$, we take the log-probability each model assigns to $x_t$ and define:

$ \Delta_t ;=; \log \pi^{+}(x_t \mid s_t) ;-; \log \pi^{-}(x_t \mid s_t), \qquad \pi^{\pm}=\mathrm{softmax}(z^{\pm}).\tag{5} $

Tokens with large $\Delta_t$ are those most strongly reinforced by the capability direction. To identify which reasoning steps each contrast reinforces, we collect the top-1% highest- $\Delta$ tokens across 200 math-reasoning traces and label every token with one of eight problem-solving episodes ([27]) using ThinkARM ([28]), an automatic episode classifier. Table 6 shows the token distribution. Relative to the scale contrast, the post-RL and hint contrasts place more weight on the reasoning framework such as the Plan and Monitor episodes that structure and track the solution, whereas the scale contrast keeps more weight on the core solving steps, Analyze and Implement. The hint contrast additionally concentrates on the final Answer tokens, which is a consequence of conditioning on a correct versus a wrong answer. Figure 5 illustrates the same division on a single trace:

the three contrasts reinforce different kinds of tokens. These emphases are complementary, resulting in a student learning to improve the reasoning by distilling different patterns.

::: {caption="Table 6: Distribution of the top-1% highest- Delta tokens over the eight Schoenfeld episodes. The first row gives the distribution of each episodes of all tokens."}

:::

5. Related Work

Section Summary: Related work in on-policy distillation has largely focused on scenarios where a stronger teacher model supervises a weaker student during its own outputs, often to transfer skills or combine specialized models. Weak-to-strong generalization research instead examines how weaker models can supervise stronger ones, though typically with less detailed guidance than token-by-token signals. W2S-OPD draws on techniques from controllable text generation to create a synthesized proxy teacher from multiple weak sources, enabling distillation in the opposite direction without needing an already superior model.

On-Policy Distillation. OPD ([7, 29, 6]) supervises the student on its own rollouts with a superior teacher. It has become a key post-training paradigm for strong-to-weak distillation ([30, 8]) and for merging multi-domain experts into one model ([5, 10, 31, 32]), with follow-ups relaxing its requirements through self-distillation from privileged information ([33, 34, 35, 36]), stabilized optimization against the student–teacher gap ([37, 13, 38, 39]), and multimodal extensions ([40, 41, 42]). These works assume a teacher at least as capable as the student. In contrast, W2S-OPD works weak-to-strong, synthesizing a teacher from weak models rather than requiring a strong superior model. Direct-OPD ([43]) is a concurrent work that transfers a weak teacher's pre-/post-RL log-ratio as a dense reward. However, its contrast is confined to RL-trained teachers, without incorporating the directions from multiple contrastive sources and only evaluated on math reasoning tasks.

Weak-to-Strong Generalization. Weak-to-Strong elicits the capabilities of a stronger model with the supervision of a weak model ([11]). This is critical when stronger model is hard to obtain ([44]). Recent work extends it to LLM reasoning and alignment ([45, 46, 47, 12, 48]). Another line casts learning by teaching where a strong model improves by instructing weaker students and turning their comprehension into a training signal ([49, 50]). However, such supervision primarily provides trajectory-level signals, while W2S-OPD instead distills a synthesized proxy teacher with token-level dense supervision.

Controllable Text Generation. Controlling the outputs of an LLM has been widely studied ([51]) along two directions: training-based methods finetune the model to elicit desired properties ([52, 53]) but are computationally costly, whereas decoding-time methods steer generation by shifting or composing output distributions at inference ([15, 54, 55]). W2S-OPD adopts this decoding-time composition to synthesize its proxy teacher, but turns it into a training signal that transfers capability from weak models to a strong student.

6. Conclusion

Section Summary: The paper presents W2S-OPD, a method that lets researchers improve a capable AI model by drawing only on weaker or smaller models rather than stronger ones. It works by extracting a useful skill direction from pairs of weak models and then aligning that direction with the target model so the resulting guidance stays close to the model’s own behavior. Experiments show this approach boosts performance on math and coding tasks, works across several sources of weak signals, and opens a path for frontier models to keep advancing using supervision that is cheaper and less capable than themselves.

We introduce W2S-OPD, a weak-to-strong on-policy distillation framework that improves a strong student using only smaller or weaker models. By isolating the capability direction between a contrast pair of weak models and re-anchoring it onto the student's base model, W2S-OPD synthesizes a proxy teacher that couples the isolated capability while staying distributionally adjacent to the student. Across math reasoning and code generation benchmarks, W2S-OPD substantially outperforms OPD, surpasses the weak teacher itself, and remains effective when the contrast pair comes from RL training, model scale, or contrastive hints, while preserving out-of-domain capabilities.

Discussion. As models approach the frontier, the assumption of an ever-stronger teacher no longer holds, and where the supervision for OPD should come from becomes an open question. Weak supervision sources are abundant, yet how far they can push a stronger student before weak-to-strong supervision saturates, and how to elicit more informative signals from them, remain to be explored. Future work may develop weak-to-strong learning into a sustained paradigm for OPD and post-training more broadly, in which frontier models continue to improve from supervision sources cheaper and weaker than themselves.

Appendix

Section Summary: The appendix explains two technical refinements for distilling knowledge from teacher models into a student policy. The first part shows that full-vocabulary KL divergence is impractical, so a single-token advantage or a top-K truncation of the teacher’s distribution is used instead; the latter preserves nearly all the probability mass at far lower communication cost and lower gradient variance. The second part reframes logit-arithmetic combinations of positive and negative models as an optimization problem whose solution is a “proxy teacher” that tilts the student’s own base distribution toward capability differences while a tunable KL penalty keeps the result close to the student, yielding a product-of-experts view that is order-invariant and naturally bounded.

A. KL-Divergence Estimation

noindentSingle-token OPD. An alternative estimator, adopted by several recent OPD implementations ([7, 56]), extracts a single scalar of teacher information per sampled token: the log-ratio on the sampled token is treated as a token-level advantage,

$ A_t ;=; \log \pi_{T}(y_t \mid s_t) ;-; \log \pi_{S}^{\mathrm{old}}(y_t \mid s_t),\tag{6} $

and plugged into a clipped policy-gradient update, where $\pi_{S}^{\mathrm{old}}$ denotes the policy that generated the rollouts. This variant reuses the RL infrastructure, but conveys only 1 scalar per position and thus exhibits higher variance than the dense top- $K$ objective, which backpropagates the teacher's distribution at every position through a directly differentiable divergence and requires no advantage estimation or clipping. We therefore adopt the dense top- $K$ objective for both distillation arms in all experiments.

noindentTop- $K$ OPD. Materializing the teacher's full distribution requires storing and transferring $|\mathcal{V}|$ logits (roughly 150K for the Qwen3 family) at every response position, which is prohibitively expensive to communicate across training workers. Since the teacher distribution is highly peaked, its top- $K$ tokens already carry the majority of probability mass at nearly all positions. The teacher worker therefore ships only the top- $K$ token indices and log-probabilities per position, and the divergence in Equation 1 is estimated on this truncated support:

$ \mathcal{L}^{\text{top-}K}{\mathrm{OPD}}(\pi{S}) ;=; \mathbb{E}{x \sim \mathcal{D}, ; y \sim \pi{S}(\cdot \mid x)} \bigg[\sum_{t=1}^{|y|} \mathrm{KL}\big(\bar{\pi}_{S}^{\mathcal{K}t} , \big|, \bar{\pi}{T}^{\mathcal{K}_t} \big) \bigg],\tag{7} $

where $\mathcal{K}_t$ denotes the top- $K$ support of the teacher distribution at position $t$, and $\bar{\pi}^{\mathcal{K}_t}$ denotes the corresponding distribution restricted to $\mathcal{K}_t$ and renormalized. The truncation error is bounded by the tiny probability mass that falls outside the teacher's top- $K$ set, so the estimation preserves the supervision almost losslessly while reducing the cost.

B. From logit arithmetic to a KL-constrained objective

Let $r(s, a):=\log\pi^{+}(a\mid s)-\log\pi^{-}(a\mid s)$ denote the capability reward carried by the contrast pair: a token-level, relative quantity measuring how much more the positive model favors $a$ than the negative one does, in which the absolute competence of either source never appears. Since $e^{z^{+}(a)-z^{-}(a)}$ equals $\pi^{+}(a\mid s)/\pi^{-}(a\mid s)$ up to a factor independent of $a$, which the softmax normalizer absorbs, Equation 2 is an exponential tilting of the student's own base model,

$ \pi_{T, \alpha}(a\mid s)=\frac{\pi_{\mathrm{base}}(a\mid s), e^{\alpha r(s, a)}}{Z_{\alpha}(s)}, \qquad Z_{\alpha}(s)=\mathbb{E}{a\sim\pi{\mathrm{base}}(\cdot\mid s)}\big[e^{\alpha r(s, a)}\big].\tag{8} $

Substituting Equation 8 into $\mathrm{KL}(q, |, \pi_{T, \alpha})$ and rearranging gives, for every $q\in\Delta(\mathcal{V})$,

$ \underbrace{\mathbb{E}{a\sim q}\big[r(s, a)\big] -\tfrac{1}{\alpha}\mathrm{KL}\big(q, |, \pi{\mathrm{base}}(\cdot\mid s)\big)}{\textstyle \mathcal{J}{\alpha}(q, ;s)} ;=;\tfrac{1}{\alpha}\log Z_{\alpha}(s);-;\tfrac{1}{\alpha}\mathrm{KL}\big(q, |, \pi_{T, \alpha}(\cdot\mid s)\big).\tag{9} $

The first term on the right does not depend on $q$ and the second is nonnegative and vanishes only at $q=\pi_{T, \alpha}$. The proxy teacher is therefore the unique closed-form maximizer

$ \pi_{T, \alpha}(\cdot\mid s);=;\arg\max_{q\in\Delta(\mathcal{V})} \Big{\underbrace{\mathbb{E}{a\sim q}\big[r(s, a)\big]}{\text{inject capability}} ;-;\tfrac{1}{\alpha}\underbrace{\mathrm{KL}\big(q, |, \pi_{\mathrm{base}}(\cdot\mid s)\big)}_{\text{stay adjacent to the student}}\Big},\tag{10} $

where $1/\alpha$ is the strength of a trust region centered at the student rather than at any weak model.

A compositional view. Equation 8 can be stated without reference to logits at all,

$ \pi_{T, \alpha}(a \mid s);\propto;\pi_{\mathrm{base}}(a \mid s), \Big(\frac{\pi^{+}(a\mid s)}{\pi^{-}(a\mid s)}\Big)^{\alpha},\tag{11} $

which exhibits the proxy teacher as a product of experts over the shared vocabulary: the student's base model is the anchor, $\pi^{+}$ enters as a conjoined expert and $\pi^{-}$ as a negated one, and $\alpha$ is the exponent controlling how sharply the pair is applied. The capability direction is thus a negation operation, which divides out what the two weak models agree on and the trust region in Equation 10 is precisely the price of keeping the resulting product near the anchor, with $\tfrac{1}{\alpha}\log Z_{\alpha}(s)$ its free energy at $s$. With $K$ contrast pairs the same reading gives:

$ \pi_{T, \boldsymbol{\alpha}}(a \mid s);\propto;\pi_{\mathrm{base}}(a\mid s) \prod_{k=1}^{K}\Big(\frac{\pi^{+}{k}(a \mid s)}{\pi^{-}{k}(a\mid s)}\Big)^{\alpha_{k}},\tag{12} $

so merging capability directions is multiplication of experts. In particular the result is invariant to the order in which the pairs are introduced, unlike sequentially fine-tuning or distilling one teacher after another, where the outcome depends on the schedule and earlier skills may be overwritten.

Three design choices follow: i) Adjacency: the reference measure in Equation 10 is the student's own base model, so the proxy teacher is a reweighting of a distribution the student already realizes, and $\alpha$ sets how far it may move; $\alpha!\to!0$ recovers $\pi_{\mathrm{base}}$ and supplies no supervision, whereas $\alpha!\to!\infty$ removes the trust region and collapses onto $\arg\max_{a} r(s, a)$, so a moderate $\alpha$ is optimal (Figure 4). ii) Contrast: the weak models enter only through the ratio $r$, so a pair with no capability gap leaves the anchor untouched and the student is never ceilinged at the sources, unlike direct OPD whose optimum is $\pi^{+}$ itself. iii) Composition: rewards add where policies do not, so Equation 4 is the same maximizer with composite reward $\sum_{k}\alpha_{k}r_{k}$, and $K$ capability directions are merged by a single distillation run.

C. Implementation Details

All experiments are implemented on top of the verl framework, and each distillation run is conducted on 2 NVIDIA B200 GPUs. Table 7 lists the training hyperparameters shared by W2S-OPD and OPD. For the proxy teacher, the divergence is estimated on the teacher's top- $K$ support with $K=32$ (Appendix A). Table 8 reports the GRPO recipe used to obtain domain experts in the pre-RL / post-RL setting, and Table 9 shows the rollout instructions used during OPD training.

\begin{tabular}{ll}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
Algorithm & On-Policy Distillation \\
Student Model & Qwen3-8B \\
Positive / Negative & Qwen3-4B-RL / Qwen3-4B \\
Amplification Coeff. ($\alpha$) & 1.0 / 0.75 \\
Teacher Top- $K$ & 32 \\
Max Prompt Length & 2048 \\
Max Response Length & 16384 \\
Learning Rate & 2e-6 \\
LR Scheduler & Constant \\
Sampling Temp. / Top- $p$ & 1.0 / 1.0 \\
Batch Size & 64 \\
Training Steps & 100 \\
\bottomrule
\end{tabular}

\begin{tabular}{ll}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
Algorithm & GRPO \\
Base Models & Qwen3-4B \\
Reasoning Mode & Non-Thinking \\
Max Prompt Length & 2048 \\
Max Response Length & 16384 \\
KL Loss Coefficient & 0 \\
Learning Rate & 1e-6 \\
LR Scheduler & Constant \\
Sampling Temp. / Top- $p$ & 1.0 / 1.0 \\
Batch Size & 128 \\
Mini-batch Size & 128 \\
Rollout Number ($G$) & 8 \\
Training Steps & 500 / 300 \\
\midrule
Math Data & DeepMath-103K (L6) \\
Code Data & Eurus \\
\bottomrule
\end{tabular}

::: {caption="Table 9: On-policy rollout prompts used during OPD training."}

:::

D. Additional Experimental Results

D.1 Runtime Analysis

: Table 10: Average wall-clock time per training step (s) in the pre-RL / post-RL setting; W2S-OPD adds only 20% over OPD.

Method	Training time per step (s)
OPD	868
W2S-OPD	1043

Training efficiency matters for practical deployment. To quantify the overhead W2S-OPD introduces, we record the average wall-clock time per training step in the pre-RL / post-RL setting and report it in Table 10. W2S-OPD incurs only a 20% increase in per-step time over OPD, despite forwarding 3 frozen models (the 8B anchor and the 4B contrast pair) instead of a single 4B teacher. The overhead stays modest because the per-step cost is dominated by on-policy rollout generation, which the two methods share, while the additional teacher-side scoring is forward-only. These scoring passes are also independent of one another and can be further parallelized across devices.

::: {caption="Table 11: Average math reasoning accuracy of W2S-OPD with base-model contrast pairs of different capability gaps. The student is Qwen3-8B."}

:::

D.2 Performance with Different Scales of Base Models

To investigate how the choice of base models affects the Smaller / Larger setting, we fix the Qwen3-8B student and the Qwen3-4B positive model, and vary the negative model between Qwen3-0.6B and Qwen3-1.7B. As shown in Table 11, both pairs improve the student, and the gain grows with the capability gap within the pair: the wider 4B-0.6B pair yields the larger improvement of 6.0 % on math reasoning, whereas the narrower 4B-1.7B pair yields 4.7. We attribute this to the amount of capability the subtraction isolates, where a larger gap spans a longer stretch of the direction along which capability grows with scale, so the difference extracts a stronger and more informative signal, and re-anchoring it onto the student base provides supervision that pushes the student further. These results suggest instantiating W2S-OPD with the most separated base-model pair available.

E. Example Generations

We provide qualitative examples shown in Figure 6–Figure 9. Figure 6 and Figure 7 come from the pre-RL / post-RL setting and compare W2S-OPD against OPD; Figure 8 and Figure 9 come from the smaller/larger setting and compare the Qwen3-8B student before and after W2S-OPD distillation. In each case the W2S-OPD student succeeds while the baseline fails, with the baseline's errors marked in red.

Math case study under Pre-RL/ Post-RL setting, with the errors of direct OPD marked in red.

Code case study under the pre-RL/post-RL setting with the errors of direct OPD marked in red.

Math case study from under the smaller/larger setting, with the errors of the Qwen3-8B student marked in red.

Code case study under the smaller/larger setting, with the errors of the Qwen3-8B student marked in red.

References

Section Summary: This section lists dozens of recent academic papers and technical reports focused on advancing large language models through techniques like on-policy distillation, reinforcement learning, and self-distillation to boost reasoning skills. Many entries examine specific models from families such as DeepSeek and Qwen, alongside methods for transferring knowledge from stronger to weaker systems and integrating capabilities across domains. The references also include benchmarks and datasets for evaluating performance on math competitions, code generation, and graduate-level question answering.

[1] Guo et al. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

[2] Zhihong Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. CoRR. abs/2402.03300. doi:10.48550/ARXIV.2402.03300. https://doi.org/10.48550/arXiv.2402.03300.

[3] Yu et al. (2024). Flow of reasoning: Training llms for divergent reasoning with minimal examples. arXiv preprint arXiv:2406.05673.

[4] Gu et al. (2024). Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations. pp. 32694–32717.

[5] Xiao et al. (2026). Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780.

[6] Agarwal et al. (2024). On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations. pp. 21246–21263.

[7] Kevin Lu and Thinking Machines Lab (2025). On-Policy Distillation. Thinking Machines Lab: Connectionism. doi:10.64434/tml.20251026.

[8] Yang et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.

[9] Ma et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv preprint arXiv:2606.30406.

[10] Xu et al. (2026). Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348.

[11] Burns et al. (2023). Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390.

[12] Yang et al. (2024). Weak-to-strong reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 8350–8367.

[13] Ko et al. (2026). Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137.

[14] Li et al. (2026). Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016.

[15] Liu et al. (2021). DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 6691–6706.

[16] Liu et al. (2024). Tuning language models by proxy. arXiv preprint arXiv:2401.08565.

[17] Pan et al. (2026). RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation. arXiv preprint arXiv:2606.11709.

[18] AI-MO (2024). AIMO Validation AIME. https://huggingface.co/datasets/AI-MO/aimo-validation-aime. 90 problems from AIME 2022–2024, extracted from the AoPS wiki.

[19] Zhang, Yifan and Math-AI, Team (2025). American Invitational Mathematics Examination (AIME) 2025.

[20] Balunovic et al. (2025). Matharena: Evaluating llms on uncontaminated math competitions, February 2025. URL https://matharena. ai. 8.

[21] Liu et al. (2023). Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems. 36. pp. 21558–21572.

[22] Jain et al. (2025). Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations. pp. 58791–58831.

[23] Rein et al. (2023). Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022.

[24] Pyatkin et al. (2026). Generalizing verifiable instruction following. Advances in Neural Information Processing Systems. 38.

[25] He et al. (2025). Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456.

[26] Cui et al. (2025). Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456.

[27] Schoenfeld, Alan H (2014). Mathematical problem solving. Elsevier.

[28] Li et al. (2026). Schoenfeld’s anatomy of mathematical reasoning by language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 32773–32802.

[29] Song, Mingyang and Zheng, Mao (2026). A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626.

[30] Zeng et al. (2026). Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763.

[31] Chen et al. (2026). Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation. arXiv preprint arXiv:2605.27115.

[32] Yang et al. (2026). OPRD: On-Policy Representation Distillation. arXiv preprint arXiv:2606.06021.

[33] Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. arXiv preprint arXiv:2601.18734.

[34] Shenfeld et al. (2026). Self-Distillation Enables Continual Learning. arXiv preprint arXiv:2601.19897.

[35] Hübotter et al. (2026). Reinforcement Learning via Self-Distillation. arXiv preprint arXiv:2601.20802.

[36] Ye et al. (2026). On-policy context distillation for language models. arXiv preprint arXiv:2602.12275.

[37] Jin et al. (2026). Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079.

[38] Xing et al. (2026). Trust Region On-Policy Distillation. arXiv preprint arXiv:2606.01249.

[39] Jang et al. (2026). Stable on-policy distillation through adaptive target reformulation. In Findings of the Association for Computational Linguistics: ACL 2026. pp. 42217–42227.

[40] Li et al. (2026). Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation. arXiv preprint arXiv:2602.02994.

[41] Liu et al. (2026). Visual-Advantage On-Policy Distillation for Vision-Language Models. arXiv preprint arXiv:2605.21924.

[42] Yoon et al. (2026). Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding. arXiv preprint arXiv:2606.00564.

[43] Feng et al. (2026). Weak-to-Strong Generalization via Direct On-Policy Distillation. arXiv preprint arXiv:2607.05394.

[44] Christiano et al. (2018). Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575.

[45] Yao et al. (2025). Revisiting weak-to-strong generalization in theory and practice: Reverse KL vs. forward KL. In Findings of the Association for Computational Linguistics: ACL 2025. pp. 2860–2888.

[46] Yao et al. (2025). On weak-to-strong generalization and f-divergence. arXiv preprint arXiv:2506.03109.

[47] Yuan et al. (2026). Incentivizing strong reasoning from weak supervision. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7138–7156.

[48] Zhao et al. (2024). Weak-to-strong jailbreaking on large language models. arXiv preprint arXiv:2401.17256.

[49] Ning et al. (2024). Can llms learn by teaching for better reasoning? a preliminary study. Advances in Neural Information Processing Systems. 37. pp. 71188–71239.

[50] Cetin et al. (2026). Reinforcement learning teachers of test time scaling. Advances in Neural Information Processing Systems. 38. pp. 107533–107567.

[51] Prabhumoye et al. (2020). Exploring controllable text generation techniques. In Proceedings of the 28th International Conference on Computational Linguistics. pp. 1–14.

[52] Keskar et al. (2019). Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.

[53] Chan et al. (2020). Cocon: A self-supervised approach for controlled text generation. arXiv preprint arXiv:2006.03535.

[54] Krause et al. (2021). Gedi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 4929–4952.

[55] Qin et al. (2022). Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems. 35. pp. 9538–9551.

[56] Zhu et al. (2026). Hybrid policy distillation for llms. arXiv preprint arXiv:2604.20244.