DISTILLM: Towards Streamlined Distillation for Large Language Models

Jongwoo Ko $^{1}$
Sungnyun Kim $^{1}$
Tianyi Chen $^{2}$
Se-Young Yun $^{1}$

https://github.com/jongwooko/distillm

$^{1}$ KAIST AI, Seoul, Republic of Korea
$^{2}$ Microsoft, Redmond, Washington, USA

Correspondence to: Se-Young Yun [email protected].

Abstract

Knowledge distillation (KD) is widely used for compressing a teacher model to a smaller student model, reducing its inference cost and memory footprint while preserving model capabilities. However, current KD methods for auto-regressive sequence models (e.g., large language models) suffer from missing a standardized objective function. Moreover, the recent use of student-generated outputs to address training-inference mismatches has significantly escalated computational costs. To tackle these issues, we introduce DISTILLM, a more effective and efficient KD framework for auto-regressive language models. DISTILLM comprises two components: (1) a novel skew Kullback-Leibler divergence loss, where we unveil and leverage its theoretical properties, and (2) an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs. Extensive experiments, including instruction-following tasks, demonstrate the effectiveness of DISTILLM in building high-performing student models while achieving up to 4.3 $\times$ speedup compared to recent KD methods.

Executive Summary: DISTILLM introduces a streamlined knowledge-distillation framework for auto-regressive language models. Large models deliver strong generative performance yet incur high inference cost and memory use. Distillation can transfer capability to smaller students, but current methods rely on asymmetric Kullback-Leibler divergence (KLD) that produces overly smooth or collapsed student distributions and on fixed training sets that create exposure bias during inference. Recent fixes that add student-generated outputs (SGOs) raise training time dramatically and still require task-specific loss selection.

The work therefore set out to supply a single, theoretically grounded objective together with an efficient way to use SGOs. The authors derive a skew KLD (SKL or SRKL) whose gradient remains bounded and whose mini-batch estimator converges faster than standard KLD or Jensen-Shannon divergence. They pair it with an adaptive off-policy scheduler that decides when to draw from a replay buffer of past SGOs, keeping generation cost low while still correcting the train-inference mismatch.

Across instruction-following, summarization, and translation benchmarks the new losses alone raise ROUGE-L and GPT-4 scores over KLD, reverse KLD, and JSD baselines. Adding the adaptive scheduler yields further gains and reduces total training time by factors of 2.5–4.3 relative to on-policy or mixed-SGO methods. The same recipe works for GPT-2, OPT, LLaMA-scale, and T5 families and does not require an already fine-tuned student.

These results matter because they cut both the cost of producing a deployable student and the cost of running it, without sacrificing output quality. Organizations can therefore compress frontier models for production use on modest hardware while preserving most of the original capability.

The framework remains limited to KLD-based objectives, supervised fine-tuning settings, and teacher-student pairs that share a tokenizer. Future work could combine skew divergence with total-variation or Earth-mover losses, extend the approach to preference optimization, and handle mismatched tokenizers. Within these bounds the reported gains are consistent across random seeds and model sizes, giving senior decision makers a clear route to faster, cheaper student models today.

1. Introduction

Section Summary: Recent advances in large language models have produced strong text-generation capabilities, but their size drives up memory use and inference costs, creating a need to shrink them while retaining performance. Traditional knowledge distillation methods that transfer knowledge from a large teacher model to a smaller student often rely on fixed datasets and standard divergence losses, which can produce overly smooth or overly peaked outputs and cause mismatches between training and real generation. To overcome these drawbacks, the paper presents DistiLLM, which combines a skew Kullback-Leibler divergence loss with an adaptive off-policy strategy that uses student-generated examples more efficiently, delivering state-of-the-art student-model results at several times the training speed of prior approaches.

Recent advancements in auto-regressive language models (LMs, [1, 2]) such as large language models (LLMs) have significantly improved the quality of text generation in a variety of generative tasks such as task-agnostic instruction-following tasks [3]. This success is often attributed to the increased scale of training data and model parameters (e.g., 175B parameters for GPT-3; [4]). However, expanding the parameter count brings associated costs, limiting the deployment of these models due to either high inference costs or large memory footprints. Therefore, a crucial objective for the practical use of these high-capacity models is to compress them by reducing the number of parameters while preserving their performance to the greatest extent possible.

As the necessity of reducing the demands on computational resources becomes important, KD [5] emerges as a promising method. It involves the transfer of knowledge from a teacher model with large capacity (i.e., LLMs) to a smaller student LM. Most approaches of KD have employed the Kullback-Leibler divergence (KLD) loss, enforcing the student model to generate outputs that mirror the teacher model's outputs on a fixed dataset [6, 7]. Such methods of KD have significantly enhanced the performance of student models, making them competitive with teacher models while increasing efficiency, especially for various classification tasks [8, 9].

These approaches using KLD on fixed datasets, however, have two primary shortcomings in applying to auto-regressive LMs. First, using the KLD can lead to sub-optimal results. Given the complexity of generative tasks compared to classification tasks, this can result in the student distribution becoming overly smooth and consequently failing to fully capture the teacher distribution, or becoming overly concentrated in high-probability distributions. This issue—referred to as mode averaging or mode collapse—arises due to the asymmetric nature of the KLD [10, 11]. Second, the use of a fixed dataset in the training phase can cause a distribution mismatch between the sequences observed during training and those generated by the student in auto-regressive inference, leading to exposure bias problems [12].

Recent studies have explored various divergence losses [10, 13] or the incorporation of student-generated output (SGO, [14, 13]) to address the existing problems. However, these methods often lack standardized objective functions and are less efficient due to the continuous SGO generation. For instance, [13] employed on-policy distillation with SGOs, but it faces low sample efficiency and high generation time as it constantly prompts the student to produce new training sequences. Also, their experiments indicated the optimal divergence seems to be task-dependent, which requires additional efforts to inconveniently select a proper loss function. [11] introduced a policy optimization method to minimize reverse KLD between student and teacher distributions, yet this also compromises training efficiency by requiring generation from both models in every iteration.

Contributions.

In this paper, we introduce $\textsc{DistiLLM}$[^1], featuring a novel skew KLD loss and an adaptive off-policy approach, focusing on both distillation effectiveness and training efficiency. We provide both theoretical and empirical evidence that the components of $\textsc{DistiLLM}$ work well individually and synergistically with each other. Our detailed contributions include:

[^1]: $\textsc{DistiLLM}$ is pronounced as distill-LLM, merging the word "distill" with "LLM".

• Skew KLD: We focus on the two key issues of existing objective functions for auto-regressive LMs: instability from potential gradient explosions in optimizing the KLD loss, and lack of emphasis on generalizability and convergence. To address these limitations, we introduce skew KLD, a new objective function with a strong theoretical foundation, optimized for stable gradients and minimal approximation errors, empirically leading to faster convergence and superior performance.
• Adaptive off-policy approach: While using SGOs in KD is generally effective in improving performance, this approach significantly increases training time (Figure 2) and makes it challenging to find the optimal proportion for using SGOs. To this end, we propose an adaptive off-policy approach module for adaptively and efficiently leveraging SGOs to consider the data perspective of KD.
• Advanced performance and efficiency: $\textsc{DistiLLM}$ accomplishes state-of-the-art performances for the student LMs on various generative tasks (e.g., instruction-following or text summarization), while achieving the 2.5 $\sim$ 4.3 $\times$ training speedup compared to recent KD techniques [11, 13].

2. Background

Section Summary: Knowledge distillation trains a smaller student language model to match the output probabilities of a larger fixed teacher model on text sequences, most often by minimizing Kullback-Leibler divergence between their token-by-token predictions. This standard objective tends to force the student to cover every possibility the teacher assigns any probability to, producing an overly smoothed distribution when the student lacks sufficient capacity. Recent fixes such as reverse KL or self-generated student outputs reduce some mismatches but introduce new problems of misguidance and high computational cost, leaving performance sensitive to the choice of divergence and data source.

2.1 KD for Auto-regressive Generative LMs

We provide preliminary information on the KD for auto-regressive generative LMs. Given a source and target sequence pair, denoted as $(\mathbf{x}, \mathbf{y})$, KD minimizes divergence $D$ between the distributions of a fixed teacher model $p(\mathbf{y} | \mathbf{x})$ and a parameterized student model $q_{\theta}(\mathbf{y} | \mathbf{x})$. The training data pairs $(\mathbf{x}, \mathbf{y})$ are either sampled from a fixed ground-truth dataset [5] or from teacher-generated outputs [6].

Conventionally, KLD, denoted as ${D}_{\text{KL}}$, is the most widely used loss in KD due to its simplicity and tractability. The sequence-level distillation using the KLD is accurately decomposed into a sum of token-wise distillation:

$ \begin{aligned} \textstyle &{D}{\text{KL}}(p, q\theta) = \mathbb{E}{\mathbf{x}} \mathbb{E}{\mathbf{y} \sim p(\cdot|\mathbf{x})} \left[\textstyle \log \frac{p(\mathbf{y}|\mathbf{x})}{q_{\theta}(\mathbf{y}|\mathbf{x})} \right] \quad\text{(a)} \ &\approx \textstyle \frac{1}{|\mathcal{D}|} \sum_{(\mathbf{x}, \mathbf{y}) \in \mathcal{D}} p(\mathbf{y}|\mathbf{x}) \log \frac{p(\mathbf{y}|\mathbf{x})}{q_{\theta}(\mathbf{y}|\mathbf{x})} \quad\text{(b)} \ &= \textstyle \frac{1}{|\mathcal{D}|} \sum_{\mathbf{x}\in\mathcal{D}{X}}\sum{t}^{|\mathbf{y}|} \sum_{y_{t} \in V} p(y_{t}|\mathbf{y}{<t}, \mathbf{x}) \log \frac{p(y{t}|\mathbf{y}{<t}, \mathbf{x})}{q{\theta}(y_{t}|\mathbf{y}_{<t}, \mathbf{x})} \quad\text{(c)} \end{aligned}\tag{1} $

where $V$ is the vocabulary token set and $\mathbf{y}{<t} \coloneqq (y{1}, y_{2}, \ldots, y_{t-1})$ represents the sequence of tokens up to index $t-1$. We focus solely on tractable KLD, as other divergences like total variation distance (TVD, [10]) do not effectively decompose sequence-level distillation into token-level components. While the explicit definition of KLD is given in Eq. 1a [6, 10], most recent studies, such as [13] and [11], approximate the distribution matching by minimizing Eq. 1b, under the assumption that the teacher's distribution is similar to its training dataset $\mathcal{D}$. For the sake of training efficiency, our method utilizes the definition provided in Eq. 1b, while Eq. 1a is used for theoretical analysis of our proposed distillation objective in Thm. Theorem 1.

2.2 Pitfalls of Existing Distillation

Limitation of objective functions.

The KLD objective in KD, primarily due to its asymmetric nature [10], often forces the student distribution to cover the entire support set of the teacher distribution, leading to significant limitations. This becomes evident when a sampled data point is included in the teacher distribution's support but falls outside the student distribution, i.e., $\exists(\mathbf{x}, \mathbf{y})$ such that $p(\mathbf{y} | \mathbf{x})\gg0$ and $q_\theta(\mathbf{y} | \mathbf{x}) \approx 0$. The limitation becomes pronounced if the student model lacks the capacity to match all support sets of the teacher distribution accurately. Consequently, this results in the student model exhibiting a mode-averaging problem, where it learns an overly smooth distribution in an attempt to cover the teacher's entire support set, as highlighted by recent studies [10, 11].

Such recent studies have partially addressed this issue by applying the reverse KLD (RKLD, [11, 13]), defined as ${D}{\text{RKL}}(p, q\theta):= {D}{\text{KL}}(q\theta, p)$, or generalized JSD [13] by introducing an interpolation parameter $\beta \in [0, 1]$, defined as

$ \begin{split} D_{\text{JSD}}^{(\beta)}(p, q_{\theta}) &:= \beta, D_{\text{KL}}(p, \beta p + (1-\beta) q_{\theta}) \ &+ (1-\beta), D_{\text{KL}}(q_{\theta}, \beta p + (1-\beta) q_{\theta}). \end{split}\tag{2} $

These approaches have shown empirical success in auto-regressive LMs, but there is a need for systematic study to provide a standard distillation objective grounded in comprehensive theoretical and experimental analyses. The lack of such backing for these recently proposed objective functions leads to sub-optimal performance and task-dependent variability [13].

Limitations of utilizing SGO.

Previous KD methods for auto-regressive LMs have encountered a training-inference mismatch between the samples from fixed datasets that are used during training and those produced by the student model during inference. Recent studies [14, 13] explored addressing this challenge by prompting the student model to generate SGOs and then training from the feedback of the teacher model on these sequences. This approach addresses the mismatch by training the student model on its familiar, self-generated sequences. These efforts have significantly improved the performance of distilling LLMs [14, 13].

Despite its effectiveness, we identify two main issues with the current utilization of SGO. First, teacher models may experience a distribution mismatch between their training data and unfamiliar or inaccurate SGOs, potentially leading to misguidance on $q_{\theta}$. As depicted in Fig. Figure 1, such a mismatch can result in the teacher model assigning low validation loss to incorrect but shorter generations and high validation loss to longer but correct ones. Second, as shown in Figure 2, generating SGOs for every iteration proves computationally inefficient. Across all experiments, irrespective of the maximum sequence length (ranging from 64 to 256) of SGOs or the model size (from GPT-2 to OpenLLaMA2-3B), the SGO generation accounts for a considerable portion of the total training time, reaching up to 80%.

However, to the best of our knowledge, there has been limited comprehensive effort to address these challenges simultaneously. For instance, MiniLLM [11] suggests a method that mixes the distributions of teacher and student to alleviate the first challenge. However, this method notably increases training computation due to the requirement of a large teacher model. These challenges motivate us to develop an approach that adaptively balances the positive effect of reduction of training-inference mismatch [13] and the negative effect of performance degradation from noisy feedback (as shown in Table 2). Meanwhile, we also aim to improve the sample efficiency of SGO, thereby enhancing computational efficiency.

Input: initial prob. $\phi$, student $q_{\theta_0}$ with parameters $\theta_0$, teacher $p$, total training iterations $T$, training & validation dataset $\mathcal{D}$, $\mathcal{D}_{val}$, empty replay buffer $\mathcal{D}_{R}$
Output: Student model $q_{\theta_T}$ with trained parameters $\theta_T$
while $t \leq T$ do
  Randomly sample $u \sim \text{Unif}(0, 1)$
  ***/ Linearly Decreasing Replay Ratio /***
  if $u < \lambda_{R} := \phi (1 - \frac{t}T)$ then
    */* Generate SGO & Update $\mathcal{D}_{R}$ */*
    Generate SGO $\{\tilde{\mathbf{y}}_{i}\}_{i=1}^{B}$ from $\{q_{\theta_t}(\cdot|\mathbf{x}_{i})\}_{i=1}^{B}$
    Store SGO into $\mathcal{D}_{R}$; $\mathcal{D}_{R} \leftarrow \mathcal{D}_{R} \cup \{(\mathbf{x}_{i}, \tilde{\mathbf{y}}_{i})\}_{i=1}^{B}$
  end if
  if $u < \phi$ then
    ***/* Use SGO in Off-policy Approach (Fig. Figure 4(c))***
    Sample mini-batch $\{(\mathbf{x}_{i}, \tilde{\mathbf{y}}_{i})\}_{i=1}^{B}$ from $\mathcal{D}_{R}$
  else
    ***/ Use Sample from Fixed Dataset (Fig. Figure 4(a)) /***
    Sample mini-batch $\{(\mathbf{x}_{i}, \mathbf{y}_{i})\}_{i=1}^{B}$ from $\mathcal{D}$
  end if
  ***/ Use S(R)KL /***
  Update $\theta_t$ by S(R)KL $D_{\text{SKL}}^{(\alpha)}(\cdot, \cdot)$
  if do validation then
    $\mathcal{L}_{prev}, \phi \leftarrow$ `SGO(Scheduler)`($\mathcal{L}_{prev}$, $\mathcal{D}_{val}$, $q_{\theta_{t}}$)
  end if
end while
**/ Adaptive SGO Scheduler /**
def `SGO(Scheduler)`($\mathcal{L}_{\tilde{t}-1}$, $\mathcal{D}_{val}$, $q_{\theta}$):
**/ Compute Loss for Validation Set /**
$\mathcal{L}_{\tilde{t}} \leftarrow \frac{1}{|D_{val}|} \sum_{\mathbf{x}_{val}, \mathbf{y}_{val}}$ `Loss`($q_\theta, \mathbf{x}_{val}, \mathbf{y}_{val}$)
if $\mathcal{L}_{\tilde{t}} > \mathcal{L}_{\tilde{t}-1} + \varepsilon$ then
Update $\phi_{\tilde{t}} \leftarrow \min(\phi_{\tilde{t}-1} + 1/N_{\phi}, 1.0)$
else
$\mathcal{L}_{\tilde{t}}, \phi_{\tilde{t}} \leftarrow \mathcal{L}_{\tilde{t}-1}, \phi_{\tilde{t}-1}$
end if
return $\mathcal{L}_{\tilde{t}}$, $\phi_{\tilde{t}}$

3. DistiLLM

Section Summary: DistiLLM is a new knowledge distillation technique that improves how a smaller student language model learns from a larger teacher by replacing standard loss functions with a skew Kullback-Leibler divergence. This modified loss mixes the two models' output distributions in a controlled ratio, which produces steadier gradients and smaller estimation errors so training stays stable and the student generalizes better. The method further pairs this loss with an adaptive off-policy sampling scheme that uses a replay buffer and a gradually shrinking on-policy ratio, delivering both higher efficiency and stronger final performance.

In this section, we present the technical details of $\textsc{DistiLLM}$, addressing the limitations of previous methods. Our proposed method includes: (1) Skew KLD (Sec. Section 3.1), which significantly improves optimization stability and generalizability. The skew KLD loss addresses the pitfalls of previous objective functions that may lead the student model to sub-optimal, lacking analytical grounding. (2) Adaptive off-policy approach (Sec. Section 3.2), which comprises a novel adaptive SGO scheduler to balance the trade-off between noisy feedback and training-inference mismatch by minimally utilizing SGO, and off-policy strategy to improve the sample efficiency of SGO with maintaining the performance. We present the overall pipeline of $\textsc{DistiLLM}$ in Algorithm 1.

3.1 Skew (Reverse) KLD

We mathematically present our motivation that skewing such KLD is highly effective in improving the performance of student models with a more favorable optimization process. The definition of skew KLD (SKL, [15]) employs the parameter $\alpha$ that controls the mixing ratio of two distributions. The $\alpha$-SKL between $p$ and $q_{\theta}$ is defined as the KLD between $p$ and the mixture of distributions $\alpha p + (1-\alpha) q_{\theta}$:

$ D_{\text{SKL}}^{(\alpha)}(p, q_{\theta}) = D_{\text{KL}} \left(p, \alpha p + (1-\alpha) q_{\theta}\right). $

We similarly define the $\alpha$-SRKL by $D_{\text{SRKL}}^{(\alpha)}(p, q_{\theta}) = D_{\text{KL}}(q_\theta, (1-\alpha) p + \alpha q_{\theta})$. Here, following our thorough analysis, we present a comprehensive insight suggesting that S(R)KL is superior to other loss functions, owing to its more stable gradient and smaller approximation error.

Stable gradient.

To provide stable optimization of SKL, we first analyze the gradients of KLD and SKL to parameter $\theta$. Given a context-target sequence pair $(\mathbf{x}, \mathbf{y})$, we define the gradient of KLD w.r.t. $\theta$ [16]:

$ \nabla_{\theta} D_{\text{KL}}(p, q_{\theta}) = - \mathbf{r}{p, q{\theta}} \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}),\tag{3} $

where $\mathbf{r}{p{1}, p_{2}}$ is the ratio between arbitrary distribution $p_{1}$ and $p_{2}$. The result is the model probability's negative gradient, weighted inversely by its value. If $q_{\theta}(\mathbf{y}|\mathbf{x}) \approx 0$, the gradient norm grows large, causing a significant, potentially noisy parameter update step. These ingredients can adversely affect gradient updates, impacting the optimization process.

We now compute the gradient of SKL w.r.t. $\theta$:

$ \nabla_{\theta} D_{\text{SKL}}^{(\alpha)}(p, q_{\theta}) = - \underbrace{(1-\alpha) \mathbf{r}{p, \tilde{q}{\theta}}}{\text{coefficient}} \nabla{\theta} q_{\theta}(\mathbf{y}|\mathbf{x}),\tag{4} $

where $\tilde{q}{\theta}(\mathbf{y}|\mathbf{x}) = \alpha p(\mathbf{y}|\mathbf{x}) + (1-\alpha) q{\theta}(\mathbf{y}|\mathbf{x})$. SKL offers a reduced gradient norm compared to KLD, due to $p$ and $q_{\theta}$ interpolation preventing the denominator of $\mathbf{r}{p, \tilde{q}{\theta}}$ from reaching zero.

This results in a more stable gradient for SKL. The gradient analysis for RKLD and SRKL reveals similar trends.

$ \begin{aligned} \nabla_{\theta} D_{\text{KL}}(q_{\theta}, p) &= - \left(\log \mathbf{r}{q{\theta}, p} + 1 \right) \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}), \ \nabla_{\theta} D_{\text{SKL}}^{(\alpha)}(q_{\theta}, p) &= -\underbrace{\left(\log \mathbf{r}{q{\theta}, \tilde{p}} + 1 - \alpha \mathbf{r}{q{\theta}, \tilde{p}} \right)}{\text{coefficient}} \nabla{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}), \nonumber \end{aligned}\tag{5} $

where $\tilde{p}(\mathbf{y}|\mathbf{x}) = (1-\alpha) p(\mathbf{y}|\mathbf{x}) + \alpha q_{\theta}(\mathbf{y}|\mathbf{x})$. All derivations for the gradients are in Appendix B.1. We visualize the gradient coefficient distribution in Fig. Figure 3(a) and Fig. Figure 3(b) which verify our analysis for gradient. As $\alpha$ becomes large, the coefficient is effectively small for both SKL and SRKL.

Small approximation error.

We show that the empirical estimator of SKL from mini-batch training has a bounded L2 norm. This bounded norm ensures that rapid convergence, with minimal error between the estimator and true divergence, yields high generalizability by accurately reflecting the full distribution from the empirical estimator.

########## {caption="Theorem 1"}

Let $p^{1}{n}$ and $p^{2}{n}$ be empirical distributions of $n$ i.i.d. samples from $p^{1}$ and $p^{2}$, respectively. Under mild assumptions, we have an upper bound for the L2 norm of $\alpha$-SKL estimator $D_{\text{SKL}}^{(\alpha)}(p^{1}_n, p^{2}n)$ for $D{\text{SKL}}^{(\alpha)}(p^{1}, p^{2})$:

$ \begin{aligned} \mathbb{E}[|D_{\text{SKL}}^{(\alpha)}&(p^{1}n, p^{2}n) - D{\text{SKL}}^{(\alpha)}(p^{1}, p^{2})|^{2}] \ &\leq \frac{c{1}(\alpha)}{n^{2}} + \frac{c_{2} \log^{2}(\alpha n)}{n} + \frac{c_3 \log^{2}(c_4 n)}{\alpha^{2}n}, \end{aligned} $

for $c_{1}(\alpha)=\min\left{\frac{1}{\alpha^{2}}, \frac{\chi^{2}(p^{1}, p^{2})^{2}}{(1-\alpha)^{2}}\right}$ and positive constants $c_{2}, c_{3}, c_{4}$ that are independent of $n$, $\alpha$, and $D_{\text{KL}}(p^{1}, p^{2})$, where $\chi^{2}(p^{1}, p^{2})$ is chi-square divergence between $p^{1}$, $p^{2}$.

The proof is in Appendix B.2. Thm. Theorem 1 states that a large $\alpha$ lowers the L2 norm between empirical and true objectives. We show in Fig. Figure 3(c) that $\alpha$-S(R)KL reduces the error between the value for each mini-batch and their moving average more effectively than (R)KLD. However, considering the gradient scale reduction in Eq. 4–Equation 5, the benefit of a reduced L2 norm from Theorem 1 is negated by compensating the reduced gradient scale of modern optimizers [17]. We further provide a statement considering the reverse of approximated gradient coefficient, $\frac{1}{(1-\alpha)}$, especially when $\mathbf{r}_{\cdot, \cdot}$ averages near 1:

########## {caption="Remark"}

By considering the reverse of approximated gradient scale, we have:

$ \begin{aligned} \textstyle \mathbb{E}[|\frac{1}{(1-\alpha)}&(D_{\text{SKL}}^{(\alpha)}(p^{1}n, p^{2}n) - D{\text{SKL}}^{(\alpha)}(p^{1}, p^{2}))|^{2}] \ &\leq \frac{c^{*}{1}(\alpha)}{n^{2}} + \frac{c_{2} \log^{2}(\alpha n)}{(1-\alpha)^{2}n} + \frac{c_3 \log^{2}(c_4 n)}{\alpha^{2}(1-\alpha)^{2} n}, \end{aligned} $

for $c_{1}^{*}(\alpha) = \min \left{\frac{1}{\alpha^{2}(1-\alpha)^{2}}, \frac{\chi^{2}(p^{1}, p^{2})^{2}}{(1-\alpha)^{4}}\right}$.

Overall, selecting $\alpha$ involves a trade-off: the relationship between the upper bound of the normalized L2 norm and $\alpha \in [0, 1]$ appears to be convex, underscoring the importance of balancing gradient and L2 norm scales, as shown in Fig. Figure 3(d). From these results, we also discern a fundamental difference between S(R)KL and JSD: S(R)KL with a mild $\alpha$ achieves an appropriate L2 norm value, whereas $D_{\text{JSD}}^{(\beta)}(p, q_\theta) := \beta D_{\text{SKL}}^{(\beta)}(p, q_\theta) + (1-\beta) D_{\text{SKL}}^{(1-\beta)}(q_\theta, p)$ cannot simultaneously moderate skew values for both terms. Our experiments indicate that $\alpha$-SKL and $\alpha$-SRKL are most effective with $\alpha=0.1$, surpassing KLD, RKLD, and JSD in performance, as demonstrated in Tab. Table 1 and Fig. Figure 8.

3.2 Adaptive Off-policy Approach

In Sec. Section 2.2, we have discussed two main issues of naïvely using SGO: (1) the risk of noisy feedback due to the teacher model's unfamiliarity with the SGO and (2) the significant increase in training time. For instance, employing SGO at every training iteration (on-policy, [13]) can lead to a substantial increase in runtime, up to 5.5 $\times$ (refer to Fig. Figure 7), and may also result in performance degradation (refer to Fig. Figure 5). To tackle these issues, we propose (1) an adaptive SGO scheduler to conservatively utilize SGO in KD, guided by the validation loss of student models, thus mitigating the risk of noisy feedback; and (2) an efficient off-policy strategy to improve the sample efficiency of SGO.

Adaptive SGO scheduler.

We define the probability of using SGOs, denoted as $\phi$. We apply SGOs with a probability of $\phi$, i.e., using samples from a fixed dataset with a probability of $1-\phi$ (refer to Fig. Figure 4(a)). Unlike previous methods that maintain a consistently high $\phi$ [14, 13], our scheduler starts with low $\phi$ value, gradually increasing during training. This strategy prevents student models from being overwhelmed by noisy feedback (as shown in Fig. Figure 1). To manage the increase of $\phi$, we primarily rely on validation loss as a metric. Our observations indicate that training on a diverse range of SGOs, rather than solely on a fixed dataset, mitigates training-inference mismatch and consequently lowers validation loss. We adjust $\phi$ by comparing the current and previous validation losses; an increase in validation loss leads to an increase in $\phi$. This method effectively improves student model performance by striking a balance between managing noisy feedback and minimizing training-inference mismatch issues. For further details, please refer to Appendix C and Algorithm 1.

Off-policy approach for sample efficiency.

To enhance efficiency, we replace the recently adopted on-policy approach [11, 13] with an off-policy approach, employing a replay buffer [21, 22], as illustrated in Fig. Figure 4(c). In this buffer, we store SGOs from student models at a probability of $\lambda_{R}$, as indicated by the frequency of the dashed blue arrows. Subsequently, we randomly draw samples from this pool. We also replace the oldest samples of $\mathcal{D}_{R}$ with new ones once it reaches its maximum capacity. This off-policy strategy significantly improves the sample efficiency of KD with SGO, saving more resources than on-policy which constantly requires new data.

Off-policy reinforcement learning is prone to high bias error [23, 24], particularly when there is a significant divergence between past and current policies, using samples from the past policy becomes suboptimal for updating the current policy. To address this, we set $\lambda_{R} := \phi (1 - \frac{t}{T})$, where $t$ represents the current training iteration and $T$ is the total number of iterations. To explain the philosophy of our design, we define $\zeta := (1 - \frac{t}{T})$ as the replay ratio:

• In the early training phase (i.e., when adaptive probability $\phi$ is small), where student model parameters rapidly evolve, we focus on using current SGOs with a high replay ratio to minimize bias error.
• In the late stages of training (i.e., when $\phi$ is larger), as the student model nears convergence, we predominantly reuse stored SGOs from $\mathcal{D}_{R}$ with a small replay ratio.

Hence, we can consistently maintain a small $\lambda_{R}$ by applying a large $\zeta$ for relatively smaller $\phi$ values and vice versa, effectively balancing bias error reduction with sample efficiency. Our design of $\zeta$ shows higher efficiency and comparable performance with its alternatives as shown in Tab. Table 9.

Synergy with SKL.

Off-policy approach's success stems from the fast convergence speed of S(R)KL while other loss functions cannot be achieved. As Fig. Figure 6 shows, both SKL and SRKL have a significant early-stage improvement, effectively leveraging the off-policy approach without high bias issues. This advantage is also evident in Tab. Table 4, where, unlike other baselines [14, 13] that suffer performance drops when switching from on-policy to off-policy, our method maintains its efficacy. As a result, we verify that our off-policy approach significantly improves the training efficiency with a negligible performance drop as depicted in Tab. Table 2 and Fig. Figure 7.

4. Experiments

Section Summary: The experiments section evaluates DistiLLM on instruction-following, summarization, and translation tasks by applying the SRKL loss with an adaptive off-policy sampling strategy. It compares this approach against several prior knowledge distillation baselines, including standard fine-tuning, KL divergence methods, and policy-gradient techniques, using fixed datasets or teacher-generated outputs. Results in accompanying tables and a figure demonstrate consistent gains in ROUGE-L scores across multiple benchmarks when using the proposed skew losses and replay buffer.

We evaluate $\textsc{DistiLLM}$ on instruction-following, text summarization, and machine translation tasks. We apply $\textsc{DistiLLM}$ with SRKL and the off-policy approach with initial probability as zero and replay buffer size of 1000 as SRKL with $\alpha$ of 0.1 as determined through our ablation studies in Sec. Section 5. We compare our approach with previous KD: (1) supervised fine-tuning (SFT) directly fine-tunes the student on fixed datasets; (2) KD [5] uses KLD on fixed datasets; (3) SeqKD [6] applies SFT to teacher-generated output; (4) ImitKD [14] employs KLD on SGO; (5) MiniLLM [11] utilizes a policy gradient approach on SGO; and (6) GKD [13] uses JSD on a mixture of SGOs and a fixed dataset. Further details on the experimental setup are found in Appendix D.

\begin{tabular}{l|c|c|c|c|c}
\toprule[0.1em]
  Loss Function & \multicolumn{1}{c|}{Dolly Eval\, ($\uparrow$)} & \multicolumn{1}{c|}{Self-Instruct\, ($\uparrow$)} & \multicolumn{1}{c|}{Vicuna Eval\, ($\uparrow$)} & \multicolumn{1}{c|}{Super-Natural\, ($\uparrow$)} & Unnatural\, ($\uparrow$) \\ \midrule
  KLD & 23.52 (0.22) & 11.23 (0.46) & 15.92 (0.41) & 20.68 (0.16) & 23.38 (0.13) \\
  RKLD & 23.82 (0.34) & 10.90 (0.58) & \underline{16.11 (0.46)} & 22.47 (0.21) & 23.03 (0.11) \\
  Generalized JSD & 24.34 (0.35) & 12.01 (0.54) & 15.21 (0.61) & 25.08 (0.36) & 27.54 (0.07) \\ \midrule
  SKL & \underline{24.80 (0.12)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{12.86 (0.34)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{16.20 (0.57)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{26.26 (0.41)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{28.06 (0.08)} \textbf{\textcolor{#02BF3D}{$\bullet$}} \\
  SRKL & \textbf{25.21 (0.27)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{12.98 (0.24)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & 15.77 (0.39) \textbf{\textcolor{#FF003F}{$\bullet$}} & \underline{25.83 (0.15)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{28.62 (0.10)} \textbf{\textcolor{#02BF3D}{$\bullet$}} \\
\bottomrule[0.1em]
\end{tabular}

\begin{tabular}{l|c|c|c|c|c}
\toprule[0.1em]
  Generation & \multicolumn{1}{c|}{Dolly Eval\, ($\uparrow$)} & \multicolumn{1}{c|}{Self-Instruct\, ($\uparrow$)} & \multicolumn{1}{c|}{Vicuna Eval\, ($\uparrow$)} & \multicolumn{1}{c|}{Super-Natural\, ($\uparrow$)} & Unnatural\, ($\uparrow$) \\ \midrule[0.1em]
  Skew KLD & 24.80 (0.12) & 12.86 (0.34) & 16.20 (0.57) & 26.26 (0.41) & 28.06 (0.08) \\ \midrule
  $\llcorner$ On-policy & 24.27 (0.46) \textbf{\textcolor{#FF003F}{$\bullet$}} & \underline{13.13 (0.44)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & 16.39 (0.21) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 25.87 (0.18) \textbf{\textcolor{#FF003F}{$\bullet$}} & 26.49 (0.09) \textbf{\textcolor{#FF003F}{$\bullet$}} \\
  $\llcorner$ Mixed & 25.27 (0.35) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 12.24 (0.69) \textbf{\textcolor{#FF003F}{$\bullet$}} & 17.19 (0.29) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 25.30 (0.33) \textbf{\textcolor{#FF003F}{$\bullet$}} & 26.51 (0.11) \textbf{\textcolor{#FF003F}{$\bullet$}} \\
  $\llcorner$ Adaptive (ours) & \textbf{25.90 (0.20)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{13.24 (0.30)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{17.59 (0.44)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{27.62 (0.05)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{28.30 (0.11)} \textbf{\textcolor{#02BF3D}{$\bullet$}} \\
  \;\;+ Off-policy (ours) & \underline{25.79 (0.28)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & 13.03 (0.29) \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{17.41 (0.15)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{27.32 (0.09)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{28.13 (0.21)} \textbf{\textcolor{#02BF3D}{$\bullet$}} \\ \midrule[0.1em]
  Skew RKLD & 25.21 (0.27) & 12.98 (0.24) & 15.77 (0.39) & 25.83 (0.15) & 28.62 (0.10) \\ \midrule
  $\llcorner$ On-policy & 26.04 (0.33) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 12.93 (0.54) \textbf{\textcolor{#FF003F}{$\bullet$}} & 17.45 (0.37) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 27.29 (0.12) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 28.72 (0.10) \textbf{\textcolor{#02BF3D}{$\bullet$}} \\
  $\llcorner$ Mixed & 26.01 (0.61) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 12.24 (0.69) \textbf{\textcolor{#FF003F}{$\bullet$}} & 17.19 (0.29) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 26.40 (0.34) \textbf{\textcolor{#02BF3D}{$\bullet$}} & 29.02 (0.14) \textbf{\textcolor{#02BF3D}{$\bullet$}} \\
  $\llcorner$ Adaptive (ours) & \textbf{26.37 (0.21)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{13.14 (0.37)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{18.32 (0.17)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{28.24 (0.22)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{30.11 (0.04)} \textbf{\textcolor{#02BF3D}{$\bullet$}} \\
  \;\;+ Off-policy (ours) & \underline{26.11 (0.68)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{13.14 (0.69)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \textbf{18.46 (0.53)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{27.51 (0.03)} \textbf{\textcolor{#02BF3D}{$\bullet$}} & \underline{29.35 (0.07)} \textbf{\textcolor{#02BF3D}{$\bullet$}} \\
\bottomrule[0.1em]
\end{tabular}

4.1 Task-Agnostic Instruction-Following

Implementation details.

Our implementation follows the experiment setup of [11]. We first construct the training data from databricks-dolly-15k [25], wherein we randomly select 14K samples for training and equally leave 500 samples for validation and testing, respectively. We evaluate the trained models on five instruction-following benchmarks: Dolly evaluation, Self-Instruct [3], Vicuna evaluation, Super-Natural Instructions [26], and Unnatural Instruction [27]. Similar to [28] and [11], we add a language modeling [29] loss to the OpenWebText [30] corpus for all experiments. Employing such an additional loss function on the pre-training corpus has been shown to effectively improve instruction-tuning performance, as demonstrated in the work of [11]. For evaluation, we adopt two metrics: ROUGE-L [31] and GPT-4 feedback [32].

Main results.

Fig. Figure 5 illustrates the instruction-following performances, demonstrating $\textsc{DistiLLM}$'s superiority to state-of-the-art methods, across diverse teacher-student combinations and evaluation metrics. Notably, GKD and MiniLLM, which recently incorporated SGO in distillation, are less effective or even lead to a performance decline in smaller-sized student models, particularly in GPT-4 feedback and some ROUGE-L scores. Further details and results can be found in Appendix E.1 and Tab. Table 11–Table 13.

For evaluating $\textsc{DistiLLM}$ on larger-sized LLMs, we utilize low-rank adaptation (LoRA; [33]) for training. Specifically, we employ OpenLLaMA2-7B [20] as the teacher model and OpenLLaMA2-3B as the student model. The results, as depicted in Fig. Figure 5(c), reveal that $\textsc{DistiLLM}$ significantly surpasses other baseline methods in performance. Notably, while other supervised KD techniques are less effective than MiniLLM in LLM applications, $\textsc{DistiLLM}$ uniquely achieves superior performance compared to MiniLLM. This outcome, particularly on task-agnostic instruction-following datasets, underscores $\textsc{DistiLLM}$'s effectiveness in general-purpose LLMs. Moreover, as we can observe in Fig. Figure 5(d), $\textsc{DistiLLM}$ requires only 1.6 $\times$ the training time of naïve KD, whereas other methods take 3 $\sim$ 7 $\times$. This demonstrates the efficiency of the proposed $\textsc{DistiLLM}$ and its broad applicability to expensive LLMs.

::: {caption="Table 3: Text summarization and machine translation results, evaluated with ROUGE-L and BLEU scores, respectively. Using the T5-XL (3B) teacher model, we fine-tune the student models, T5-Base (0.2B) and T5-Small (0.06B)."}

:::

4.2 Text Summarization and Machine Translation

We evaluate the effectiveness of task-specific LMs on summarization and translation tasks using two datasets, SAMSum [34] and IWSLT 2017 [35]. For the SAMSum dataset, we use T5-XL v1.1 [36] as the teacher model and T5-Base/-Small v1.1 as the student models. For the IWSLT dataset, we employed mT5-XL [37] as the teacher model and mT5-Base/-Small v1.1 as the student models.

Tab. Table 3 displays ROUGE-L and BLEU [38] scores for student models. We observe that $\textsc{DistiLLM}$ outperforms other baselines, but has a smaller performance margin in single-task scenarios than in general-purpose instruction-following tasks. In the SAMSum, students trained with ImitKD outperform those trained with GKD, while in the IWSLT, GKD outperforms ImitKD. These findings align with the results in [13], which indicate that the effectiveness of objective functions and the use of SGO are task-dependent. Despite these variations, $\textsc{DistiLLM}$ consistently shows superiority across different tasks, attributed to its adaptive use of the SGO scheduler and skew divergence. Additional details on the performance in the XSum [39] and CNN/DM [40] datasets are provided in Appendix E.2, further emphasizing the superior performance of $\textsc{DistiLLM}$.

5. Analysis and Discussion

Section Summary: The experiments demonstrate that the new skew-based loss functions outperform standard divergence measures by delivering faster training convergence and stronger generalization on instruction-following tasks. The adaptive off-policy sampling approach further improves results and efficiency, providing substantial speed gains over on-policy or mixed baselines while reducing the impact of noisy feedback. Additional tests confirm that these components work best together, achieve near-optimal performance without manual tuning, and allow effective distillation even when the student model starts from a pre-trained rather than fine-tuned state.

We conduct experimental analyses to verify the effectiveness of each component of $\textsc{DistiLLM}$, distilling GPT-2 XL $\rightarrow$ GPT-2 in instruction-following datasets.

5.1 Effect of Skew Divergence

In Table 1, we compare the performance of various models trained with different objective functions: conventional KLD, RKLD, and JSD with a $\beta$ of 0.9 [13], as well as our SKL and SRKL with a $\alpha$ of 0.1. The results show that our proposed objective functions generally outperform the others. Notably, as Figure 6 illustrates, both SKL and SRKL achieve remarkably high validation ROUGE-L scores during the entire training phase, consistently demonstrating the rapid convergence and strong generalization capabilities of our proposed loss functions. These empirical results verify our theoretical analysis in Sec. Section 3.1 and indicate that even our simple modification leads to significant performance enhancements.

5.2 Effect of Adaptive Off-policy Approach

In Tab. Table 2 and Figure 7, we confirm the effectiveness and efficiency of our adaptive off-policy approach by comparing it with an on-policy [14] and a mixed strategy [13]. In the mixed strategy, we use the on-policy approach with a probability of 0.5; otherwise, we sample from the fixed dataset, following the original work. The results indicate that our adaptive SGO scheduler effectively balances the trade-off between the risk of noisy feedback and training-inference mismatch. Notably, while the baselines suffer from performance degradation when applying SKL, our proposed adaptive strategy consistently improves the performance for all datasets. Moreover, the off-policy approach leads to a minimal performance drop while significantly improving computational efficiency. It achieves 2.2 $\times$ to 3.4 $\times$ faster training speed compared to the on-policy or mixed strategy.

::: {caption="Table 4: Application of our off-policy method to the existing KD methods. Off-policy significantly reduces the performance of ImitKD and GKD, as opposed to our proposed DistiLLM."}

:::

::: {caption="Table 5: Comparison of the performance from the adaptive SGO scheduler and its terminal probability (0.4 for SKL and 0.3 for SRKL) and the best performance from manually tuned probability and corresponding value (i.e., results in parenthesis)."}

:::

5.3 Additional Ablation Studies on DistiLLM

Skew values $\alpha$.

As we highlight the importance of proper choice $\alpha$ in Section 3.1, we empirically evaluate the performance of $\alpha$-SKL under the wide range of $\alpha$. Figure 8 illustrates that both SKL and SRKL achieve the best performance on the $\alpha$ value of 0.1. These results are highly consistent with the result in Figure 3, where both normalized L2 norms have the smallest values in $\alpha=0.1$ and prove the validity of our theoretical analyses in Section 3.1. We observed that SKL shows a mild performance reduction as $\alpha$ increases above 0.1, the performance reduction of the SRKL is comparably severe which is discussed in detail in Appendix E.4.

Combining off-policy with existing KD methods.

To verify the synergy between our proposed objectives and the off-policy approach, we replaced the on-policy approach in GKD and ImitKD with an off-policy method. Given that the JSD and KLD exhibit slower training speeds compared to our SRKL, as shown in Figure 6, we observe that the effectiveness of the off-policy approach is significantly lower than that of our proposed $\textsc{DistiLLM}$ as reported in Table 4. These results validate the substantial effectiveness of combining our two proposed components.

Appropriateness of adaptive probability.

Table 5 compares the performance of probability values from the adaptive SGO scheduler with the best performance among those obtained from manually defined probabilities. Our results show that, in most cases, the performance using our method exceeds that of the manually selected probabilities. Furthermore, the probability values derived from our adaptive SGO scheduler are close to the optimal manually defined probabilities, with the differences being no more than 0.2 for SKL and 0.1 for SRKL.

One-stage Distillation

::: {caption="Table 6: Comparison of the performance of GPT-2 student across different KD methods and initialization (with or without fine-tuning before KD)."}

:::

One significant issue with previous KD methods using SGO is their reliance on beginning with fine-tuned student models. Without fine-tuning, student models may produce degenerated SGOs, leading to substantial noisy feedback [11]. A key advantage of our proposed $\textsc{DistiLLM}$ is its rapid convergence speed and innovative adaptive SGO scheduler, which eliminates the need for such fine-tuned student models. Consequently, we evaluated the performance of students distilled from pre-trained parameters without any prior fine-tuning. Table 6 demonstrates $\textsc{DistiLLM}$'s relative robustness to the initial state of the student model (i.e., regardless of whether the student was fine-tuned before KD). These findings confirm that $\textsc{DistiLLM}$ can maintain efficiency with only a minor reduction in performance, a feat not matched by other methods.

6. Related Work

Section Summary: Researchers have applied knowledge distillation to shrink large language models by letting a powerful "teacher" model guide a smaller "student" one, either by generating labeled examples or explanations for fine-tuning or by directly aligning the student's output patterns with the teacher's. Earlier work has explored both using inaccessible models like ChatGPT to create training data and, when the teacher is available, testing different mathematical measures to reduce differences in how the two models generate text. The present paper builds on these ideas with a new method called DistiLLM that refines both the training objectives and the way data are used.

KD [5] effectively compresses neural networks, allowing smaller student models to match the performance of larger teacher models. Recently, KD has been extended to compressing auto-regressive LMs, such as GPT-3 [4], to address the challenges posed by the large scale of current LLMs [2, 41], making them more viable in compute-intensive frameworks. One popular direction of KD for auto-regressive LMs is to harness LLMs as supervising data generators where only the teacher predictions are accessible like ChatGPT [1] APIs. This line of research employed LLMs for guided annotations of unlabeled data [42, 43] or for imparting reasoning capabilities [44, 45], where the resulting generated data are used for fine-tuning smaller LMs.

Another noteworthy approach, when the teacher model is accessible, entails matching the student model's generation distribution with that of the teacher model through divergence loss functions. Recent studies [14, 10, 11, 13] have focused on finding the proper objectives or using datasets during the distillation for auto-regressive LMs. ImitKD [14] demonstrated the effectiveness of SGO in distillation, leading to [13] propose on-policy approach of SGO with diverse objectives like RKLD and JSD. [10] examined various f-divergences, including total variation distance and JSD, in auto-regressive LMs, while [11] proposed a policy gradient-based method addressing the high variance issues in RL-based methods. Building on this research, we introduce an effective, efficient KD method, $\textsc{DistiLLM}$, with comprehensive analysis in objective function and data utilization.

7. Conclusion

Section Summary: Researchers have introduced a new method called DistiLLM to improve how knowledge is transferred to large language models that generate text one word at a time. The approach combines a carefully designed loss function grounded in mathematical analysis with an adaptive training strategy that reduces misleading signals and reuses past examples more efficiently. Experiments across multiple text-generation tasks showed that this method trains faster and produces stronger results than previous techniques.

We have proposed $\textsc{DistiLLM}$ to address the challenges of KD frameworks for auto-regressive LMs. Our approach incorporates two key components: (1) SKL which is based on mathematically in-depth analyses and empirical evidence; (2) an adaptive off-policy approach that enhances the utility of SGO by reducing the noisy feedback introduced by SGO and improving the sample efficiency with a replay buffer. Through extensive experiments on various generation tasks, we have demonstrated the superior performance of $\textsc{DistiLLM}$, achieving significant training efficiency and performance improvement.

Impact Statement

Section Summary: The paper focuses on making knowledge distillation more effective for training smaller versions of large language models such as LLaMA-2, while also cutting down on the computing power required. This dual benefit could help researchers develop capable models without needing costly high-end hardware. The authors state that the work raises no direct ethical concerns but are open to further review if needed.

This paper aims to improve the efficiency and effectiveness of KD for auto-regressive LMs, such as open-source LLMs (e.g., LLaMA-2 [2] and Falcon [46]). A notable feature of our methodology is its dual efficacy in improving performance and computational efficiency. This approach is significant, particularly as it reduces reliance on high-end computational resources for training such small-sized LLMs. We believe that this work does not present any direct ethical concerns. Therefore, a detailed ethical discussion is not considered necessary at this stage. However, we recognize the importance of ethical considerations and are open to further discussion should our work undergo an ethics review.

Acknowledgements

Section Summary: This research was funded by two grants from a Korean government agency focused on information and communications technology. One grant supported an artificial intelligence graduate program at KAIST and covered 10 percent of the work, while the other supported development of autonomous AI agents with enhanced knowledge and covered the remaining 90 percent.

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), 10%) and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration, 90%).

$\textsc{DistiLLM}$: Towards Streamlined Distillation for Large Language Models Supplementary Material

\

Appendix

Section Summary: The appendix first outlines key limitations of DistiLLM, including its reliance on a Kullback-Leibler divergence objective that can still suffer from mode collapse or training mismatches, its design for supervised fine-tuning rather than preference-based optimization, and its assumption that teacher and student models share the same tokenizer. It then supplies detailed gradient derivations for standard and skew variants of KL divergence and reverse KL divergence, demonstrating mathematically and empirically how the skew operation smooths gradient norms and improves optimization stability. Additional discussion covers normalized gradient distributions under common optimizers like AdamW to further support these stability benefits.

A. Limitation

While $\textsc{DistiLLM}$ shows effectiveness in terms of computational efficiency of training and students' performances compared to recent baselines [11, 13], we acknowledge a couple of limitations:

• KLD-based Objective: Our focus is mainly on (R)KLD because of its tractability that allows decomposing sequence-level distillation into token-wise distillation (as shown in Eq. (1a b)-Equation (1a c)). However, (R)KLD also entails limitations previously identified in [16, 47]: mode averaging (or mode collapse) and train-inference mismatch. Although DistiLLM, especially S(R)KL, effectively mitigated these issues, additionally combining with objective functions based on TVD [16] or EMD [47] could further enhance performance. We expect that by linearly interpolating between SKL and EMD (or TVD), we can leverage the strengths of both: the rapid convergence of SKL during the early stages of training and the superior performance of EMD or TVD towards the late training compared to KLD.
• Supervised Fine-tuning Approach: $\textsc{DistiLLM}$ is is primarily designed for a supervised fine-tuning setup, which has recently shown effectiveness [48], Meanwhile, many contemporary chat LLMs utilize preference optimization [28, 49]; thus, an extension of $\textsc{DistiLLM}$ to accommodate human preference optimization setups might be considered as future work.
• Same Tokenizer between Teacher & Student: Our method focuses on the scenario where the teacher and student models share the same tokenizer, which is also a common setup for white-box KD methods. However, with recent techniques [50, 51] that facilitate the transfer of knowledge between models with different tokenizers, we can designate this for future work to be explored in conjunction with these new methods.

B. Further Discussion on Skew KLD

Here, we provide the derivation of our theoretical results described in Section 3.1 and further (empirical) discussion on our proposed SKL.

B.1 Details for Gradient Analysis

Derivation of Section 3.1.

We derive the sample-wise gradient of KLD, SKL, RKLD, and SRKL to support our argument that simple skew operation on KLD improves the stability of the optimization.

• Firstly, we compute the gradient of KLD for a single sample $(\mathbf{x}, \mathbf{y})$:

$ \begin{aligned} \nabla_{\theta} D_{\text{KL}} (p, q_{\theta}) &= -\nabla_{\theta} p(\mathbf{y}|\mathbf{x}) \log q_{\theta} (\mathbf{y}|\mathbf{x}) \ &= -\frac{p(\mathbf{y}|\mathbf{x})}{q_{\theta}(\mathbf{y}|\mathbf{x})} \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}). \end{aligned} $

• Secondly, we compute the gradient of SKL for $(\mathbf{x}, \mathbf{y})$:

$ \begin{aligned} \nabla_{\theta} &D_{\text{SKL}}^{(\alpha)} (p, q_{\theta}) \ &= - \nabla_{\theta} p(\mathbf{y}|\mathbf{x}) \log \left(\alpha p(\mathbf{y}|\mathbf{x}) + (1-\alpha) q_{\theta} (\mathbf{y}|\mathbf{x}) \right) \ &= - \nabla_{\theta} p(\mathbf{y}|\mathbf{x}) \log \tilde{q}{\theta} (\mathbf{y}|\mathbf{x}) \ &= - \frac{p(\mathbf{y}|\mathbf{x})}{\tilde{q}{\theta} (\mathbf{y}|\mathbf{x})} \nabla_{\theta} \tilde{q}{\theta} (\mathbf{y}|\mathbf{x}) \ &= - \frac{p(\mathbf{y}|\mathbf{x})}{\tilde{q}{\theta} (\mathbf{y}|\mathbf{x})} \cdot (1-\alpha) \cdot \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}). \end{aligned} $

• Thirdly, we compute the gradient of RKLD for $(\mathbf{x}, \mathbf{y})$:

$ \begin{aligned} \nabla_{\theta} &D_{\text{KL}} (q_{\theta}, p) \ &= \log p(\mathbf{y}|\mathbf{x}) \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}) - \nabla_{\theta}!\left(q_{\theta} (\mathbf{y}|\mathbf{x}) \log q_{\theta} (\mathbf{y}|\mathbf{x}) \right) \ &= \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}) \cdot (\log {p(\mathbf{y}|\mathbf{x})}{q(\mathbf{y}|\mathbf{x})} - 1). \end{aligned} $

• Lastly, with a definition of $\tilde{p}(\mathbf{y}|\mathbf{x}) = (1-\alpha) p(\mathbf{y}|\mathbf{x}) + \alpha q_{\theta}(\mathbf{y}|\mathbf{x})$, we compute the gradient of SRKL for $(\mathbf{x}, \mathbf{y})$:

$ \begin{aligned} &\nabla_{\theta} D_{\text{SKL}}^{(\alpha)} (q_{\theta}, p) \ &= \nabla_{\theta} (q_{\theta} (\mathbf{y}|\mathbf{x}) \log \tilde{p} (\mathbf{y}|\mathbf{x})) - \nabla_{\theta} (q_{\theta} (\mathbf{y}|\mathbf{x}) \log q_{\theta} (\mathbf{y}|\mathbf{x})) \ &= \log \tilde{p}(\mathbf{y}|\mathbf{x}) \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}) + \frac{q_{\theta} (\mathbf{y}|\mathbf{x})}{\tilde{p}(\mathbf{y}|\mathbf{x})} \nabla_{\theta} \tilde{p}(\mathbf{y}|\mathbf{x}) \ &- \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}) \log q_{\theta} (\mathbf{y}|\mathbf{x}) - \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x})\ &= -\left(\log \frac{q_{\theta}(\mathbf{y}|\mathbf{x})}{\tilde{p}(\mathbf{y}|\mathbf{x})} + 1 - \alpha \frac{q_{\theta}(\mathbf{y}|\mathbf{x})}{\tilde{p}(\mathbf{y}|\mathbf{x})} \right) \nabla_{\theta} q_{\theta} (\mathbf{y}|\mathbf{x}), \end{aligned} $

As we described in Section 3.1, we can prevent the undesired gradient norm explosion due to smoothed distributions $\tilde{q}_{\theta}(\mathbf{y}|\mathbf{x})$ and $\tilde{p}(\mathbf{y}|\mathbf{x})$ for SKL and SRKL, respectively.

Further empirical discussion.

Recent optimizers, such as Adam [52] and AdamW [17], adaptively adjust learning rates based on the gradient scale. Hence, it is also important to observe the distribution of the normalized gradient divided by their scale. In Figure 9, we additionally plot the normalized version of the gradient coefficient, obtained by dividing the corresponding median values of the coefficient. SKL shows consistently reduced variance as $\alpha$ increases not only for original values but also normalized values. However, SRKL shows the smallest variance at $\alpha$ of 0.1 for both original and normalized values, but when $\alpha$ is larger than 0.1, the variance becomes larger as $\alpha$ grows. These results are also related to those in Figure 8, where SKL shows higher robustness than SRKL across different $\alpha$ values. Still, our mathematical analysis is valid given that the gradient scale becomes smaller as $\alpha$ increases.

B.2 Proof of Theorem 1

Theorem 1 (Restated). *Assume $p^{1}$, $p^{2}$ be two probability distributions such that $\mathbb{V}_{p^{2}}[dp^{1}/dp^{2}] < \infty$ and $p^{1}$ is absolutely continuous with respect to $p^{2}$ (i.e., the Radon-Nikodym derivative $dp^{1}/dp^{2}$ exists). Then, for $\forall \alpha < 1/8$, the variance of $\alpha$-SKL estimator satisfies

$ \begin{aligned} \mathbb{E}[|D_{\text{SKL}}^{(\alpha)}&(p^{1}n, p^{2}n) - D{\text{SKL}}^{(\alpha)}(p^{1}, p^{2})|^{2}] \ &\leq \frac{c{1}(\alpha)}{n^{2}} + \frac{c_{2} \log^{2}(\alpha n)}{n} + \frac{c_3 \log^{2}(c_4 n)}{\alpha^{2}n}, \end{aligned} $

for $c_{1}(\alpha) = \min{\frac{1}{\alpha^2}, \frac{\chi^{2}(p^{1}, p^{2})^2}{(1-\alpha)^2}}$ and constants $c_{2}, c_{3}, c_{4} > 0$ that are independent of $n$, $\alpha$, and $D(p^{1}, p^{2})$, where $\chi^{2}(p^{1}, p^{2}) \coloneqq \mathbb{E}_{p^{2}} [(dp^{1}/dp^{2})^{2}]$.* Here, we denote $p^{1} \ll p^{2}$ as $p^{1}$ is absolutely continuous with respect to $p^{2}$.

In this section, we state and define the regularity assumptions to derive the asymptotic upper bounds for the variance of $\alpha$-skew KL divergence. Formally, the $f$-divergence of two distributions is defined as

$ D_{f} (p^{1}, p^{2}) = \mathbb{E}{\mathbf{y} \sim p^{1}} \left[f\left(\frac{p^{1}(\mathbf{y}|\mathbf{x})}{p^{2}(\mathbf{y}|\mathbf{x})}\right) \right] \coloneqq \mathbb{E}{p^{1}} \left[f\left(\frac{dp^{1}}{dp^{2}}\right) \right], $

where $dp^{1}$ and $dp^{2}$ are the probability densities of probability $p^{1}$ and $p^{2}$.

The KL divergence is a $f$-divergence generated by $f(t) = t \log t - t + 1$, and the $\alpha$-skew KL divergence is a $f$-divergence generated by

$ f^{(\alpha)}(t) = t \log \left(\frac{t}{\alpha t + 1 - \alpha} \right) - (1-\alpha)(t-1). $

By following [53] and [54], we state the following regularity assumptions on the functions $f$ and $f^*$.

########## {caption="Assumption 2"}

The generator $f$ is twice continuously differentiable with $f'(1)=0$. Moreover

One can observe that both KLD and RKLD do not satisfy Assumption 2 because KLD is unbounded. On the other hand, the $\alpha$-skew KL divergence satisfies Assumption 2 from the following proposition.

########## {caption="Lemma 3: [53]"}

The $\alpha$-skew KL divergence generated by $f^{(\alpha)}$ satisfies Assumption 2 with

$ \begin{aligned} C_0 = 1 - \alpha, ;; C_0^* = \log \frac{1}{\alpha} - 1 + \alpha, ;; C_1 = 1, \ C_1^* = \frac{(1-\alpha)^{2}}{\alpha}, ;; C_2 = \frac{1}{2}, ;; C_2^* = \frac{1-\alpha}{8\alpha}. \end{aligned} $

For general $f$-divergences which satisfy Assumption 2, the following concentration bound holds.

########## {caption="Lemma 4: [53]"}

Assume $f$ satisfies Assumption 2, and let $p^{1}$ and $p^{2}$ be two distributions with $p^{1} \ll p^{2}$. Let $p^{1}_{m}$ be $m$ i.i.d samples from $p^{1}$ and $p^{2}_n$ be $n$ i.i.d samples from $p^{2}$. Then the $f$-divergence $D_f$ satisfies following:

$ \begin{aligned} &\mathbb{P}[|D_f(p^{1}{m}, p^{2}{n}) - \mathbb{E}[D_f(p^{1}{m}, p^{2}{n})]| > \epsilon] \leq \ &2 \exp \left(-\frac{\epsilon^{2}}{\frac{2}{m}(C_1 \log m + c_1)^{2} + \frac{2}{n} (C_1^* \log n + c_2)^{2}} \right) \end{aligned} $

where $c_1 = \max { C_0^*, C_2}$ and $c_2 = \max { C_0, C_2^* }$.

Thus, the following lemma derives a concentration bound for the $\alpha$-skew KL divergence by plugging the constants in Lemma 3 to Lemma 4.

########## {caption="Lemma 5: [54]"}

For $\alpha < \frac{1}{8}$, the following holds:

$ \begin{aligned} &\mathbb{P}[|D_{\text{SKL}}^{(\alpha)}(p^{1}{m}, p^{2}{n}) - \mathbb{E}[D_{\text{SKL}}^{(\alpha)}(p^{1}{m}, p^{2}{n})]| > \epsilon] \ &\leq 2 \exp \left(-\frac{\epsilon^{2}}{\frac{2}{m} \log^{2}(\alpha m) + \frac{2}{\alpha^{2} n} \log^{2} (e^{1/8}n)} \right) \end{aligned} $

Proof: Note that $C_{0}^{} = \log (1/\alpha) - 1 + \alpha \geq C_2 = 1/2$, and $C_0 = 1 - \alpha \leq C_2^ = \frac{1-\alpha}{8\alpha}$ for $\alpha < \frac{1}{8}$. Then the concentration bound follows from Lemma 5.

Lastly, we present the following upper bound on the bias of the empirical estimator of KLD.

########## {caption="Lemma 6: [55]"}

Suppose $p^{1} \ll p^{2}$, and $\mathbb{V}_{p^{1}}[dp^{1}/dp^{2}] < \infty$. Then we have

$ |\mathbb{E}[D_{\text{KL}}(p^{1}{m}, p^{2}{n})] - D_{KL} (p^{1}, p^{2}) | \leq \frac{\chi^{2}(p^{1}, p^{2})}{\min { n, m }}. $

########## {caption="Lemma 7: [54]"}

For $\alpha \in (0, 1)$, the following holds:

$ |\mathbb{E}[D_{\text{SKL}}^{(\alpha)}(p^{1}{m}, p^{2}{n})] - D_{\text{SKL}}^{(\alpha)} (p^{1}, p^{2}) | \leq \frac{c(\alpha)}{\min { n, m }}, $

where $c(\alpha) \coloneqq \min \left{ \frac{1}{\alpha}, \frac{\chi^{2}(p^{1}{m}, p^{2}{n})}{1-\alpha} \right}$.

Proof: From Lemma 6, we have

$ |\mathbb{E}[D_{\text{SKL}}^{(\alpha)}(p^{1}{m}, p^{2}{n})] - D_{\text{SKL}}^{(\alpha)} (p^{1}, p^{2}) | \leq \frac{\chi^{2}(p^{1}, \alpha p^1 + (1-\alpha) p^2}{\min { n, m }}, $

where $\chi^2 (p^1, \alpha p^1 + (1-\alpha) p^2) = \int \frac{d^2 p^1}{\alpha d p^1 + (1-\alpha) dp^2} \leq \int \frac{1}{\alpha} dp^1 = \frac{1}{\alpha}$, or $\int \frac{d^2 p^1}{\alpha d p^1 + (1-\alpha) dp^2} \leq \frac{1}{1-\alpha} \int \frac{d^2 p^1}{dp^2} = \frac{\chi^2 (p^1, p^2)}{1-\alpha}$. Therefore, we have

$ |\mathbb{E}[D_{\text{SKL}}^{(\alpha)}(p^{1}{m}, p^{2}{n})] - D_{\text{SKL}}^{(\alpha)} (p^{1}, p^{2}) | \leq \frac{c(\alpha)}{\min {n, m}} $

for $c(\alpha) \coloneqq \min \left{ \frac{1}{\alpha}, \frac{\chi^2 (p^1, p^2)}{1-\alpha} \right}$

Now we present the proof of Theorem 1 in the main paper.

Proof: Define

$ \begin{aligned} &B_1 \coloneqq D_{\text{SKL}}^{(\alpha)}(p^{1}{n}, p^{2}n) - \mathbb{E}[D{\text{SKL}}^{(\alpha)}(p^{1}{n}, p^{2}n)] \ &B_2 \coloneqq \mathbb{E}[D{\text{SKL}}^{(\alpha)}(p^{1}{n} | p^{2}{n})] - D_{\text{SKL}}^{(\alpha)} (p^{1}, p^{2}). \end{aligned} $

Then, by using bias-variance decomposition [56], we have

$ \begin{aligned} \mathbb{E}{p^{1}, p^{2}} [| D{\text{SKL}}^{(\alpha)}(p^{1}n, p^{2}n) - D{\text{SKL}}^{(\alpha)} (p^{1}, p^{2}) |^{2}] \ = \underbrace{\mathbb{E}{p^{1}, p^{2}} [| B_{1} |^{2}]}{\text{Variance}} + \underbrace{\mathbb{E}{p^{1}, p^{2}} [| B_{2} |^{2}]}_{\text{Bias}^{2}}. \end{aligned} $

Since the following holds for any random variable $X$,

$ \begin{aligned} \mathbb{V}(X) &= \mathbf{E}[(X - \mathbb{E}X)^{2}] \ &= \int_{0}^{\infty} \mathbb{P} [| X - \mathbb{E}X|^{2} > t] dt \ &= \int_{0}^{\infty} \mathbb{P} [|X - \mathbb{E}X| > \sqrt{t}] dt, \end{aligned} $

by Lemma 4, we have a variance for estimator $D_{\text{SKL}}^{(\alpha)}(p^{1}_n, p^{2}_n)$ as follows:

$ \begin{split} &\mathbb{V}{p^{1}, p^{2}}[D{\text{SKL}}^{(\alpha)}(p^{1}_n, p^{2}n)] \leq \ &\int{0}^{\infty} 2 \exp \left(-\frac{t}{\frac{2}{n} \left(\log^{2}(\alpha n) + \frac{1}{\alpha^{2}} \log^{2} (e^{1/8}n) \right)}\right) dt. \end{split}\tag{6} $

As we can directly compute the bias term through Lemma 7, we have

$ \begin{aligned} \mathbb{E}{p^{1}, p^{2}} [| D{\text{SKL}}^{(\alpha)}(p^{1}n, p^{2}n)] - D{\text{SKL}}^{(\alpha)} (p^{1}, p^{2}) |^{2}] \notag \ \leq \underbrace{\frac{c{1}(\alpha)}{n^{2}}}{\text{from Lemma 7}} + \underbrace{\frac{c{2} \log^{2}(\alpha n)}{n} + \frac{c_{3} \log^{2}(c_{4}n)}{\alpha^{2}n}}_{\text{from Eq.~Equation (6)}}, \end{aligned} $

where $c_{1}(\alpha) = \min\left{\frac{1}{\alpha^{2}}, \frac{\chi^{2}(p^{1}, p^{2})^{2}}{(1-\alpha)^{2}}\right}$ and constants $c_{2}, c_{3}, c_{4} > 0$ that are independent of $n$, $\alpha$, and $D_{\text{KL}}(p^{1}, p^{2})$, where $\chi^{2}(p^{1}, p^{2}) := \mathbb{E}_{p^{2}} \left[(dp^{1}/dp^{2})^{2} \right]$.

C. Details of DistiLLM Algorithm

We describe $\textsc{DistiLLM}$ in detail, especially for the adaptive off-policy approach, which could not be fully explained in Section 3.2 due to lack of margin.

Instead of using $\phi$ defined in Section 3.2, we further define the probability of using SGOs determined at each validation iteration $\tilde{t}$, denoted as $\phi_{\tilde{t}}$. We adjust the probability by using the following rule with validation loss $\mathcal{L}{\tilde{t}}$ and $\mathcal{L}{\tilde{t}-1}$ for iteration $\tilde{t}$ and $\tilde{t}-1$, respectively:

$ \phi_{\tilde{t}} = \begin{cases} \phi_{\tilde{t}-1} & \text{if } \mathcal{L}{\tilde{t}} \leq \mathcal{L}{\tilde{t}-1} + \varepsilon \ \min (\phi_{\tilde{t}-1} + 1 / N_{\phi}, 1.0)& \text{otherwise} \end{cases}, $

where $\varepsilon$ is the loss tolerance, a hyperparameter introduced to mitigate unexpected fluctuations in loss values and enhance the stability of the process. We set this value as 0.1 for all experiments. $N_{\phi}$ denotes the total number of stages for adjusting the probability, allowing the probability to adopt any value in the set $\left{\frac{i}{N_{\phi}} , \big| , , i = 0, 1, \ldots, N_{\phi} \right}$. In our experiments, we set an initial validation loss $\mathcal{L}{0}$ as validation loss of initialized student models, $\phi{0}$ as 0.0, and $N_{\phi}$ as 10. If a newly computed validation loss exceeds the preceding one, we increment $\phi_{\tilde{t}}$ by one stage.

Here, we present empirical evidence to justify using validation loss as the primary metric. Fig. Figure 10 shows that the validation loss for both SKL and SRKL consistently increases over iterations, as depicted by the blue lines. This trend indicates that the student models may overly fit to the training dataset and overlook other general sequences, leading to a training-inference mismatch. To create a more versatile student model that performs well on both training and validation datasets, we introduce additional SGOs whenever an increase in validation loss is observed. This approach effectively mitigates the mismatch between training and inference, as well as reduces the risk of noisy feedback.

D. Experimental Setup

We elaborate the detailed experimental setup regarding the datasets used (Appendix D.1), training details (Appendix D.2), and evaluation details (Appendix D.3).

D.1 Dataset Description

We apply $\textsc{DistiLLM}$ on various generation tasks including task-agnostic instruction-following, text summarization, and machine translation. We provide detailed descriptions of the datasets used.

• databricks-dolly-15k (instruction-following, [25]): databricks-dolly-15k is an open-source dataset of instruction-following records generated by thousands of Databricks employees in several behavioral categories that are outlined in [28], including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.
• Self-Instruct (instruction-following, [26]): Self-Instruct is a framework designed to enhance the language model's instruction-following capabilities by leveraging the model's own outputs to generate a vast set of instructional data. It includes 52K instructions and 82K inputs and outputs for tuning, along with 252 expert-written tasks aimed at practical applications and additional 50K examples from public datasets for benchmarking.
• Vicuna (instruction-following, [57]): We also use 80 challenging questions that were used for evaluating Vicuna, following [11].
• Super-Natural Instruction (instruction-following, [26]): Super-Natural Instruction is introduced as a benchmark of 1, 616 diverse NLP tasks and their expert-written instructions. The collection covers 76 distinct task types. Its test set consists of 9K samples ranging from 119 tasks.
• Unnatural Instruction (instruction-following, [27]): Unnatural Instruction uses AI to create 240K instructions with little human help, showing that AI-made data can be as good as human-made data for training language models. The core set of this dataset contains 60K samples.
• SAMSum (text summarization, [34]): SAMSum consists of 16K messenger-like conversations, annotated with a summary for providing a concise overview of the conversation's content by the third person.
• XSum (text summarization, [39]): XSum comprises over 200K news articles, each accompanied by a one-sentence summary designed for the evaluation of abstractive single-document summarization systems, focusing on extreme summarization to capture the essence of articles in a single sentence.
• CNN/DM (text summarization, [40]): CNN/ DM consists of over 300K English news articles that were originally designed for machine-reading and comprehension as well as abstractive question answering, but it now also supports extractive and abstractive summarization.
• IWSLT 2017 (machine translation, [35]): IWSLT 2017 addresses text translation, using a single machine translation system for multiple language directions such as English and German. Here, we specifically focus on an English-to-German (En-De) translation task.

D.2 Training Details

For training the teacher and student models, we used four A100 40GB GPUs for the instruction-following task and four RTX 3090 GPUs for the text summarization and machine translation tasks.

Instruction-following experiments.

Our experimental setup for training LMs on databricks-dolly-15k primarily follows the experimental setup for [11]. For models within 1B parameters, we search for the learning rates in 5e-4, 1e-4, 5e-5, the batch sizes in 8, 16, 32 within the possible maximum batch size for A100 40GB GPUs, and train these models for 20 epochs. For models that have more than 1B parameters, we search for the learning rate in 5e-5, 1e-5, 5e-6, the batch sizes of 8, and train these models for 10 epochs. We fully use the distillation loss for the instruction-following dataset and language modeling loss for OpenWebText [30] corpus. The checkpoints of each student are selected by the ROUGE-L scores on the validation set.

For MiniLLM [11], we follow the original setup except for the number of GPUs. For ImitKD [14], MiniLLM, and GKD [13], we initialize the student models with the fine-tuned ones according to their original methods, ensuring a fair comparison in our method as well. However, $\textsc{DistiLLM}$ demonstrates effectiveness even without the need for such fine-tuned student models, unlike other methods that utilize SGOs. The corresponding results are available in. We conduct validation at the end of every training epoch. For MiniLLM, we have used the original code ^2 [11], while for other baselines, we have re-implemented them. To train the OpenLLaMA2 [20], we use LoRA for query and value weights with a rank of 16.

Task-specific experiments.

For text summarization tasks (SAMSum, XSum, and CNN/DM), we train all teacher and student models for 10 epochs. In contrast, for the IWSLT 2017 En-De dataset, we train them for 2 epochs. Since the official code for MiniLLM [11] is not available on such tasks, we do not consider conducting experiments with MiniLLM on these tasks. Moreover, as the other methods such as SeqKD [6], ImitKD [14], and GKD [13] highly increase the training time from obtaining the SGOs or teacher-generated outputs, we only use the 20K of random samples for XSum and CNN/DM, as we described in Section 4.2. However, due to the difficulty of machine translation, we use the full dataset of IWSLT 2017. We use a fixed learning rate of 1e-4 and use possible maximum batch size within 8, 32, 64 for RTX 3090 GPUs. We also conducted ten validations for all experiments. For the training teacher model, we utilize LoRA [33] for all weights for query, key, value, and output with rank of 16.

D.3 Evaluation

For evaluating the teacher and student models, we applied a single A100 40GB GPU for the instruction-following task and a single RTX 3090 GPU for the text summarization and machine translation tasks.

Instruction-following.

Our evaluation setup for the instruction-following task also follows the [11]. During the evaluation, we sample the responses from each model using a temperature of 1.0, a max-length limit of 512, and five random seeds (i.e., ${10, 20, 30, 40, 50}$). We adopt a prompt wrapper as shown in Figure 11. However, for GPT-4 feedback, instead of using the prompt introduced in [11], we use a more popular prompt introduced in [32] which is illustrated in Figure 12 with setting the temperature of 0.7. We also report the ratio of the total score of model responses and ground truth answers by following [11].

Task-specific experiments.

During the evaluation, we sample the responses from each model using a greedy sampling, and a max-length limit of 128. We use ROUGE-L [31] and BLEU [38] for text summarization and machine translation, respectively.

E. Additional Results

In this section, we provide additional experimental results to demonstrate the effectiveness of our proposed method and its components.

E.1 Full Results of Instruction-Following (Figure 5)

In Fig. Figure 14, we describe the full version of the main result in Sec. Section 4.1 and Fig. Figure 14. Our proposed $\textsc{DistiLLM}$ consistently outperform the baselines such as MiniLLM [11] and GKD [13] in most of the datasets (Dolly Evaluation, Self-Instruct, Vicuna Evaluation, Super-Natural, and Unnatural) and metrics (ROUGE-L and GPT-4 feedback) regardless of model sizes. The numerical results are reported in Table 11, Table 12, and Table 13. These results demonstrate that $\textsc{DistiLLM}$ achieves the best performance (bold number) in most cases, except for a few second best performances (underlined number).

E.2 Full Results of Task-Specific KD (Tab. Table 3)

::: {caption="Table 7: The full results of Table 3, which is the performance comparison of KD methods trained on text summarization and machine translation datasets. We report the ROUGE-L and BLEU scores for the distilled student."}

:::

In addition to the results on SAMSum and IWSLT 2017 datasets in Sec. Section 4.2 and Tab. Table 3, we further conduct experiments on XSum and CNN/DM datasets. As the existing KD methods using SGO require up to 5 $\times$ training time compared to naïve KD, we randomly sample 20K from the training dataset. Tab. Table 7 further shows the performance of XSum and CNN/DM for $\textsc{DistiLLM}$ and other KD baselines. Our $\textsc{DistiLLM}$ consistently outperforms other baselines in terms of ROUGE-L and BLEU scores.

E.3 Terminal Probability by Adaptive SGO Scheduler

In the following list, we report the terminal probability values determined by our adaptive SGO scheduler. We observe that the final probability values are varying across the different tasks. This highlights the importance of using our novel SGO scheduler which can adaptively balance the SGOs and a fixed dataset.

• GPT-2 family(databricks-dolly-15k): 0.4 (Base), 0.4 (Medium), 0.5 (Large)
• OPT family (databricks-dolly-15k): 0.2 (125M), 0.3 (350M), 0.5 (1.3B)
• OpenLLaMA2 (databricks-dolly-15k): 0.6 (3B)
• T5 (SAMSum): 0.7 (Small), 0.8 (Base)
• T5 (IWSLT 2017): 0.5 (Small), 0.7 (Base)
• T5 (XSum): 0.3 (Small), 0.6 (Base)
• T5 (CNN/DM): 0.4 (Small), 0.6 (Base)

E.4 Sensitivity Study for $\alpha$

::: {caption="Table 8: Comparison of the ROUGE-L score of GPT-2 student using different alpha. All results show the concave form in terms of ROUGE-L scores."}

:::

As shown in Table 8, we also report the numerical values of the results in Figure 6 for all datasets. The results for Vicuna and Unnatural are similar to the results for Dolly, Self-Instruct, and Super-Natural Instruction datasets (i.e., making concave form in terms of ROUGE-L scores as $\alpha$ increases). These results highly relate to our theoretical analysis in Section 3.1 that the normalized approximation error dividing by gradient showed the convexity. This also provides the importance of the skew value $\alpha$ which has an important role in balancing the approximation error and gradient scale.

\begin{tabular}{l|c|ccccc}
\toprule[0.1em]
  Replay ratio $\zeta$ & Time & Dolly & Self & Vicuna & SNI & UNI \\ \midrule
  Constant (0.5) & $\times$ 2.13 & 26.32 & 12.99 & 17.31 & 27.45 & 28.64 \\ 
  Increasing (${t}/{T}$) & $\times$ 2.39 & \textbf{26.44} & 12.42 & 17.60 & 26.80 & \textbf{29.90} \\
  Decreasing ($1-{t}/{T}$) & $\times$ \textbf{1.67} & 26.11 & \textbf{13.14} & \textbf{18.46} & \textbf{27.51} & 29.35 \\
\bottomrule[0.1em]
\end{tabular}

E.5 Design of Replay Ratio

Table 9 summarizes the training time and performance according to the different scheduling strategies for replay ratio in the off-policy training. Despite the high bias error [24] of off-policy training, the performance of the constant or increasing $\zeta$ is similarly compared to our proposed decreasing manner thanks to our adaptive probability of using SGO. Furthermore, our scheduling strategy not only demonstrates the highest training efficiency but also consistently delivers the best performance overall, achieving the best results in three out of five cases.

E.6 Synergy of SKL and SRKL

Similar to generalized JSD [13], we consider the interpolation between the SKL and SRKL using the coefficient $\beta \in [0, 1]$. Here, we define interpolated SKL (ISKL) as follows:

$ D_{\text{ISKL}}^{(\beta)}(p, q_\theta) = \beta D_{\text{SKL}}^{(0.1)}(p, q_\theta) + (1-\beta) D_{\text{SRKL}}^{(0.1)}(p, q_\theta). $

We set the $\alpha$ for 0.1 based on previously described results. Figure 13 shows the results of ISKL across different $\beta$ values. Note that we do not use SGO for these results. We observe that there is no consistent tendency between the $\beta$ and corresponding performance. However, the performance for ISKL for all $\beta$ still outperforms the other loss functions previously used in auto-regressive KD (i.e., JSD and RKLD).

E.7 Qualitative Evaluation

We provide some responses generated by the models distilled by different methods based on OpenLLaMA2-3B in Tab. Table 14 and Tab. Table 15. The prompts are sampled from the databricks-dolly-15k, Self-Inst, and Vicuna. Our results demonstrate that $\textsc{DistiLLM}$ produces more detailed and accurate responses compared to other KD baselines. Specifically, $\textsc{DistiLLM}$ excels at comprehending the precise instructions given (for example, following alphabetical order in Case #1 from Tab. Table 14, and identifying categorization candidates in Case #2 from Tab. Table 15). Additionally, we observe that $\textsc{DistiLLM}$ achieves high-quality output across various types of tasks, including code generation (as in Case #1 in Tab. Table 15) and mathematical reasoning (as in Case #3 in Tab. Table 15).

E.8 Replay Buffer Capacity

\begin{tabular}{l|ccccc}
\toprule[0.1em]
  $\mathcal{D}_{R}$ Capacity & Dolly & Self & Vicuna & SNI & UNI \\ \midrule
  250 & 24.71 & 11.95 & 16.79 & 24.88 & 26.73 \\ 
  500 & 25.32 & 12.46 & 17.64 & 25.69 & 27.00 \\
  1000 & 26.11 & \textbf{13.14} & \textbf{18.46} & \textbf{27.51} & \textbf{29.35} \\
  2000 & \textbf{26.48} & 12.88 & 17.49 & 26.51 & 28.61 \\
  4000 & 25.58 & 12.65 & 17.22 & 25.65 & 27.55 \\
\bottomrule[0.1em]
\end{tabular}

We also conduct experiments to confirm the effect of the capacity of the replay buffer on our off-policy approach. Table 10 summarizes the performance associated with different capacities of the replay buffer, $\mathcal{D}{R}$. We observe that determining the appropriate capacity involves a trade-off. A capacity that is too small may lead to overfitting on a limited number of samples within $\mathcal{D}{R}$. Conversely, a capacity that is too large results in the inclusion of outdated SGO in the replay buffers, potentially introducing a high bias issue as noted by Lee (2023) [24]. In our experiments, a capacity value of 1000 demonstrates the most balanced performance overall.

::: {caption="Table 11: Comparison with state-of-the-art KD methods, fine-tuned GPT-2 model families [18] on databricks-dolly-15k dataset. All results are based on our re-implementation. The bold and underline markings indicate the best and second-best results, respectively, among those from the same evaluation dataset and student model."}

:::

::: {caption="Table 12: Comparison with state-of-the-art KD methods, fine-tuned OPT model families [19] on databricks-dolly-15k dataset. All results are based on our re-implementation. The bold and underline markings indicate the best and second-best results, respectively, among those from the same evaluation dataset and student model."}

:::

::: {caption="Table 13: Comparison of state-of-the-art KD methods using OpenLLaMA2-7B [20] and OpenLLaMA2-3B as teacher and student models, respectively. We fine-tune the models on databricks-dolly-15k dataset. All results are based on our re-implementation. The bold and underline markings indicate the best and second-best results, respectively, among those from the same evaluation dataset and student model."}

:::

colourcmarkpinegreen colourxmarkred colourtmarkorange

References

Section Summary: This section provides a numbered bibliography of 21 academic papers, technical reports, and preprints that underpin the discussed research on language models. The cited works focus primarily on large-scale models such as GPT-4 and Llama 2, along with methods for knowledge distillation, model compression, and related training techniques in natural language processing. They draw from sources published between 2015 and 2024 by organizations including OpenAI and various university research groups.

[1] OpenAI. Gpt-4 technical report, 2023.

[2] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

[3] Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484–13508, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi:10.18653/v1/2023.acl-long.754. URL https://aclanthology.org/2023.acl-long.754.

[4] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

[5] Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL https://api.semanticscholar.org/CorpusID:7200347.

[6] Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi:10.18653/v1/D16-1139. URL https://aclanthology.org/D16-1139.

[7] Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108, 2019. URL https://api.semanticscholar.org/CorpusID:203626972.

[8] Sun, S., Cheng, Y., Gan, Z., and Liu, J. Patient knowledge distillation for BERT model compression. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4323–4332, Hong Kong, China, November 2019. Association for Computational Linguistics. doi:10.18653/v1/D19-1441. URL https://aclanthology.org/D19-1441.

[9] Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191–5198, 2020.

[10] Wen, Y., Li, Z., Du, W., and Mou, L. f-divergence minimization for sequence-level knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10817–10834, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:10.18653/v1/2023.acl-long.605. URL https://aclanthology.org/2023.acl-long.605.

[11] Gu, Y., Dong, L., Wei, F., and Huang, M. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=5h0qf7IBZZ.

[12] Arora, K., El Asri, L., Bahuleyan, H., and Cheung, J. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 700–710, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.findings-acl.58. URL https://aclanthology.org/2022.findings-acl.58.

[13] Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3zKtaqxLhW.

[14] Lin, A., Wohlwend, J., Chen, H., and Lei, T. Autoregressive knowledge distillation through imitation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6121–6133, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.494. URL https://aclanthology.org/2020.emnlp-main.494.

[15] Lee, L. On the effectiveness of the skew divergence for statistical language analysis. In International Workshop on Artificial Intelligence and Statistics, pp. 176–183. PMLR, 2001.

[16] Ji, H., Ke, P., Hu, Z., Zhang, R., and Huang, M. Tailoring language generation models under total variation distance. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=VELL0PlWfc.

[17] Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.

[18] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.

[19] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.

[20] Geng, X. and Liu, H. Openllama: An open reproduction of llama, May 2023. URL https://github.com/openlm-research/open_llama.

[21] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.

[22] Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W. Revisiting fundamentals of experience replay. In International Conference on Machine Learning, pp. 3061–3071. PMLR, 2020.

[23] Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.

[24] Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., and Yun, C. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

[25] Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction-tuned llm, 2023. URL [https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable- instruction-tuned-llm](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable- instruction-tuned-llm).

[26] Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., Pathak, E., Karamanolakis, G., Lai, H., Purohit, I., Mondal, I., Anderson, J., Kuznia, K., Doshi, K., Pal, K. K., Patel, M., Moradshahi, M., Parmar, M., Purohit, M., Varshney, N., Kaza, P. R., Verma, P., Puri, R. S., Karia, R., Doshi, S., Sampat, S. K., Mishra, S., Reddy A, S., Patro, S., Dixit, T., and Shen, X. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085–5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.emnlp-main.340. URL https://aclanthology.org/2022.emnlp-main.340.

[27] Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14409–14428, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:10.18653/v1/2023.acl-long.806. URL https://aclanthology.org/2023.acl-long.806.

[28] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.

[29] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. OpenAI blog, 2018.

[30] Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Openwebtext corpus, 2019.

[31] Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013.

[32] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=uccHPGDlao.

[33] Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.

[34] Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70–79, 2019.

[35] Cettolo, M., Federico, M., Bentivogli, L., Niehues, J., Stüker, S., Sudoh, K., Yoshino, K., and Federmann, C. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation, pp. 2–14, Tokyo, Japan, December 14-15 2017. International Workshop on Spoken Language Translation. URL https://aclanthology.org/2017.iwslt-1.1.

[36] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.

[37] Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mT5: A massively multilingual pre-trained text-to-text transformer. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483–498, Online, June 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.naacl-main.41. URL https://aclanthology.org/2021.naacl-main.41.

[38] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi:10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040.

[39] Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797–1807, 2018.

[40] See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. In Barzilay, R. and Kan, M.-Y. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073–1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:10.18653/v1/P17-1099. URL https://aclanthology.org/P17-1099.

[41] Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.

[42] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.

[43] Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4, 2023.

[44] Wang, P., Wang, Z., Li, Z., Gao, Y., Yin, B., and Ren, X. Scott: Self-consistent chain-of-thought distillation. arXiv preprint arXiv:2305.01879, 2023a.

[45] Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023.

[46] Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Alhammadi, M., Daniele, M., Heslow, D., Launay, J., Malartic, Q., Noune, B., Pannier, B., and Penedo, G. The falcon series of language models: Towards open frontier models. https://arxiv.org/abs/2311.16867, 2023.

[47] Bibliography entry for citation key "ren2024emo" was not supplied with the source.

[48] Bibliography entry for citation key "chen2024selfplay" was not supplied with the source.

[49] Bibliography entry for citation key "rafailov2024direct" was not supplied with the source.

[50] Bibliography entry for citation key "wan2024knowledge" was not supplied with the source.

[51] Bibliography entry for citation key "boizard2024towards" was not supplied with the source.

[52] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

[53] Liu, L., Pillutla, K., Welleck, S., Oh, S., Choi, Y., and Harchaoui, Z. Divergence frontiers for generative models: Sample complexity, quantization effects, and frontier integrals. Advances in Neural Information Processing Systems, 34:12930–12942, 2021.

[54] Lee, K. and Shin, J. Rényicl: Contrastive representation learning with skew rényi divergence. Advances in Neural Information Processing Systems, 35:6463–6477, 2022.

[55] Rubenstein, P., Bousquet, O., Djolonga, J., Riquelme, C., and Tolstikhin, I. O. Practical and consistent estimation of f-divergences. Advances in Neural Information Processing Systems, 32, 2019.

[56] Pedro, D. A unified bias-variance decomposition and its applications. In 17th International Conference on Machine Learning, pp. 231–238, 2000.

[57] Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.