Learning Dynamics of LLM Finetuning

Yi Ren
University of British Columbia
[email protected]

Danica J. Sutherland
University of British Columbia & Amii
[email protected]

Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique "squeezing effect" to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance. Code for experiments is available at https://github.com/Joshua-Ren/Learning_dynamics_LLM.

Deep neural networks usually acquire new knowledge by updating their parameters via gradient descent (GD). This procedure can be described by learning dynamics, which links changes in the model's predictions to the gradients generated by learning specific examples. With the help of learning dynamics, researchers have not only explained many interesting phenomena during training, e.g., the "zig-zag" learning path ([1]) and the formation of compositional concept space ([2]), but used these insights to propose novel, improved algorithms in different problems (e.g. [3,4,5]).

The study of large language models (LLM) is gaining popularity due to their surprising capabilities on various tasks. To ensure the LLMs follow human instructions and align well with human preferences, finetuning has attracted much recent attention. Practitioners often start with instruction tuning, where the model learns extra knowledge necessary for the downstream task, and then preference tuning, where the model aligns its outputs to human preference ([6]). Various finetuning algorithms have been proposed to fit into this pipeline, with differing explanations as to why they improve the model's performance.

Contrary to most existing analyses of LLM finetuning, which use the perspective of their training targets, their status at the end of training, or their relationships to reinforcement learning (e.g. [7,8,9]), this paper tries to understand LLMs' evolution from a dynamical perspective. Specifically, we formalize the learning dynamics of LLM finetuning by decomposing the change of the model's prediction into three terms which play different roles.

This framework can be easily adapted to various finetuning algorithms with different goals, including supervised finetuning (SFT, [10]), direct preference optimization (DPO, [11], and its variants) and even reinforcement learning based methods (e.g., PPO, [12]). This framework helps explain several interesting and counter-intuitive observations during training -- including the "repeater" phenomenon after preference tuning ([13]), hallucination

([14]), the decay in confidence of all responses during off-policy DPO ([8]), and more.

Moreover, we also provide a new perspective on understanding why off-policy DPO and other variants underperform their on-policy counterparts ([15]). Our explanation starts by observing an interesting "squeezing effect, " which we demonstrate is a consequence of gradient ascent (as in DPO and similar algorithms) on models with cross-entropy loss following a softmax layer.

In short, for each token's prediction, the negative gradient will push down the model's predictions on (almost) all possible output labels, moving this probability mass to the most-likely labels. This can be detrimental to the alignment we are trying to achieve. This effect is most serious when the negative gradient is imposed on an already-unlikely label, which is why the confidence of almost all responses decreases during off-policy DPO. Inspired by this, we propose a simple but very effective method to further improve alignment performance.

Learning dynamics is usually an umbrella term describing how the change of a specific factor influences the model's prediction. In this paper, we narrow down it to describe ``how the change in model's parameter $\theta$ influences the corresponding change in $f_\theta$ '', i.e., the relationship between $\textcolor{cyan}{\Delta\theta}$ and $\textcolor{orange}{\Delta f_\theta}$. When the model updates its parameters using gradient descent (GD), we have

where the update of $\theta$ during step $t\rightarrow t+1$ is given by one gradient update on the sample pair $(\textcolor{cyan}{\bm{\mathsf{x}}_u}, \textcolor{cyan}{\bm{\mathsf{y}}_u})$ with learning rate $\eta$. In short, the learning dynamics in this paper address the question:

Learning dynamics can shed light on many important problems in deep learning and also help to understand various counterintuitive phenomena. The origin of it might be the "stiffness" ([16]) or "local elasticity" ([17,18]) of neural networks. See Appendix A for more discussions.

As a warm-up, we first consider a standard supervised learning problem, where the model learns to map $\bm{\mathsf{x}}$ to predictions $\bm{\mathsf{y}}=\{y_1, \dots, y_L\}\in\mathcal{V}^L$, where $\mathcal V$ is the vocabulary of size $V$.

The model usually outputs a probability distribution by first generating a matrix of logits $\bm{\mathsf{z}}=h_\theta(\bm{\mathsf{x}})\in\mathbb{R}^{V\times L}$ and then takes the $\operatorname{\mathsf{Softmax}}$ of each column. We can track the change in the model's confidence by observing $\log \pi_\theta(\bm{\mathsf{y}}\mid \bm{\mathsf{x}})$.

The learning dynamics of Equation 1 become,

For simplicity, we start from the $L=1$ scenario, where the $\Delta\theta$ and $\Delta\log\pi$ can be linked by the following result, a version of a result of [1] proved and further discussed in Appendix B. For multi-label classification ($L>1$), the updates separate; we can calculate $L$ different $\Delta\log\pi^t$ and stack them together.

Let $\pi= \operatorname{\mathsf{Softmax}}(\bm{\mathsf{z}})$ and $\bm{\mathsf{z}}=h_\theta(\bm{\mathsf{x}})$. The one-step learning dynamics decompose as

where $\mathcal{A}^t(\textcolor{orange}{\bm{\mathsf{x}}_o}) = \nabla_{\bm{\mathsf{z}}}\log\pi_{\theta^t}(\textcolor{orange}{\bm{\mathsf{x}}_o}) = I - \bm{\mathsf{1}} \pi_{\theta^t}^\top(\textcolor{orange}{\bm{\mathsf{x}}_o})$, $\mathcal{K}^t(\textcolor{orange}{\bm{\mathsf{x}}_o}, \textcolor{cyan}{\bm{\mathsf{x}}_u}) = (\nabla_{\theta} \bm{\mathsf{z}}(\textcolor{orange}{\bm{\mathsf{x}}_o})|_{\theta^t})(\nabla_{\theta} \bm{\mathsf{z}}(\textcolor{cyan}{\bm{\mathsf{x}}_u})|_{\theta^t})^\top$ is the empirical neural tangent kernel of the logit network $\bm{\mathsf{z}}$, and $\mathcal{G}^t(\textcolor{cyan}{\bm{\mathsf{x}}_u}, \textcolor{cyan}{\bm{\mathsf{y}}_u}) =\nabla_{\bm{\mathsf{z}}}\mathcal{L}(\textcolor{cyan}{\bm{\mathsf{x}}_u}, \textcolor{cyan}{\bm{\mathsf{y}}_u})|_{\bm{\mathsf{z}}^t}$.

$\mathcal{A}^t(\textcolor{orange}{\bm{\mathsf{x}}_o})= I - \bm{\mathsf{1}} \pi_{\theta^t}^\top(\textcolor{orange}{\bm{\mathsf{x}}_o})$ only depends on the model's current predicted probability.

The matrix $\mathcal{K}^t$ is the empirical neural tangent kernel (eNTK, [19]) of the model, the product of the model's gradients with respect to $\textcolor{orange}{\bm{\mathsf{x}}_o}$ and $\textcolor{cyan}{\bm{\mathsf{x}}_u}$. The analysis in this paper relies on the following assumption:

The common "lazy eNTK" assumption discussed in [20] is a sufficient but not necessary condition for this paper. Appendix C provides a more detailed discussion and experimental verification for both MNIST and LLM settings. We can then think of $\mathcal{K}^t$ as a model-specific similarity measurement between different input samples: larger $\|\mathcal{K}^t\|_F$ means the update of $\bm{\mathsf{x}}_u$ likely influences the model's prediction on $\bm{\mathsf{x}}_o$ more. Finally, $\mathcal{G}^t$ is determined by the loss function $\mathcal{L}$, which provides the energy and direction for the model's adaptation. For example, for cross-entropy loss $\mathcal{L}_\text{CE} \triangleq - \bm{\mathsf{y}}_u^\top \log\pi(\bm{\mathsf{y}}\mid \bm{\mathsf{x}}_u)$, we have $\mathcal{G}_\text{CE}^t=\pi_{\theta^t}(\bm{\mathsf{y}}\mid \bm{\mathsf{x}}_u) - \bm{\mathsf{y}}_u$, a length- $V$ vector that points from the model's current predictive distribution to the desired supervisory distribution. For typical "hard" labels, $\bm{\mathsf{y}}_u$ is a one-hot vector $\bm{\mathsf{e}}_{\bm{\mathsf{y}}_u}$.

Proposition 1 describes how the update of $\textcolor{cyan}{\bm{\mathsf{x}}_u}$ changes the model's prediction on $\textcolor{orange}{\bm{\mathsf{x}}_o}$ for each learning step. Since a real model updates its parameters for many steps,

it is important to ask about accumulation of these per-step influences over time.

We start by analyzing a simple example of training a LeNet on the MNIST dataset ([21]).

See Figure 1-(a), where the network $\pi_{\theta^t}$ is updating its parameters using the loss calculated on one training example $(\textcolor{cyan}{\bm{\mathsf{x}}_u}, \textcolor{cyan}{\bm{\mathsf{y}}_u}= \bm{\mathsf{e}}_\texttt{4})$.

The residual term $\mathcal{G}^t_\text{CE}(\textcolor{cyan}{\bm{\mathsf{x}}_u}, \textcolor{cyan}{\bm{\mathsf{y}}_u})$ is then represented by the red arrows, which all start from $\pi_{\theta^t}(\bm{\mathsf{y}}\mid \textcolor{cyan}{\bm{\mathsf{x}}_u})$ and point to $\bm{\mathsf{e}}_{\texttt{4}}$. We can then ask how the model's predictions on different $\textcolor{orange}{\bm{\mathsf{x}}_o}$ change after this update. As in Figure 1-(b), for an $\textcolor{orange}{\bm{\mathsf{x}}_o}$ in the same class with $\textcolor{cyan}{\bm{\mathsf{x}}_u}$ (i.e., the identical case), the predicted probability of this correct label is "pulled up" by this update, as expected.

On the other hand, if this $\textcolor{orange}{\bm{\mathsf{x}}_o}$ is similar to $\textcolor{cyan}{\bm{\mathsf{x}}_u}$ (i.e., $\|\mathcal{K}^t\|_F$ is reasonably large) but comes from another class, then the predicted probability on $\textcolor{cyan}{\bm{\mathsf{x}}_u}$ 's class (not the correct label of $\textcolor{orange}{\bm{\mathsf{x}}_o}$) would be "pulled up, " as in the second panel of Figure 1-(b). Last, for examples that look dissimilar to $\textcolor{cyan}{\bm{\mathsf{x}}_u}$ (small $\| \mathcal K^t \|_F$), this update will not change the model's prediction on $\textcolor{orange}{\bm{\mathsf{x}}_o}$ much, as in the bottom panel in Figure 1-(b).

The interactions among the updates of different inputs then form an interesting pattern for the learned predictions. As illustrated in Figure 1-(c), when making predictions on images coming from class 4, the model tends to assign higher confidence on class 9. That is because the examples in class 9 on average look more similar to class 4 than examples in other classes. Hence the update of examples in classes 4 and 9 will reinforce their mutual influence and lead to a bump in their predictions. To verify this, we plot the average of $\pi(\bm{\mathsf{y}}\mid \bm{\mathsf{x}})$ for $\bm{\mathsf{x}}$ from each of the classes in Figure 1-(d).

The values of some off-diagonal patches are significantly higher than others, which means the examples in those classes look more similar, like 4 and 9, 5 and 3, 8 and 5, etc.

Although learning dynamics have been applied to many deep learning systems, extending this framework to LLM finetuning is non-trivial. The first problem is the high dimensionality and the sequence nature of both the input and output signals. The high-dimensional property makes it hard to observe the model's output, and the sequential nature makes the distributions on different tokens mutually dependent, which is more complicated than a standard multi-label classification problem considered by most previous work. Furthermore, as there are many different algorithms for LLM finetuning -- SFT ([10]), RLHF ([6]), DPO ([11]), etc. -- analyzing them under a uniform framework is challenging. Finally, compared with the training-from-scratch scenario, where a roughly uniform distribution over all possible outputs is usually assumed, LLMs' finetuning dynamics heavily rely on the pretrained base model, which could make the analysis harder. For example, the pretrained model usually assigns little probability mass to unlikely tokens, which is good for most applications but leads to risk of the "squeezing effect" we show later.

We now tackle these problems and propose a unified framework for different finetuning methods.

The typical loss function used during supervised finetuning is the negative log-likelihood (NLL) of a given completion $\bm{\mathsf{y}}^+_u=[y^+_1, \dots, y^+_L] \in \mathcal{V}^L$, conditioned on the prompt $\bm{\mathsf{x}}_u$:

Note that compared with the multi-label classification problem discussed before, where the joint distribution of all labels can be factorized as $\pi(\bm{\mathsf{y}}\mid \bm{\mathsf{x}})=\prod_l \pi(y_l\mid \bm{\mathsf{x}})$, the sequential nature of language modeling makes the analysis more complicated, because we must have $\pi(\bm{\mathsf{y}}\mid \bm{\mathsf{x}})=\prod_l \pi(y_l\mid \bm{\mathsf{x}}, \bm{\mathsf{y}}_{<l})$.

To solve this problem, we can merge this factorization into the definition of the backbone $h_\theta$ while keeping the format of Proposition 1. Specifically, letting $\bm{\mathsf{\chi}}$ be the concatenation of $\bm{\mathsf{x}}$ and $\bm{\mathsf{y}}$, the prediction of all tokens of $\bm{\mathsf{y}}$ is

Here $\bm{\mathsf{z}}$ is a $V\times L$ matrix where each column contains the logits of the prediction of the $l$ th token. Our $h_{\theta}$, even though it takes the entire sequence ${\bm{\mathsf{\chi}}}$ as its input, will force the model not to refer to the future tokens $\bm{\mathsf{y}}_{>l}$ when making predictions on the $l$ -th token, commonly implemented via "causal masking" (proposed in [22], details in Figure 10a of Appendix D). Then, we can calculate $(\nabla_{\theta} \bm{\mathsf{z}}_l(\bm{\mathsf{\chi}}_o)|_{\theta^t})(\nabla_{\theta} \bm{\mathsf{z}}_l(\bm{\mathsf{\chi}}_u)|_{\theta^t})^\top$ on each column of $\bm{\mathsf{z}}$ and stack them to form a $V\times V \times M\times L$ tensor $\mathcal{K}^t(\bm{\mathsf{\chi}}_o, \bm{\mathsf{\chi}}_u)$. The calculation of $\mathcal{G}^t$ and $\mathcal{A}^t$ follows a similar procedure. Thanks to the causal mask implemented in $h_\theta$, the resulting decomposition is almost identical to that in a multi-label classification problem. Assuming have a response $\bm{\mathsf{y}}_u$ of length $L$ associated with $\bm{\mathsf{x}}_u$, stacked into $\bm{\mathsf{\chi}}_u$, and $\bm{\mathsf{y}}_o$ of length $M$ associated with $\bm{\mathsf{x}}_o$, stacked into $\bm{\mathsf{\chi}}_o$. The change of the model's prediction on the $m$ -th token of $\bm{\mathsf{y}}_o$ can be represented as, when $\bm{\mathsf{z}}$ gradients have bounded norm,

where $\mathcal{G}^t_\text{SFT}\left(\bm{\mathsf{\chi}}_u\right)=\pi_{\theta^t}(\bm{\mathsf{y}}\mid \bm{\mathsf{\chi}}_u) - \bm{\mathsf{y}}_u$. Compared with Proposition 1, the main difference is that the eNTK term also depends on the responses $\bm{\mathsf{y}}_u$ and $\bm{\mathsf{y}}_o$, which allows us to answer questions like

When tracking the model's confidence on different responses given the question $\bm{\mathsf{x}}_u$, learning from $\bm{\mathsf{y}}_u^+$ will impose a strong "upwards" pressure on $\bm{\mathsf{y}}_u^+$, as illustrated in the first panel of Figure 2. At the same time, the confidence of "similar responses" will also be slightly pulled up, like how learning a 4 influences the prediction on 9 in the MNIST example. We will discuss how to understand the "similarity" between two sequences of responses in the next section.

Preference finetuning, which teaches the model to provide responses that align better with human preferences, is also an important phase in LLM finetuning pipelines. Different from the SFT stage above, where the training tells the model "what to do", many preference finetuning methods also teach the model "what not to do, " which makes the learning dynamics more complex. For intuition, we start by analyzing a typical method, i.e., DPO (direct preference optimization, an RL-free method), under a similar framework. Following [11], the loss function of off-policy DPO is

where ${\bm{\mathsf{y}}_u^+}$ and ${\bm{\mathsf{y}}_u^-}$ are pre-generated responses, and $\pi_\text{ref}$ is the reference model, typically the result of SFT. $\bm{\mathsf{\chi}}_u^+$ and $\bm{\mathsf{\chi}}_u^-$ share the same $\bm{\mathsf{x}}$ but different $\bm{\mathsf{y}}$. In the loss function, the $\pi_{\theta^t}$ terms are also calculated using teacher forcing. Hence we can decompose the learning dynamics for DPO similarly to Equation 5,

where $a$ is the margin (i.e., the $\sigma(\cdot)$ value) for the $l$ -th token, which represents how well the current policy separates $\bm{\mathsf{y}}_{u}^+$ and $\bm{\mathsf{y}}_{u}^-$ compared with the reference policy. Due to the monotonicity of $\sigma(\cdot)$, a larger margin leads to larger $a$, which in turn restrains the strength of $\mathcal{G}^t_\text{DPO+/-}$. In other words, $\mathcal{G}^t_\text{DPO+/-}$ automatically provides less energy on the examples that are already well separated. We then check the role of $\beta$, which controls the regularizing effect on the KL distance between $\pi_{\theta^t}$ and $\pi_\text{ref}$ in the original RL loss ([11]). When the margin is negative, larger $\beta$ leads to a smaller $a$ and hence provides stronger $\mathcal{G}^t_\text{DPO+/-}$ for the model to "catch up" the separating ability of the reference model faster. But when the model is better and has a positive margin, increasing $\beta$ will increase $a$ and hence create a negative influence on $\beta(1-a)$, which makes the model update less. This aligns well with the claims of [11]: the stronger regularizing effect tends to "drag $\pi_{\theta}$ back towards $\pi_\text{ref}$ " when its predictions deviate from $\pi_\text{ref}$ too much. The derivation and the $\mathcal G^t$ functions for other RL-free methods are given in Appendix B.2.2.

These analyses make no assumptions on where $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ come from. Hence our framework can be directly extended to on-policy RL-free methods, which often perform better than their off-policy counterparts ([9,15]). The main difference between these algorithms is how the supervisory responses are generated. Off-policy methods typically use a fixed pre-collected dataset, with $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ are generated by another LLM or humans. In other words, it is likely that both the chosen and rejected responses come from the "less likely" region of the model's prediction. On-policy responses, though, are more likely to have higher predicted probabilities under this model, as they were sampled from it. We will show next that imposing large negative pressure on an unlikely prediction will lead to unexpected behavior, giving a new explanation for why on-policy sampling is important for algorithms with large negative gradients.

As demonstrated by the first two panels in Figure 2, the use of large negative gradients is the main difference between the learning dynamics of SFT and DPO. We will show later that this difference is the key to understanding why the learning curves of SFT and DPO behave so differently. For example, [8,9,26] (and our Figure 4) reported that the confidence of both $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ gradually decreases when conducting DPO, while the confidence of $\bm{\mathsf{y}}_u^+$ rarely drops during SFT. This trend becomes more serious if $\pi_{\theta^0}$ is finetuned for longer before conducting DPO (reported in Figure 3 of [8] and verified by our Figure 17). Furthermore, we also find that for all the $\pi_{\theta^t}(\bm{\mathsf{y}}\mid \bm{\mathsf{\chi}}_u)$ we track (various responses similar to $\bm{\mathsf{y}}_u^+$ or $\bm{\mathsf{y}}_u^-$; details in the next section), none of them increase during the DPO phase. This is different from SFT and quite counter-intuitive:

To answer this question, we first identify a phenomenon we call the squeezing effect, which occurs when using negative gradients from any model outputting a distribution with $\operatorname{\mathsf{Softmax}}$ output heads, even in a simple multi-class logistic regression task. Specifically, in the $L=1$ case when we impose a negative gradient on label $\bm{\mathsf{y}}_u^-$, we can describe the changing of model's predictive distribution $\pi_{\theta^{t+1}}$ as follows:

Appendix E illustrates the squeezing effect and analytically proves its existence for logistic regression models, by directly computing $\pi_{\theta^{t+1}} / \pi_{\theta^t}$ in different situations. Section 4.2 also experimentally verifies the analysis above in real LLM experiments.

Note that in practical settings, where both positive and negative pressures and the auto-regressive nature are strictly considered, the squeezing effect can become more complicated. The differences between the two eNTK terms in Equation 7 also influence the relative strength and direction of these two pressures. [27] analyze a similar problem in a token-level setting, and their conclusions align with ours well. We left a more detailed analysis to our future work.

We can now get a high-level overview of the learning dynamics of a typical off-policy DPO algorithm. Since both $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ are not sampled from the model's distribution, $\bm{\mathsf{y}}^*$ sometimes can be dissimilar to $\bm{\mathsf{y}}_u^+$, and the $\bm{\mathsf{y}}_u^-$ are likely located in a valley region of the model's prediction. Then its learning dynamics would look like the sketch in the second panel of Figure 2: the confidence on almost all $\bm{\mathsf{y}}$ are pushed down. At the same time, all the decreased probability mass is squeezed to $\bm{\mathsf{y}}^*$, which might make the model keep generating repeated phrases, as reported by [13]. Variants of DPO algorithms often unintentionally mitigate this squeezing effect by constraining the strength of the negative gradient or the positions of $\bm{\mathsf{y}}_u^-$, which partially explains their benefits. The last four panels in Figure 2 and Appendix B.2.2 have more details.

We now verify our analysis in practical settings. We first create the training set $\mathcal{D}_\text{train}$ by randomly selecting 5000 examples from the training split of the dataset. We consider two common datasets, Antropic-HH ([28]) and UltraFeedback ([29]), in all experiments. Each example in $\mathcal{D}_\text{train}$ contains three components: the prompt (or question) $\bm{\mathsf{x}}$, the preferred response $\bm{\mathsf{y}}^+$, and the less preferred response $\bm{\mathsf{y}}^-$. SFT finetunes with $\bm{\mathsf{x}}$ and $\bm{\mathsf{y}}^+$, while DPO uses all three (subscripts of $\bm{\mathsf{x}}$ and $\bm{\mathsf{y}}$ are removed for conciseness). We repeat the experiments on two series of models: pythia-410M/1B/1.4B/2.8B ([30]) and Qwen1.5-0.5B/1.8B ([31]).

To get a more detailed observation of the learning dynamics, we further create a probing dataset $\mathcal{D}_\text{prob}$ by randomly selecting 500 examples from $\mathcal{D}_\text{train}$, and generate several typical responses based on the corresponding $\bm{\mathsf{x}}$, $\bm{\mathsf{y}}^+$, or $\bm{\mathsf{y}}^-$. (We also study another probing dataset where all $\bm{\mathsf{x}}$ come from the test set in an ablation study in the appendix.) Then for each $\bm{\mathsf{x}}$ in $\mathcal{D}_\text{prob}$, we can observe how $\log\pi_{\theta^t}(\bm{\mathsf{y}}\mid \bm{\mathsf{\chi}})$ gradually changes on different types of $\bm{\mathsf{y}}$. For example, one extended response type can be a rephrase of $\bm{\mathsf{y}}^+$, an irrelevant response answering another question $\bm{\mathsf{x}}'$, or just a randomly generated English sentence with the same number of words with $\bm{\mathsf{y}}^+$. We explain why we need these extended responses and how they are generated in detail in Appendix D.1. In short, $\mathcal{D}_\text{prob}$ helps us to get a more fine-grained inspection of the learning dynamics, which can not only support our analysis above, but also shed more light on how the model's prediction evolves on the entire $\mathcal{Y}\in\mathbb{R}^{V\times L}$, a very sparse and huge space.

The main lesson we learn from the analysis in Section 3.1 is that learning from $\bm{\mathsf{y}}_u^+$ not only increases the model's confidence on $\bm{\mathsf{y}}_u^+$, but also indirectly "pulls up" responses similar to $\bm{\mathsf{y}}_u^+$ with a smaller strength (scaled roughly by $\|\mathcal{K}^t\|_F$), similar to how learning a '4`'' influences the prediction of a '9`'' in the MNIST example. At the same time, the increase of $\pi_{\theta^t}(\bm{\mathsf{y}}_u^+| \bm{\mathsf{\chi}}_u)$ naturally "pushes down" all $\bm{\mathsf{y}}\neq \bm{\mathsf{y}}_u^+$, because the model's predicted probability to all responses in $\mathcal{Y}$ -space must sum to one. The model's behavior on different $\bm{\mathsf{y}}$ is mostly a trade-off among these pressures. To verify this claim, we finetune the model for several epochs and evaluate the model's prediction on all responses in $\mathcal{D}_\text{prob}$ every 25 updates (with a training batch size of 4, the probing occurs every 100 examples). For each type of response, we average the model's confidence on all 500 examples and report the mean value of their log-likelihood.

As demonstrated in the first panel of Figure 3, the model's confidence on $\bm{\mathsf{y}}_u^+$ keeps increasing throughout the whole learning process, which is straightforward because the main "pull-up" pressure is imposed directly on $\bm{\mathsf{y}}_u^+$. However, the behavior of some responses similar to $\bm{\mathsf{y}}_u^+$ is non-trivial. For example, we draw the following types of responses in the same panel, i.e., the less preferred response for the same question ($\bm{\mathsf{y}}_u^-$), two types of rephrases of $\bm{\mathsf{y}}_u^+$ generated by ChatGPT ($\bm{\mathsf{y}}_\text{gpts}^+$ and $\bm{\mathsf{y}}_\text{gptf}^+$), another preferred response randomly selected from the test set ($\bm{\mathsf{y}}_\text{test}^+$), or even a randomly generated English sentence ($\bm{\mathsf{y}}_\text{hum}$). The model's confidence in these responses are all slightly increased at the beginning of training, and then gradually decrease as the training goes on, even though the model never sees them during SFT. This counter-intuitive behavior can be well explained by the learning dynamics we discussed before. Since all these examples are "similar" to $\bm{\mathsf{y}}_u^+$ to some extent (at least, they are all common "standard English" sentences), their $\|\mathcal{K}^t\|_F$ are reasonably large. Then learning $\bm{\mathsf{y}}_u^+$ will indirectly increase the model's confidence of these similar $\bm{\mathsf{y}}$. That is why the corresponding $\pi_{\theta^t}(\bm{\mathsf{y}}| \bm{\mathsf{\chi}}_u)$ are slightly increased at the beginning of training. However, as the training goes on, the model's confidence on $\bm{\mathsf{y}}_u^+$ keeps increasing and the update energy, the norm of $\mathcal{G}_\text{SFT}^t$ in Equation 5, gradually decreases. That means the indirect "pull-up" pressures are also diminished accordingly. Then, the "push-down" pressure on all $\bm{\mathsf{y}}\neq \bm{\mathsf{y}}_u^+$ becomes dominant and all the related curves start going down.

To verify the existence of this global "push-down" pressure, we observe two types of responses; both have the same number of words as their $\bm{\mathsf{y}}_u^+$. One is a purely random English word sequence $\bm{\mathsf{y}}'_\text{rnd}$. Another is a random permutation of all the words in $\bm{\mathsf{y}}_u^+$, which is called $\bm{\mathsf{y}}_\text{urnd}^+$. Since both are not natural language, we expect the $\|\mathcal{K}^t\|_F$ between them and $\bm{\mathsf{y}}_u^+$ to be very small, which means learning from $\bm{\mathsf{y}}_u^+$ imposes almost no "pull-up" pressure on them; thus the "push-down" pressure will dominate through the whole training procedure. These analyses are well supported by the second panel in Figure 3, in which we see these $\pi_{\theta^t}(\bm{\mathsf{y}}| \bm{\mathsf{\chi}}_u)$ all start from a very small value, and keep decreasing throughout the training.

Another interesting type of responses is $\bm{\mathsf{y}}_{j\neq u}^+$, a preferred response for another question $\bm{\mathsf{x}}_{j\neq u}$ in the training set. For these responses, the model's prediction on $\pi_{\theta^t}(\bm{\mathsf{y}}_{j\neq u}^+ | \bm{\mathsf{\chi}}_u)$ will be kept influenced by two "pull-up" pressures: one is from learning $[\bm{\mathsf{x}}_u; \bm{\mathsf{y}}_u^+]$, another is from learning $[\bm{\mathsf{x}}_{j\neq u}; \bm{\mathsf{y}}_{j\neq u}^+]$, where the latter might be even stronger as the gradient is directly calculated by observing $\bm{\mathsf{y}}_{j\neq u}^+$. That explains why we see the confidence on $\bm{\mathsf{y}}_{j\neq u}^+$ keeps increasing with a smaller rate compared with $\bm{\mathsf{y}}_u^+$ in the third panel. Because the "pull-up" pressure is always strong enough to counter the "push-down" one. These observations provide us with a unique explanation of why specific types of hallucinations are amplified after SFT. Specifically, the increase of $\pi_{\theta^t}(\bm{\mathsf{y}}_{j\neq u}^+ | \bm{\mathsf{\chi}}_u)$ means if we ask the model to answer a question $\bm{\mathsf{x}}_u$, it might provide a response from (or partially from) another unrelated question $\bm{\mathsf{x}}_{j\neq u}$ in the training set.

Last, to further explore the "similarity" between different responses from the model's perspective. we SFT the model using more types of responses and observe how $\pi_\theta(\bm{\mathsf{y}}'\mid \bm{\mathsf{\chi}}_u)$ changes accordingly. The results are demonstrated in Figure 3, where the blue and orange colors represent the positive and negative influence respectively. The x-axis is the updating response while the y-axis denotes the observing response. Hence the first column resembles how different $[\bm{\mathsf{x}}_u; \bm{\mathsf{y}}']$ changes when we SFT the model using $[\bm{\mathsf{x}}_u; \bm{\mathsf{y}}_u^+]$. One interesting finding is that all responses generated by ChatGPT are considered very similar to each other, regardless of how semantically different they are. Probably, LLM has its preferred idioms or phrases, which could be considered as a type of "fingerprint". We left this interesting problem for our future work.

To verify our framework also explains the model's behavior in preference tuning, we conduct similar experiments for DPO. Recall the residual term $\mathcal{G}^t_\text{DPO}$ introduces a pair of arrows on both $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$, with different directions. To show how these two pressures influence the model, we check two types of rephrases of $\bm{\mathsf{y}}_u^+$ or $\bm{\mathsf{y}}_u^-$ ($\bm{\mathsf{y}}_\text{gpts}^+$, $\bm{\mathsf{y}}_\text{gptf}^+$, $\bm{\mathsf{y}}_\text{gpts}^-$, and $\bm{\mathsf{y}}_\text{gptf}^-$, used in the previous experiment). See the three curves in the first panel in Figure 4, where the two rephrases decrease at a similar speed, faster than the decay of $\bm{\mathsf{y}}_u^+$. That is because the upward pressure is directly imposed on $\bm{\mathsf{y}}_u^+$ rather than these rephrases. Similarly, in the second panel, we observe that $\bm{\mathsf{y}}_u^-$ decays faster than its rephrases, because $\mathcal{G}^t_\text{DPO}$ directly imposes a negative pressure on $\bm{\mathsf{y}}_u^-$. Then in the third panel, we find the rephrases of $\bm{\mathsf{y}}_u^+$ consistently decay slower than those of $\bm{\mathsf{y}}_u^-$, although none of them ever occur during training. That is because these responses are close to $\bm{\mathsf{y}}_u^+$ or $\bm{\mathsf{y}}_u^-$ in $\mathcal{Y}$, which means their $\|\mathcal{K}^t\|_F$ is relatively large. Hence the pressures imposed on $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ also introduce a non-negligible influence on them. Last, in the fourth panel, the margin $\pi_{\theta^t}(\bm{\mathsf{y}}_u^+| \bm{\mathsf{\chi}}_u)-\pi_{\theta^t}(\bm{\mathsf{y}}_u^-| \bm{\mathsf{\chi}}_u)$ keeps increasing, which means the model is gaining the ability to separate $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ as the training goes on.

Although $\mathcal{G}^t_\text{DPO}$ directly imposes a "pull-up" pressure on $\bm{\mathsf{y}}_u^+$, the value of $\pi_{\theta^t}(\bm{\mathsf{y}}_u^+ | \bm{\mathsf{\chi}}_u)$ does not increase a lot as it does in SFT. The downward arrow on $\bm{\mathsf{y}}_u^-$ indeed introduces a "push-down" pressure on responses that are similar to $\bm{\mathsf{y}}_u^-$, but the influence is unlikely to be that strong (it will be weakened by $\|\mathcal{K}^t\|_F$) to make the confidence on almost every observing responses decrease so fast, as demonstrated in the last panel of Figure 3 where we use similar $\eta$ for both SFT and DPO. Then, where has the probability mass gone during DPO? The key to answering this question is the squeezing effect discussed in Section 3.3: since the big negative gradient is imposed on $\bm{\mathsf{y}}_u^-$, which is at this point probably in a region of low $\pi_{\theta^t}(\bm{\mathsf{y}}| \bm{\mathsf{\chi}}_u)$, the confidence of most $\bm{\mathsf{y}}$ will be decreased and $\pi_{\theta^t}(\bm{\mathsf{y}}^*| \bm{\mathsf{\chi}}_u)$ will increase very fast.

To verify this, we report the log-likelihood of $\bm{\mathsf{y}}$ chosen by greedy decoding: each token is chosen by maximizing the conditional probability given $[\bm{\mathsf{x}}_u; \bm{\mathsf{y}}_{<l}^+]$ in real-time, where $\bm{\mathsf{y}}_{<l}^+$ is a sub-sequence of $\bm{\mathsf{y}}_u^+$. As illustrated by the last panel of Figure 4, the confidence of this "teacher forcing" greedy $\bm{\mathsf{y}}$ increases very fast (from -113 to -63), which is even faster than the increase of $\pi_{\theta^t}(\bm{\mathsf{y}}_u^+| \bm{\mathsf{\chi}}_u)$ during SFT (from -130 to -90), within 8 epochs. However, the tokens with the highest confidence do not necessarily form a preferred response: it will reinforce the prior bias in $\theta^0$. This could be a reasonable explanation of the "degeneration" reported in recent work (e.g. [13]): as $\pi_{\theta^t}$ becomes more peaky at its most confident predictions, it is easier to sample sequences with repeated phrases. Note that such behavior could also be understood as a special type of self-bias amplifying ([32]), which would bring more serious consequences if it is combined with a multiple-generation self-improving algorithm, e.g., self-reward ([33]), iterative DPO ([34]), etc.

In summary, the behaviors of different types of responses all match our analyses well. More subtle trends of different responses support our story well (both for SFT and DPO). Due to space constraints, we explain these (and the full results on other models and datasets) in Appendix D.

Since the "squeezing effect" caused by the big negative gradient on unlikely predictions can damage the model's performance during DPO, we can first train the model on both $[\bm{\mathsf{x}}_u; \bm{\mathsf{y}}_u^+]$ and $[\bm{\mathsf{x}}_u; \bm{\mathsf{y}}_u^-]$ during the SFT stage (making the negative response more likely), and then run the usual DPO. Following the analysis above, we can expect during this new SFT stage, the region of those responses similar to $\bm{\mathsf{y}}_u^+$ or $\bm{\mathsf{y}}_u^-$ will be "pulled up" simultaneously. This is what we want because in many cases, both $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ are reasonably good responses for the question $\bm{\mathsf{x}}_u$; the new SFT design hence helps to pull up a larger region that contains more suitable responses compared with the baseline SFT. After that, the "push-down" pressure imposed during DPO can efficiently decrease the model's confidence on $\bm{\mathsf{y}}_u^-$ and its similar responses. Since $\bm{\mathsf{y}}_u^-$ is no longer so unlikely before DPO, the squeezing effect should not be as strong as in the baseline procedure.

We call our training pipeline "extend" and compare its learning dynamics with the baseline setting in Figure 5. It is clear that the squeezing effect is mitigated, because the confidence of other responses all decays slower during DPO, and we also observe a big drop in the greedy-decoding response when DPO starts. To further show that mitigating the squeezing effect indeed brings benefits, we compare the responses generated by models trained using different methods by feeding them to ChatGPT and Claude3. Specifically, we first SFT the model for two epochs using two methods discussed above and call the resulting policy network $\pi_\text{base}$ and $\pi_\text{extend}$. Then, we conduct identical DPO training on both $\pi_\text{base}$ and $\pi_\text{extend}$ for several epochs. The win rate of the proposed method against the baseline one is provided in Table 1. It is clear that before DPO, $\pi_\text{base}$ is better, because $\pi_\text{extend}$ is explicitly trained on those $\bm{\mathsf{y}}^-$. However, the $\pi_\text{extend}$ performs better after DPO several epochs since the squeezing effect is efficiently mitigated. Please refer to Appendix F for more details. In the future, this simple method inspired by our analysis could be further improved by introducing more responses, e.g., rephrases of $\bm{\mathsf{y}}_u^+$, etc., during both stages, and also by combining with many existing RL-free methods we mentioned before.

Learning dynamics, which depict how the model's prediction changes when it learns new examples, provide a powerful tool to analyze the behavior of models trained with gradient descent. To better utilize this tool in the context of LLM finetuning, we first derive the step-wise decomposition of LLM finetuning for various common algorithms. Then, we propose a unified framework for understanding LLM predictions' behaviors across different finetuning methods. The proposed analysis successfully explains various phenomena during LLM's instruction tuning and preference tuning, some of them are quite counter-intuitive. We also shed light on how specific hallucinations are introduced in the SFT stage, as previously observed ([35]), and where the improvements of some new RL-free algorithms come from compared with the vanilla off-policy DPO. The analysis of the squeezing effect also has the potential to be applied to other deep learning systems which apply big negative gradients to already-unlikely outcomes. Finally, inspired by this analysis, we propose a simple (but counter-intuitive) method that is effective in improving the alignment of models.

This research was enabled in part by support provided by the Canada CIFAR AI Chairs program, WestGrid, and Compute Canada. We thank Shangmin Guo, Noam Razin, Wonho Bae, and Hamed Shirzad for their valuable discussions and feedback. We also appreciate the constructive comments from the anonymous reviewers, which helped improve this work.

Beyond their application to LLMs, learning dynamics are widely utilized in analyzing various machine learning problems. For example, if we consider $\textcolor{cyan}{\bm{\mathsf{x}}_u}$ from the training set, and $\textcolor{orange}{\bm{\mathsf{x}}_o}$ from the test set, this form of learning dynamics provides a new perspective on generalization: the model generalizes better if the loss of $f_\theta(\textcolor{orange}{\bm{\mathsf{x}}_o})$ keeps decreasing when it learns from $\textcolor{cyan}{\bm{\mathsf{x}}_u}$. By studying the influence of different $\textcolor{cyan}{\bm{\mathsf{x}}_u}$ at different stages during supervised learning, [1] explain a "zigzag" pattern of the learning path,

which sheds light on why the model can spontaneously pursue better supervisory signals and correct noisy labels in the early stage of training (see also [36]). [37,3] apply learning dynamics to explain why directly finetuning a well-trained backbone with a randomly initialized task head might harm the out-of-distribution generalization ability. [38,39,40] also explains where the simplicity bias favoring compositional representations comes from during knowledge distillation ([41]), providing a new perspective of understanding why successive knowledge transferring can improve the model's systematic generalization ability.

The network's local elasticity ([17]) and stiffness ([16]) are also correlated with this topic. It reveals that neural networks operate like adaptive local learners, influencing only nearby points in feature space during training. This gives them a unique edge over linear models in terms of memorization, stability, and the emergence of meaningful internal structure—all without explicit regularization. The authors of [18] further link this behavior to the model's generalization ability. Extending their theoretical framework to more complicated settings like LLMs' finetuning might be a promising direction.

Besides explaining the model's behavior, learning dynamics is also helpful for evaluating the quality or the effectiveness of different training samples. For example, [4] propose a quantitative metric called TracIn to compute the influence of a training example on the predictions made by the model. This metric is then applied by [5] to search for the most influential examples in LLM instruction finetuning. By expanding Equation 1 in the neural tangent kernel (NTK) regime, [42] propose a metric called lpNTK to measure the relative difficulty among different training samples. These metrics and analyses inspired by learning dynamics are expected to be helpful in many related fields, like coreset selection ([43]), active learning ([44]) (see, e.g., [45]), and dataset distillation ([46]).

In this paper, we broadly define finetuning as any in-weight learning on top of a pretrained base model, including supervised finetuning (SFT), direct policy optimization (DPO, [11]) and its variants, etc. Since the analysis throughout this paper relies on the "teacher forcing" mechanism and the relatively stable eNTK assumption, our framework cannot be directly applied to algorithms with token-wise supervision like reinforcement learning with human feedback (RLHF, [6]) and proximal policy optimization (PPO, [12]). We leave the study of the token-wise learning dynamics, which aligns better with the "squeezing effect" in real settings, to future work.

We also identify several related works that report similar observations on the phenomena discussed in this paper. For example, [47,35] mentioned that learning new facts during SFT tends to make the model hallucinate more, which aligns with our finding that the model tends to use $\bm{\mathsf{y}}_{j\neq i}^+$ when answering question $i$. [13] related the peakiness of the model's distribution to LLM's "repeater phenomena", which also indirectly supports our claims well: more DPO leads to a more serious squeezing effect, hence the model's prediction becomes peakier on most tokens, which makes the aforementioned phenomena more common.

Furthermore, the "confidence decaying on $\bm{\mathsf{y}}_u^+$ " attracts more attention in the community, because it is quite counter-intuitive and the vanilla off-policy DPO algorithm works reasonably well in most cases. Many related works study this phenomenon by analyzing the major discrepancy between off-policy DPO and PPO, i.e., where the samples used to train the model comes from, e.g., [8,48,15]. They showed that when the responses are off-policy sampled, the learning process may fail to benefit from the contrastive information in the data. In other words, we should be more careful when working on the "valley" region of the model's distribution. Other works try to analyze this problem by inspecting the token-level influence between responses. For example, [26] assumes $\bm{\mathsf{y}}_u^+$ and $\bm{\mathsf{y}}_u^-$ are identical expect one token. Under this assumption, the model's confidence of $\bm{\mathsf{y}}_u^+$ after the identical token is guaranteed to decrease. They propose a solution by significantly enhancing the learning rate (roughly x50 larger when their $\lambda=50$) of the positive part when detecting $\bm{\mathsf{y}}_u$ located in a low-confidence region. [27] takes the similarity between the hidden embeddings and the geometry of the readout layer of different responses into account. Most of the conclusions of their paper align with ours well. The main discrepancy lies in the squeezing effect part, which we will discuss in our future work (they do not contradict each other, but need a more detailed analysis to understand the whole story).

The "squeezing effect" can negatively impact our analysis when it is strongly imposed in a valley region of the model. However, a well-regulated negative gradient is both beneficial and commonly observed in many deep-learning systems. For example, it is common in many "machine unlearning" algorithms, e.g., in [49]. Moreover, even in the field of LLM finetuning, we can find many mechanisms in different popular algorithms that can mitigate this effect. For example, the typical learning rate of DPO is usually smaller than that used in SFT, which unintentionally mitigates the harmful squeezing effect. The on-policy counterpart of the DPO-like algorithms is shown to perform better than their off-policy counterparts, which also supports our claims. Furthermore, we find the PPO loss automatically avoids imposing a big negative gradient (when its $\hat{A}_t$ is negative) on the valley region (when its $\pi_\theta$ is small).

On the other hand, the effect that negative gradients make the model's distribution peakier is independently reported in many related works. For example, Equation 1 in [50] shows that we are minimizing a negative thing in a standard GAN loss, which might explain why peakiness occurs. Furthermore, in Table 1 and Table 2 of [51], we see the peakiness (measured by $\Delta p_{top10}, \Delta p_{mode}$) of the “PG-average” method is stronger than the standard PG method. Note that the “PG-average” method will map a reward ranging from 0 to 1 to a centered one ranging from -0.5 to 0.5. Since the negative reward can introduce a negative gradient, the peakiness increases.

Proof: ¹ Suppose we want to observe the model's prediction on an "observing example" $\textcolor{orange}{\bm{\mathsf{x}}_o}$. Starting from Equation 2, we first approximate $\log \pi^{t+1}(\bm{\mathsf{y}}\mid \textcolor{orange}{\bm{\mathsf{x}}_o})$ using first-order Taylor expansion (we use $\pi^t$ to represent $\pi_{\theta^t}$ interchangeably for notation conciseness):

Then, assuming the model updates its parameters using SGD calculated by an "updating example" $(\textcolor{cyan}{\bm{\mathsf{x}}_u}, \textcolor{cyan}{\bm{\mathsf{y}}_u})$, we can rearrange the terms in the above equation to get the following expression:

where $d$ is the number of parameters of the model. To evaluate the leading term, we plug in the definition of SGD and repeatedly use the chain rule:

For the higher-order term, using as above that

and noting that, since the residual term $\mathcal{G}^t$ is usually bounded (and the practical algorithms will also use gradient clip to avoid too large gradient), we have that

In the decomposition, using $\{\pi_1, \dots, \pi_V\}$ to represent the model's prediction on different dimensions, we can write our $\mathcal{A}^t$ as:

The second term in this decomposition, $\mathcal{K}^t(\textcolor{orange}{\bm{\mathsf{x}}_o}, \textcolor{cyan}{\bm{\mathsf{x}}_u})$, is the product of gradients at $\textcolor{orange}{\bm{\mathsf{x}}_o}$ and $\textcolor{cyan}{\bm{\mathsf{x}}_u}$. Intuitively, if their gradients have similar directions, the Frobenius norm of this matrix is large, and vice versa. This matrix is known as the empirical neural tangent kernel, and it can change through the course of training as the network's notion of "similarity" evolves. For appropriately initialized very wide networks trained with very small learning rates, $\mathcal K^t$ remains almost constant during the course of training, the kernel it converges to is known as the neural tangent kernel ([19,20]). Note that the assumption that $\mathcal{K}^t(\textcolor{orange}{\bm{\mathsf{x}}_o}, \textcolor{cyan}{\bm{\mathsf{x}}_u})$ is unchanged (usually used in theoretical analysis) might be too strong in the LLM's finetuning. Hence as stated in the main context, our qualitative analysis only assumes that "during the training, the relative influence of learning $\bm{\mathsf{x}}_u$ on all other different $\bm{\mathsf{x}}_o$ is relatively stable". We will validate this assumption using experiments in Appendix C.

As stated in Section 3, one of the conundrums of decomposing the learning dynamics of LLM is its auto-regression nature of the output sequence. Different from the multi-label classification problem, where $y_l$ for different $l$ is independently generated as long as the shared network is fixed, the $y_l$ for the LLM's output depends on $\bm{\mathsf{y}}_{<l}$, which is usually sampled from the model's prediction iteratively. However, in most of the finetuning cases where the supervisory signal $\textcolor{cyan}{\bm{\mathsf{y}}_u}$ is given, the model will apply the so-called "teacher forcing" mechanism when calculating the predicting probabilities. In other words, when generating the output of each $y_l$, the $\bm{\mathsf{y}}_{<l}$ is given rather than sampled on-policy. This mechanism makes it possible for us to define $\bm{\mathsf{\chi}}=[\bm{\mathsf{x}}; \bm{\mathsf{y}}]$ and hence merge the auto-regressive nature of the sequence prediction into the shared $\mathcal{K}^t(\textcolor{orange}{\bm{\mathsf{\chi}}_o}, \textcolor{cyan}{\bm{\mathsf{\chi}}_u})$. After this step, the decomposition of LLM's finetuning learning dynamics then becomes similar to a multi-label classification task.

Here we derive the residual term, i.e., $\mathcal{G}^t$ for different algorithms in LLM's finetuning. We first rewrite Equation 5 here:

where $m\in\{1, \dots, M\}$, $l\in\{1, \dots, L\}$, and $\mathcal{G}^t(\textcolor{cyan}{\bm{\mathsf{\chi}}_u})=\nabla_{\bm{\mathsf{z}}}\mathcal{L}(\textcolor{cyan}{\bm{\mathsf{\chi}}_u})|_{\bm{\mathsf{z}}^t}$ is a $V\times L$ matrix. As the auto-regression nature of the SFT loss is already encoded in the causal mask used in $h_\theta$, as demonstrated in Figure 10a. the columns in $\mathcal{G}^t(\textcolor{cyan}{\bm{\mathsf{\chi}}_u})$ are independent of each other, which can be separately calculated. Plus, the summation over $l$ can also be achieved by left-multiplying a length- $L$ all-one vector $\bm{\mathsf{1}}$. Specifically, the SFT loss for each $l$ is:

where $y_u^+$ is for the $l$ -th dimension of $\bm{\mathsf{y}}_u^+$. The gradient of $\mathcal{L}$ on $\bm{\mathsf{z}}$ can be then calculated as:

where $\oslash$ is element-wise division.

To calculate the equation above, we first recall the NLL loss of the $l$ -th token is $[\mathcal{L}_\text{SFT}] _l\triangleq\mathcal{L}=-\log\pi(y_l=y_l^+)=- \bm{\mathsf{e}}_{y_l^+}^\top\log\pi$, where $\pi=\mathsf{Softmax}(\bm{\mathsf{z}})$. Then, $\underbrace{\nabla_{\bm{\mathsf{z}}}\mathcal{L}}_{1\times V}=\underbrace{\nabla_{\pi}\mathcal{L}}_{1\times V}\underbrace{\nabla_{\bm{\mathsf{z}}}\pi}_{V\times V}$. For each dimension of $\nabla_{\bm{\mathsf{z}}}\mathcal{L}_l$, we have $\frac{\partial{\mathcal{L}}}{\pi_i}=0$ if $\pi_i\neq y_l^+$ and $\frac{\partial{\mathcal{L}}}{\pi_i}=-\frac{1}{\pi_i}$ if $\pi_i=y_l^+$. By writing it in vector form, we have $\nabla_{\bm{\mathsf{z}}}\mathcal{L}=-(\bm{\mathsf{e}}_{y_l^+}\oslash\pi)^\top\nabla_{\bm{\mathsf{z}}}\pi$. For $\nabla_{\bm{\mathsf{z}}}\pi$, we have:

Combining this matrix and the $1\times V$ vector $(\bm{\mathsf{e}}_{y_l^+}\oslash\pi)^\top$, where the only non-zero term is $\frac{1}{\pi_k}$ at the $k=y_l^+$ position. So, left multiplying by this vector is actually first selecting the $k$ -th row of $\nabla_{\bm{\mathsf{z}}}\pi$, and then multiplying $\frac{1}{\pi_k}$ to it. In summary, we have:

By stacking the terms with different $l\in [L]$, we can get

Direct Preference Optimization (DPO, [11]) is usually considered the first RL-free alignment algorithm for preference finetuning. Different from the standard RLHF (reinforcement learning with human feedback ([52])), the training of off-policy DPO is more similar to SFT, where the model keeps learning from a pre-generated preference dataset. Hence, we start from DPO to analyze the learning dynamics of different preference finetuning algorithms (the on-policy versions of these algorithms could also be explained using the proposed framework).

Following [11], the training loss of DPO is:

Before calculating the residual term $\mathcal{G}^t_\text{DPO}$, we need to re-calculate the learning dynamics decomposition, because the loss term now depends on both $\pi_{\theta^t}(\bm{\mathsf{y}}_u^+ \mid \bm{\mathsf{\chi}}_u^+)$ and $\pi_{\theta^t}(\bm{\mathsf{y}}_u^- \mid \bm{\mathsf{\chi}}_u^-)$, which involves two different $\bm{\mathsf{z}}$ terms. Specifically, we define $\pi_{\theta^t}(\bm{\mathsf{y}}_u^+ \mid \bm{\mathsf{\chi}}_u^+)= \operatorname{\mathsf{Softmax\_column}}(\bm{\mathsf{z}}^+)$ and $\pi_{\theta^t}(\bm{\mathsf{y}}_u^- \mid \bm{\mathsf{\chi}}_u^-)= \operatorname{\mathsf{Softmax\_column}}(\bm{\mathsf{z}}^-)$, where $\bm{\mathsf{z}}^+=h_{\theta}(\bm{\mathsf{\chi}}_u^+)$ and $\bm{\mathsf{z}}^-=h_{\theta}(\bm{\mathsf{\chi}}_u^-)$ respectively ($\bm{\mathsf{\chi}}_u^+=[\textcolor{cyan}{\bm{\mathsf{x}}_u}; \bm{\mathsf{y}}_u^+]$ and $\bm{\mathsf{\chi}}_u^-=[\textcolor{cyan}{\bm{\mathsf{x}}_u}; \bm{\mathsf{y}}_u^-]$). Then, starting from $L=1$, the decomposition for the DPO loss (similar to Equation 8 for SFT) could be written as: