Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs cover

📄 Executive Summary

Reinforcement learning (RL), especially reinforcement learning with verifiable rewards, has become an essential technique for improving reasoning, safety, and alignment in large language models. Historically, practitioners have relied on the AdamW optimization algorithm because it is the standard for pretraining and supervised fine-tuning, despite its heavy memory footprint. However, RL introduces a fundamentally distinct learning environment where models sample their own data on the fly and receive sparse feedback. This non-stationary process raises the question of whether the complex, memory-intensive mechanisms within AdamW—specifically momentum and adaptive per-parameter learning rates—are genuinely necessary for RL.

The article evaluated whether basic Stochastic Gradient Descent (SGD), which tracks no historical optimizer states and uses substantially less memory, can match or exceed the performance of AdamW in reinforcement learning fine-tuning. The authors investigated the individual contributions of momentum and adaptive learning rates across multiple model families, tasks, and reinforcement learning algorithms.

To test this, the researchers conducted extensive empirical experiments using Qwen (1.7-billion and 8-billion parameters) and Llama (8-billion parameters) models across mathematical reasoning, competitive programming, and synthetic coding benchmarks. They tested two reinforcement learning frameworks—Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO)—and systematically ablated optimizer components by comparing standard AdamW, RMSProp (adaptive rates without momentum), SGD with momentum (momentum without adaptive rates), and vanilla SGD.

The analysis yielded several key findings. First, contrary to long-standing wisdom that SGD fails on large transformers, SGD matched or outperformed AdamW across all evaluated reinforcement learning benchmarks. In coding evaluations on the 1.7-billion parameter model, SGD produced an average accuracy improvement of 8.7 percentage points over AdamW. Second, momentum and adaptive learning rates provided no consistent benefit in RL; in fact, momentum frequently degraded performance because past gradient directions misalign with continuously shifting policy updates. Third, full model fine-tuning with SGD updated only 0.01% to 0.46% of parameters (over 99.9% update sparsity) without any explicit sparsity constraints, modifying over 1,000 times fewer parameters than AdamW. Finally, eliminating optimizer memory states in SGD delivered immediate hardware savings, reducing peak GPU memory usage by 15.7 gigabytes on a 1.7-billion parameter model.

These findings indicate that optimization principles established in supervised learning do not directly apply to reinforcement learning. By showing that reinforcement learning fine-tuning operates within a highly localized, low-dimensional parameter subspace, the results help explain why reinforcement learning suffers far less from catastrophic forgetting than supervised learning. Operationally, the dramatic memory reduction lowers computing costs and hardware barriers, enabling organizations to train larger models or utilize larger batch sizes without sacrificing accuracy.

Organizations training reasoning-focused language models should consider piloting SGD in place of AdamW for reinforcement learning workloads to capture substantial memory and efficiency gains. Practitioners adopting SGD must account for the requirement of significantly higher learning rates (typically on the order of 0.1 rather than the nominal 1e-6 used in AdamW) to achieve proper update scale. While the findings are robust across multiple model scales, algorithms, and extended training horizons, confidence is bounded by the evaluated model range (up to 8 billion parameters) and focus on verifiable reasoning tasks. Teams should validate SGD dynamics on very large frontier models and subjective alignment tasks before undertaking full-scale production migration.

📚 What to read before this

Open this section in ChapterPal to generate recommendations.

📚 What to read after this

Open this section in ChapterPal to generate recommendations.

Do We Need Adam?

Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs

Sagnik Mukherjee
University of at Illinois, Urbana-champaign, USA

Lifan Yuan
University of at Illinois, Urbana-champaign, USA

Pavan Jayasinha
University of Waterloo, Ontario, Canada

Dilek Hakkani-Tür
University of at Illinois, Urbana-champaign, USA

Hao Peng
University of at Illinois, Urbana-champaign, USA

Correspond to: Sagnik Mukherjee [email protected]

Keywords: Machine Learning, ICML

Abstract

Reinforcement learning (RL), particularly RL from verifiable reward (RLVR), has become a crucial phase of training large language models (LLMs) and a key focus of current scaling efforts. However, optimization practices in RL largely follow those of next-token-prediction stages (e.g., pretraining and supervised fine-tuning), despite the fundamental differences between RL and these stages emphasized by recent work. One such practice is the use of the AdamW optimizer, which is widely adopted for training large-scale transformers despite its high memory overhead. Our analysis shows that both momentum and adaptive learning rate of AdamW are less influential in RL than in SFT, leading us to hypothesize that RL benefits less from Adam’s per-parameter adaptive learning rates and momentum. Confirming our hypothesis, our experiments demonstrate that the substantially more memory-efficient SGD, which is known to perform poorly in supervised learning of large-scale transformers, matches or even outperforms AdamW in RL for LLMs. Remarkably, full fine-tuning with SGD updates fewer than 0.02% of model parameters without any sparsity-promoting regularization, more than 1,000$\times$ fewer than AdamW. Our analysis offers potential reasons for this update sparsity. Our findings provide fresh insights into the optimization dynamics of RL in LLMs and demonstrate that RL can be substantially more parameter-efficient than previously recognized. [^1]

[^1]: Code: [

https://github.com/SagnikMukherjee/sgd_adam_rlvr](https://github.com/SagnikMukherjee/sgd_adam_rlvr)

Executive Summary: Reinforcement learning from verifiable rewards has emerged as a critical stage for improving large language models on reasoning and alignment tasks, yet training practices continue to rely on the AdamW optimizer developed for next-token prediction stages such as supervised fine-tuning. This choice incurs substantial memory overhead from maintaining momentum and second-moment statistics, even though recent analyses indicate that RL produces sparser, lower-dimensional updates and operates in a non-stationary landscape where past gradient information may misalign with current gradients.

The article set out to test whether momentum and per-parameter adaptive learning rates remain necessary under RLVR, or whether the simpler SGD optimizer could achieve comparable results. Researchers ran controlled experiments across mathematical reasoning, coding, and adaptive verifiable environments using GRPO and PPO algorithms on Qwen and Llama model families ranging from 1.7B to 8B parameters, directly comparing SGD, SGD with momentum, RMSProp, and AdamW while measuring task performance, memory usage, and the sparsity and rank of parameter updates.

SGD matched or exceeded AdamW on nearly all benchmarks, with average pass@1 gains of roughly 8 percentage points on coding tasks and similar or higher scores on math evaluations; neither momentum nor adaptive rates consistently improved outcomes, and momentum frequently reduced performance. SGD produced extremely sparse updates, modifying fewer than 0.02 percent of parameters without any sparsity-inducing regularization—often more than 1,000 times fewer than AdamW—while also yielding lower effective rank updates. These patterns held under PPO and during extended training of 500 steps. The resulting optimizer-state savings reached 15.7 GB peak memory for the 1.7B model.

These results indicate that RLVR fine-tuning is far more parameter-efficient than previously assumed and that optimization heuristics validated on supervised regimes do not transfer directly. Practitioners can immediately reduce GPU memory requirements and potentially train larger models or larger batches by switching to SGD, while the observed sparsity offers a mechanistic explanation for reduced catastrophic forgetting in RL-trained models. The authors recommend adopting SGD for RLVR workloads and pursuing further research into optimizers explicitly designed for the distinct dynamics of reinforcement learning in language models.

The findings rest on experiments with models up to 8B parameters and three specific domains; results may differ at larger scales or with different reward structures, and SGD requires substantially higher nominal learning rates than AdamW. remains high within the tested settings but warrants verification on frontier-scale models and additional RL algorithms before broad adoption.

1. Introduction

Section Summary: Reinforcement learning with verifiable rewards has driven major gains in large language models on reasoning and alignment tasks, but it differs from standard supervised training because it continuously generates its own data and receives far sparser feedback per example. These traits create a simpler, lower-dimensional optimization problem in which the complex Adam optimizer commonly used for language models proves unnecessary. Simpler methods such as plain stochastic gradient descent match or exceed Adam’s performance while updating only a tiny fraction of parameters and cutting memory use substantially.

"The important thing is not to stop questioning."

— Albert Einstein

Reinforcement learning (RL) [1, 2, 3], particularly its verifiable-reward variant (RLVR; [4, 5]), has been a major driver behind the widely recognized success of large language models (LLMs) on complex reasoning tasks [6, 7, 8], as well as their alignment with human values and adherence to safety protocols [9]. Compared to other LLM training paradigms based on next-token prediction (NTP), such as supervised fine-tuning (SFT) and pretraining, RL constitutes a fundamentally different training regime.

Two key differences are particularly relevant. (1) Unlike SFT, online RL samples training data from the most recent version of the policy, causing both the data distribution and the effective optimization landscape to co-evolve with the policy throughout training. (2) RL updates incorporate only $O(1)$ bits of information from the environment per episode, substantially sparser than the $O(#\text{tokens})$ information in SFT ([10]). These differences have significant impact on the model behaviors as well as the training dynamics. [11, 12] demonstrate that RL-trained models generalize better than those trained with SFT, and [13] attribute RL's better generalization to reduced catastrophic forgetting of on-policy learning. [14] show that RL fine-tuning updates only about 20% of the parameters which are significantly sparser than those from SFT. Moreover, [15] show that RL updates concentrate in off-principal directions of the parameter space, while inducing only minimal spectral drift. Both findings suggest that the effective optimization problem in RLVR is both low-dimensional and geometrically constrained, with learning confined to a subspace of the parameter space.

\begin{tabular}{@ cccccc @}
\toprule
\textbf{Optimizer} & \textbf{Momentum} & \textbf{Adaptive LR} & \textbf{Final Update} & \textbf{Optim. State } \\
\midrule
SGD 
 {} & N/A 
 {} & N/A
 {} & $\theta_{t+1} = \theta_t - \eta g_t$ 
 {} & $O(n)$ \\[4pt]

SGD + Momentum 
 {} & $m_{t} = \mu m_{t-1} + g_t$
 {} & N/A
 {} & $\theta_{t+1} = \theta_t - \eta m_{t}$ 
 {} & $O(2n)$ \\[4pt]

RMSProp 
 {} & N/A
 {} & $v_{t} = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$
 {} & $\theta_{t+1} = \theta_t - \eta \frac{g_t}{\sqrt{v_{t}}+\varepsilon}$ 
 {} & $O(2n)$ \\[4pt]

AdamW
 {} & $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t$
 {} & $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$
 {} & $\theta_{t+1} = \theta_t - \eta \frac{m_t}{\sqrt{ v_t}+\varepsilon}$
 {} & $O(3n)$ \\

\bottomrule
\end{tabular}

These differences motivate a closer examination of optimization practices in RL for LLMs, which largely follow those established for NTP stages. Among them, perhaps the most important is the use of the Adam optimizer [16], in particular its AdamW variant ([17, 18, 19, 20]). Our analysis suggests that both momentum and per-parameter adaptive learning rates, the two key ingredients of AdamW, are less influential in RL than in SFT (§ 4.2).

These findings lead us to hypothesize that RLVR benefits less from AdamW than SFT, which is supported by our experimental results (§ 4): ablating from AdamW the first moment (effectively yielding RMSProp; [21]), or the second moment (yielding SGD with momentum), or both (yielding SGD) performs on par with or even stronger than AdamW.

Among these findings the most surprising one is the strong performance by SGD in RLVR: SGD has long been considered ill-suited for training large transformers [22, 23, 24, 25, 26] and only works under restrictive settings such as using a very small batch size [27]. Our findings suggest that these prior conclusions, usually drawn in supervised learning, may not fully carry over to RLVR for LLMs.

Beyond its strong performance, SGD in RLVR produces highly sparse parameter updates without any explicit regularization promoting sparsity (§ 5). Across three verifiable domains (namely mathematical reasoning, coding and RLVE; [28]), two model families (Qwen and Llama), and two RL algorithms (PPO and GRPO), SGD updates $0.02% - 0.46%$ of model parameters, which is sometimes nearly $500\times$ fewer than AdamW. Our analysis partially attributes SGD's sparser updates to its lack of adaptive learning rates (§ 5).

:::: {.figure cols="2"}

Figure 1: Training reward (left) and validation reward on MATH (right) comparing SGD and AdamW. ::::

Our findings yield several broader insights and implications. First, the pronounced update sparsity with SGD suggests that RL in LLMs can be highly parameter-efficient. The fact that only a small fraction of model parameters are updated offers a mechanistic perspective that complements prior work showing that RL suffers less from reduced catastrophic forgetting [12, 13, 11] and has a strong dependence on the capabilities of pretrained base models [29, 30, 31]. Second, our comparison between AdamW and SGD highlights that optimization decisions depend on the training regime, and that conclusions drawn from SFT may not carry over to RL. From a practical standpoint, forgoing AdamW's momentum terms yields immediate memory savings. For example, when training the Qwen3-1.7B model, SGD reduces GPU memory usage by 15.7 GB compared to AdamW without losing accuracy (§ 4.3). Collectively, our findings motivate further investigation into optimization techniques specifically tailored to RL for LLMs, particularly with respect to their potential to reduce forgetting and improve efficiency and scalability.

2. Background

Section Summary: This section reviews policy gradient methods used to train models by maximizing expected rewards from sampled outputs, along with the SGD, AdamW, and RMSProp optimizers that update model parameters during training. It explains how the policy gradient relies on a reward signal minus a baseline to reduce variance, noting that the sampled outputs create a shifting optimization problem and that each training step receives only limited new information from the environment. The discussion of optimizers contrasts their use of momentum and adaptive learning rates to identify which features matter most for effective reinforcement learning from rule-based rewards.

In this section, we review policy gradient methods [32] as well as the SGD, AdamW, and RMSProp optimizers, which provide the necessary background for later sections.

Policy Gradient

To optimize a policy $\pi_\theta$ that maximizes expected rewards, policy gradient methods derive updates by differentiating the objective. For a prompt $\mathbf{x}$, the gradient takes the form:

$ \nabla_\theta J(\theta) = \mathbb{E}{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi\theta}\left[(R(\mathbf{x}, \mathbf{y}) - b) \nabla_\theta \log \pi_\theta(\mathbf{y}|\mathbf{x})\right] $

$\theta$ denotes parameters of the policy $\pi_\theta$, and $R$ the return. and $b$ is a baseline for variance reduction, and can be instantiated in various ways: value function estimates, group-averaged returns, or leave-one-out statistics [33, 34]. Equation 1 highlights two key attributes of the RL objective: (1) the output trajectory $\mathbf{y}$ is sampled from the evolving policy, creating a non-stationary optimization landscape, and (2) the reward signal $R$ is a rule based reward gained from the environment. In this sense, each episode gains $O(1)$ bits of external information from the environment [10].

SGD, SGD with Momentum, RMSProp, and AdamW

The update rules and state requirements for these optimizers are summarized in Table 1. AdamW maintains two auxiliary states: the first moment (momentum) and the second moment (used for adaptive learning rates). SGD, in contrast, tracks no auxiliary state and has the simplest update rule. Intuitively, RMSProp ([21]) and SGD with momentum each retain exactly one of AdamW's components: RMSProp can be viewed as AdamW without momentum (or equivalently, SGD with adaptive learning rates), while SGD with momentum can be viewed as AdamW without adaptive learning rates.[^2] Hence, by comparing all four optimizers we can identify which component, if either, is essential for effective RLVR training.

[^2]: Up to bias correction and weight decay.

3. Do We Need Adam?

Section Summary: The section questions whether AdamW's core features—per-parameter adaptive learning rates and momentum—remain useful when training language models with reinforcement learning from verifiable rewards (RLVR), given that RL differs fundamentally from standard supervised fine-tuning. It shows that RLVR produces far less variation in gradient magnitudes across parameters than supervised training does, reducing the value of adaptive step sizes, and that its constantly shifting data distribution and loss landscape cause momentum to accumulate outdated or misaligned directions. These observations lead to the hypothesis that simpler optimizers without these mechanisms may work just as well, or better, for RLVR.

As we can see above, the main difference between AdamW and SGD consists of two components: momentum and adaptive learning rate. They are considered beneficial by default, as prior works have extensively demonstrated the superiority of AdamW over SGD. For example, [35] argued that adaptive methods provably outperform SGD under heavy-tailed stochastic gradient noise. [22] attributed AdamW's advantage to favorable directional sharpness properties. [23] went further, showing that among common optimizers, SGD uniquely underperforms others for LLM training. A common thread across these explanations is that transformers induce a complex, heterogeneous loss landscape, one with highly varying curvature across parameters, where per-parameter adaptivity becomes essential. However, in light of recent findings that suggest RL has a fundamentally different training dynamics [14, 15], we revisit this belief. And more specifically we ask:


Are adaptive learning rates and momentum needed for RLVR training ?

Adaptive Learning rate might not be required

AdamW adapts the learning rate for each parameter by normalizing updates using an exponential moving average of squared gradients $\sqrt{v}$.

$**Figure 2:** Comparison of $\sqrt{v}$ distributions between SFT and RLVR at step 50. RLVR concentrates in a narrower, low-magnitude regime. The standard deviation is ${\sim}22\times$ higher in SFT ($\sigma = 5.11 \times 10^{-6}$) than RLVR ($\sigma = 2.29 \times 10^{-7}$).$

This increases the effective step size for parameters with historically small gradients and decreasing it for those with larger ones. When $\sqrt{v}$ varies substantially across parameters, different parameters experience different effective step sizes. Conversely, if $\sqrt{v}$ is similar across parameters, tracking this auxiliary state confers little benefit beyond single global step size. We first compare the standard deviation in $\sqrt{v}$ between SFT and RLVR training runs on the same model using AdamW (Details in Appendix C). As shown in Figure 2, SFT exhibits approximately $22\times$ higher standard deviation in $\sqrt{v}$ compared to RLVR ($\sigma_{\text{SFT}} = 5.11 \times 10^{-6}$ vs. $\sigma_{\text{RL}} = 2.29 \times 10^{-7}$). This difference suggests that the second-moment, central to AdamW's adaptive learning rates may be far less load-bearing in RLVR than in SFT, motivating us to ablate it later in § 4.2.

Momentum could be counter-productive

Further, we make a crucial observation that RL is fundamentally non-stationary: both the data distribution and the loss landscape evolve throughout training as the policy updates [1]. Momentum computes a moving average of past gradients, encoding a memory of previous loss landscapes. However, when data distribution shifts with policy update, the optimization landscape may change substantially between updates, causing accumulated moment estimates to point in directions misaligned with the current policy gradient. This phenomenon ([36]) has been shown to hinder optimization in temporal-difference learning and policy gradient methods ([37, 38, 39]). Given that RLVR inherits this non-stationarity, the efficacy of momentum-based optimizers in this setting warrants careful investigation.

In order to empirically verify this, we computed cosine similarity between the accumulated momentum $m_{t-1}$ and current step's gradient $g_t$ in SFT and RL in the AdamW optimizer with the code setup discussed in § 4. Our analysis (in Appendix D) reveals a striking contrast: in SFT, gradient largely aligns with the accumulated momentum directionally, with a cosine similarity of 0.997 between $g_t$ and $m_{t-1}$; In contrast, in RL, the cosine similarity drops to near-zero ($-0.007$), suggesting substantially weaker directional alignment. These findings provide evidence that RL's non-stationary landscape can make momentum less effective. These two observations lead to our key hypothesis:


Momentum and adaptive learning rates are less essential in RLVR than in SFT.

4. Can SGD Match AdamW in RLVR?

Section Summary: This section tests whether basic stochastic gradient descent can perform as well as the more complex AdamW optimizer when training language models with reinforcement learning on verifiable tasks such as math and coding problems. Experiments across multiple model sizes and domains show that SGD matches or exceeds AdamW's results on standard benchmarks, even though it lacks momentum and adaptive learning rates. The findings also reveal that SGD creates much sparser parameter updates spread evenly across layers rather than concentrating changes in specific areas.

::: {caption="Table 2: Performance comparison of different optimizers on GRPO across model families. SGD achieves comparable or better performance than AdamW. Adaptive learning rates and momentum do not consistently improve over SGD."}

:::

This section empirically evaluates Hypothesis 1. If it holds, we expect SGD, which uses neither momentum nor adaptive learning rates, to achieve similar performance to AdamW (§ 4.1). We also examine the individual contributions of momentum and adaptive learning rates through additional comparisons with SGD with momentum and RMSProp (§ 4.2).

Experimental setup

RL Algorithm: We experiment with two widely-used RL algorithms: Group Relative Policy Optimization (GRPO; [33]) and Proximal Policy Optimization (PPO; [40]). Unless otherwise specified, results are reported using GRPO. Experiments in this section are performed with GRPO. PPO results are presented in § 6.
Domains: In order to ensure generalizability of our observations, we experiment across three domains: (1) mathematical reasoning, (2) coding, and (3) RL with Adaptive Verifiable Environments (RLVE; [28]), which contains LeetCode-style synthetic tasks and enables prolonged training.
Training datasets: For math, our training dataset comprises of the NuminaMath-CoT dataset [41] (randomly sampled 35K examples). For coding tasks we use the code split (all 25K samples) of the post-training dataset as used by [7], where the problems are sourced from APPS [42], CodeContests [43], TACO [44], and Codeforces [45]. Further, for RLVE, we use 260 out of 400 tasks, where each task starts with a difficulty level of 0 and automatically evolves to be more challenging throughout the training.
Base models: We experiment with Qwen3-1.7B, Qwen3-8B [46] and Llama-3.1-8B-Instruct [20] to study the robustness of our observation across model families and scales.
Evaluation: For math tasks, our evalaute on MATH-500, AMC [47], AIME [48, 49], OlympiadBench [50] and GPQA Diamond [51]. For OlympiadBench we used the $\textsc{OE_MM_maths_en_COMP}$ subset which is comprising of 675 competition level math questions in english. For coding tasks, we evaluate pass@1 and pass@10 on HumanEval [52], HumanEval+ [53], MBPP [54], and MBPP+ [53]. Pass@K metrics computed with a 0.2 temperature and 10 samples.

Additional details are provided in Appendix A.1. Additionally, SGD requires a much larger learning rate than AdamW, we provide a detailed discussion on that in Appendix A

:::: {.figure cols="2"}

Figure 3: SGD updates are distributed across the model rather than concentrated in specific layers. Across all layers, SGD produces significantly sparser updates than AdamW. ::::

\begin{tabular}{c c c c c c c c c c c c c}
\toprule
 {} & & & \multicolumn{2}{c}{\textbf{HumanEval}} & \multicolumn{2}{c}{\textbf{HumanEval+}} & \multicolumn{2}{c}{\textbf{MBPP}} & \multicolumn{2}{c}{\textbf{MBPP+}} & & \\
\cmidrule(lr){4-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9} \cmidrule(lr){10-11}
\textbf{Model} & \textbf{Res Len} & \textbf{Optim.} & @1 & @10 & @1 & @10 & @1 & @10 & @1 & @10 & $\Delta$@1 & $\Delta$@10 \\
\midrule
\multirow{4}{*}[6pt]{\textsc{Qwen 3 1.7b}} 
 {} & \multirow{2}{*}{4K} 
 {} & AdamW 
 {} & 44.0 & 54.9 & 37.0 & 47.0 & 39.3 & 64.4 & 46.8 & 72.2 & & \\
 {} & 
 {} & SGD 
 {} & \textbf{49.3} & \textbf{56.1} & \textbf{44.1} & \textbf{50.6} & \textbf{49.6} & \textbf{65.2} & \textbf{59.0} & \textbf{73.0} & +8.7 & +1.6 \\
\cmidrule(lr){2-13}
 {} & \multirow{2}{*}{8K} 
 {} & AdamW 
 {} & 43.7 & 54.9 & 37.0 & 46.3 & 40.9 & 65.2 & 49.0 & 73.5 & & \\
 {} & 
 {} & SGD 
 {} & \textbf{49.0} & \textbf{56.7} & \textbf{44.5} & \textbf{50.0} & \textbf{50.1} & \textbf{65.4} & \textbf{60.5} & \textbf{75.1} & +8.3 & +1.8 \\
\bottomrule
\end{tabular}

\begin{tabular}{@ l *{2}{cccc} cc cc @}
\toprule
 {} & \multicolumn{8}{c}{\textbf{Math}} & \multicolumn{2}{c}{\textbf{Code}} & \multicolumn{2}{c}{\textbf{RLVE}} \\
\cmidrule(lr){2-9} \cmidrule(lr){10-11} \cmidrule(lr){12-13}
 {} & \multicolumn{4}{c}{Qwen3-8B} & \multicolumn{4}{c}{Qwen3-1.7B} & \multicolumn{2}{c}{Qwen3-1.7B} & \multicolumn{2}{c}{Qwen3-1.7B} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-11} \cmidrule(lr){12-13}
 {} & A & S & S+M & R
 {} & A & S & S+M & R
 {} & A & S
 {} & A & S \\
\midrule
Sparsity 
 {} & 91.30 & 99.99 & 99.99 & 86.47
 {} & 91.09 & 99.94 & 99.94 & 86.43
 {} & 92.01 & 99.94
 {} & 86.69 & 99.84 \\

Rank 
 {} & 88.48 & 26.11 & 25.92 & 84.97
 {} & 87.79 & 24.30 & 24.47 & 87.79
 {} & 87.87 & 23.58
 {} & 86.99 & 25.58 \\
\bottomrule
\end{tabular}

4.1 How Does SGD Fare?

Math

Table 2 summarizes the results. We evaluate two settings for the maximum rollout length: (1) 3K, matching the training length, and (2) 8K. Across all experiments, SGD closely matches and often outperforms AdamW. Notably, under the 3K rollout setting, SGD consistently outperforms AdamW across all cases.

Coding

Table 3 presents our results on code generation. We report pass@1 and pass@10. SGD consistently outperforms AdamW across all benchmarks and evaluation settings. Under a 4K maximum response length, SGD achieves an average pass@1 improvement of 8.7% over AdamW. Complementing these results, Figure 1 present the learning curves of training and validation rewards while training the Qwen3-1.7B model. They indicate that training with SGD either matches or outperforms AdamW by the end of the training. While these experiments focus on GRPO training for 270 steps, we will soon show in § 6 that these findings generalize to extended training duration and PPO.


Despite established wisdom that SGD is ill-suited for transformers [22, 23],
it performs on par and often outperforms AdamW in training transformers with RLVR.

4.2 Do Momentum and Adaptive Learning Rates Help?

As discussed earlier in § 2 and Table 1, RMSProp can be intuitively viewed as AdamW ablating momentum, while SGD with momentum can be viewed as AdamW ablating adaptive learning rates. Accordingly, comparisons with them help disentangle the respective contributions of momentum and adaptive learning rates in AdamW, which this section focuses on. Details on the experimental setup for RMSProp and SGD with momentum is detailed in Appendix B .

Table 2 summarizes the results in math reasoning. Comparing SGD + Momentum vs. SGD, we see that momentum hurts the performance in all but one case, with the only exception of Qwen 3 1.7B. This suggests that momentum provides limited benefit in RLVR and may even be counterproductive, consistent with the analysis in § 3. RMSProp shows mixed results. It slightly outperforms SGD on Qwen 3 8B and Qwen 3 1.7B at longer response lengths, but underperforms on Llama 3.1 8B. Overall, these results indicate that neither momentum nor adaptive learning rates consistently help in RLVR.


Neither momentum nor adaptive learning rates improves performance.

4.3 Memory Footprint of SGD

Memory consumption during the policy update phase of RL is dominated by model weights, activations, and optimizer state. While the first two depend on the model architecture and token count, the optimizer state presents an opportunity for significant memory reduction.

For a model with with $p$ trainable parameters, AdamW requires approximately $12p$ bytes of persistent optimizer state: 4 bytes each for the FP32 master weights, $m$, and $v$. In contrast, SGD requires only $4p$ bytes, as it only maintains the FP32 master weights. Thus SGD reduces memory consumption by $2 \times p \times d_{\text{optim}}$ bytes, where $d_{\text{optim}}$ is the number of bytes per optimizer state element (typically 4 for FP32).

For Qwen3-1.7B, this translates to savings of roughly 13.6 GB in optimizer states. In practice, we observe a 15.7 GB reduction in peak memory usage compared to AdamW on Qwen3-1.7B. The additional savings beyond 13.6 GB reflect the reduced communication buffer overhead incurred by FSDP. This total memory reduction enables training larger models or fitting larger batch sizes within the same hardware constraints.

5. SGD Induces Sparse and Low-rank Updates

Section Summary: SGD produces far sparser updates than AdamW during reinforcement learning from verifiable rewards, typically changing only about 0.02 percent of a model’s parameters while AdamW changes roughly 10 percent. These sparse changes occur across all layers rather than in isolated parts of the network, and the resulting update matrices also show markedly lower effective rank. The difference appears to stem mainly from SGD’s lack of per-parameter adaptive learning rates, and the sparsity level stays high throughout training.

Now that we have established that SGD achieves competitive performance in RLVR, we next take a closer look at its parameter updates and ask:


How do parameter updates induced by SGD compare with those of AdamW in RLVR?

Following [14], we investigate this question through the lens of update sparsity. Let $\theta^{0}$ and $\theta^{1}$ denote the model parameters before and after RLVR, respectively. Update sparsity is

$ \mathrm{sparsity}(\theta^{0}, \theta^{1}) := 1 -\frac{\lVert \theta^{1} - \theta^{0} \rVert_{0}}{n} $

$n$ is the number of parameters. The $\ell_0$ norm is computed using a threshold of $10^{-5}$, accounting for numerical precision, taking the bfloat16 data type into account.[^3]

[^3]: All models are trained with a bflaot16 precision. PyTorch uses $10^{-5}$ as the default tolerance.

SGD induces sparser updates than AdamW

Our results, summarized in Table 4, reveal a striking difference between SGD and AdamW. Across model families and scales, SGD produces orders-of-magnitude sparser parameter updates than AdamW. For example, in the Qwen-3-8B model, AdamW updates approximately 10% of model parameters (corresponding to 90% sparsity), whereas SGD updates only 0.01% of parameters (99.99% sparsity). Similar trends are observed for Qwen-3-1.7B and Llama-3.1-8B, and these observations hold consistently across domains. As shown in Figure 4, update sparsity under AdamW decreases as training proceeds, while that under SGD barely does.

While SGD with momentum yields update sparsity similar to that of SGD, RMSProp produces sparsity levels comparable to AdamW. These results suggest that the sparsity observed with SGD partly stems from the absence of per-parameter adaptive learning rates. See § 7 for a more in-depth discussion.

:::: {.figure cols="2"}

Figure 4: Update sparsity of SGD barely decreases as training proceeds. Following plots are from the math experiments ::::

Layerwise analysis of update sparsity

Consistent with the observations of [14], we find that these sparse updates are not concentrated in specific layers or submodules. Figure 3 illustrate the layerwise sparsity, showing that SGD consistently produces significantly sparser updates than AdamW across all layers.

SGD updates have Low Effective Rank

Another major difference between SGD and AdamW lies in the effective rank of their parameter update matrices: SGD produces updates with substantially lower rank than AdamW (Table 4). To quantify update rank, we first extract the update matrices corresponding to all two-dimensional weight tensors in the transformer. Then for each update matrix, we perform singular value decomposition (SVD) and compute the number of singular values required to explain 99% of the spectral energy, defined as the sum of squared singular values. This gives us an effective rank per parameter matrix, and we report the mean across all parameter matrices. The results are shown in Table 4. Our observations indicate that models trained with SGD exhibit substantially lower effective rank than their AdamW-trained counterparts. While this difference is less pronounced for the Llama model considered, the overall trend holds consistently across models and settings.


In RLVR, SGD produces highly sparse updates, often modifying only about $0.02\%$ parameters, orders of magnitude fewer than AdamW.

The effective rank of SGD updates is also significantly lower than that of AdamW.

6. Validation with PPO and Extended Training

Section Summary: To test whether their earlier findings hold more broadly, the authors swapped GRPO for the PPO algorithm and also ran much longer training sessions. In both cases SGD matched AdamW’s performance on math benchmarks while producing far sparser updates, often leaving over 99 % of parameters unchanged. The same sparsity advantage appeared even in the PPO critic network and persisted across hundreds of extra training steps, showing that the pattern is not limited to one optimizer or short runs.

To test whether our observations generalize across RL algorithms, we first replace GRPO with PPO and also examine whether they hold under extended training durations. In both settings, we find that SGD achieves performance comparable to AdamW while inducing substantially sparser updates.

\begin{tabular}{@ lcccccc @}
\toprule
\textbf{Model} & \textbf{Opt.} & \textbf{Math} & \textbf{AMC} & \textbf{Oly.} & \textbf{GP.} & \textbf{Mean} \\
\midrule
\multirow{2}{*}{\textsc{Qwen 8b}} & AdamW & 81.8 & 56.6 & 45.6 & 31.8 & 54.0 \\
  {} & SGD & 80.0 & 55.5 & 44.7 & 46.5 & 56.7 \\
\cmidrule{1-7}
\multirow{2}{*}{\textsc{Qwen 1.7b}} & Adam & 73.2 & 43.4 & 34.2 & 14.6 & 41.4 \\
  {} & SGD & 73.2 & 37.4 & 30.8 & 26.3 & 41.9 \\
\bottomrule
\end{tabular}

SGD vs. AdamW under PPO

The PPO training setup largely follows that of § 4, except for a shorter maximum response length of 1K due to PPO’s higher computational cost. We train both the policy and the critic using the same optimizer. As shown in Table 5, the SGD vs. AdamW comparison under PPO mirrors the trend observed with GRPO, with SGD consistently performing on par with AdamW. The update sparsity observed with GRPO also hold for PPO, where Qwen-3-8B and Qwen-3-1.7B exhibit 99.92% and 99.98% sparse updates (resp.) under SGD and 87.9 and 89.02% sparse updates under AdamW (resp.). Interestingly, the critic in PPO also exhibits substantial sparsity under SGD: for Qwen-3-8B, AdamW updates approximately 61.1% of critic parameters, whereas SGD updates only 8%. This further highlights a qualitative difference between the optimization behavior of SGD and AdamW in RLVR.

::: {caption="Table 6: Results of training on RLVE with GRPO."}

:::

Effects of extended training duration with RLVE

We further evaluate our observations under a setup with substantially more optimization steps to examine whether it exposes potential brittleness of SGD. We choose RLVE because the environment automatically evolves, so models can always train on their capability frontier, preventing the update to stall due to lack of useful supervision signals from data ([28]). We train with RLVE for 500 steps. Table 6 shows that SGD effectively matches AdamW performance. Notably, even under extended training duration, SGD maintains highly sparse updates: 99.8% of parameters remain unchanged, compared to 86.7% when using AdamW. The competitive performance and substantial sparsity demonstrates the potential of SGD for training over substantially more optimization steps. Further investigation reveals that across training steps the sparsity decays much slower in SGD as compared to AdamW (Figure 4). Which implies even after prolonged training SGD trained checkpoints will continue to be significantly sparser as compared to AdamW, as also verified in our experiment with the RLVE environment.


The competitive performance and substantial sparsity of SGD generalize to PPO and persist under extended training.

7. Why are SGD Updates Sparser?

Section Summary: Even with exact math, neural network gradients would rarely hit zero, but in real training many updates are already tiny and get rounded away to nothing by ordinary floating-point hardware. SGD produces especially sparse updates because it applies the same fixed step size to every parameter, leaving the smallest changes too weak to survive rounding. In contrast, AdamW and similar adaptive methods automatically boost those weak updates, so far more of them remain visible after the same rounding step.

As noted in prior work ([14, 15]), if backpropagation were performed with unlimited numerical precision, the resulting gradients would be very unlikely to be sparse. The update sparsity observed in practice therefore arises from a combination of two factors: (1) many updates having magnitudes close to zero, and (2) these small updates being suppressed by floating-point rounding when applied to the parameters. The latter is an inherent constraint of modern computing hardware and system, it is therefore of particular interest to examine how algorithmic optimization choices in RLVR contribute to the former. Prior work has studied both the inherently small gradients in RL for LLMs [15] and the sparse updates produced by AdamW [14]. We build on these findings and inquire: why does SGD produce substantially sparser updates than AdamW? We discuss this next.

The absence of adaptive learning rates

AdamW adapts the learning rate for each parameter by normalizing updates with an exponential moving average of squared gradients, increasing the effective step size for parameters with historically small gradients and decreasing it for those with larger ones. We conjecture that this effectively amplifies updates with small magnitudes that would otherwise be suppressed by floating-point rounding. This conjecture is further supported by the results in Table 4: both SGD and SGD with momentum, which lack adaptive learning rates, produce highly sparse updates; AdamW and RMSProp, both using adaptive learning rates, induce substantially denser updates.

8. Related Work

Section Summary: Recent work has positioned reinforcement learning with verifiable rewards (RLVR) as an important approach for improving large language models, offering advantages over earlier methods by avoiding certain training instabilities and enabling better reasoning performance, which has led major labs to invest heavily in it. At the same time, researchers have only begun to examine how RLVR actually changes model weights during training, with early findings suggesting that updates remain highly localized and operate differently from standard supervised fine-tuning. In parallel, studies of optimizers have reinforced why adaptive methods such as AdamW are routinely preferred over plain stochastic gradient descent when training transformer-based models, citing issues like gradient noise and varying curvature across parameter blocks.

8.1 RLVR in LLMs

RLVR has emerged as a key paradigm in the training of LLMs. Removing the noisy reward models, as used in RLHF [2], it alleviates issues such as reward hacking ([55, 56, 57]). Further recent work has also established that online RL, as compared to its counterpart supervised finetuning, does not suffer from catastrophic forgetting ([13, 11]). Owing to these benefits as well as recent algorithmic improvements such as GRPO [33], DAPO [58] etc, this training paradigm has yielded in significant improvements in expanding reasoning boundaries of LLMs. Recent reports from leading frontier labs report significant effort being spent in RL based post-training of frontier LLMs [59, 46, 4, 20].

8.2 Training Dynamics in RLVR

Despite this progress, the training dynamics induced by RLVR in the weight space of LLMs is poorly understood. Some early evidences as discovered by [14] show that RL updates are considerably sparser as compared to SFT, where often only 5 to 20% of model weights accumulate any update as part of training. [15] argued that this localization happens because online RL causes rotation in the weight space, only altering off-principle eigenvectors. These findings combined paints a picture that RL finetuning happens in a very distinct regime than SFT. Further indicating that borrowing design principles might prove to be suboptimal.

8.3 SGD vs AdamW for Training LLMs

Conventional wisdom suggests that, under standard training setups, SGD often underperforms adaptive optimizers such as Adam when training Transformer models. This phenomenon has been well studied, and is often attributed to (i) heavy-tailed distribution of the noise in stochastic gradients [35], (ii) directional sharpness [22] (curvature of the function along the update direction), (iii) [26] provides evidence that this gap in performance is much more visible under a high batch setting, (iv) [25] attributed this to the block heterogeneity, i.e. the dramatic difference in the hessian spectrum in the parameter blocks in transformers. Together, these findings provide compelling evidence as to AdamW has become the de facto optimizer for training large (often) transformer based language models.

9. Conclusion

Section Summary: Recent research shows that the simple SGD optimizer performs as well as or better than the more complex AdamW method when fine-tuning large language models with reinforcement learning, across various models, tasks, and algorithms. Momentum and adaptive learning rates provide little benefit and can even hurt results, while SGD naturally updates fewer than 0.02 percent of parameters, indicating that effective training occurs in a surprisingly low-dimensional space. These findings also cut memory use by up to 15.7 GB and suggest that optimization ideas from standard supervised learning do not transfer directly to reinforcement learning.

We demonstrate that SGD, long considered ill-suited for training large transformers, matches or outperforms AdamW in RLVR across multiple models and domains and RL algorithms. Neither momentum nor adaptive learning rates consistently improve performance, with momentum often proving detrimental. The observations hold true even under prolonged training. Remarkably, SGD updates fewer than 0.02% of parameters without any explicit sparsity regularization, revealing that effective RL fine-tuning operates in a surprisingly low-dimensional subspace. These findings yield immediate practical benefits—SGD reduces memory usage by up to 15.7 GB compared to AdamW—while highlighting that optimization principles from supervised learning do not directly transfer to RL. We hope this work motivates further investigation into optimization methods tailored to the distinct dynamics of reinforcement learning in LLMs.

10. Impact Statement

Section Summary: This work shows that switching from the common AdamW optimizer to simpler SGD during reinforcement learning for large language models can cut memory use substantially, making such training more feasible for researchers without access to massive computing resources. It also reveals that these methods produce highly localized changes in the model, which may explain reduced forgetting of prior knowledge and point toward more efficient ways to adapt models in the future. The authors see no new societal risks from this approach and suggest it could help spread the ability to align and improve language models more widely across the research community.

From a practical standpoint, our work offers immediate memory savings for practitioners training LLMs with reinforcement learning. By eliminating the need for AdamW's momentum buffers, SGD reduces GPU memory footprint, potentially democratizing access to RL-based LLM training for researchers with limited computational resources. The extreme parameter sparsity we observe with SGD also provides mechanistic insights into how RL modifies pretrained models, suggesting that effective reasoning capabilities may emerge from surprisingly localized changes. This understanding could inform future work on efficient fine-tuning methods and help explain why RL-trained models exhibit reduced catastrophic forgetting compared to supervised approaches. We do not foresee specific negative societal consequences arising directly from this work beyond those already associated with LLM training more broadly. Our contributions are primarily methodological and do not introduce new capabilities that would amplify existing risks. If anything, reducing the computational requirements for RL training may help distribute the ability to align and improve LLMs more broadly across the research community.

Appendix

Section Summary: The appendix provides extra details on the training setup for Qwen and Llama models, including the use of the verl framework on multiple high-memory GPUs, shared settings such as batch size and sequence lengths, and differing configurations for optimizers like GRPO and PPO along with their rollout and KL penalty choices. It explains why SGD needs learning rates orders of magnitude higher than AdamW's nominal rate by analyzing the wide distribution of AdamW's effective per-parameter rates, and it includes ablation experiments on momentum and adaptive methods plus data collection on gradients and optimizer moments during supervised and reinforcement learning runs. Supporting figures illustrate SGD learning rate sweeps and the spread of effective rates.

A.1 Additional Details of Training Setup

Models:

For Qwen models, we enabled thinking mode during training, which enables generation of long Chain of thoughts.

Hyperparameters:

All experiments are implemented using the verl framework.^4 Models are trained using four 96 GB NVIDIA GH200 GPUs. Unless otherwise specified, all runs share the following settings: a training batch size of 256, a maximum prompt length of 1,024 tokens and two training epochs. While training models with GRPO we keep the maximum sequence length of 3072 for math (rollouts=4), 4096 for code and 8192 for RLVE. However, in PPO we had to keep the response length to 1024 and 8 rollouts. For experiments with GRPO, we set the KL penalty coefficient to 0.001. All experiments use the KL term as a loss shaping term, and not a reward shaping term. Proximal Policy Optimization (PPO) experiments use Generalized Advantage Estimation (GAE) with a dedicated critic network (learning rate $= 10^{-5}$) and 8 rollout samples. All other training configurations and hyperparameters are held constant, enabling a direct comparison between the two optimizers. For all experiments with AdamW we used a learning rate of $10^{-6}$. Similarly for all experiments with SGD, we used a learning rate of $10^{-1}$. We observed that a much higher learning rate than AdamW is typically required for SGD. Only the PPO experiment with Qwen-3-1.7b required a lower learning rate of $10^{-2}$. However, since AdamW uses a per-parameter adaptive learning rate, the learning rate is not comparable across SGD and AdamW.

$**Figure 5:** **SGD Learning Rate Ablation on Qwen3-8B (NuminaMath- CoT).** **Left:** Training reward curves over optimization steps. SGD converges to comparable or higher rewards than AdamW, provided the learning rate is sufficiently high. **Right:** Final training reward as a function of learning rate (log scale). SGD requires learning rates orders of magnitude larger than AdamW ($10^{-6}$) to achieve peak performance in the RLVR setting.$

$**Figure 6:** **Distribution of AdamW's effective per-parameter learning rates.** We compute the effective learning rate $\eta_{\text{eff}} = \frac{\eta}{\sqrt{v} + \epsilon}$ for each parameter using moment estimates extracted at step 50 of a GRPO code training run (see § 4 for setup details). Despite AdamW's nominal learning rate of $10^{-6}$ (red dashed line), the distribution of effective learning rates spans roughly $10^{-1}$ to $10^{2}$, with the bulk concentrated between $10^{0}$ and $10^{1}$. SGD's learning rate of $10^{-1}$ (green dashed line) falls near the lower tail of this distribution, explaining why SGD requires learning rates $10^{5}$ – $10^{6}\times$ larger than AdamW's nominal rate to achieve comparable update magnitudes.$ {width=80%}

A. SGD requires a high learning rate

Our earlier observations indicated that SGD needs a much higher learning rate as compared to AdamW, where the optimal learning for SGD is 0.1 while for AdamW it is $10^{-6}$.

To determine the best operating point for SGD LR, we analyze AdamW's distribution of effective per-parameter learning rates. We compute effective learning rate as $\frac{\eta}{\sqrt{v}+\epsilon}$, where $\eta$ is the nominal learning rate of AdamW, $v$ is the second-moment estimate, and $\epsilon=\text{1e-8}$. Figure 6 shows this distribution extracted at step 50 of a GRPO code training run. Despite AdamW's nominal learning rate of $10^{-6}$, the effective rates span $10^{-1}$ to $10^{2}$, with the bulk concentrated between $10^{0}$ and $10^{1}$. This explains why SGD requires learning rates $10^5$-$10^6\times$ larger than AdamW's nominal rate to achieve comparable update magnitudes. This is to be expected since AdamW rescales updates using per-parameter second-moment estimates–often leading to much larger effective step sizes than suggested by the nominal learning rate.

To empirically confirm the need for high SGD LR, we swept SGD over LR $\in {10^{-3}, 10^{-2}, 10^{-1}, 1, 10}$ using the same setup (math tasks) on Qwen3-8B detailed in Section 4. Figure 5 shows that SGD converges to comparable final reward at LR$=0.1$ and LR$=1$, only crashing at LR$=10$. Notably, SGD underperforms at low learning rates (LR=$10^{-2}$), indicating that RL fine-tuning's loss landscape not only tolerates but benefits from aggressive step sizes.

B. Experimental Details for ablating momentum and Adaptive Learning Rate

We train Qwen and Llama models with the mentioned optimizers on the math domain, in a training setup same as that described earlier in § 4. For SGD+momentum (momentum=0.9), we sweep over three different learning rates ($10^{-1}$, $10^{-2}$, and $10^{-3}$), and report the highest average validation performance. Similarly, for RMSProp, we sweep learning rates ($10^{-5}$ and $10^{-6}$) and report the best mean performance.

C. Experimental Details for the Adaptive Learning Rate Analysis

SFT was performed on Qwen3-1.7B using the OpenCodeInstruct dataset ([60]); RLVR used the same model with the setup described in Section 4. Both training runs used identical AdamW hyperparameters ($\beta_1=0.9$, $\beta_2=0.999$, $\text{lr}=10^{-6}$, weight decay $=0.01$, global batch size $=256$) and were distributed across 4 GPUs using FSDP. At training step 50, we captured the full optimizer state for all 430M trainable parameters on rank 0: gradients $g_t$ immediately before optimizer.step(), and AdamW's first moment $m_t$ (exponential moving average of gradients) and second moment $v_t$ (exponential moving average of squared gradients) immediately after. We then compared the distributions of gradient magnitudes $|g|$, first moment magnitudes $|m|$, and second moment roots $\sqrt{v}$ between the two training regimes.

D. Experimental Details for the Momentum Analysis

To quantify the role of momentum in AdamW's first moment estimator, we profiled optimizer state at training step 50 under two regimes: supervised fine-tuning (SFT) on high-quality code completions from nvidia/OpenCodeInstruct [60] (filtered to test scores $\geq 0.9$) and train with GRPO with KL regularization on code generation tasks. Both experiments used Qwen3-1.7B with identical hyperparameters ($\beta_1 = 0.9$, $\beta_2 = 0.999$, $\text{lr} = 10^{-6}$, batch size $= 256$, 4 GPUs with FSDP). We captured gradients $g_t$ before optimizer.step() and first moments $m_t$ after, then recovered the momentum buffer $m_{t-1} = (m_t - (1-\beta_1)g_t)/\beta_1$ to isolate accumulated history from the current gradient. We computed two metrics across 430M parameters: history ratio $r_t = |m_{t-1}| / |g_t|$ (whether momentum materially affects the update) and directional alignment $\cos \phi_t = \langle m_{t-1}, g_t \rangle / (|m_{t-1}| \cdot |g_t|)$ (whether momentum reinforces or opposes the gradient).

References

Section Summary: This references section compiles foundational texts and recent research papers on reinforcement learning and its applications to training large language models. It includes classic works on RL algorithms and optimization methods like Adam, alongside numerous 2022–2025 studies examining how RL with human feedback, process rewards, and related techniques enhance model reasoning, generalization, and instruction-following. Many entries focus on empirical comparisons of training approaches and practical insights from teams at OpenAI, DeepSeek, and academic labs.

[1] Sutton et al. (1998). Reinforcement learning: An introduction. MIT press Cambridge.

[2] Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155. arXiv:2203.02155.

[3] Daniel M. Ziegler et al. (2020). Fine-Tuning Language Models from Human Preferences. https://arxiv.org/abs/1909.08593. arXiv:1909.08593.

[4] Guo et al. (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature. 645(8081). pp. 633–638. doi:10.1038/s41586-025-09422-z. http://dx.doi.org/10.1038/s41586-025-09422-z.

[5] OpenAI et al. (2024). OpenAI o1 System Card. https://arxiv.org/abs/2412.16720. arXiv:2412.16720.

[6] Hunter Lightman et al. (2023). Let's Verify Step by Step. https://arxiv.org/abs/2305.20050. arXiv:2305.20050.

[7] Ganqu Cui et al. (2025). Process Reinforcement through Implicit Rewards. https://arxiv.org/abs/2502.01456. arXiv:2502.01456.

[8] Hongru Wang et al. (2025). Acting Less is Reasoning More! Teaching Model to Act Efficiently. https://arxiv.org/abs/2504.14870. arXiv:2504.14870.

[9] DeepSeek-AI et al. (2025). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. https://arxiv.org/abs/2512.02556. arXiv:2512.02556.

[10] John Schulman and Thinking Machines Lab (2025). LoRA Without Regret. Thinking Machines Lab: Connectionism. doi:10.64434/tml.20250929.

[11] Idan Shenfeld et al. (2025). RL's Razor: Why Online Reinforcement Learning Forgets Less. https://arxiv.org/abs/2509.04259. arXiv:2509.04259.

[12] Tianzhe Chu et al. (2025). SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. https://arxiv.org/abs/2501.17161. arXiv:2501.17161.

[13] Howard Chen et al. (2025). Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting. https://arxiv.org/abs/2510.18874. arXiv:2510.18874.

[14] Sagnik Mukherjee et al. (2025). Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. https://arxiv.org/abs/2505.11711. arXiv:2505.11711.

[15] Hanqing Zhu et al. (2025). The Path Not Taken: RLVR Provably Learns Off the Principals. https://arxiv.org/abs/2511.08567. arXiv:2511.08567.

[16] Diederik P. Kingma and Jimmy Ba (2017). Adam: A Method for Stochastic Optimization. https://arxiv.org/abs/1412.6980. arXiv:1412.6980.

[17] Ilya Loshchilov and Frank Hutter (2019). Decoupled Weight Decay Regularization. https://arxiv.org/abs/1711.05101. arXiv:1711.05101.

[18] Nathan Lambert et al. (2025). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. https://arxiv.org/abs/2411.15124. arXiv:2411.15124.

[19] Team OLMo et al. (2025). 2 OLMo 2 Furious. https://arxiv.org/abs/2501.00656. arXiv:2501.00656.

[20] Aaron Grattafiori et al. (2024). The Llama 3 Herd of Models. https://arxiv.org/abs/2407.21783. arXiv:2407.21783.

[21] Sebastian Ruder (2017). An overview of gradient descent optimization algorithms. https://arxiv.org/abs/1609.04747. arXiv:1609.04747.

[22] Yan Pan and Yuanzhi Li (2023). Toward Understanding Why Adam Converges Faster Than SGD for Transformers. https://arxiv.org/abs/2306.00204. arXiv:2306.00204.

[23] Rosie Zhao et al. (2025). Deconstructing What Makes a Good Optimizer for Language Models. https://arxiv.org/abs/2407.07972. arXiv:2407.07972.

[24] Akiyoshi Tomihari and Issei Sato (2025). Understanding Why Adam Outperforms SGD: Gradient Heterogeneity in Transformers. https://arxiv.org/abs/2502.00213. arXiv:2502.00213.

[25] Yushun Zhang et al. (2024). Why Transformers Need Adam: A Hessian Perspective. https://arxiv.org/abs/2402.16788. arXiv:2402.16788.

[26] Frederik Kunstner et al. (2023). Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be. https://arxiv.org/abs/2304.13960. arXiv:2304.13960.

[27] Teodora Srećković et al. (2025). Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. https://arxiv.org/abs/2506.12543. arXiv:2506.12543.

[28] Zhiyuan Zeng et al. (2025). RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments. https://arxiv.org/abs/2511.07317. arXiv:2511.07317.

[29] Kanishk Gandhi et al. (2025). Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs. https://arxiv.org/abs/2503.01307. arXiv:2503.01307.

[30] Lifan Yuan et al. (2025). From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones. https://arxiv.org/abs/2509.25123. arXiv:2509.25123.

[31] Shivam Agarwal et al. (2025). The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=UfFTBEsLgI.

[32] Sutton et al. (1999). Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in Neural Information Processing Systems. pp. . https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf.

[33] Zhihong Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300. arXiv:2402.03300.

[34] Arash Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. https://arxiv.org/abs/2402.14740. arXiv:2402.14740.

[35] Jingzhao Zhang et al. (2020). Why ADAM Beats SGD for Attention Models. https://openreview.net/forum?id=SJx37TEtDH.

[36] Emmanuel Bengio et al. (2021). Correcting Momentum in Temporal Difference Learning. https://arxiv.org/abs/2106.03955. arXiv:2106.03955.

[37] Kavosh Asadi et al. (2023). Resetting the Optimizer in Deep RL: An Empirical Study. https://arxiv.org/abs/2306.17833. arXiv:2306.17833.

[38] Benjamin Ellis et al. (2024). Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps. https://arxiv.org/abs/2412.17113. arXiv:2412.17113.

[39] Alexander David Goldie et al. (2025). Can Learned Optimization Make Reinforcement Learning Less Difficult?. https://arxiv.org/abs/2407.07082. arXiv:2407.07082.

[40] John Schulman et al. (2017). Proximal Policy Optimization Algorithms. https://arxiv.org/abs/1707.06347. arXiv:1707.06347.

[41] Jia LI et al. (2024). NuminaMath. https://github.com/project-numina/aimo-progress-prize.

[42] Dan Hendrycks et al. (2021). Measuring Coding Challenge Competence With APPS. https://arxiv.org/abs/2105.09938. arXiv:2105.09938.

[43] Yujia Li et al. (2022). Competition-Level Code Generation with AlphaCode. https://arxiv.org/abs/2203.07814. arXiv:2203.07814.

[44] Rongao Li et al. (2023). TACO: Topics in Algorithmic COde generation dataset. https://arxiv.org/abs/2312.14852. arXiv:2312.14852.

[45] Codeforces (2024). Codeforces. Online programming competition platform. https://codeforces.com/.

[46] An Yang et al. (2025). Qwen3 Technical Report. https://arxiv.org/abs/2505.09388. arXiv:2505.09388.

[47] Dan Hendrycks et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/abs/2103.03874. arXiv:2103.03874.

[48] Zhang, Yifan and Math-AI, Team (2024). American Invitational Mathematics Examination (AIME) 2024.

[49] Zhang, Yifan and Math-AI, Team (2025). American Invitational Mathematics Examination (AIME) 2025.

[50] Chaoqun He et al. (2024). OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. https://arxiv.org/abs/2402.14008. arXiv:2402.14008.

[51] David Rein et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/abs/2311.12022. arXiv:2311.12022.

[52] Mark Chen et al. (2021). Evaluating Large Language Models Trained on Code. https://arxiv.org/abs/2107.03374. arXiv:2107.03374.

[53] Jiawei Liu et al. (2023). Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. https://arxiv.org/abs/2305.01210. arXiv:2305.01210.

[54] Jacob Austin et al. (2021). Program Synthesis with Large Language Models. https://arxiv.org/abs/2108.07732. arXiv:2108.07732.

[55] Weng, Lilian (2024). Reward Hacking in Reinforcement Learning.. lilianweng.github.io. https://lilianweng.github.io/posts/2024-11-28-reward-hacking/.

[56] Dario Amodei et al. (2016). Concrete Problems in AI Safety. https://arxiv.org/abs/1606.06565. arXiv:1606.06565.

[57] Leo Gao et al. (2022). Scaling Laws for Reward Model Overoptimization. https://arxiv.org/abs/2210.10760. arXiv:2210.10760.

[58] Qiying Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476. arXiv:2503.14476.

[59] xAI (2025). Grok: AI Assistant. Accessed: 2025-09-24. https://x.ai/grok.

[60] Wasi Uddin Ahmad et al. (2025). OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs. https://arxiv.org/abs/2504.04030. arXiv:2504.04030.