Normalized Low-Rank Adaptation

Jiale Kang$^{1,3}$, Ziyin Yue, Zheng Zhan$^{2}$, Yangyi Huang$^{3}$, Weiyang Liu$^{3,4}$
$^{1}$ Yuanshi Intelligence
$^{2}$ Microsoft Research
$^{3}$ The Chinese University of Hong Kong
$^{4}$ Shenzhen Loop Area Institute

spherelab.ai/NoRA

Abstract

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.

Executive Summary: The paper addresses the challenge of making low-rank adaptation (LoRA) more reliable and effective when fine-tuning large language models. Although LoRA reduces trainable parameters by factoring weight updates into low-rank matrices, its early training dynamics remain unstable because the down-projection matrix starts randomly while the up-projection starts at zero. This imbalance often slows convergence, reduces final performance, and increases forgetting of the original model.

The work set out to test whether explicitly controlling the scale of the down-projection matrix could improve optimization without adding parameters or breaking LoRA’s ability to merge weights at inference time. The authors introduced Normalized Low-Rank Adaptation (NoRA), which normalizes each column of the down-projection along the rank dimension throughout training, and a lighter variant (NoRA-init) that applies the same normalization only once at initialization. Both approaches were evaluated on 340-million-parameter models during pretraining, on Llama-3.2-3B during supervised fine-tuning, and on a 1.5-billion-parameter model during reinforcement learning with verifiable rewards.

Across these regimes, NoRA produced consistent gains. In supervised fine-tuning it raised average accuracy by 5.4 points over standard LoRA while cutting forgetting on retained benchmarks. In reinforcement learning it improved overall scores by 1.6 points and remained stable where spectral-initialization methods such as PiSSA degraded. In pretraining it prevented the sharp performance collapse seen with ordinary low-rank updates and kept gradient norms closer to those of full fine-tuning. Applying normalization only at initialization captured most of the benefit; enforcing it continuously yielded modest additional gains in stability and retention.

These results indicate that rank-dimension normalization corrects an otherwise arbitrary scale imbalance that acts as a hidden, poorly conditioned preconditioner on the input coordinates. Because the change requires no extra parameters, no change to the optimizer, and no inference overhead, it can be adopted immediately as a drop-in replacement for existing LoRA implementations. Practitioners should prefer NoRA or NoRA-init whenever parameter-efficient adaptation is used, especially in reinforcement-learning or long-horizon pretraining settings where stability matters. Further work could examine the interaction with other scaling heuristics and test the approach at larger model sizes. The main limitations are that experiments were conducted on models up to a few billion parameters and that the recommended scaling factor requires modest per-task tuning to balance adaptation against retention.

1. Introduction

Section Summary: As large language models grow in size, their training becomes increasingly expensive, which has made parameter-efficient methods like LoRA popular for adapting them with far fewer trainable parameters. Standard LoRA relies on random initialization for one of its update matrices, which can lead to unstable or suboptimal early training, so the paper explores whether normalizing that matrix along its rank dimension can improve optimization dynamics without losing LoRA’s efficiency or exact mergeability into the original model. The resulting approach, called NoRA, applies this normalization either continuously during training or only at initialization and shows faster convergence, greater stability, and stronger results across pretraining, fine-tuning, and reinforcement learning.

As large language models (LLMs) continue to scale, their training costs grow rapidly, increasing the need for parameter-efficient learning. Among parameter-efficient finetuning (PEFT) approaches, Low-rank adaptation (LoRA) ([1]) has become particularly popular due to its simplicity, efficiency, and strong empirical performance. By representing weight updates through low-rank factorization, LoRA greatly reduces the number of trainable parameters and has been successfully applied to supervised finetuning, pretraining, and reinforcement learning ([2, 3]). Despite its effectiveness, LoRA’s optimization dynamics remain poorly understood. Standard LoRA parameterizes $\Delta\bm{W}=\alpha\bm{B}\bm{A}$ with randomly initialized $\bm{A}$ and zero-initialized $\bm{B}$. Although this preserves the pretrained initialization, early optimization depends entirely on the random features induced by $\bm{A}$. This observation motivates the question below:

Does the regularization of the down-projection matrix $\bm{A}$ provide a useful design dimension

for improving LoRA optimization?

Existing work suggests that initialization can substantially affect low-rank optimization. Methods such as PiSSA ([4]) and MiLoRA ([5]) construct the initial low-rank subspace from the spectral structure of pretrained weights and often converge faster than random initialization. However, they require singular-value decomposition and modify the initial decomposition of the pretrained weights, making them less convenient for large-scale training and potentially fragile in reinforcement-learning settings ([6]). Other approaches improve LoRA through modified scaling ([7]), constrained parameterizations ([8]), or fixed projections ([9]). However, how the regularization of the down-projection matrix affects or improves LoRA's optimization remains poorly understood.

Our investigation begins with the observation that the normalized latent bottleneck used in Multi-head Latent Attention (MLA) substantially stabilizes training and also improves convergence. Similar observations have also been made in multiple training settings ([10, 11, 12]). These phenomena suggest that the numerical structure of the normalized latent representation presented to the up-projection can play an important role in stable optimization. Directly introducing latent normalization into LoRA changes the standard linear update from $\bm B\bm A\bm x$ to $\bm{B} \cdot \text{Norm}(\bm A\bm x)$. Because the normalization depends on the input, the resulting transformation is nonlinear with respect to $\bm x$ and can no longer be exactly merged into the pretrained weight matrix like LoRA. Inspired by MLA, we therefore study whether the useful numerical effect of latent normalization can instead be encoded as a regularization of the linear down-projection such that we can have both LoRA's exact mergeability and MLA's favorable optimization benefits.

To this end, we adapt MLA's normalization principle from the latent feature to the low-rank projection itself. Each column of $\bm{A}\in\mathbb{R}^{r\times k}$ represents how an input coordinate is projected into the $r$-dimensional latent space. Guided by the normalization dimension in MLA, we propose to normalize these projection vectors along the rank dimension, yielding the linear form $\bm{B}\cdot\text{Norm}(\bm{A})\bm{x}$. We refer to this formulation as Normalized LoRA (NoRA), which constrains the input-to-latent projection magnitudes throughout training while preserving exact weight mergeability. This normalization scheme also ensures that the gradient norm of NoRA can be better aligned with full finetuning, bridging the learnability gap between LoRA and full finetuning.

NoRA requires the down-projection matrix $\bm{A}$ to be normalized along the output rank dimension. This principle motivates two ways of applying normalization: (1) as a training constraint (i.e., NoRA) and (2) as an initialization strategy (i.e., NoRA-init). In the former, NoRA continuously normalizes the down-projection matrix $\bm{A}$ along the low-rank dimension $r$ throughout training. In the latter, NoRA-init normalizes $\bm{A}$ only once at initialization and subsequently optimizes it without constraints during training. For NoRA-init, we propose two initialization variants: (1) random normalization, where $\bm{A}$ is randomly initialized and then normalized; and (2) Block Identity Matrix Initialization (BIMI), where $\bm{A}$ is constructed as a block-identity matrix that inherently satisfies the rank-dimensional normalization requirement. NoRA improves training by removing undesirable scale imbalance across the low-rank dimensions, providing a better-conditioned parameterization than standard LoRA.

We evaluate both NoRA and NoRA-init across pretraining, supervised finetuning, and reinforcement learning with verifiable rewards. Across a diverse range of base models, tasks, and training configurations, both methods consistently accelerate convergence, improve training stability, and enhance downstream performance. They also mitigate catastrophic forgetting during supervised adaptation and remain robust in reinforcement-learning settings where other spectral initialization methods can be fragile. In general, we can conclude that these results establish rank-dimension normalization of the down-projection as an important design principle for improving LoRA optimization. Our contributions are summarized as follows:

• We identify the down-projection matrix $\bm{A}$ as a previously underexplored yet important design dimension in LoRA, showing that its magnitude can play a critical role in shaping optimization dynamics and ultimately improving training efficiency and downstream performance.
• We propose NoRA, which normalizes the down-projection matrix $\bm{A}$ along the rank dimension in the LoRA parameterization. We further introduce NoRA-init, an initialization-only variant that applies this normalization at initialization and requires no re-normalization during training, enabling seamless performance improvements without modifying the optimization procedure.
• Extensive experiments across pretraining, SFT, and RLVR demonstrate NoRA's improved convergence, stability, downstream performance, and resistance to catastrophic forgetting.

2. Preliminaries

Section Summary: LoRA adapts a pretrained neural network by adding a low-rank update to its weight matrices, expressed as the product of two smaller matrices A and B, where B begins at zero so the change starts from nothing. At the very first training step the gradient with respect to A is zero, which means the initial direction of learning is controlled entirely by how A projects the input into a smaller “latent feature” space. The paper therefore presents a single formulation in which this latent feature can be constructed in different ways—random, structured, or normalized—and shows that many existing methods are simply alternative choices for building the same intermediate representation before it is passed to B.

LoRA parameterizes the weight update of a pretrained matrix as

$ \Delta\bm W=\bm{W}-\bm{W}_0=\alpha\bm B\bm A,\tag{1} $

where $\bm A\in\mathbb{R}^{r\times k}$, $\bm B\in\mathbb{R}^{d\times r}$, and $r\ll\min(d, k)$. Standard LoRA initializes $\bm B^{(0)}=\bm0$ to preserve the pretrained model. Let $\bm G=\partial\mathcal L/\partial\bm W$ denote the gradient of the corresponding full weight matrix. At initialization, we have the following gradients with respect to $\bm{B}$ and $\bm{A}$:

$ \left. \frac{\partial\mathcal L}{\partial\bm B} \right|_{t=0}

\alpha \bm G (\bm A^{(0)})^\top, \qquad \left. \frac{\partial\mathcal L}{\partial\bm A} \right|_{t=0}

\bm0.\tag{2} $

Therefore, the earliest optimization of LoRA is entirely determined by the latent representation induced by the initial projection $\bm A^{(0)}$.

Motivated by this observation, we consider a unified latent-feature formulation:

$ \Delta\bm y

\alpha \bm B \bm\phi(\bm x),\tag{3} $

where $\bm\phi(\bm x)\in\mathbb{R}^r$ denotes the latent feature presented to the up-projection. Existing low-rank methods can be interpreted as constructing this latent feature in different ways:

$ \bm\phi(\bm x)= \begin{cases} \bm A\bm x, & \text{random projection (LoRA)}, \ \bm A_{init}\bm x, & \text{structured projection (e.g., PiSSA, MiLoRA)}, \ \text{Norm}(\bm A\bm x), & \text{normalized projection (e.g., MLA)}. \end{cases}\tag{4} $

Although these approaches adopt different implementations, they can all be viewed as designing the latent feature presented to the up-projection. This latent-projection perspective motivates us to revisit how to regularize the latent projection without losing the exact mergeability of LoRA.

::: {caption="Table 1: Overview of representative PEFT methods categorized by their primary mechanism, including forward parameterization and initialization strategy."}

:::

3. NoRA: Normalized Low-Rank Adaptation

Section Summary: NoRA modifies standard low-rank adaptation by normalizing each column of the down-projection matrix along the rank dimension so that every input coordinate is mapped through a unit-length direction. This transfers the beneficial scale control seen in certain latent-feature methods into a fixed, parameter-only constraint that remains linear in the input and can still be merged exactly into the pretrained weights after training. The same principle also explains the effectiveness of related approaches such as MiSS and motivates a simpler initialization-only variant that captures most of the optimization advantage.

3.1 From Normalized Latent Features to Normalized Low-Rank Projection

The normalized latent projection in MLA suggests that the representation presented to the up-projection plays an important role in training dynamics. Specifically, MLA constructs the low-dimensional latent feature as

$ \bm{\phi}(\bm{x}) = \text{Norm}(\bm{A}\bm{x}),\tag{5} $

where the normalization operator explicitly regulates the projected feature before it is mapped back to the output space. Directly introducing this operation into LoRA changes the standard linear update from $\bm{B}\bm{A}\bm{x}$ to $\bm{B}\text{Norm}(\bm{A}\bm{x})$. Since the normalization depends on the input, this transformation becomes nonlinear with respect to $\bm{x}$ and can no longer be exactly merged into the pretrained weight matrix. We therefore seek to transfer the scale-control effect from the latent feature to the down-projection matrix itself. Modern Transformer architectures are typically pre-normalized, such that the magnitude of the input $\bm{x}$ to each linear layer is already well controlled by LayerNorm. In this setting, variations in the column norms of $\bm{A}$ introduce additional scale imbalance in how different input coordinates are projected into the low-rank latent space. We therefore normalize each column of $\bm{A}$ along the LoRA rank dimension. For $\bm{A}\in\mathbb{R}^{r\times k}$, we write

$ \bm{A} = \left[\bm{a}_1, , \bm{a}_2, , \dots, , \bm{a}_k \right], \qquad \bm{a}_j\in\mathbb{R}^{r},\tag{6} $

where $\bm{a}_j$ denotes the latent projection vector associated with the $j$-th input coordinate. We define

$ \text{Norm}(\bm{A}) = \left[\frac{\bm{a}_1} {\max(\lVert\bm{a}_1\rVert_2, \epsilon)}, , \dots, , \frac{\bm{a}_k} {\max(\lVert\bm{a}_k\rVert_2, \epsilon)} \right],\tag{7} $

where $\epsilon$ is a small constant for numerical stability. Thus, we have that

$ \left| \left[\text{Norm}(\bm{A}) \right] _{:, j} \right|_2 = 1, \qquad j=1, \dots, k.\tag{8} $

Based on this rank-dimension normalization, we introduce normalized low-rank adaptation, which parameterizes the low-rank update as the following form:

$ \Delta\bm{y} = \alpha \bm{B}\text{Norm}(\bm{A})\bm{x}.\tag{9} $

Unlike latent-feature normalization in MLA, $\text{Norm}(\cdot)$ depends only on the LoRA parameters rather than the input. NoRA therefore imposes a normalization constraint on the down-projection throughout training while remaining linear with respect to $\bm{x}$. After training, the normalized down-projection can be absorbed into the low-rank update, preserving exact weight mergeability. In this way, every input coordinate is continuously projected through a unit-norm latent direction, removing arbitrary column-wise scale variation in $\bm{A}$. The resulting transition can be summarized as

$ \underbrace{ \bm{B}\text{Norm}(\bm{A}\bm{x}) }{\text{\textbf{MLA}: input-dependent latent normalization}} \quad\rightarrow\quad \underbrace{ \bm{B}\text{Norm}(\bm{A})\bm{x} }{\text{\textbf{NoRA}: normalized low-rank projection}}.\tag{10} $

Our initialization and gradient-norm analyses further suggest that much of the benefit of NoRA is already established at the beginning of optimization. This motivates a simpler initialization-only variant, which we denote as NoRA-init. Specifically, we initialize

$ \bm{A}^{(0)} = \text{Norm}(\bm{A}_{\text{init}}), \qquad \bm{B}^{(0)} = \bm{0},\tag{11} $

and subsequently optimize $\bm{A}$ and $\bm{B}$ using the standard LoRA parameterization without performing normalization throughout training. $\bm{A}_\text{init}$ denotes the random initialization in LoRA. Despite removing the persistent constraint during training, NoRA-Init retains most of the optimization benefit of NoRA in our experiments. This observation suggests that controlling the scale of the input-to-latent projection at initialization is particularly important for the early optimization of low-rank adaptation.

3.2 Connection to MiSS and Block Identity Matrix Initialization

Having established the effectiveness of rank-dimension normalization, we further investigate alternative approaches for achieving this form of normalization. MiSS ([14]) is a recent PEFT method that improves parameter efficiency through a matrix-shard sharing strategy. Consider an input $\bm{x}\in\mathbb{R}^{k}$ with $k=br$, partitioned into $b$ blocks $\bm{x}^{(i)}\in\mathbb{R}^{r}$. The MiSS update is written as

$ \Delta \bm{y}_{\mathrm{MiSS}}=\Delta \bm{W}\bm{x}

\alpha \cdot \operatorname{expand}(\bm{B})\bm{x}

\alpha \bm{B}\sum_{i=1}^{b}\bm{x}^{(i)}

\alpha \bm{B} \underbrace{ \left[\bm{I}r, , \dots, , \bm{I}r \right] }{\bm{A}{\mathrm{fix}}} \bm{x}.\tag{12} $

Thus, MiSS is equivalent to a LoRA-style update with a fixed down-projection $\bm{A}_{\mathrm{fix}}$ composed of repeated identity matrices. Importantly, each column of this projection has unit norm:

$ \left| (\bm{A}{\mathrm{fix}}){:, j} \right|_2

1, \qquad \forall j.\tag{13} $

MiSS therefore can be interpreted as a special case of NoRA, as its down-projection matrix is normalized by a block-identity construction (i.e., $\bm{A}$ is fixed to the constant $\bm{A}_{\text{fix}}$). As shown in Figure 1, NoRA produces substantially stronger early gradients than standard LoRA, with gradient norms approaching those of full finetuning. Interestingly, MiSS also exhibits a highly similar pattern. These results well justifies the empirical effectiveness of NoRA and its special case MiSS.

Motivated by the connection to MiSS, we introduce Block Identity Matrix Initialization (BIMI) as a simple structured realization of NoRA-init. Rather than fixing the block-identity projection throughout training, BIMI uses it only to initialize the LoRA down-projection matrix and allows it to remain fully trainable afterward. Specifically, for $k=br+q$, where $0\leq q<r$, we initialize

$ \bm{A}_{\text{bimi}}

\left[\underbrace{ \bm{I}_r, , \bm{I}_r, , \dots, , \bm{I}r }{b\ \text{blocks}}, \bm{E}_q \right] \in\mathbb{R}^{r\times k},\tag{14} $

where $\bm{E}q\in\mathbb{R}^{r\times q}$ contains the first $q$ columns of $\bm{I}r$ and is omitted when $q=0$. By construction, we have $|(\bm{A}{\text{bimi}}){:, j}|_2=1, \forall j, $, making BIMI directly a special case of NoRA-init. The resulting update retains the standard LoRA form, $\Delta \bm{y}=\alpha \bm{B}\bm{A}\bm{x}$, with both $\bm{A}$ and $\bm{B}$ optimized during training.

3.3 Why Does NoRA Work? A Preconditioning Perspective

The early optimization of LoRA is dictated by the initialization of the down-projection matrix $\bm{A}$. This is because LoRA can be viewed as performing full-finetuning gradient descent under a hidden preconditioner that is determined by $\bm{A}$ and acts on the input coordinates. Let $\bm{G}=\partial\mathcal{L}/\partial\bm{W}\in \mathbb{R}^{d\times k}$ be the full-finetuning gradient. A gradient step of size $\eta$ on $\bm{B}$ at initialization is $\Delta\bm{B}=-\eta\alpha, \bm{G}\bm{A}^{\top}$, while $\bm{A}$ receives no gradient and stays fixed. The induced change of the merged weight is therefore

$ \Delta\bm{W}= \alpha, \Delta\bm{B}, \bm{A} = -\eta, \bm{G}, \bm{P}, \qquad \bm{P} = \alpha^{2}\bm{A}^{\top}\bm{A}\in \mathbb{R}^{k\times k}.\tag{15} $

Full finetuning takes the step $\Delta {\bm{W}}=-\eta, {\bm{G}}$, i.e., equation 15 with ${\bm{P}}= {\bm{I}}$. Hence, at initialization, and approximately for as long as ${\bm{B}}$ remains small (the neglected term $\alpha {\bm{B}}, \Delta {\bm{A}}=-\eta\alpha^{2}{\bm{B}} {\bm{B}}^{\top}{\bm{G}}$ is second order in ${\bm{B}}$), LoRA is full finetuning with the gradient right-multiplied by a positive semidefinite matrix of rank $r$ that acts on the input coordinates. Right-multiplication by a $k\times k$ matrix is exactly where curvature-based methods ([18]) place an input-side preconditioner. Moreover, we observe that column norms are per-coordinate learning rates:

$ \Delta {\bm{W}}{:, j}=-\eta\Big[; \underbrace{\alpha^{2}| {\bm{a}}j|2^{2}; {\bm{G}}{:, j}}{\text{coordinate }j\text{'s own update}} ;+; \underbrace{\sum\nolimits{i\neq j}\alpha^{2}({\bm{a}}i^{\top}{\bm{a}}j), {\bm{G}}{:, i}}{\text{crosstalk from the rank bottleneck}} \Big].\tag{16} $

The first term is the full-finetuning update of the weight column with respect to input coordinate $j$, scaled by the squared length of its latent $\bm{a}_j$. The second term is crosstalk between input coordinates induced by the rank bottleneck, and its coefficients are inner products of latent directions.

Random initialization of $\bm{A}$ makes learning rates unbalanced at each coordinate. For example, when ${\bm{A}}^{(0)}$ has zero-mean i.i.d. entries of variance $\sigma^{2}\propto1/k$, $\mathbb{E}| {\bm{a}}_j|_2^{2}=r\sigma^{2}\propto r/k\ll1$. Therefore, at any moderate $\alpha$, every learning rate is far too small, so gradient flows through the adapter at a small fraction of the full-finetuning rate. This is empirically verified by the small gradient norm of LoRA in Figure 1. The learning rates are also unbalanced. $| {\bm{a}}_j|_2^{2}$ fluctuates around its mean with relative spread of order $1/\sqrt{r}$, so at small rank each coordinate gets its learning rate randomly, unrelated to data or curvature. A preconditioner should remove distortion, not add its own.

Normalization is exactly the fix. Setting every $| {\bm{a}}_j|_2=1$ at unit scaling (rank-dimension normalization in NoRA), $\alpha=1$ (equivalently $\alpha=r$ under the $\alpha/r$ implementation convention), fixes the problems. Specifically, this leads to $\text{Diag}(\bm{P})= {\bm{I}}$ deterministically, $\mathbb{E}[\bm{P}]=\bm{I}$ over the random directions, and the adapter's gradient norm can match full finetuning's gradient norm in expectation, independently of $r$. Therefore, we can get the flat NoRA curve in Figure 1. In contrast, row normalization ($\text{Norm}_k$) leaves $\text{Diag}(\bm{P})$ random of order $r/k$, and brings no empirical gain (see Table 3). Moreover, BIMI uses unit basis vectors as columns (the same diagonal through a completely different crosstalk pattern) and performs similarly to rank-dimension normalization of random initializations (see Table 3), which points to the diagonal of $\bm{P}$ as the decisive quantity.

Initialization does most of the work; the constraint adds stability. Since ${\bm{A}}$ 's gradient, $\alpha {\bm{B}}^{\top}{\bm{G}}$, vanishes with ${\bm{B}}$, ${\bm{A}}^{(0)}$ alone fixes the learning rates and the input subspace during the decisive early phase, correcting the rates once at initialization thus captures most of the benefit. This is exactly what NoRA-init does. Performing $\text{Norm}({\bm{A}})$ in the forward pass adds two things: (1) the loss becomes invariant to each column's scale, so gradients are tangential and, under gradient descent, norms can only grow while the effective step anneals automatically, as in weight-normalized training ([10, 19, 11]); and (2) $\text{Diag}({\bm{P}})= {\bm{I}}$ is enforced for the whole run on a compact set of directions, so the latent scale can neither collapse nor explode and the update stays linear in ${\bm{x}}$ and can be merged exactly. To summarize, NoRA sets the diagonal gains of LoRA's hidden preconditioner to one, and keeps them throughout training.

4. Experiments and Results

Section Summary: The experiments evaluate the NoRA method through pretraining, supervised fine-tuning, and reinforcement learning stages, along with targeted ablations on normalization choices and weight initializations. Results indicate that row-wise normalization of the down-projection matrix consistently boosts performance and training stability across random and structured initializations, yielding faster convergence and higher accuracy on downstream tasks compared with standard low-rank approaches. In pretraining especially, the method avoids common optimization failures such as vanishing gradients when adapting attention layers, while maintaining the efficiency benefits of low-rank updates.

To thoroughly evaluate our method, we conduct experiments across pretraining, supervised finetuning (SFT), and reinforcement learning. We further perform ablation studies to investigate the effect of the normalization dimension and initialization distribution. Evaluation settings are in Table 2.

Ablation Study. We investigate the effect of the normalization dimension of the down-projection matrix $\bm{A}$ by comparing column-wise and row-wise normalization, and further evaluate the effectiveness of NoRA across different random or deterministic initialization schemes.

Pretraining. We begin with controlled studies on the MLA architecture under the L24-D1024 setting to examine the effect of intermediate normalization within low-rank latent representations. Building on these observations, we further evaluate NoRA on standard multi-head attention (MHA), where all attention projection layers are adapted with NoRA.

Supervised finetuning. We evaluate NoRA on Llama 3.2-3B across a diverse set of downstream tasks, including mathematical reasoning and code generation, as well as its robustness against catastrophic forgetting ([20]). More detailed settings are given in Appendix B.

Reinforcement learning with verifiable rewards. To assess its effectiveness in post-training optimization, we additionally evaluate NoRA under reinforcement learning using the ReRL framework.

::: {caption="Table 2: Overview of all the training settings in our experiments."}

:::

4.1 Effect of the Normalization Dimension

::: {caption="Table 3: Effect of different normalization dimensions under supervised finetuning. The average is computed over accuracies on GSM8K and Math."}

{width=60%}

:::

We first study whether the effect of the normalization dimension under different initialization schemes. Given $\bm A\in\mathbb{R}^{r\times k}$, $\text{Norm}_k$ normalizes each row, while $\text{Norm}r$ normalizes each column $\bm A{:, j}\in\mathbb{R}^r$. As shown in Table 3, $\text{Norm}_k$ provides little improvement, whereas $\text{Norm}_k$ consistently improves Random Uniform, Gaussian, and Kaiming Uniform initializations. After applying NoRA, all three initialization schemes achieve similar performance, indicating that its benefit is largely independent of the initialization distribution. BIMI achieves comparable performance and satisfies NoRA by construction, serving as a structured and deterministic instance of NoRA.

4.2 LLM Pretraining

We first conduct LLM pretraining experiments using the standard MLA architecture. Normalizing the latent representation $\bm{A}\bm{x}$ consistently improves model performance and accelerates convergence during pretraining. Motivated by these results, we extend the same normalization principle to the linear low-rank formulation by applying normalization to the down-projection $\bm{A}$, yielding NoRA. To evaluate the effectiveness of NoRA for LLM pretraining, we adopt NoRA-init and compare it with the standard low-rank parameterization. As shown in Table 4, NoRA-init achieves faster convergence and better downstream performance. Although its performance remains slightly below that of directly normalizing $\bm{A}\bm{x}$, NoRA-init preserves the linear structure $\bm{B}\bm{A}\bm{x}$, thereby retaining the parameter mergeability and deployment efficiency of conventional low-rank parameterizations.

We further evaluate NoRA-init under the standard MHA architecture for LLM pretraining. Specifically, we parameterize all linear projections in the attention module, including $\bm{q}{\mathrm{proj}}$, $\bm{k}{\mathrm{proj}}$, $\bm{v}{\mathrm{proj}}$, and $\bm{o}{\mathrm{proj}}$, using low-rank factors with rank $r=64$. As shown in Table 4, the standard low-rank parameterization exhibits a severe performance collapse on LAMBADA, along with substantial degradation on several other benchmarks. An examination of the optimization dynamics reveals that its gradient norms diminish to abnormally low levels during training, suggesting insufficient optimization. In contrast, NoRA-init achieves significantly better performance than standard low-rank parameterization and preserves well-scaled gradient magnitudes throughout training, thereby avoiding such optimization collapse and yielding more stable loss trajectories and downstream performance.

This observation is related to the capacity-limited behavior of low-rank adaptation discussed in [44], where insufficient low-rank capacity can largely constrain the flexibility of optimization trajectory. Our results further show that capacity alone does not fully determine optimization behavior. Under the same rank constraint, modifying the initialization of the low-rank projection substantially changes the training dynamics. This suggests that controlling the scale and structure of the input-to-latent projection provides a more favorable optimization geometry under limited-rank parameterizations, thereby alleviating performance degradation in pretraining.

::: {caption="Table 4: Zero-shot performance of 340M models trained on SlimPajama ([24]). Commonsense reasoning tasks are evaluated with lm-evaluation-harness ([45]); the recall-intensive task follows prefix-linear-attention ([46]) with 2K input tokens. '-' indicates a collapse in perplexity."}

:::

::: {caption="Table 5: Comparison of PEFT methods under supervised finetuning and retained benchmark settings."}

:::

4.3 Supervised Finetuning

Table 5 compares NoRA with representative PEFT methods under supervised finetuning. NoRA achieves the best overall SFT performance among the compared methods, improving the average score from 37.93 with standard LoRA to 43.37, a gain of 5.44 points. In particular, NoRA substantially improves performance on GSM8K and HumanEval, reaching 61.63 and 42.10, respectively, while also outperforming PiSSA ([4]), OFT ([47, 48, 49]), RSLoRA ([7]), and MiSS ([14]) in average performance. These results demonstrate the effectiveness of rank-dimension normalization for improving LoRA.

We further compare NoRA with its initialization-only variant, NoRA-init. NoRA-init applies normalization constraint only to the initial down-projection and then follows standard LoRA training, whereas NoRA continuously enforces the normalization constraint throughout training. NoRA-init already improves the SFT average from 37.93 to 42.38, capturing a large portion of the gain achieved by NoRA. With the normalization constraint maintained throughout training, NoRA further improves the average score to 43.37. The comparison indicates that controlling the input-to-latent projection magnitude at initialization accounts for a large portion of the gain, while persistent normalization provides additional improvements. Comparison of training dynamics is given in Figure 2. NoRA shows stronger learnability than LoRA without introducing extra parameters.

NoRA also outperforms MiSS, which achieves an average score of 42.70. Recall that MiSS can be reformulated as a fixed block-identity down-projection satisfying the normalization constraint. The strong performance of both MiSS and NoRA-init therefore provides further evidence that normalized low-rank projections play an important role in their optimization. Notably, NoRA-init achieves comparable performance to MiSS while keeping the down-projection fully trainable, avoiding the constraint of fixing the projection throughout training.

Finally, we examine whether the NoRA principle generalizes beyond the standard LoRA parameterization. We apply $\text{Norm}_r$-based normalization to DoRA and obtain consistent improvements: NoRA-init-DoRA improves the SFT average from 38.30 for DoRA to 41.40. This result demonstrates that the benefit of rank-dimension normalization is not specific to LoRA, but can also be transferred to other low-rank adaptation variants. Together with the improvements observed for LoRA, this suggests that NoRA captures a general optimization principle for low-rank adaptation. NoRA also exhibits favorable knowledge retention, achieving an average change of +0.02 on MMLU, AGIEval, and ARC-C, compared with $-0.56$ for LoRA and -0.70 for MiSS. Thus, NoRA improves adaptation performance while maintaining the pretrained model's retained knowledge.

4.4 RLVR on Mathematical Reasoning

We further evaluate NoRA under reinforcement learning with verifiable rewards (RLVR) on challenging mathematical reasoning benchmarks. As shown in Table 6, standard LoRA improves the overall average score from 41.0 for the base model to 42.8, while NoRA further increases it to 44.4, outperforming the base model and standard LoRA by 3.4 and 1.6 points, respectively. NoRA yields broad improvements across the evaluated benchmarks, with particularly clear gains on AMC, MATH500, and Minerva. These results suggest that rank-dimension normalization remains effective under RLVR and facilitates more effective optimization of low-rank adapters.

A notable difference emerges for initialization methods based on pretrained-weight spectral decomposition. Although PiSSA ([4]) and MiLoRA ([5]) are effective under supervised finetuning, their performance degrades substantially in the RLVR setting. This behavior is consistent with prior observations that spectral initialization methods can suffer from optimization instability during reinforcement learning ([6]). In contrast, NoRA remains stable and consistently improves the overall performance of standard LoRA without relying on singular-value decomposition of the pretrained weights. These results highlight the robustness of NoRA across different optimization regimes. Unlike PiSSA and MiLoRA, NoRA does not depend on the spectral structure of pretrained weights, while preserving the lightweight low-rank parameterization and deployment efficiency of standard LoRA. Together with the supervised finetuning results, the RLVR experiments suggest that controlling the scale of the input-to-latent projection provides a simple and broadly effective mechanism for improving low-rank optimization.

::: {caption="Table 6: Comparison of accuracy and pass rate on the RLVR task. All numbers are reported in percentages."}

:::

5. Related Work and Concluding Remarks

Section Summary: Researchers have explored many ways to adapt large language models to new tasks without updating all parameters, with the LoRA method becoming popular for its simplicity and efficiency. Other recent variants try to improve LoRA by using special initialization techniques such as singular value decomposition or gradient information, yet these often require extra computation or data. In contrast, the present work introduces NoRA, which normalizes the down-projection matrix to create a better-conditioned optimization space, yielding steadier training and stronger results across pretraining, fine-tuning, and reinforcement learning without added overhead or loss of mergeability.

Parameter-efficient finetuning. As LLMs continue to grow in scale and capability, there has been increasing interest in adapting them to downstream tasks in a parameter-efficient manner ([50, 51, 1, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 47, 48, 49, 73, 74]). Among them, LoRA ([1]) is widely adopted due to its simplicity, efficiency, and mergeability at inference time. Subsequent variants improve LoRA from different perspectives, such as adaptive rank allocation ([15]), low-bit quantization ([75]) and weight-decomposed low-rank updates (DoRA) ([8]). While these methods improve the capacity, efficiency, or stability of low-rank adaptation, our work focuses on a complementary question: how the low-rank subspace should be initialized for better optimization.

LoRA initialization and optimization. Recent studies show that LoRA is sensitive to initialization, scaling, and early-stage optimization dynamics. PiSSA initializes LoRA from principal singular components of pretrained weights ([4]), OLoRA ([76]) adopts orthogonal initialization, LoRA-GA ([77]) uses gradient information to guide initialization, EVA leverages activation statistics, and rsLoRA improves stability through rank-dependent scaling ([7]). These methods demonstrate the importance of initialization, but often rely on SVD, gradient estimation, activation statistics, or scaling heuristics. In contrast, NoRA requires no additional data, SVD, or test-time overhead while preserving the mergeability of LoRA.

Concluding remarks. This work shows that LoRA’s effectiveness depends not only on its rank, but also critically on the geometry and scale of its down-projection. Inspired by the normalized latent representations in MLA, we ask whether the benefits of latent normalization can be transferred to LoRA without sacrificing its linear structure and exact mergeability. This leads to NoRA, which normalizes the down-projection along the rank dimension and thereby controls the scale of the input-to-latent mapping. From a preconditioning perspective, LoRA can be viewed as full finetuning under an implicit low-rank, input-side preconditioner determined by the down-projection; NoRA's rank-dimension normalization corrects undesirable scale imbalance in this preconditioner and yields a better-conditioned optimization geometry. NoRA can consistently improve convergence, training stability and downstream performance across pretraining, supervised finetuning, and RLVR.

Appendix

Section Summary: The appendix outlines the experimental setups used across three stages of model development. Pretraining runs were performed on the FineWeb-10BT dataset for a fixed number of steps using identical AdamW optimizer settings, learning-rate schedules, batch sizes, and sequence lengths to enable direct comparisons. Supervised fine-tuning experiments individually tuned hyperparameters for each low-rank adaptation method, with a default scaling factor chosen to balance task adaptation against retention of prior knowledge, while reinforcement-learning trials applied consistent LoRA configurations and DAPO training parameters on a math-focused dataset using fixed hardware and sampling settings.

A. Pretraining Settings

We conduct the pre-training experiments on FineWeb-10BT. Unless otherwise specified, all models are trained for 20, 480 optimization steps with a sequence length of 2, 048 and a global batch size of 256. We use AdamW with a peak learning rate of 3 x 10^-4 and $\epsilon=10^{-15}$. The learning rate is warmed up for the first 1, 024 steps and then decayed using a cosine schedule to $10%$ of the peak learning rate. The gradient norm is clipped at 1.0. All experiments use the same training configuration and random seed for controlled comparison.

\begin{tabular}{lc}
\textbf{Hyperparameter} & \textbf{Value} \\
\noalign{\global\savewidth\arrayrulewidth
  \global\arrayrulewidth 1pt}\hline\noalign{\global\arrayrulewidth\savewidth}
Dataset & FineWeb-10BT \\
Optimizer & AdamW \\
Learning rate & $3\times10^{-4}$ \\
Adam $\epsilon$ & $10^{-15}$ \\
LR scheduler & Cosine decay \\
Warmup steps & 1, 024 \\
Minimum LR ratio & 0.1 \\
Global batch size & 256 \\
Sequence length & 2, 048 \\
Training steps & 20, 480 \\
Gradient clipping & 1.0 \\
Random seed & 42 \\
\end{tabular}

B. Supervised Finetuning Settings

For the SFT experiments, we tune the hyperparameters of each method individually to ensure competitive performance. For NoRA, we recommend using $\alpha=r$ as the default setting, under which the early gradient norm is close to that of full finetuning. Although a larger scaling factor, such as $\alpha=2r$, can yield better loss fitting in some cases, we observe that it may also lead to increased forgetting of the pretrained knowledge. Considering this trade-off between adaptation and retention, we use and recommend $\alpha:r=1:1$ as the default configuration.

\begin{tabular}{lcccccc}
\textbf{Hyperparameters} & \textbf{LoRA} & \textbf{DoRA} & \textbf{PiSSA} & \textbf{rsLoRA} & \textbf{OFT} & \textbf{NoRA} \\
\noalign{\global\savewidth\arrayrulewidth
  \global\arrayrulewidth 1pt}\hline\noalign{\global\arrayrulewidth\savewidth}
rank & 32 & 32 & 32 & 32 & 32 & 32 \\
$\alpha$ & 64 & 64 & 32 & 64 & - & 32 \\
\hline
dropout & \multicolumn{6}{c}{0.0} \\
optimizer & \multicolumn{6}{c}{AdamW} \\
lr & \multicolumn{6}{c}{2e-5} \\
lr scheduler & \multicolumn{6}{c}{Cosine decay} \\
batch size & \multicolumn{6}{c}{128} \\
warmup ratio & \multicolumn{6}{c}{0.3} \\
epochs & \multicolumn{6}{c}{1} \\
target\_modules & \multicolumn{6}{c}{Q, K, V, O, Up, Down, Gate} \\
\end{tabular}

C. Reinforcement Learning Settings

We conduct reinforcement learning experiments on DeepSeek-R1-Distill-Qwen-1.5B using the DAPO objective and the DAPO-Math-17K dataset. All experiments are performed with 8 GPUs using bfloat16 precision. We train for 1, 024 optimization steps with a global batch size of 128 and a learning rate of 1 x 10^-5 under a cosine learning-rate schedule without warmup. For each prompt, we sample 8 responses with a maximum completion length of 16, 384 tokens. The maximum prompt length is set to 512 tokens.

For parameter-efficient training, we use a rank of 32 and set $\alpha=64$. The low-rank modules are applied to all attention projections ($q$, $k$, $v$, and $o$) and MLP projections (up, down, and gate). We use the same training configuration across different methods to ensure a controlled comparison.

\begin{tabular}{lc}
\textbf{Hyperparameter} & \textbf{Value} \\
\noalign{\global\savewidth\arrayrulewidth
  \global\arrayrulewidth 1pt}\hline\noalign{\global\arrayrulewidth\savewidth}
Model & DeepSeek-R1-Distill-Qwen-1.5B \\
Dataset & DAPO-Math-17K \\
RL objective & DAPO \\
Precision & bfloat16 \\
Learning rate & $1\times10^{-5}$ \\
LR scheduler & Cosine decay \\
Warmup ratio & 0 \\
Global batch size & 128 \\
Training steps & 1, 024 \\
LoRA rank $r$ & 32 \\
LoRA $\alpha$ & 64 \\
LoRA dropout & 0.05 \\
Responses per prompt & 8 \\
Maximum prompt length & 512 \\
Maximum completion length & 16, 384 \\
$\epsilon_{\mathrm{high}}$ & 0.28 \\
$\beta$ & 0.0 \\
Random seed & 42 \\
\end{tabular}

References

Section Summary: This section lists dozens of academic papers, preprints, and datasets that support the work, with a heavy focus on methods for efficiently adapting large language models through low-rank updates and related fine-tuning techniques. It also includes references to benchmarks for mathematical and commonsense reasoning, evaluation tools, and foundational work on transformers and optimization. Together, these citations show the research landscape the authors drew from when developing and testing their approach.

[1] Hu et al. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

[2] Wang et al. (2026). Tina: Tiny reasoning models via lora. In ICLR.

[3] Shen et al. (2026). On the Geometry of On-Policy Distillation. arXiv preprint arXiv:2606.07082.

[4] Meng et al. (2024). Pissa: Principal singular values and singular vectors adaptation of large language models. In NeurIPS.

[5] Wang et al. (2025). Milora: Harnessing minor singular components for parameter-efficient llm finetuning. In NAACL-HLT.

[6] Yin et al. (2025). Evaluating parameter efficient methods for rlvr. arXiv preprint arXiv:2512.23165.

[7] Kalajdzievski, Damjan (2023). A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732.

[8] Liu et al. (2024). Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353.

[9] Kopiczko et al. (2024). Vera: Vector-based random matrix adaptation. In ICLR.

[10] Liu et al. (2017). Deep hyperspherical learning. In NeurIPS.

[11] Liu et al. (2018). Decoupled networks. In CVPR.

[12] Loshchilov et al. (2025). ngpt: Normalized transformer with representation learning on the hypersphere. In ICLR.

[13] Hayou et al. (2024). Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354.

[14] Kang, Jiale and Yin, Qingyu (2026). Miss: Revisiting the trade-off in lora with an efficient shard-sharing structure. In ICLR.

[15] Zhang et al. (2023). Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.

[16] Zhang et al. (2023). LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning. arXiv preprint arXiv:2308.03303.

[17] Ye et al. (2025). Vera: Explainable video anomaly detection via verbalized learning of vision-language models. In CVPR.

[18] Martens, James and Grosse, Roger (2015). Optimizing neural networks with kronecker-factored approximate curvature. In ICML.

[19] Salimans, Tim and Kingma, Durk P (2016). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NeurIPS.

[20] Huang et al. (2026). PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective. arXiv preprint arXiv:2605.28819.

[21] Yang, Songlin and Zhang, Yu (2024). FLA: A Triton-Based Library for Hardware-Efficient Implementations of Linear Attention Mechanism. https://github.com/fla-org/flash-linear-attention.

[22] Liu et al. (2024). Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434.

[23] Vaswani et al. (2017). Attention is all you need. Advances in neural information processing systems. 30.

[24] Soboleva et al. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.

[25] Paperno et al. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. In ACL.

[26] Merity et al. (2016). Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.

[27] Clark et al. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

[28] Zellers et al. (2019). Hellaswag: Can a machine really finish your sentence?. In ACL.

[29] Bisk et al. (2020). Piqa: Reasoning about physical commonsense in natural language. In AAAI.

[30] Mihaylov et al. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.

[31] Sakaguchi et al. (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM. 64(9). pp. 99–106.

[32] Grattafiori et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

[33] Yu et al. (2024). Metamath: Bootstrap your own mathematical questions for large language models. In ICLR.

[34] Zheng et al. (2024). Opencodeinterpreter: Integrating code generation with execution and refinement. In Findings of ACL.

[35] Cobbe et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

[36] Chen et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

[37] Austin et al. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

[38] Yu et al. (2026). Dapo: An open-source llm reinforcement learning system at scale. In NeurIPS.

[39] Zhang, Yifan and Math-AI, Team (2025). American Invitational Mathematics Examination (AIME) 2025.

[40] Lightman et al. (2024). Let's verify step by step. In ICLR.

[41] Lewkowycz et al. (2022). Solving quantitative reasoning problems with language models. In NeurIPS.

[42] Li et al. (2024). Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 13(9). pp. 9.

[43] Balunović et al. (2025). Matharena: Evaluating llms on uncontaminated math competitions. arXiv preprint arXiv:2505.23281.

[44] John Schulman and Thinking Machines Lab (2025). LoRA Without Regret. Thinking Machines Lab: Connectionism. doi:10.64434/tml.20250929.

[45] Gao et al. (2024). The Language Model Evaluation Harness. doi:10.5281/zenodo.12608602. https://zenodo.org/records/12608602.

[46] Arora et al. (2024). Just read twice: closing the recall gap for recurrent language models. arXiv preprint arXiv:2407.05483.

[47] Qiu et al. (2023). Controlling Text-to-Image Diffusion by Orthogonal Finetuning. In NeurIPS.

[48] Qiu et al. (2025). Orthogonal Finetuning Made Scalable. In EMNLP.

[49] Liu et al. (2024). Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. In ICLR.

[50] Houlsby et al. (2019). Parameter-efficient transfer learning for NLP. In ICML.

[51] Aghajanyan et al. (2020). Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255.

[52] Edalati et al. (2022). Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650.

[53] Wang et al. (2022). Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. In EMNLP.

[54] Gheini et al. (2021). Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation. In EMNLP.

[55] Zaken et al. (2022). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In ACL.

[56] Guo et al. (2020). Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463.

[57] Sung et al. (2021). Training neural networks with fixed sparse masks. NeurIPS.

[58] Ansell et al. (2022). Composable Sparse Fine-Tuning for Cross-Lingual Transfer. In ACL.

[59] Lester et al. (2021). The power of scale for parameter-efficient prompt tuning. In EMNLP.

[60] Li, Xiang Lisa and Liang, Percy (2021). Prefix-tuning: Optimizing continuous prompts for generation. In ACL.

[61] Vu et al. (2022). SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer. In ACL.

[62] He et al. (2021). Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366.

[63] Mao et al. (2021). Unipelt: A unified framework for parameter-efficient language model tuning. arXiv preprint arXiv:2110.07577.

[64] Karimi Mahabadi et al. (2021). Compacter: Efficient low-rank hypercomplex adapter layers. In NeurIPS.

[65] Liu et al. (2022). Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NeurIPS.

[66] Sung et al. (2022). Lst: Ladder side-tuning for parameter and memory efficient transfer learning. In NeurIPS.

[67] Chen et al. (2023). Parameter-Efficient Fine-Tuning Design Spaces. In ICLR.

[68] Jia et al. (2022). Visual prompt tuning. In ECCV.

[69] Chen et al. (2022). Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS.

[70] Zhang et al. (2022). Neural prompt search. arXiv preprint arXiv:2206.04673.

[71] Jie, Shibo and Deng, Zhi-Hong (2023). Fact: Factor-tuning for lightweight adaptation on vision transformer. In AAAI.

[72] Lian et al. (2022). Scaling & shifting your features: A new baseline for efficient model tuning. In NeurIPS.

[73] Wu et al. (2024). Mixture-of-subspaces in low-rank adaptation. arXiv preprint arXiv:2406.11909.

[74] Zi et al. (2023). Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices. arXiv preprint arXiv:2309.02411.

[75] Dettmers et al. (2023). Qlora: Efficient finetuning of quantized llms. In NeurIPS.

[76] Büyükakyüz, Kerim (2024). Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2406.01775.

[77] Wang et al. (2024). Lora-ga: Low-rank adaptation with gradient approximation. In NeurIPS.