Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta
Facebook
{armenag, lsz, sonalgupta}@fb.com

Abstract

Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine-tuning through the lens of intrinsic dimension provides us with empirical and theoretical intuitions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; in other words, there exists a low dimension reparameterization that is as effective for fine-tuning as the full parameter space. For example, by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre-training implicitly minimizes intrinsic dimension and, perhaps surprisingly, larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness. Lastly, we connect intrinsic dimensionality with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalization bounds that are independent of the full parameter count.

Executive Summary: Pre-trained language models with hundreds of millions of parameters achieve strong results on NLP tasks after fine-tuning on datasets that often contain only hundreds or thousands of examples. This outcome is surprising because standard gradient descent appears sufficient without aggressive regularization or other safeguards against overfitting, and the underlying reasons have remained unclear.

The paper set out to explain this behavior by measuring the intrinsic dimension of fine-tuning objectives. Intrinsic dimension is the smallest number of free parameters needed to reach a solution that is within 90 percent of the performance obtained by optimizing all model weights. The authors treat pre-training as implicitly learning a compact framework for representing typical NLP tasks and test whether that framework reduces the number of parameters actually required at fine-tuning time.

They evaluated this idea empirically on multiple GLUE tasks using BERT and RoBERTa models. A random-projection method (Fastfood transform) re-parameterized the full weight space into a much smaller subspace; only a few hundred trainable parameters were optimized while the rest remained fixed. They tracked how intrinsic dimension changed during pre-training from scratch, compared dozens of existing pretrained models that differ in size and method, and derived generalization bounds that depend only on the reduced dimension.

The central results are that common tasks can be solved to 90 percent of full performance with a few hundred parameters (207 for RoBERTa-Large on MRPC), that continued pre-training steadily lowers intrinsic dimension even though the language-modeling objective never sees the downstream tasks, and that larger models consistently exhibit lower intrinsic dimension. In addition, lower intrinsic dimension reliably predicts smaller generalization gaps on held-out data, and standard compression arguments yield generalization bounds whose size depends on the intrinsic dimension rather than the total number of model parameters.

These findings indicate that pre-training succeeds largely by compressing the average downstream task into a low-dimensional subspace inside the model. Larger models appear advantageous precisely because they make that compression easier. The work therefore supplies both an explanation for why simple fine-tuning succeeds and a quantitative link between model scale, task complexity, and generalization.

The analysis rests on upper bounds obtained from random projections and on a modest set of models and tasks; tighter estimates or broader coverage could alter the exact numbers. The theoretical bounds apply directly only to the subspace-training procedure, not yet to ordinary fine-tuning. Readers should therefore treat the reported dimensions as useful but conservative reference points rather than exact minima.

For decision makers, the results support continued investment in larger pretrained models when the goal is rapid adaptation to new tasks with limited labels. They also suggest that future efficiency efforts should target further reduction of intrinsic dimension, for example through architecture choices or pre-training objectives that explicitly encourage compressibility. Additional work is needed to confirm whether ordinary gradient descent implicitly optimizes within similarly low-dimensional subspaces.

1. Introduction

Section Summary: Pre-trained language models with hundreds of millions of parameters are routinely fine-tuned on tiny labeled datasets using ordinary gradient descent, yet it is not obvious why this succeeds or how much of the full model is truly necessary. The authors introduce intrinsic dimensionality—the smallest number of adjustable parameters needed to reach near-optimal performance on a task—as a way to measure this, demonstrating that many NLP problems can be solved to within 90 percent of full accuracy using only a few hundred dimensions that are then projected back into the larger space. They further show that pre-training itself reduces this dimensionality, that bigger models tend to require even fewer effective parameters, and that the approach supports generalization bounds independent of total model size.

Pre-trained language models ([1, 2, 3, 4, 5]) provide the defacto initialization for modeling most existing NLP tasks. However, the process of fine-tuning them on often very small target task datasets remains somewhat mysterious. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples?

We propose intrinsic dimensionality as a new lens through which fine-tuning can be analyzed ([6]). An objective function's intrinsic dimensionality describes the minimum dimension needed to solve the optimization problem it defines to some precision level. In the context of pre-trained language models, measuring intrinsic dimensional will tell us how many free parameters are required to closely approximate the optimization problem that is solved while fine-tuning for each end task. For example, we will show that 200 parameters (randomly projected back into the full parameter space) are enough to represent the problem of tuning a RoBERTa model to within 90% of the performance of the full model. More generally, we also describe a set of strong empirical and theoretical connections between intrinsic dimensionality, number of parameters, pre-training, and generalization.

We first empirically show that standard pre-trained models can learn a large set of NLP tasks with very few parameters and that the process of pre-training itself implicitly minimizes the intrinsic dimension of later tuning for different NLP tasks. We continue by conducting a study across over a dozen various pre-trained models to show that number of parameters strongly inversely correlates with intrinsic dimensionality, at least in part to justify the extreme effectiveness of such models. We interpret pre-training as providing a framework that learns how to compress the average NLP task. Finally, we connect intrinsic dimensional with low dimensional task representations and compression based generalization bounds to provide intrinsic-dimension-based generalization bounds that are independent of the full parameter count, further justifying why these methods generalize so well in practice across tasks.

The contributions of our paper are the following:

• We empirically show that common NLP tasks within the context of pre-trained representations have an intrinsic dimension several orders of magnitudes less than the full parameterization.
• We propose a new interpretation of intrinsic dimension as the downstream fine-tuning task's minimal description length within the framework of the pre-trained model. Within this interpretation, we empirically show that the process of pre-training implicitly optimizes the description length over the average of NLP tasks, without having direct access to those same tasks.
• We measure the intrinsic dimension of a large set of recently developed pre-training methods. We discover that there exists a fortuitous trend where larger models tend to have a smaller intrinsic dimension.
• Lastly, we show that compression based generalization bounds can be applied to our intrinsic dimension framework to provide generalization bounds for large pre-trained models independent of the pre-trained model parameter count.

2. Related Work

Section Summary: Previous research calculated the intrinsic dimension of neural network objective functions to study how model architecture affects optimization difficulty. Related analyses of pre-trained models such as BERT have shown they contain substantial redundancy, so they can be heavily pruned or fine-tuned using only a tiny fraction of parameters while retaining most performance; the intrinsic-dimension approach offers a continuous view of these sparsity questions. At the same time, existing work leaves open why standard fine-tuning so reliably produces good generalization and why different pre-training schemes yield broadly useful representations.

Calculating the intrinsic dimension of an objective function was proposed [6]. In their paper, they analyzed the impact of various architectures on the intrinsic dimensionality of their objective. Our work is a direct extension of this paper, focusing on analyzing pre-trained representations instead.

There is a large collection of literature analyzing pre-trained models from the perspective of capacity. For example, a recent line of work has shown that pre-trained models such as BERT are redundant in their capacity, allowing for significant sparsification without much degradation in end metrics ([7, 8, 9]). [10] showed that fine-tuning top layers of pre-trained models is not effective and that alternate methods allow fine-tuning effectively with a couple of percent of the parameters. Furthermore, we can view computing the intrinsic dimensionality as a continuous relaxation of the sparsification problem.

Moreover, standard approaches towards fine-tuning seem to have non-trivial effects on the generalization of pre-trained representations ([11]). A holistic explanatory picture of the successes of fine-tuning has not yet been painted. A clear understanding of the underlying mechanisms which lead to the incredible generalization of fine-tuned pre-trained representations is currently missing. Moreover, we still do not understand why various pre-training methodology manifests in universally useful representations.

3. Intrinsic Dimensionality of Finetuning

Section Summary: The section explains that intrinsic dimensionality refers to the smallest number of adjustable parameters needed to fine-tune a large model to near its full performance, by optimizing within a random low-dimensional subspace (via efficient projections like the Fastfood transform) rather than the original high-dimensional weight space. Standard methods treat all parameters uniformly and search for the minimal such dimension that recovers 90% of full accuracy, but this ignores how different layers contribute distinctly to the task. To address this, the authors propose a structure-aware variant that introduces per-layer scaling factors, allowing the low-dimensional optimization to allocate capacity more effectively across layers.

Background

An objective function's intrinsic dimension measures the minimum number of parameters needed to reach satisfactory solutions to the respective objective ([6]). Alternatively, the intrinsic dimension represents the lowest dimensional subspace in which one can optimize the original objective function to within a certain level of approximation error. Computing the exact intrinsic dimensional of the objective function is computation intractable; therefore, we resort to heuristic methods to calculate an upper bound. Let $\theta^{D}=\left[\theta_0, \theta_1, ..., \theta_m\right]$ be a set of $D$ parameters that parameterize some model $f(\cdot, \theta)$. Instead of optimizing the empirical loss in the original parameterization ($\theta^{D}$), the subspace method fine-tunes the model via the following re-parametrization in the lower-dimensionsal $d$-dimensions:

$ \theta^{D} = \theta^{D}_0 + P(\theta^{d})\tag{1} $

where $P: \mathbb{R}^d \rightarrow \mathbb{R}^D$ projects from a parameter from a lower dimensional $d$ to the higher dimensional $D$. Intuitively, we do an arbitrary random projection onto a much smaller space; usually, a linear projection, we then solve the optimization problem in that smaller subspace. If we reach a satisfactory solution, we say the dimensionality of that subspace is the intrinsic dimension. This methodology was proposed in the seminal paper by [6]. Concretely [6] proposed 3 various actualizations of $P$; a random linear dense projection ($\theta^{d}W$), random linear sparse projection($\theta^{d}W_{\text{sparse}}$) and random linear projection via the Fastfood transform ([12]).

We will primarily use the Fastfood transform, defined as:

$ \begin{aligned} \theta^{D} = \theta^{D}_0 + \theta^{d}M && M=HG\Pi HB \end{aligned}\tag{2} $

The factorization of $M$ consists of $H$, a Hadamard matrix, $G$, a random diagonal matrix with independent standard normal entries, $B$ a random diagonal matrix with equal probability $\pm 1$ entries, and $\Pi$ a random permutation matrix. Furthermore, the matrix multiplication with a Hadamard matrix can be computed in $\mathcal{O}(D \log{d})$ via the Fast Walsh-Hadamard Transform. Note that everything but $\theta_d$ is fixed; therefore, the optimization problem lies only in $d$-dimensions. Note that if we place a constraint of $M$ being a binary matrix, we recover the sparsification problem; therefore, we can view finding intrinsic dimensionality as a continuous relaxation of the sparsification problem.

The standard method of measuring the intrinsic dimensionality of an objective as proposed by [6] requires searching over various $d$, training using standard SGD over the subspace reparameterization $\theta^{D}$ and selecting the smallest $d$ which provides us with a satisfactory solution ($d_{90}$). [6] defined the satisfactory solution as being 90% of the full training metric. For example, if we reach 85% accuracy training a model with all of its parameters, the goal is to find the smallest $d$, which would reach $0.9 * 85% = 76.5%$ accuracy; we call this dimension $d_{90}$. Let us also note that by merely initializing $\theta^d = 0$ we recover the original parameterization $\theta^{D}_0$ which in the context of fine-tuning represents the original weights of the pre-trained model.

The way [6] define a satisfactory solution reduces the dependence of the dataset's size on the calculation of intrinsic dimension. For a small dataset, we will generally have worse end metrics; therefore, we have a lower $d_{90}$ cut-off; inversely, a larger dataset will require a more non-trivial $d_{90}$ cut-off.

Structure Aware Intrinsic Dimension

Due to the large size of pre-trained language models (generally in the hundreds of millions of parameters), the only computationally reasonable subspace optimization method is one that utilizes the Fastfood transform. For example, if we are interested in subspace training with $d=1000$ for the RoBERTa-Large model using a dense matrix, we would require 1.42 terabytes of memory to store just the projection matrix.

Unfortunately, the method of finding the intrinsic dimension proposed by [6] is unaware of the layer-wise structure of the function parameterized by $\theta$. Existing literature argues that in attention-based pre-trained models, individual layers specialize separately ([13]); therefore, it is useful to incorporate a notion of structure when computing $d_{90}$. We define Structure-Aware Intrinsic Dimension (SAID) as the following

$ \theta^{D}i = \theta^{D}{0, i} + \lambda_i P(\theta^{d-m})_i\tag{3} $

For $m$ layers, we trade $m$ parameters from our subspace parameter $\theta_d$ to allow for layer-wise scaling through jointly learned $\lambda$, thus $\theta_d$ becomes $\left[\theta_{d-m}, \lambda\right]$. This allows the SAID method to focus a larger capacity of $\theta^{d-m}$ towards specific layers what might carry more relevant information for the task at hand. Conversely, we will refer to the layer unaware method Equation (2) as the Direct Intrinsic Dimension (DID) method.

4. Intrinsic Dimensionality of Common NLP Tasks

Section Summary: Researchers measured the intrinsic dimensionality of fine-tuning for BERT and RoBERTa models on GLUE sentence-pair tasks by optimizing the models inside random low-dimensional subspaces. They found remarkably low values: RoBERTa-Large reached 90 percent of full fine-tuning accuracy on MRPC with roughly 200 parameters and on QQP with roughly 800, and the structured SAID method consistently outperformed the unstructured baseline. These results indicate that current large language models can solve common NLP tasks in solution spaces far smaller than their total parameter counts.

4.1 Sentence Prediction

We first empirically calculate the intrinsic dimension of various pre-trained models on a set of sentence prediction tasks from the GLUE Benchmark ([14]). We focus on analyzing BERT ([2]) and RoBERTa ([3]) at both the base and large model sizes.

We chose to experiment with MRPC ([15]) and QQP ([16]) as reference examples of small and large tuning datasets. MRPC is a binary classification task for predicting semantic equivalency for two paraphrases with roughly 3700 training samples, while QQP is a binary classification task for predicting semantic equality of two questions, with roughly 363k samples. For every dataset and every model, we run 100 subspace trainings with $d$ ranging from 10 to 10000 on a log scale. For every training run, we do a small hyperparameter search across four learning rates. We initialize every $\theta_d$ to the zero vector to allow for our starting point to be the original pre-trained model. Our subspace optimization method also operates over the randomly initialized sentence classification head to ensure we have exactly $d$ parameters to optimize.

We use both the SAID and DID subspace optimization methods, which we implemented in the Huggingface Transformers library ([17]). We present the results in Figure 1.

\begin{tabular}{lrrrr}\\\toprule 
  & \multicolumn{2}{c}{SAID} & \multicolumn{2}{c}{DID} \\ \cmidrule(lr){2-3}\cmidrule(lr){4-5} 
Model & MRPC & QQP & MRPC & QQP \\\midrule
BERT-Base & 1608 & 8030 & 1861 & 9295 \\  
BERT-Large & 1037 & 1200& 2493 & 1389\\ \midrule
RoBERTa-Base & 896 & 896& 1000 & 1389 \\  
RoBERTa-Large & \bf{207} & \bf{774}& 322 & \bf{774}\\ \bottomrule
\end{tabular}

4.2 Analysis

The first takeaway is the incredible low dimensionality of viable solutions. With RoBERTa-Large, we can reach 90% of the full fine-tuning solution of MRPC using roughly 200 parameters and 800 parameters for QQP (Table 1). Recall that our approximation of intrinsic dimension is necessarily crude by using random projections and restricting them to the use of Fastfood transform; therefore, it is likely that the true intrinsic dimension is much lower.

Furthermore, RoBERTa consistently outperforms BERT across various subspace dimensions $d$ while having more parameters. We leave a more in-depth analysis of model parameter size on intrinsic dimensionality to a later section (§ 5.2).

Lastly we see that adding a notion of structure in the computation of intrinsic dimension is beneficial with the SAID method consistently improving over the structure unaware DID method.

5. Intrinsic Dimension, Pre-Training, and Generalization Gap

Section Summary: Pre-training large language models appears to lower the intrinsic dimensionality of typical NLP tasks, allowing them to be represented with far fewer parameters relative to the underlying model. Experiments that track RoBERTa checkpoints show this dimensionality dropping steadily during pre-training, with easier tasks consistently requiring smaller values than harder ones. The authors interpret this as evidence that pre-training learns more compressible representations of language, reducing the “description length” needed to encode new tasks and improving generalization.

One interpretation of the intrinsic parameter vector is that it encodes the task at hand with respect to the original pre-trained representations. Therefore, we can interpret $d$ as the minimal description length of the task within the framework dictated by the pre-trained representations ([18]). Under this interpretation of intrinsic dimensionality, we hypothesize that pre-training is implicitly lowering the intrinsic dimensionality of the average NLP task, and therefore compress the minimal description length of those same tasks.

What do we more precisely mean by intrinsic parameter encoding a task within the framework provided by the pre-trained representations? Traditionally, a finetuned model (e.g. for a classification tasks) simply consists of a classification head $g$, parameterized by $w_g$ applied to fine-tuned representations $f$, parameterized by $w_f$ per sample $x$. Therefore, to fully describe a task, we need to pack together parameterizations and weights $\left{g, f, w_g, w_f\right}$. This model description is completely decoupled from the original weights of the pre-trained representation $w_{f_0}$, therefore to represent $n$ classification tasks, we need to maintain $n \left{w_g, w_f\right}$; additionally, the task representation is incredibly high dimensional. Conversely, fine-tuning utilizing SAID in $d$-dimensions requires storing only $\theta_d$ per task, a single random seed used to generate $M$ and the original pre-trained weights $w_{f_0}$. Therefore, we can represent arbitrary NLP tasks within a single pre-trained model framework with $d+1$ parameters.

For example, in the last section, we represented MRPC with roughly 200 parameters, which translates to needing less than a kilobyte of data to encode a complex natural language task within the framework provided by RoBERTa.

We hypothesize that the better the pre-trained models are, the fewer bits (description length) are needed to represent the average NLP task, as we will demonstrate empirically in the next section.

5.1 Pre-Training Intrinsic Dimension Trajectory

To verify our hypothesis of pre-training optimizing intrinsic dimension, we retrain a RoBERTa-Base from scratch and measure various NLP tasks' intrinsic dimensions using the SAID method across various checkpoints. We completely replicate the setting as described by ([3]) apart from only training for a total of 200k steps (instead of 500k) with half the batch size (1k). To calculate the intrinsic dimension more efficiently, we reuse the best learning rates discovered in Section 4 for $d < 10000$ and use a fixed learning rate for anything else. To find $d_{90}$ we do a binary search across $d$ per each checkpoint, with a minimum $d$ of 100 and a maximum of 4 million. The "full solution" that we use when deciding $d_{90}$ cut-off is computed by fine-tuning the checkpointed model in the standard way. We compute SAID on six datasets; MRPC, QQP, Yelp Polarity ([19]), SST-2 ([20]), MNLI ([21]) and ANLI using all rounds of data ([22]).

We present our results in Figure 2. We see that the intrinsic dimensionality of RoBERTa-Base monotonically decreases as we continue pre-training. We do not explicitly optimize for intrinsic dimensionality, specifically during pre-training (the language model does not have access to downstream datasets!), but none-the-less the intrinsic dimension of these downstream tasks continues to decrease.

More so, tasks that are easier to solve consistently show lower intrinsic dimensionality across all checkpoints, for example, Yelp Polarity vs. the notoriously tough ANLI dataset. The correlation between tasks traditionally hard for RoBERTa and their large intrinsic dimension hints at a connection between generalization and intrinsic dimension. We will discuss generalization further in Section § 5.3.

Given our task representation interpretation of intrinsic dimensionality, we argue that the large scale training of Masked Language Models (MLM) learns generic and distributed enough representations of language to facilitate downstream learning of highly compressed task representations. Furthermore, we argue for another perspective of pre-training learning representations that form a compression framework with respect to various NLP tasks.

5.2 Parameter Count and Intrinsic Dimension

We would also like to measure the relationships between the parameter count of arbitrary pre-trained models and the intrinsic dimension of downstream NLP tasks. The optimal experiment to run would be to fix the pre-training method, e.g., MLM RoBERTa style, vary the architecture size from small to very big, and compute the intrinsic dimension of a group of tasks at every size of the model. Unfortunately, such an experiment is computationally infeasible due to the need to train many RoBERTa models.

Due to these constraints, we opt to do an empirical study over existing pre-trained models, regardless of the pre-training method. We show that the trend is strong enough to overcome differences in training methodology. We select the following pre-trained models in our study: BERT ([2]), RoBERTa ([3]), BART ([4]), Electra ([23]), Albert ([24]), XLNet ([25]), T5 ([26]), and XLM-R ([27]). Furthermore, we selected various sizes of these models, as available publicly within the HuggingFace Transformers library ([17]).

We used the MRPC dataset and computed intrinsic dimension for every pre-trained model utilizing the same binary search methodology mentioned in the previous section with additional small hyper-parameter searches across learning rate (due to the wide range of learning rates needed by various models).

We present our results in Figure 3. We see a strong general trend that as the number of parameters increases, the intrinsic dimension of fine-tuning on MRPC decreases. We ran this experiment on other datasets to ensure that this is not an artifact of the dataset. Our experiments showed the same trend; we refer to the Appendix for all trends per dataset.

Within the same window of number of parameters, pre-training methodology becomes essential. For example, in the regime of $10^8$ parameters, the RoBERTa method of pre-training dominates similar sized pre-training methods. However, there does not seem to be a method that can overcome the limitations induced by the number of parameters. Interpreting these results through the lens of learning a compression framework for NLP tasks is straightforward; the more parameters we have in the model, the less we need to represent a task.

5.3 Generalization Bounds through Intrinsic Dimension

We have shown strong empirical evidence connecting pre-training, fine-tuning, and intrinsic dimensionality. However, we have yet to argue the connection between intrinsic dimensionality and generalization. Given that we have seen pre-training minimize intrinsic dimension, we hypothesize that generalization improves as the intrinsic dimension decreases.

To do so, we will empirically experiment with the connections between $d_{90}$ and evaluation set performance by looking at various checkpoints from our RoBERTa experiments in Section § 5.1. We also plot the relative generalization gap (delta between train time performance and test time performance).

In Figure 4 we plot the evaluation accuracy's achieved by our pre-training experiment in Section § 5.1. A lower intrinsic dimension is strongly correlated with better evaluation performance. Additionally we are interested in measuring relative generalization gap ($\frac{acc_{train}-acc_{eval}}{1-acc_{eval}}$) across intrinsic dimension. We select the training accuracy that provides us with the best evaluation metrics when computing this figure.

We present our results in Figure 5. Lower intrinsic dimension once again correlates strongly with a smaller relative generalization gap. If we interpret the intrinsic dimension as a measure of complexity, we expect the generalization gap to decrease with intrinsic dimension.

5.3.1 Generalization Bounds

By applying standard compression based generalization bounds, we can provide theoretical backing to the empirical connection between intrinsic dimension and generalization ([28]).

Consider the following definition of multi-class classification loss with an optional margin over our supervised dataset $D$.

$ \mathcal{L}{\gamma}(f) = \mathbb{P}{(x, y)\sim D}\left[f(x)[y] \le \gamma + \max_{i \ne y} f(x)[j] \right]\tag{4} $

When $\gamma = 0$, $\mathcal{L}0$ recovers the standard classification loss. Furthermore, Let $\hat{\mathcal{L}}{\gamma}(f)$ be an unbiased empirical estimate of the margin loss.

########## {caption="Theorem 1"}

Let $f$ be a function which is parameterized by $\theta^D$ as described in Equation 1 with a total of $d$ trainable intrinsic parameters on a dataset with $m$ samples. Then with a high probability, we can state the following asymptotic generalization bound

$ \mathcal{L}{0}(f) \leq \hat{\mathcal{L}}{0}(f) + \mathcal{O}\left(\sqrt{\frac{d}{m}}\right)\tag{5} $

Proof: We defer the proof Section § A.1 in the Appendix. We note that this is an extension of the well-known compression based generalization bound explored by [28].

This generalization bound is independent of the underlying parameter count ($D$) of the pre-trained model but depends on the ability to compress the downstream task ($d$). Moreover, given that our previous section shows larger models compress better, our bounds are aligned with general intuition and recent empirical evidence that larger pre-trained models generalize better. Explicitly, these bounds only apply to pre-trained methods trained with the intrinsic dimension subspace method; research has yet to show that standard SGD optimizes in this low dimensional space (although experimentally, this seems to be confirmed). We leave the theoretical contribution of showing SGD optimizes in this space, resembling something such as intrinsic subspace, for future work.

We want to highlight that generalization is not necessarily measured by the pre-trained model's parameter count or measure of complexity, but the pre-trained model's ability to facilitate the compression of downstream tasks. In some sense, if we want to compress downstream tasks better, we must expect pre-trained representations to have a considerable measure of complexity.

6. Conclusion

Section Summary: The authors argue that pre-training and fine-tuning can be understood through the concept of intrinsic dimensionality, which measures how many parameters are truly needed to learn a task. Their experiments show that pre-trained models often solve common language tasks using only a few hundred adjustable parameters, because pre-training effectively compresses language data and reduces the intrinsic dimension of new tasks. This reduction also improves generalization, as models with lower intrinsic dimensions tend to achieve higher accuracy and smaller gaps between training and test performance, suggesting a path toward more efficient large-scale models.

In conclusion, we proposed viewing the various phenomena surrounding fine-tuning and pre-training through the lens of intrinsic dimensionality. We empirically showed that common natural language tasks could be learned with very few parameters, sometimes in the order of hundreds, when utilizing pre-trained representations. We provided an interpretation of pre-training as providing a compression framework for minimizing the average description length of natural language tasks and showed that pre-training implicitly minimizes this average description length.

We continued by doing an empirical study of existing pre-training methods and their respective intrinsic dimension, uncovering the phenomena that intrinsic dimensionality decreases as we increase the number of pre-trained representation parameters. This phenomenon provides some intuitions to the trend of growing pre-trained representations. We connected intrinsic dimensionality with generalization by first showing that pre-trained models with lower intrinsic dimensions across various tasks achieve higher evaluation accuracies and lower relative generalization gaps. Furthermore, we explain these empirical results by applying well-known generalization bounds to the intrinsic dimension to get generalization bounds that grow on the order of the intrinsic dimension, not on the pre-trained model's parameter count.

Intrinsic dimensionality is a useful tool for understanding the complex behavior of large models. We hope that future work will make explicit theoretical connections between SGD and optimizing the intrinsic dimension as well as explain exactly why pre-training methods optimize the intrinsic dimensionailty of tasks before not seen.

Appendix

Section Summary: The appendix recalls a definition of (γ, S)-compressible classifiers that can be approximated to within γ by a low-parameter model that also receives a short helper string s. It then notes that networks whose weights are confined to a random low-dimensional subspace (as in the DID and SAID methods) meet this definition with γ = 0, because the subspace itself is fully determined by a random seed plus the starting pretrained weights. Substituting this lossless compressibility into a known generalization theorem and discretizing the weights yields the final bound relating true risk to empirical risk through a term that scales with the square root of the subspace dimension rather than the full parameter count.

A.1 Proofs

[28] define $(\gamma, S)$ compressible using helper string $s$ as the following.

########## {caption="Definition 2"}

$(\gamma, S)$ compressible using helper string $s$

Suppose $G_{\mathcal{A}, s}= \left{g_{\theta, s}|\theta \in \mathcal{A}\right}$ is a class of classifiers indexed by trainable parameters A and fixed strings s. A classifier $f$ is $(\gamma, S)$-compressible with respect to $G_{\mathcal{A}}$ using helper string s if there exists $\theta \in \mathcal{A}$ such that for any $x \in S$, we have for all y

$ |f(x)[y] - g_{\theta, s}(x)[y]| \leq \gamma\tag{6} $

########## {caption="Remark"}

If we parameterize $f(x; \theta)$ via the intrinsic dimension approach as defined in Equation 1, then $f$ is compressible losslessly using a helper string consisting of the random seed used to generate the static random projection weights and the initial pre-trained representation $\theta^D_0$. Therefore we say $f$ parameterized by either DID or SAID is $\left(0, S\right)$ compressible.

Theorem $2.1$ in [28] states given a compression consisting of $r$ discrete states we achieve the following generalization bound.

$ \mathcal{L}{0}(f) \leq \hat{\mathcal{L}}{\gamma}(f) + O\left(\sqrt{\frac{d \log{r}}{m}}\right)\tag{7} $

We can trivially represent our parameters $\theta_d$ in a discrete fashion through discretization (as was done in [28]), and the number of states is dependent on the level of quantization but is static once chosen (FP32 vs. FP16).

We then connect the fact that models trained in low dimensional subspace using SAID/DID methods are (0, S)-compressible to derive the final asymptotic bound.

$ \mathcal{L}{0}(f) \leq \hat{\mathcal{L}}{0}(f) + \mathcal{O}\left(\sqrt{\frac{d}{m}}\right)\tag{8} $

References

Section Summary: This section compiles a numbered list of academic citations, mostly from recent years, that cover influential work on large-scale language models and pre-training methods such as GPT, BERT and its variants, BART, and related transformer architectures. It also includes references on the lottery ticket hypothesis for model pruning, parameter-efficient fine-tuning strategies, and standard NLP benchmarks and datasets like GLUE, SNLI, and Quora question pairs. Additional entries point to practical tools like the Hugging Face library and foundational studies in representation learning and text classification.

[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.

[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.

[4] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.

[5] Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing, 2020.

[6] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838, 2018.

[7] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in Neural Information Processing Systems, 33, 2020.

[8] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00561, 2020.

[9] Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv:1910.12708, 2019.

[10] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. arXiv preprint arXiv:1902.00751, 2019.

[11] Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. arXiv preprint arXiv:2008.03156, 2020.

[12] Quoc Le, Tamás Sarlós, and Alex Smola. Fastfood-approximating kernel expansions in loglinear time. In Proceedings of the international conference on machine learning, volume 85, 2013.

[13] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341, 2019.

[14] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi:10.18653/v1/W18-5446. URL https://www.aclweb.org/anthology/W18-5446.

[15] William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.

[16] Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs, 2017. URL https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs.

[17] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, pp. arXiv–1910, 2019.

[18] Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in neural information processing systems, 6:3–10, 1993.

[19] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. arXiv:1509.01626 [cs], September 2015.

[20] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013.

[21] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/N18-1101.

[22] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.

[23] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020.

[24] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.

[25] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753–5763, 2019.

[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.

[27] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.

[28] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296, 2018.