LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

ChapterPal

Tim Dettmers$^{\lambda *}$

Mike Lewis$^{\dagger}$

Younes Belkada$^{\S\mp}$

Luke Zettlemoyer$^{\dagger\lambda}$

University of Washington$^{\lambda}$
Facebook AI Research$^{\dagger}$
Hugging Face$^{\S}$
ENS Paris-Saclay$^{\mp}$

Abstract

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open source our software.

Executive Summary: Large language models, which power advanced natural language processing tasks like text generation and question answering, have grown massively in size, often exceeding 100 billion parameters. This scale drives impressive performance but creates a major barrier: these models demand enormous GPU memory for inference, the process of using a trained model to generate outputs. Running a 175-billion-parameter model typically requires multiple high-end GPUs or specialized servers, limiting access for researchers, small organizations, and even some enterprises. With models like OPT-175B and BLOOM-176B now publicly available, reducing memory needs without sacrificing accuracy has become urgent to democratize AI capabilities and enable broader innovation.

This document introduces LLM.int8(), a new method to quantize—compress—key parts of transformer-based language models to 8-bit precision. The goal is to halve the memory footprint for inference while maintaining full original performance, allowing models up to 175 billion parameters to run on modest hardware without retraining or fine-tuning.

The authors analyzed how quantization breaks down in large models by studying hidden states—internal representations—across various pretrained transformers ranging from 125 million to 175 billion parameters. They examined language modeling perplexity, a measure of how well models predict text (lower is better), on the C4 dataset, and zero-shot accuracy on diverse tasks using models like OPT. Data came from public sources trained on vast text corpora, with evaluations on NVIDIA A40 GPUs over sequences up to 2048 tokens. Key assumptions included focusing on feed-forward and attention projection layers, which account for most parameters and computation, while treating outliers—unusually large values in hidden states—as the main quantization challenge.

The core findings reveal why prior 8-bit methods fail and how LLM.int8() succeeds. First, standard quantization techniques, like absolute maximum or zero-point methods, preserve performance only up to about 2.7 billion parameters but degrade sharply beyond 6.7 billion, sometimes worsening as models grow larger. This stems from emergent outlier features: starting around 6.7 billion parameters, a small set of hidden dimensions (about 0.1% of total, or roughly 6-7 per model) produce values up to 20 times larger than normal, affecting all layers and 75% of sequences. Removing these outliers halves attention probabilities and boosts perplexity by 600-1000%, far more than removing random features (under 0.3% impact), proving their critical role in predictions. Second, these outliers follow an asymmetric pattern, favoring certain quantization types temporarily but failing at scale due to precision loss. Third, LLM.int8() counters this with two steps: vector-wise quantization, which uses per-row and per-column scaling for finer precision across matrix operations, and mixed-precision decomposition, which handles outliers in 16-bit while quantizing 99.9% of values to 8-bit. This combo yields no perplexity increase or accuracy drop up to 175 billion parameters, unlike baselines that collapse to random levels. Finally, it achieves about a 2x memory reduction—for instance, loading BLOOM-176B uses 1.96 times less space—while offering up to 2x faster matrix multiplications for the largest models, though smaller ones see minor slowdowns from overhead.

These results mean large language models can now run on single servers with consumer-grade GPUs, slashing costs and hardware barriers. For example, OPT-175B, previously needing 8+ high-end GPUs, fits on one node with four A100s using LLM.int8(). This boosts accessibility for low-resource users, accelerating research in areas like few-shot learning, but it also risks widening gaps if big organizations scale deployments more efficiently. Unlike expectations that finer quantization alone suffices, the findings highlight outliers as a phase-shift phenomenon tied more to model perplexity than size, challenging assumptions in prior work on smaller models (under 1 billion parameters) and explaining why methods like nuQmm or ZeroQuant top out at 20 billion.

Adopt LLM.int8() for inference on models over 6 billion parameters to enable immediate use without performance hits; the authors have open-sourced it via GitHub and integrated it with Hugging Face Transformers for easy deployment on hosted models. For training or fine-tuning, explore hybrids—initial tests show promise for feed-forward layers but degradation in attention without adjustments. Next, validate on emerging models beyond 175 billion parameters and develop 8-bit attention handling, possibly via pilots on specific tasks.

Confidence in these findings is high for inference on autoregressive transformers up to 175 billion parameters, backed by consistent trends across diverse models and frameworks. Uncertainties include untested scales above 175 billion, where new outliers might emerge, and the focus on integer 8-bit (Int8) rather than floating-point alternatives, which could offer further gains but lack hardware support. Data gaps on non-English tasks or varied architectures warrant caution before broad policy or investment decisions.

1. Introduction

Section Summary: Large language models, which power many natural language processing tasks, require vast amounts of memory to run, especially for versions with billions of parameters, where most of the computational load comes from specific layers. To address this, researchers have developed an 8-bit quantization technique called LLM.int8() that shrinks model size without sacrificing performance, even for massive 175-billion-parameter models like OPT and BLOOM. By handling unusual "outlier" features through vector-wise scaling for smaller models and mixed-precision math for larger ones—keeping 16-bit precision on just 0.1% of dimensions—it enables these huge models to run efficiently on everyday server GPUs, with the software now freely available.

Large pretrained language models are widely adopted in NLP ([1, 2, 3, 4]) but require significant memory for inference. For large transformer language models at and beyond 6.7B parameters, the feed-forward and attention projection layers and their matrix multiplication operations are responsible for 95%[^1] of consumed parameters and 65-85% of all computation ([5]). One way to reduce the size of the parameters is to quantize them to less bits and use low-bit-precision matrix multiplication. With this goal in mind, 8-bit quantization methods for transformers have been developed ([6, 7, 8, 9]). While these methods reduce memory use, they degrade performance, usually require tuning quantization further after training, and have only been studied for models with less than 350M parameters. Degradation-free quantization up to 350M parameters is poorly understood, and multi-billion parameter quantization remains an open challenge.

[^1]: Other parameters come mostly from the embedding layer. A tiny amount comes from norms and biases.

In this paper, we present the first multi-billion-scale Int8 quantization procedure for transformers that does not incur any performance degradation. Our procedure makes it possible to load a 175B parameter transformer with 16 or 32-bit weights, convert the feed-forward and attention projection layers to 8-bit, and use the resulting model immediately for inference without any performance degradation. We achieve this result by solving two key challenges: the need for higher quantization precision at scales beyond 1B parameters and the need to explicitly represent the sparse but systematic large magnitude outlier features that ruin quantization precision once they emerge in all transformer layers starting at scales of 6.7B parameters. This loss of precision is reflected in C4 evaluation perplexity (Section 3) as well as zeroshot accuracy as soon as these outlier features emerge, as shown in Figure 1.

We show that with the first part of our method, vector-wise quantization, it is possible to retain performance at scales up to 2.7B parameters. For vector-wise quantization, matrix multiplication can be seen as a sequence of independent inner products of row and column vectors. As such, we can use a separate quantization normalization constant for each inner product to improve quantization precision. We can recover the output of the matrix multiplication by denormalizing by the outer product of column and row normalization constants before we perform the next operation.

To scale beyond 6.7B parameters without performance degradation, it is critical to understand the emergence of extreme outliers in the feature dimensions of the hidden states during inference. To this end, we provide a new descriptive analysis which shows that large features with magnitudes up to 20x larger than in other dimensions first appear in about 25% of all transformer layers and then gradually spread to other layers as we scale transformers to 6B parameters. At around 6.7B parameters, a phase shift occurs, and all transformer layers and 75% of all sequence dimensions are affected by extreme magnitude features. These outliers are highly systematic: at the 6.7B scale, 150, 000 outliers occur per sequence, but they are concentrated in only 6 feature dimensions across the entire transformer. Setting these outlier feature dimensions to zero decreases top-1 attention softmax probability mass by more than 20% and degrades validation perplexity by 600-1000% despite them only making up about 0.1% of all input features. In contrast, removing the same amount of random features decreases the probability by a maximum of 0.3% and degrades perplexity by about 0.1%.

To support effective quantization with such extreme outliers, we develop mixed-precision decomposition, the second part of our method. We perform 16-bit matrix multiplication for the outlier feature dimensions and 8-bit matrix multiplication for the other 99.9% of the dimensions. We name the combination of vector-wise quantization and mixed precision decomposition, LLM.int8(). We show that by using LLM.int8(), we can perform inference in LLMs with up to 175B parameters without any performance degradation. Our method not only provides new insights into the effects of these outliers on model performance but also makes it possible for the first time to use very large models, for example, OPT-175B/BLOOM, on a single server with consumer GPUs. While our work focuses on making large language models accessible without degradation, we also show in Appendix D that we maintain end-to-end inference runtime performance for large models, such as BLOOM-176B and provide modest matrix multiplication speedups for GPT-3 models of size 6.7B parameters or larger. We open-source our software^2 and release a Hugging Face Transformers ([10]) integration making our method available to all hosted Hugging Face Models that have linear layers.

$**Figure 2:** Schematic of LLM.int8(). Given 16-bit floating-point inputs $\mathbf{X}_{f16}$ and weights $\mathbf{W}_{f16}$, the features and weights are decomposed into sub-matrices of large magnitude features and other values. The outlier feature matrices are multiplied in 16-bit. All other values are multiplied in 8-bit. We perform 8-bit vector-wise multiplication by scaling by row and column-wise absolute maximum of $\mathbf{C}_x$ and $\mathbf{C}_w$ and then quantizing the outputs to Int8. The Int32 matrix multiplication outputs $\mathbf{{Out}}_{i32}$ are dequantization by the outer product of the normalization constants $\mathbf{C}_x\otimes \mathbf{C}_w$. Finally, both outlier and regular outputs are accumulated in 16-bit floating point outputs.$

2. Background

Section Summary: This background section examines quantization techniques for scaling up transformer models, focusing on when and why they break down in terms of model size and precision levels. It compares two main methods: absmax quantization, which is widely used and simply scales data to fit within an 8-bit symmetric range based on the largest absolute value, and zeropoint quantization, which more precisely shifts asymmetric data distributions to fully utilize the 8-bit range for better accuracy, though it often runs slower without specialized hardware support. These approaches are applied to efficient 8-bit matrix multiplications starting from 16-bit floating-point inputs, enabling lower-precision computations while approximating original results.

In this work, push quantization techniques to their breaking point by scaling transformer models. We are interested in two questions: at which scale and why do quantization techniques fail and how does this related to quantization precision? To answer these questions we study high-precision asymmetric quantization (zeropoint quantization) and symmetric quantization (absolute maximum quantization). While zeropoint quantization offers high precision by using the full bit-range of the datatype, it is rarely used due to practical constraints. Absolute maximum quantization is the most commonly used technique.

2.1 8-bit Data Types and Quantization

Absmax quantization scales inputs into the 8-bit range $[-127, 127]$ by multiplying with $s_{x_{f16}}$ which is 127 divided by the absolute maximum of the entire tensor. This is equivalent to dividing by the infinity norm and multiplying by 127. As such, for an FP16 input matrix $\mathbf{X}_{f16}\in \mathbb{R}^{s\times h}$ Int8 absmax quantization is given by:

$ \mathbf{X}{i8} = \Biggl\lfloor{\dfrac{127\cdot\mathbf{X}{f16}}{\max\limits_{ij}(|\mathbf{X}{f16{ij}}|)}}\Biggr\rceil= \Biggl\lfloor{\dfrac{127}{\lVert \mathbf{X}{f16} \rVert\infty}\mathbf{X}{f16}}\Biggr\rceil = \Bigl\lfloor{s{x_{f16}} \mathbf{X}_{f16}}\Bigr\rceil, $

where $\lfloor{} \rceil{}$ indicates rounding to the nearest integer.

Zeropoint quantization shifts the input distribution into the full range $[-127, 127]$ by scaling with the normalized dynamic range $nd_x$ and then shifting by the zeropoint $zp_x$. With this affine transformation, any input tensors will use all bits of the data type, thus reducing the quantization error for asymmetric distributions. For example, for ReLU outputs, in absmax quantization all values in $[-127, 0)$ go unused, whereas in zeropoint quantization the full $[-127, 127]$ range is used. Zeropoint quantization is given by the following equations:

$ nd_{x_{f16}} = \dfrac{2\cdot 127}{\max\limits_{ij}(\mathbf{X}{f16}^{ij}) - \min\limits{ij}(\mathbf{X}_{f16}^{ij})} $

$ zp_{x_{i16}} = \bigl\lfloor{\mathbf{X}{f16} \cdot \min\limits{ij}(\mathbf{X}_{f16}^{ij})\bigr\rceil} $

$ \mathbf{X}{i8} = \bigl\lfloor{{nd{x_{f16}} \mathbf{X}_{f16}}\bigr\rceil} $

To use zeropoint quantization in an operation we feed both the tensor $\mathbf{X}{i8}$ and the zeropoint $zp{x_{i16}}$ into a special instruction^3 which adds $zp_{x_{i16}}$ to each element of $\mathbf{X}{i8}$ before performing a 16-bit integer operation. For example, to multiply two zeropoint quantized numbers $A{i8}$ and $B_{i8}$ along with their zeropoints $zp_{a_{i16}}$ and $zp_{b_{i16}}$ we calculate:

$ C_{i32} = \text{multiply}{i16}({A}{zp_{a_{i16}}}, {B}{zp{b_{i16}}}) = ({A}{i8}+zp{a_{i16}})({B}{i8}+zp{b_{i16}}) $

where unrolling is required if the instruction multiply$_{i16}$ is not available such as on GPUs or TPUs:

$ C_{i32}= {A}{i8}{B}{i8} + {A}{i8}zp{b_{i16}} + {B}{i8}zp{a_{i16}} + zp_{a_{i16}}zp_{b_{i16}}, $

where $A_{i8}B_{i8}$ is computed with Int8 precision while the rest is computed in Int16/32 precision. As such, zeropoint quantization can be slow if the multiply${i16}$ instruction is not available. In both cases, the outputs are accumulated as a 32-bit integer $C{i32}$. To dequantize $C_{i32}$, we divide by the scaling constants $nd_{a_{f16}}$ and $nd_{b_{f16}}$.

Int8 Matrix Multiplication with 16-bit Float Inputs and Outputs.

Given hidden states ${\mathbf{X}{f16}\in \mathbb{R}^{s\times h}}$ and weights $\mathbf{W}{f16}\in \mathbb{R}^{h\times o}$ with sequence dimension $s$, feature dimension $h$, and output dimension $o$ we perform 8-bit matrix multiplication with 16-bit inputs and outputs as follows:

$ \begin{split} \mathbf{X}{f16} \mathbf{W}{f16} =\mathbf{C}{f16} & \approx \frac{1}{c{x_{f16}}c_{w_{f16}}} \mathbf{C}{i32} = S{f16}\cdot\mathbf{C}{i32}\ & \approx S{f16}\cdot\mathbf{A}{i8}\mathbf{B}{i8} = S_{f16}\cdot Q(\mathbf{A}{f16})\phantom{.} Q(\mathbf{B}{f16}), \end{split} $

Where $Q(\cdot)$ is either absmax or zeropoint quantization and $c_{x_{f16}}$ and $c_{w_{f16}}$ are the respective tensor-wise scaling constants $s_x$ and $s_w$ for absmax or $nd_x$ and $nd_w$ for zeropoint quantization.

3. Int8 Matrix Multiplication at Scale

Section Summary: To address the problem of outliers in large language models that degrade the precision of 8-bit quantization, researchers developed vector-wise quantization, which uses separate scaling factors for rows and columns in matrix multiplications to limit the impact of these outliers. For even bigger models beyond 6.7 billion parameters, they introduced mixed-precision decomposition, keeping just 0.1 percent of the most extreme features in higher 16-bit precision while handling the rest in efficient 8-bit, which cuts memory use by about half compared to full 16-bit processing. Their combined method, called LLM.int8(), maintains model performance while scaling up to 175 billion parameters, as tested through language modeling and accuracy evaluations.

The main challenge with quantization methods that use a single scaling constant per tensor is that a single outlier can reduce the quantization precision of all other values. As such, it is desirable to have multiple scaling constants per tensor, such as block-wise constants ([11]), so that the effect of that outliers is confined to each block. We improve upon one of the most common ways of blocking quantization, row-wise quantization ([12]), by using vector-wise quantization, as described in more detail below.

To handle the large magnitude outlier features that occur in all transformer layers beyond the 6.7B scale, vector-wise quantization is no longer sufficient. For this purpose, we develop mixed-precision decomposition, where the small number of large magnitude feature dimensions ($\approx$ 0.1%) are represented in 16-bit precision while the other 99.9% of values are multiplied in 8-bit. Since most entries are still represented in low-precision, we retain about 50% memory reduction compared to 16-bit. For example, for BLOOM-176B, we reduce the memory footprint of the model by 1.96x.

Vector-wise quantization and mixed-precision decomposition are shown in Figure 2. The LLM.int8() method is the combination of absmax vector-wise quantization and mixed precision decomposition.

3.1 Vector-wise Quantization

One way to increase the number of scaling constants for matrix multiplication is to view matrix multiplication as a sequence of independent inner products. Given the hidden states $\mathbf{X}{f16}\in \mathbb{R}^{b\times h}$ and weight matrix $\mathbf{W}{f16}\in \mathbb{R}^{h\times o}$, we can assign a different scaling constant $c_{x_{f16}}$ to each row of $\mathbf{X}{f16}$ and $c_w$ to each column of $\mathbf{W}{f16}$. To dequantize, we denormalize each inner product result by $1/(c_{x_{f16}}c_{w_{f16}})$. For the whole matrix multiplication this is equivalent to denormalization by the outer product $\mathbf{c}{x{f16}} \otimes \mathbf{c}{w{f16}}$, where $\mathbf{c}_x \in \mathbb{R}^s$ and $\mathbf{c}_w \in \mathbb{R}^o$. As such the full equation for matrix multiplication with row and column constants is given by:

$ \mathbf{C}{f{16}} \approx \frac{1}{\mathbf{c}{x{f16}}\otimes \mathbf{c}{w{f16}}} \mathbf{C}{i32} = S\cdot\mathbf{C}{i32}= \mathbf{S}\cdot\mathbf{A}{i8}\mathbf{B}{i8} = \mathbf{S}\cdot Q(\mathbf{A}{f16})\phantom{.} Q(\mathbf{B}{f16}), $

which we term vector-wise quantization for matrix multiplication.

3.2 The Core of LLM.int8(): Mixed-precision Decomposition

In our analysis, we demonstrate that a significant problem for billion-scale 8-bit transformers is that they have large magnitude features (columns), which are important for transformer performance and require high precision quantization. However, vector-wise quantization, our best quantization technique, quantizes each row for the hidden state, which is ineffective for outlier features. Luckily, we see that these outlier features are both incredibly sparse and systematic in practice, making up only about 0.1% of all feature dimensions, thus allowing us to develop a new decomposition technique that focuses on high precision multiplication for these particular dimensions.

We find that given input matrix $\mathbf{X}{f16}\in \mathbb{R}^{s\times h}$, these outliers occur systematically for almost all sequence dimensions $s$ but are limited to specific feature/hidden dimensions $h$. As such, we propose mixed-precision decomposition for matrix multiplication where we separate outlier feature dimensions into the set ${O ={i| i\in \mathbb{Z}, 0 \leq i \leq h}}$, which contains all dimensions of $h$ which have at least one outlier with a magnitude larger than the threshold $\alpha$. In our work, we find that $\alpha=6.0$ is sufficient to reduce transformer performance degradation close to zero. Using Einstein notation where all indices are superscripts, given the weight matrix $\mathbf{W}{f16}\in \mathbb{R}^{h\times o}$, mixed-precision decomposition for matrix multiplication is defined as follows:

$ \mathbf{C}{f16} \approx \sum\limits{h\in O} \mathbf{X}{f16}^h \mathbf{W}{f16}^h + \mathbf{S}{f16} \cdot \sum\limits{h\not\in O} \mathbf{X}{i8}^h \mathbf{W}{i8}^h $

where $\mathbf{S}{f16}$ is the denormalization term for the Int8 inputs and weight matrices $\mathbf{X}{i8}$ and $\mathbf{W}_{i8}$.

This separation into 8-bit and 16-bit allows for high-precision multiplication of outliers while using memory-efficient matrix multiplication with 8-bit weights of more than 99.9% of values. Since the number of outlier feature dimensions is not larger than 7 ($|O|\leq7$) for transformers up to 13B parameters, this decomposition operation only consumes about 0.1% additional memory.

3.3 Experimental Setup

We measure the robustness of quantization methods as we scale the size of several publicly available pretrained language models up to 175B parameters. The key question is not how well a quantization method performs for a particular model but the trend of how such a method performs as we scale.

We use two setups for our experiments. One is based on language modeling perplexity, which we find to be a highly robust measure that is very sensitive to quantization degradation. We use this setup to compare different quantization baselines. Additionally, we evaluate zeroshot accuracy degradation on OPT models for a range of different end tasks, where we compare our methods with a 16-bit baseline.

For the language modeling setup, we use dense autoregressive transformers pretrained in fairseq ([13]) ranging between 125M and 13B parameters. These transformers have been pretrained on Books ([14]), English Wikipedia, CC-News ([15]), OpenWebText ([16]), CC-Stories ([17]), and English CC100 ([18]). For more information on how these pretrained models are trained, see [19].

To evaluate the language modeling degradation after Int8 quantization, we evaluate the perplexity of the 8-bit transformer on validation data of the C4 corpus ([20]) which is a subset of the Common Crawl corpus.^4 We use NVIDIA A40 GPUs for this evaluation.

To measure degradation in zeroshot performance, we use OPT models ([4]), and we evaluate these models on the EleutherAI language model evaluation harness ([21]).

3.4 Main Results

The main language modeling perplexity results on the 125M to 13B Int8 models evaluated on the C4 corpus can be seen in Table 1. We see that absmax, row-wise, and zeropoint quantization fail as we scale, where models after 2.7B parameters perform worse than smaller models. Zeropoint quantization fails instead beyond 6.7B parameters. Our method, LLM.int8(), is the only method that preserves perplexity. As such, LLM.int8() is the only method with a favorable scaling trend.

When we look at the scaling trends of zeroshot performance of OPT models on the EleutherAI language model evaluation harness in Figure 1, we see that LLM.int8() maintains full 16-bit performance as we scale from 125M to 175B parameters. On the other hand, the baseline, 8-bit absmax vector-wise quantization, scales poorly and degenerates into random performance.

::: {caption="Table 1: C4 validation perplexities of quantization methods for different transformer sizes ranging from 125M to 13B parameters. We see that absmax, row-wise, zeropoint, and vector-wise quantization leads to significant performance degradation as we scale, particularly at the 13B mark where 8-bit 13B perplexity is worse than 8-bit 6.7B perplexity. If we use LLM.int8(), we recover full perplexity as we scale. Zeropoint quantization shows an advantage due to asymmetric quantization but is no longer advantageous when used with mixed-precision decomposition."}

:::

Although our primary focus is on saving memory, we also measured the run time of LLM.int8(). The quantization overhead can slow inference for models with less than 6.7B parameters, as compared to a FP16 baseline. However, models of 6.7B parameters or less fit on most GPUs and quantization is less needed in practice. LLM.int8() run times is about two times faster for large matrix multiplications equivalent to those in 175B models. Appendix D provides more details on these experiments.

4. Emergent Large Magnitude Features in Transformers at Scale

Section Summary: As transformers grow larger, unusual features with extremely high values suddenly appear in their internal computations, influencing every layer and causing problems with techniques that compress the models for efficiency. These outliers, limited to just a handful of specific dimensions despite numbering in the tens of thousands per input, are consistent in big models but sporadic in smaller ones, and researchers identified them by tracking values above a certain threshold that span many layers and parts of the input. Removing these features disrupts the model's attention mechanism and overall accuracy, explaining why some compression methods work well for small models but fail in larger ones unless advanced adjustments are made.

As we scale transformers, outlier features with large magnitudes emerge and strongly affect all layers and their quantization. Given a hidden state $\mathbf{X}\in \mathbb{R}^{s\times h}$ where $s$ is the sequence/token dimension and $h$ the hidden/feature dimension, we define a feature to be a particular dimension $h_i$. Our analysis looks at a particular feature dimension $h_i$ across all layers of a given transformer.

We find that outlier features strongly affect attention and the overall predictive performance of transformers. While up to 150k outliers exist per 2048 token sequence for a 13B model, these outlier features are highly systematic and only representing at most 7 unique feature dimensions $h_i$. Insights from this analysis were critical to developing mixed-precision decomposition. Our analysis explains the advantages of zeropoint quantization and why they disappear with the use of mixed-precision decomposition and the quantization performance of small vs. large models.

4.1 Finding Outlier Features

The difficulty with the quantitative analysis of emergent phenomena is two-fold. We aim to select a small subset of features for analysis such that the results are intelligible and not to complex while also capturing important probabilistic and structured patterns. We use an empirical approach to find these constraints. We define outliers according to the following criteria: the magnitude of the feature is at least 6.0, affects at least 25% of layers, and affects at least 6% of the sequence dimensions.

More formally, given a transformer with $L$ layers and hidden state $\mathbf{X}l\in \mathbb{R}^{s\times h}, l=0...L$ where $s$ is the sequence dimension and $h$ the feature dimension, we define a feature to be a particular dimension $h_i$ in any of the hidden states $\mathbf{X}{l_{i}}$. We track dimensions $h_i, 0 \leq i \leq h$, which have at least one value with a magnitude of $\alpha\geq6$ and we only collect statistics if these outliers occur in the same feature dimension $h_i$ in at least 25% of transformer layers $0...L$ and appear in at least 6% of all sequence dimensions $s$ across all hidden states $\mathbf{X}_l$. Since feature outliers only occur in attention projection (key/query/value/output) and the feedforward network expansion layer (first sub-layer), we ignore the attention function and the FFN contraction layer (second sub-layer) for this analysis.

Our reasoning for these thresholds is as follows. We find that using mixed-precision decomposition, perplexity degradation stops if we treat any feature with a magnitude 6 or larger as an outlier feature. For the number of layers affected by outliers, we find that outlier features are systematic in large models: they either occur in most layers or not at all. On the other hand, they are probabilistic in small models: they occur sometimes in some layers for each sequence. As such, we set our threshold for how many layers need to be affected to detect an outlier feature in such a way as to limit detection to a single outlier in our smallest model with 125M parameters. This threshold corresponds to that at least 25% of transformer layers are affected by an outlier in the same feature dimension. The second most common outlier occurs in only a single layer (2% of layers), indicating that this is a reasonable threshold. We use the same procedure to find the threshold for how many sequence dimensions are affected by outlier features in our 125M model: outliers occur in at least 6% of sequence dimensions.

We test models up to a scale of 13B parameters. To make sure that the observed phenomena are not due to bugs in software, we evaluate transformers that were trained in three different software frameworks. We evaluate four GPT-2 models which use OpenAI software, five Meta AI models that use Fairseq ([13]), and one EleutherAI model GPT-J that uses Tensorflow-Mesh ([22]). More details can be found in Appendix C. We also perform our analysis in two different inference software frameworks: Fairseq and Hugging Face Transformers ([10]).

4.2 Measuring the Effect of Outlier Features

To demonstrate that the outlier features are essential for attention and predictive performance, we set the outlier features to zero before feeding the hidden states $\mathbf{X}_l$ into the attention projection layers and then compare the top-1 softmax probability with the regular softmax probability with outliers. We do this for all layers independently, meaning we forward the regular softmax probabilities values to avoid cascading errors and isolate the effects due to the outlier features. We also report the perplexity degradation if we remove the outlier feature dimension (setting them to zero) and propagate these altered, hidden states through the transformer. As a control, we apply the same procedure for random non-outlier feature dimensions and note attention and perplexity degradation.

Our main quantitative results can be summarized as four main points.

(1) When measured by the number of parameters, the emergence of large magnitude features across all layers of a transformer occurs suddenly between 6B and 6.7B parameters as shown in Figure 3a as the percentage of layers affected increases from 65% to 100%. The number of sequence dimensions affected increases rapidly from 35% to 75%. This sudden shift co-occurs with the point where quantization begins to fail.

(2) Alternatively, when measured by perplexity, the emergence of large magnitude features across all layers of the transformer can be seen as emerging smoothly according to an exponential function of decreasing perplexity, as seen in Figure 3b. This indicates that there is nothing sudden about emergence and that we might be able to detect emergent features before a phase shift occurs by studying exponential trends in smaller models. This also suggests that emergence is not only about model size but about perplexity, which is related to multiple additional factors such as the amount of training data used, and data quality ([23, 24]).

(3) Median outlier feature magnitude rapidly increases once outlier features occur in all layers of the transformer, as shown in Figure 4a. The large magnitude of outliers features and their asymmetric distribution disrupts Int8 quantization precision. This is the core reason why quantization methods fail starting at the 6.7B scale – the range of the quantization distribution is too large so that most quantization bins are empty and small quantization values are quantized to zero, essentially extinguishing information. We hypothesize that besides Int8 inference, regular 16-bit floating point training becomes unstable due to outliers beyond the 6.7B scale – it is easy to exceed the maximum 16-bit value 65535 by chance if you multiply by vectors filled with values of magnitude 60.

(4) The number of outliers features increases strictly monotonically with respect to decreasing C4 perplexity as shown in Figure 4b, while a relationship with model size is non-monotonic. This indicates that model perplexity rather than mere model size determines the phase shift. We hypothesize that model size is only one important covariate among many that are required to reach emergence.

These outliers features are highly systematic after the phase shift occurred. For example, for a 6.7B transformer with a sequence length of 2048, we find about 150k outlier features per sequence for the entire transformer, but these features are concentrated in only 6 different hidden dimensions.

These outliers are critical for transformer performance. If the outliers are removed, the mean top-1 softmax probability is reduced from about 40% to about 20%, and validation perplexity increases by 600-1000% even though there are at most 7 outlier feature dimensions. When we remove 7 random feature dimensions instead, the top-1 probability decreases only between 0.02-0.3%, and perplexity increases by 0.1%. This highlights the critical nature of these feature dimensions. Quantization precision for these outlier features is paramount as even tiny errors greatly impact model performance.

4.3 Interpretation of Quantization Performance

Our analysis shows that outliers in particular feature dimensions are ubiquitous in large transformers, and these feature dimensions are critical for transformer performance. Since row-wise and vector-wise quantization scale each hidden state sequence dimension $s$ (rows) and because outliers occur in the feature dimension $h$ (columns), both methods cannot deal with these outliers effectively. This is why absmax quantization methods fail quickly after emergence.

However, almost all outliers have a strict asymmetric distribution: they are either solely positive or negative (see Appendix C). This makes zeropoint quantization particularly effective for these outliers, as zeropoint quantization is an asymmetric quantization method that scales these outliers into the full $[-127, 127]$ range. This explains the strong performance in our quantization scaling benchmark in Table 1. However, at the 13B scale, even zeropoint quantization fails due to accumulated quantization errors and the quick growth of outlier magnitudes, as seen in Figure 4a.

If we use our full LLM.int8() method with mixed-precision decomposition, the advantage of zeropoint quantization disappears indicating that the remaining decomposed features are symmetric. However, vector-wise still has an advantage over row-wise quantization, indicating that the enhanced quantization precision of the model weights is needed to retain full precision predictive performance.

5. Related work

Section Summary: This section reviews prior research on quantizing AI models, starting with 8-bit data types like the integer-based Int8, which is the only one supported by graphics processors, while floating-point variants like FP8 could perform better but lack hardware support due to limitations in handling large values. It also discusses studies on unusual "outlier" features in language models, which arise from factors like data normalization and word frequencies, and explains how this research builds on that by linking outliers to model size and their role in successful quantization. Finally, it compares the approach to similar techniques like nuQmm and ZeroQuant, which use more precise but hardware-intensive methods for smaller models, while this work enables error-free 8-bit quantization for much larger ones up to 176 billion parameters, with complementary applications in projects like GLM-130B.

There is closely related work on quantization data types and quantization of transformers, as described below. Appendix B provides further related work on quantization of convolutional networks.

8-bit Data Types. Our work studies quantization techniques surrounding the Int8 data type, since it is currently the only 8-bit data type supported by GPUs. Other common data types are fixed point or floating point 8-bit data types (FP8). These data types usually have a sign bit and different exponent and fraction bit combinations. For example, a common variant of this data type has 5 bits for the exponent and 2 bits for the fraction ([25, 26, 27, 28]) and uses either no scaling constants or zeropoint scaling. These data types have large errors for large magnitude values since they have only 2 bits for the fraction but provide high accuracy for small magnitude values. [29] provide an excellent analysis of when certain fixed point exponent/fraction bit widths are optimal for inputs with a particular standard deviation. We believe FP8 data types offer superior performance compared to the Int8 data type, but currently, neither GPUs nor TPUs support this data type.

Outlier Features in Language Models. Large magnitude outlier features in language models have been studied before ([30, 31, 32, 33]). Previous work proved the theoretical relationship between outlier appearance in transformers and how it relates to layer normalization and the token frequency distribution ([34]). Similarly, [35] attribute the appearance of outliers in BERT model family to LayerNorm, and [36] show empirically that outlier emergence is related to the frequency of tokens in the training distribution. We extend this work further by showing how the scale of autoregressive models relates to the emergent properties of these outlier features, and showing how appropriately modeling outliers is critical to effective quantization.

Multi-billion Scale Transformer Quantization. There are two methods that were developed in parallel to ours: nuQmm ([37]) and ZeroQuant ([38]). Both use the same quantization scheme: group-w2ise quantization, which has even finer quantization normalization constant granularity than vector-wise quantization. This scheme offers higher quantization precision but also requires custom CUDA kernels. Both nuQmm and ZeroQuant aim to accelerate inference and reduce the memory footprint while we focus on preserving predictive performance under an 8-bit memory footprint. The largest models that nuQmm and ZeroQuant evaluate are 2.7B and 20B parameter transformers, respectively. ZeroQuant achieves zero-degradation performance for 8-bit quantization of a 20B model. We show that our method allows for zero-degradation quantization of models up to 176B parameters. Both nuQmm and ZeroQuant suggest that finer quantization granularity can be an effective means to quantize large models. These methods are complementary with LLM.int8(). Another parallel work is GLM-130B which uses insights from our work to achieve zero-degradation 8-bit quantization ([39]). GLM-130B performs full 16-bit precision matrix multiplication with 8-bit weight storage.

6. Discussion and Limitations

Section Summary: This research shows that large AI models with billions of parameters can be compressed into a simpler Int8 format for faster computing without losing any accuracy, using a technique that handles unusual data patterns separately and applies precise quantization, tested successfully on models up to 175 billion parameters. However, the study is limited to the Int8 format and doesn't explore newer 8-bit floating-point options, which hardware isn't ready for yet, and it only covers models of this size, leaving bigger ones untested. It also skips compressing the attention mechanism fully and focuses only on running the models rather than training them, with those areas marked for future research.

We have demonstrated for the first time that multi-billion parameter transformers can be quantized to Int8 and used immediately for inference without performance degradation. We achieve this by using our insights from analyzing emergent large magnitude features at scale to develop mixed-precision decomposition to isolate outlier features in a separate 16-bit matrix multiplication. In conjunction with vector-wise quantization that yields our method, LLM.int8(), which we show empirically can recover the full inference performance of models with up to 175B parameters.

The main limitation of our work is that our analysis is solely on the Int8 data type, and we do not study 8-bit floating-point (FP8) data types. Since current GPUs and TPUs do not support this data type, we believe this is best left for future work. However, we also believe many insights from Int8 data types will directly translate to FP8 data types. Another limitation is that we only study models with up to 175B parameters. While we quantize a 175B model to Int8 without performance degradation, additional emergent properties might disrupt our quantization methods at larger scales.

A third limitation is that we do not use Int8 multiplication for the attention function. Since our focus is on reducing the memory footprint and the attention function does not use any parameters, it was not strictly needed. However, an initial exploration of this problem indicated that a solution required additional quantization methods beyond those we developed here, and we leave this for future work.

A final limitation is that we focus on inference but do not study training or finetuning. We provide an initial analysis of Int8 finetuning and training at scale in Appendix E. Int8 training at scale requires complex trade-offs between quantization precision, training speed, and engineering complexity and represents a very difficult problem. We again leave this to future work.

::: {caption="Table 3: Different hardware setups and which methods can be run in 16-bit vs. 8-bit precision. We can see that our 8-bit method makes many models accessible that were not accessible before, in particular, OPT-175B/BLOOM."}

:::

7. Broader Impacts

Section Summary: This work primarily allows researchers and organizations with limited GPU resources to use large AI models that were previously too big to run, opening up new possibilities for studies and applications that weren't feasible before. It also lets well-funded groups handle more models efficiently on their existing hardware, which could widen the gap between resource-rich and resource-poor entities. Overall, making powerful pretrained models like OPT more accessible through techniques such as Int8 inference may spur innovative research in academia but could lead to unpredictable positive and negative societal effects.

The main impact of our work is enabling access to large models that previously could not fit into GPU memory. This enables research and applications which were not possible before due to limited GPU memory, in particular for researchers with the least resources. See Table 3 for model/GPU combinations which are now accessible without performance degradation. However, our work also enables resource-rich organizations with many GPUs to serve more models on the same number of GPUs, which might increase the disparities between resource-rich and poor organizations.

In particular, we believe that the public release of large pretrained models, for example, the recent Open Pretrained Transformers (OPT) ([4]), along with our new Int8 inference for zero- and few-shot prompting, will enable new research for academic institutions that was not possible before due to resource constraints. The widespread accessibility of such large-scale models will likely have both beneficial and detrimental effects on society that are difficult to predict.

Acknowledgments

We thank Ofir Press, Gabriel Ilharco, Daniel Jiang, Mitchell Wortsman, Ari Holtzman, Mitchell Gordon for their feedback on drafts of this work. We thank JustHeuristic (Yozh) and Titus von Köller for help with Hugging Face Transformers integration.

Checklist

Appendix

Section Summary: The appendix begins by comparing memory usage, showing that the LLM.int8() method reduces the footprint enough to run massive open-source models like OPT-175B and BLOOM-176B on single consumer-grade GPUs, unlike standard 16-bit precision. It then reviews related research on quantizing smaller language models and convolutional networks, highlighting how this approach avoids performance drops without needing extra training, and details outlier features in transformers that are crucial for accurate predictions and favor certain quantization techniques. Finally, it examines inference speeds, noting that while 8-bit methods can accelerate large matrix operations on high-end hardware, overheads often limit gains for smaller setups.

A. Memory usage compared to 16-bit precision

Table 3 compares the memory footprint of 16-bit inference and LLM.int8() for different open source models. We can see, that LLM.int8() allows to run the largest open source models OPT-175B and BLOOM-176B on a single node equipped with consumer-grade GPUs.

:::

B. Additional Related Work

Quantization of Transformers with fewer than 1B Parameters

Quantization of transformers has been focused on sub-billion parameter masked language model (MLMs), including BERT ([40]) and RoBERTa ([41]). Versions of 8-bit BERT/RoBERTa include Q8BERT ([8]), QBERT ([9]), product quantization with quantization noise ([42]), TernaryBERT ([43]), and BinaryBERT ([44]). Work by [45] performs both quantization and pruning. All these models require either quantization-aware finetuning or post-training quantization to make the model usable in low-precision. In contrast with our methods, the model can be used directly without performance degradation.

If one views matrix multiplication as 1x1 convolution, vector-wise quantization is equivalent to channel-wise quantization for convolution combined with row quantization ([12]). For matrix multiplication, this was used by [46] for BERT-sized transformers (350M parameters), while we are the first to study vector-wise quantization for autoregressive and large-scale models. The only other work that we are aware of that quantizes transformers other than BERT is [6], which uses post-training quantization with zeropoint quantization in the forward pass and zeropoint-row-wise quantization in the backward pass. However, this work is still for sub-billion parameter transformers. We compare with both zeropoint and row-wise quantization in our evaluations and do not require post-training quantization.

Low-bitwidth and Convolutional Network Quantization

Work that uses less than 8-bits for data types is usually for convolutional networks (CNNs) to reduce their memory footprint and increase inference speed for mobile devices while minimizing model degradation. Methods for different bit-widths have been studied: 1-bit methods ([47, 48, 49]), 2 to 3-bit ([50, 51]), 4-bits ([52]), more bits ([53]), or a variable amount of bits ([54]). For additional related work, please see the survey of [55]. While we believe that lower than 8-bit width with some performance degradation is possible for billion-scale transformers, we focus on 8-bit transformers that do not degrade performance and that can benefit from commonly used GPUs that accelerates inference through Int8 tensor cores.

Another line of work that focuses on convolutional network quantization is to learn adjustments to the quantization procedure to improve quantization errors. For example, using Hessian information ([56]), step-size quantization ([57]), soft quantization ([54]), mixed-precision via linear programming optimization ([58]), and other learned quantization methods ([59, 60]).

C. Detailed Outlier Feature Data

Table 4 provides tabulated data from our outlier feature analysis. We provide the quartiles of the most common outlier in each transformer and the number of outliers that are one-sided, that is, which have asymmetric distributions which do not cross zero.

::: {caption="Table 4: Summary statistics of outliers with a magnitude of at least 6 that occur in at least 25% of all layers and at least 6% of all sequence dimensions. We can see that the lower the C4 validation perplexity, the more outliers are present. Outliers are usually one-sided, and their quartiles with maximum range show that the outlier magnitude is 3-20x larger than the largest magnitude of other feature dimensions, which usually have a range of [-3.5, 3.5]. With increasing scale, outliers become more and more common in all layers of the transformer, and they occur in almost all sequence dimensions. A phase transition occurs at 6.7B parameters when the same outlier occurs in all layers in the same feature dimension for about 75% of all sequence dimensions (SDim). Despite only making up about 0.1% of all features, the outliers are essential for large softmax probabilities. The mean top-1 softmax probability shrinks by about 20% if outliers are removed. Because the outliers have mostly asymmetric distributions across the sequence dimension $s$, these outlier dimensions disrupt symmetric absmax quantization and favor asymmetric zeropoint quantization. This explains the results in our validation perplexity analysis. These observations appear to be universal as they occur for models trained in different software frameworks (fairseq, OpenAI, Tensorflow-mesh), and they occur in different inference frameworks (fairseq, Hugging Face Transformers). These outliers also appear robust to slight variations of the transformer architecture (rotary embeddings, embedding norm, residual scaling, different initializations)."}

:::

D. Inference Speedups and Slowdowns

D.1 Matrix Multiplication benchmarks

While our work focuses on memory efficiency to make models accessible, Int8 methods are also often used to accelerate inference. We find that the quantization and decomposition overhead is significant, and Int8 matrix multiplication itself only yields an advantage if the entire GPU is well saturated, which is only true for large matrix multiplication. This occurs only in LLMs with a model dimension of 4096 or larger.

Detailed benchmarks of raw matrix multiplication and quantization overheads are seen in Table 5. We see that raw Int8 matrix multiplication in cuBLASLt begins to be two times faster than cuBLAS at a model size of 5140 (hidden size 20560). If inputs need to be quantized and outputs dequantized – a strict requirement if not the entire transformer is done in Int8 – then the speedups compared to 16-bit is reduced to 1.6x at a model size of 5140. Models with model size 2560 or smaller are slowed down. Adding mixed precision decomposition slows inference further so that only the 13B and 175B models have speedups.

These numbers could be improved significantly with optimized CUDA kernels for the mixed precision decomposition. However, we also see that existing custom CUDA kernels are much faster than when we use default PyTorch and NVIDIA-provided kernels for quantization which slow down all matrix multiplications except for a 175B model.

::: {caption="Table 5: Inference speedups compared to 16-bit matrix multiplication for the first hidden layer in the feed-forward of differently sized GPT-3 transformers. The hidden dimension is 4x the model dimension. The 8-bit without overhead speedups assumes that no quantization or dequantization is performed. Numbers small than 1.0x represent slowdowns. Int8 matrix multiplication speeds up inference only for models with large model and hidden dimensions."}

:::

D.2 End-to-end benchmarks

Besides matrix multiplication benchmarks, we also test the end-to-end inference speed of BLOOM-176B in Hugging Face. Hugging Face uses an optimized implementation with cached attention values. Since this type of inference is distributed and, as such, communication dependent, we expect the overall speedup and slowdown due to Int8 inference to be smaller since a large part of the overall inference runtime is the fixed communication overhead.

We benchmark vs. 16-bit and try settings that use a larger batch size or fewer GPUs in the case of Int8 inference, since we can fit the larger model on fewer devices. We can see results for our benchmark in Table 6. Overall Int8 inference is slightly slower but close to the millisecond latency per token compared to 16-bit inference.

::: {caption="Table 6: Ablation study on the number of GPUs used to run several types of inferences of BLOOM-176B model. We compare the number of GPUs used by our quantized BLOOM-176B model together with the native BLOOM-176B model. We also report the per-token generation speed in milliseconds for different batch sizes. We use our method integrated into transformers([10]) powered by accelerate library from HuggingFace to deal with multi-GPU inference. Our method reaches a similar performance to the native model by fitting into fewer GPUs than the native model."}

:::

E. Training Results

We test Int8 training on a variety of training settings and compare to 32-bit baselines. We test separate settings for running the transformer with 8-bit feed-forward networks with and without 8-bit linear projections in the attention layer, as well at the attention iteself in 8-bit and compare against 32-bit performance. We test two tasks (1) language modeling on part of the RoBERTa corpus including Books ([14]), CC-News ([15]), OpenWebText ([16]), and CC-Stories ([17]); and (2) neural machine translation (NMT) ([61]) on WMT14+WMT16 ([62, 63]).

The results are shown in Table 7 and Table 8. We can see that for training, using the attention linear projections with Int8 data types and vector-wise quantization leads to degradation for NMT and for 1.1B language model but not for 209M language modeling. The results improve slightly if mixed-precision decomposition is used but is not sufficient to recover full performance in most cases. These suggests that training with 8-bit FFN layers is straightforward while other layers require additional techniques or different data types than Int8 to do 8-bit training at scale without performance degradation.

::: {caption="Table 7: Initial results on small and large-scale language modeling. Doing attention in 8-bit severely degrades performance and performance cannot fully recovered with mixed-precision decomposition. While small-scale language models is close to baseline performance for both 8-bit FFN and 8-bit linear projects in the attention layers performance degrades at the large scale."}

:::

::: {caption="Table 8: Neural machine translation results for 8-bit FFN and linear attention layers for WMT14+16. Decomp indicates the percentage that is computed in 16-bit instead of 8-bit. The BLEU score is the median of three random seeds."}

:::

F. Fine-tuning Results

We also test 8-bit finetuning on RoBERTa-large finetuned on GLUE. We run two different setups: (1) we compare with other Int8 methods, and (2) we compare degradation of finetuning with 8-bit FFN layers as well as 8-bit attention projection layers comparel to 32-bit. We finetune with 5 random seeds and report median performance.

Table 9 compares with different previous 8-bit methods for finetuning and shows that vector-wise quantization improves on other methods. Table 10 shows the performance of FFN and/or linear attention projections in 8-bit as well as improvements if mixed-precision decomposition is used. We find that 8-bit FFN layers lead to no degradation while 8-bit attention linear projections lead to degradation if not combined with mixed-precision decomposition where at least the top 2% magnitude dimensions are computed in 16-bit instead of 8-bit. These results highlight the critical role of mixed-precision decomposition for finetuning if one wants to not degrade performance.

::: {caption="Table 9: GLUE finetuning results for quantization methods for the feedforward layer in 8-bit while the rest is in 16-bit. No mixed-precision decomposition is used. We can see that vector-wise quantization improve upon the baselines."}

:::

::: {caption="Table 10: Breakdown for 8-bit feedforward network (FFN) and linear attention layers for GLUE. Scores are median of 5 random seeds. Decomp indicates the percentage that is decomposed into 16-bit matrix multplication. Compared to inference, fine-tuning appears to need a higher decomp percentage if the linear attention layers are also converted to 8-bit."}

:::

References

[1] Vaswani et al. (2017). Attention is all you need. arXiv preprint arXiv:1706.03762.

[2] Radford et al. (2019). Language models are unsupervised multitask learners. OpenAI blog. 1(8). pp. 9.

[3] Brown et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

[4] Zhang et al. (2022). OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01068.

[5] Ilharco et al. (2020). High Performance Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts. pp. 24–27. doi:10.18653/v1/2020.emnlp-tutorials.4. https://aclanthology.org/2020.emnlp-tutorials.4.

[6] Chen et al. (2020). A statistical framework for low-bitwidth training of deep neural networks. Advances in Neural Information Processing Systems. 33. pp. 883–894.

[7] Lin et al. (2020). Towards fully 8-bit integer inference for the transformer model. arXiv preprint arXiv:2009.08034.

[8] Zafrir et al. (2019). Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS). pp. 36–39.

[9] Shen et al. (2020). Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence. pp. 8815–8821.

[10] Wolf et al. (2019). HuggingFace's Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.

[11] Dettmers et al. (2022). 8-bit Optimizers via Block-wise Quantization. 9th International Conference on Learning Representations, ICLR.

[12] Khudia et al. (2021). FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference. arXiv preprint arXiv:2101.05615.

[13] Ott et al. (2019). fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038.

[14] Zhu et al. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision. pp. 19–27.

[15] Nagel, Sebastian (2016). Cc-news.

[16] Gokaslan, Aaron and Cohen, Vanya (2019). Openwebtext corpus. urlhttp://Skylion007. github. io/OpenWebTextCorpus.

[17] Trinh, Trieu H and Le, Quoc V (2018). A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847.

[18] Wenzek et al. (2020). CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In Proceedings of the 12th Language Resources and Evaluation Conference. pp. 4003–4012. https://www.aclweb.org/anthology/2020.lrec-1.494.

[19] Artetxe et al. (2021). Efficient Large Scale Language Modeling with Mixtures of Experts. arXiv preprint arXiv:2112.10684.

[20] Raffel et al. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.

[21] Gao et al. (2021). A framework for few-shot language model evaluation. doi:10.5281/zenodo.5371628. https://doi.org/10.5281/zenodo.5371628.

[22] Shazeer et al. (2018). Mesh-tensorflow: Deep learning for supercomputers. Advances in neural information processing systems. 31.

[23] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556.

[24] Henighan et al. (2020). Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701.

[25] Naigang Wang et al. (2018). Training Deep Neural Networks with 8-bit Floating Point Numbers. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. pp. 7686–7695. https://proceedings.neurips.cc/paper/2018/hash/335d3d1cd7ef05ec77714a215134914c-Abstract.html.

[26] Xiao Sun et al. (2019). Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 4901–4910. https://proceedings.neurips.cc/paper/2019/hash/65fc9fb4897a89789352e211ca2d398f-Abstract.html.

[27] Léopold Cambier et al. (2020). Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. https://openreview.net/forum?id=Bkxe2AVtPS.

[28] Naveen Mellempudi et al. (2019). Mixed Precision Training With 8-bit Floating Point. CoRR. abs/1905.12334. http://arxiv.org/abs/1905.12334.

[29] Jin et al. (2022). F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization. arXiv preprint arXiv:2202.05239.

[30] Timkey, William and van Schijndel, Marten (2021). All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. arXiv preprint arXiv:2109.04404.

[31] Bondarenko et al. (2021). Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948.

[32] Wei et al. (2022). Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. arXiv preprint arXiv:2209.13325.

[33] Luo et al. (2021). Positional Artefacts Propagate Through Masked Language Model Embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 5312–5327. doi:10.18653/v1/2021.acl-long.413. https://aclanthology.org/2021.acl-long.413.

[34] Gao et al. (2019). Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009.

[35] Kovaleva et al. (2021). BERT busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.06990.

[36] Puccetti et al. (2022). Outliers Dimensions that Disrupt Transformers Are Driven by Frequency. arXiv preprint arXiv:2205.11380.

[37] Park et al. (2022). nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models. arXiv preprint arXiv:2206.09557.

[38] Yao et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. arXiv preprint arXiv:2206.01861.

[39] Zeng et al. (2022). GLM-130B: An Open Bilingual Pre-trained Model. arXiv preprint arXiv:2210.02414.

[40] Devlin et al. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

[41] Liu et al. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

[42] Fan et al. (2020). Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.07320.

[43] Wei Zhang et al. (2020). TernaryBERT: Distillation-aware Ultra-low Bit BERT. In EMNLP.

[44] Haoli Bai et al. (2021). BinaryBERT: Pushing the Limit of BERT Quantization. ArXiv. abs/2012.15701.

[45] Zhao et al. (2021). Automatic Mixed-Precision Quantization Search of BERT. arXiv preprint arXiv:2112.14938.

[46] Wu et al. (2020). Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602.

[47] Matthieu Courbariaux and Yoshua Bengio (2016). BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. CoRR. abs/1602.02830. http://arxiv.org/abs/1602.02830.

[48] Mohammad Rastegari et al. (2016). XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV. pp. 525–542. doi:10.1007/978-3-319-46493-0_32. https://doi.org/10.1007/978-3-319-46493-0_32.

[49] Matthieu Courbariaux et al. (2015). BinaryConnect: Training Deep Neural Networks with binary weights during propagations. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. pp. 3123–3131. https://proceedings.neurips.cc/paper/2015/hash/3e15cc11f979ed25912dff5b0669f2cd-Abstract.html.

[50] Chenzhuo Zhu et al. (2017). Trained Ternary Quantization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. https://openreview.net/forum?id=S1_pAu9xl.

[51] Jungwook Choi et al. (2019). Accurate and Efficient 2-bit Quantized Neural Networks. In Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 - April 2, 2019. https://proceedings.mlsys.org/book/268.pdf.

[52] Rundong Li et al. (2019). Fully Quantized Network for Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 2810–2819. doi:10.1109/CVPR.2019.00292. http://openaccess.thecvf.com/content_CVPR_2019/html/Li_Fully_Quantized_Network_for_Object_Detection_CVPR_2019_paper.html.

[53] Courbariaux et al. (2014). Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024.

[54] Ruihao Gong et al. (2019). Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. pp. 4851–4860. doi:10.1109/ICCV.2019.00495. https://doi.org/10.1109/ICCV.2019.00495.

[55] Haotong Qin et al. (2020). Binary Neural Networks: A Survey. CoRR. abs/2004.03333. https://arxiv.org/abs/2004.03333.

[56] Dong et al. (2019). Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 293–302.

[57] Esser et al. (2019). Learned step size quantization. arXiv preprint arXiv:1902.08153.

[58] Yao et al. (2021). Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning. pp. 11875–11886.

[59] Zhang et al. (2018). Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV). pp. 365–382.

[60] Gholami et al. (2021). A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630.

[61] Ott et al. (2018). Scaling neural machine translation. arXiv preprint arXiv:1806.00187.

[62] Macháček, Matouš and Bojar, Ondřej (2014). Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation. pp. 293–301.

[63] Sennrich et al. (2016). Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891.