Sliding-window beats linear attention

Alexia Jolicoeur-Martineau
Microsoft
Applied Sciences Group (ASG)
[email protected]

Rhea Sanjay Sukthanker
Microsoft
Applied Sciences Group (ASG)
[email protected]

Pashmina Cameron
Microsoft
Applied Sciences Group (ASG)
[email protected]

Emy Gervais
Independent
[email protected]

Abstract

Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable.

Several alternatives have been proposed to fix the quadratic scaling problem, one of which is retrofitting LLMs to use Linear Attention. This idea has attracted a lot of attention, given its promise to solve the quadratic scaling problem with state-of-the-art performance at low cost. However, this line of research has not been properly compared to simpler baselines.

In this work, we show that Sliding Window Attention (SWA) with sinks performs as well or better than post-trained Linear Attention models. We observe this across multiple LLMs on various downstream tasks. For long-context reasoning tasks (Needle-in-a-Haystack and BABILong), SWA achieves massively higher performance (2 to 10 times higher than linear attention). SWA requires no post-training, is extremely fast, and requires low memory; therefore, making it an extremely cheap and reliable solution.

To reduce inference memory cost, we strongly recommend switching to SWA instead of post-training linear models. Linear attention models may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA.

Executive Summary: This volume is designed for machine learning researchers, deep learning engineers, and natural language processing practitioners who seek efficient inference strategies for large language models. The text assumes a solid working knowledge of deep learning fundamentals, the standard Transformer architecture, self-attention mechanics, and key efficiency bottlenecks such as key-value (KV) cache memory scaling during autoregressive generation.

The work investigates methods for mitigating the quadratic computational and memory costs of standard self-attention. It establishes a direct comparative evaluation between training-free Sliding Window Attention (SWA) equipped with attention sinks and various post-trained Linear Attention techniques. Progressing from the mathematical formulation of self-attention, SWA receptive fields, and linear kernel approximations, the text moves to systematic empirical evaluations across standard reasoning benchmarks, long-context tasks, and hardware execution efficiency.

Central to the book's presentation is the concept of attention sinks: the phenomenon where autoregressive models allocate significant attention weight to initial tokens. The authors demonstrate that pairing a local sliding window with a fixed number of initial sink tokens prevents catastrophic performance drops without modifying model weights. The book contrasts this approach against linearized attention frameworks (such as LoLCATs, Hedgehog, and Liger-GLA), evaluating their respective expressivity constraints, memory footprints, and computational speeds across diverse base model families ranging from 1.3 billion to 70 billion parameters.

Readers will learn how to evaluate inference efficiency, select appropriate attention masks, and implement constant-memory inference pipelines on existing pretrained models without post-training or custom linear kernel retrofitting. After studying this text, practitioners will be able to assess the trade-offs between local windowing and linear recurrent state tracking, particularly in long-context retrieval environments like Needle-in-a-Haystack and BABILong.

The scope of the text is specifically restricted to training-free SWA and post-trained linear attention conversions. It explicitly excludes hybrid attention architectures that retain full-attention layers, learned dynamic attention sink mechanisms, multi-modal applications, and models trained from scratch with linear attention.

1. Introduction

Section Summary: Large language models require huge and growing amounts of memory because each new token adds to a key-value cache whose size and compute cost grow quadratically with context length. Linear-attention approaches promise to reduce this cost to a fixed, linear amount, and recent work such as LoLCATs has shown that pretrained models can be converted to these methods with surprisingly little extra training. The authors nevertheless find that simply switching the original model to sliding-window attention with a few attention sinks already matches or exceeds the performance of these linearized models on both short and long-context tasks, without any retraining or special kernels.

Large Language Models (LLMs) consume massive amounts of memory and energy because they scale quadratically with context-length ([11]). Every new token’s keys and values must be added to the KV cache, and they contribute to an ever-growing memory and compute cost.

Linear attention methods ([12]) propose an alternative which reduce the time and memory complexity from quadratic to linear. This removes the need to store a KV cache which makes the memory cost small and fixed at inference-time instead of having an ever-growing KV cache.

While promising, linear attention models come with significant drawbacks: 1) lower expressivity, 2) untractable problem of having to decide what to remember so that important information is not overwritten/ignored, and 3) training is expensive and most software/hardware are not built for them.

To tackle the expensive training, a solution is to take an existing pretrained models and swap its expensive quadratic attention with a linear attention. This would normally require fine-tuning with billions of tokens ([13]), but LoLCATs ([3]) showed that this could be done with as little as 40M tokens by combining Hedgehog ([2]) and Sliding Window Attention (SWA). In doing so, LoLCATs recovers most of the full attention performance on knowledge and reasoning tasks.

Given the promises of these linearizing methods, we gave them a try. What we found was striking. We discovered that changing the attention mask to Sliding Window Attention (SWA)with attention sinks (meaning that we attend to the $k$ previous tokens and the first 4 tokens) gives us better downstream performance than most linearized models. For long-context tasks, the performance gap is even more pronounced, whereas SWA obtains much higher accuracy than linear attention post-training.

The fact that SWA with sinks retains most of the performance of the original model is well known ([14, 15]), however, to our knowledge, linearizing methods have not been compared to SWA with sinks. In this work, we provide this direct comparison for a wide variety of models (ranging from 1.3B to 70B) on various benchmarks (short and long context). In doing so, we demonstrate that pretrained models can already use SWA at inference time to obtain high performance at a fixed inference memory cost without needing any post-training or specialized linear kernels.

Notation We denote Sliding Window Attention (SWA) in the following way: SWA( w , s ), where $w$ is the window-size (varies from 64 to 512) and $s$ is the number of attention sink (always fixed at 4).

2. Background

Section Summary: Transformers rely on self-attention to let each token in a sequence incorporate information from previous ones, but this process scales quadratically with length and requires storing a growing cache of keys and values, which becomes costly for long inputs. Sliding window attention addresses part of the problem by restricting each token to attend only to a fixed number of recent tokens plus a few early “sink” tokens, allowing the model’s effective context to expand gradually across layers without retraining. Linear attention offers a different fix by approximating the attention scores with a simple kernel function, which reduces the computation to a constant-time update of two fixed-size vectors and eliminates the need for an ever-expanding cache, though finding kernels that preserve performance remains difficult.

2.1 Transformers and Self-Attention (SA)

Transformers ([11]) are the backbone of modern Large Language Models (LLMs). They process a sequence of $L$ consecutive tokens. Assuming discrete tokens (with vocab-size $V$) used in language, the tokens are embedded ($ V \to D$), transforming the shape of the sequence to $[L, D]$. Transformers then process this sequence by alternating between a transformation on the dimension $D$ using Multi-Layer Perceptron (MLP) and on the dimension $L$ using Self-Attention (SA). Both MLP and SA use residual connections ($x \gets x + f(x)$ where $f$ is either the MLP or SA). The final output is then projected back into shape $[L, V]$ to predict the next-token probabilities.

Self-Attention works as follows: the data of shape $[L, D]$ is linearly projected to keys $\mathbf{k}$, queries $\mathbf{q}$, and values $\mathbf{v}$ and the output is transformed in the following way:

$ \mathbf{x}{t} = \frac{\sum{i=1}^t a_{t, i}\mathbf{v}i}{\sum{i=1}^t a_{t, i}} = \frac{\sum_{i=1}^t \exp(\mathbf{q}_t \mathbf{k}_i^\top / \sqrt{d})\mathbf{v}i}{\sum{i=1}^t \exp(\mathbf{q}_t \mathbf{k}_i^\top / \sqrt{d})}, ;;; \text{for }t\text{ in }[1, \ldots, L]\tag{1} $

Several variants of the formulation have proposed, such as Multi-Query Attention ([16]), Grouped Query Attention ([17]). Unlike MLP which has complexity $\mathrm{O}(LD^2)$, Self-Attention has complexity $\mathrm{O}(DL^2)$ and it requires a KV cache at inference time to keep tracks of all previous keys and values in order to make the inference cost per new token linear $\mathrm{O}(L)$. The quadratic scaling with respect to context-length and the ever-growing KV cache is a big problem for LLMs as it increases in memory and requires more processing time as the sequence grows.

Many engineering tricks have been exploited to make self-attention more efficient ([18, 19, 20, 21]) and compress the KV cache ([22, 23, 24]). However, the quadratic problem remains. Various solutions have been suggested, but we focus on the following two: Sliding Window Attention (SWA) and Linear Attention (LA).

2.2 Sliding Window Attention (SWA)

Instead of attending to all previous tokens, Sliding Window Attention (SWA) ([25]) proposes to only attend to the previous $w$ tokens (its window size). This may appear extremely constraining, but similar to convolutional networks ([26]), the effective receptive field grows after each self-attention layers such that after $l$ layers, the receptive field is $lw$. Empirically, SWA improves long-term memorization and length extrapolation by encouraging models to learn dependencies beyond their local receptive field ([15]). SWA can be represented as follows:

$ \mathbf{x}t = \frac{\displaystyle\sum{i=\max(1, t-w+1)}^{t} \exp!\left(\mathbf{q}_t\mathbf{k}_i^\top/\sqrt{d}\right)\mathbf{v}i} {\displaystyle\sum{i=\max(1, t-w+1)}^{t} \exp!\left(\mathbf{q}_t\mathbf{k}_i^\top/\sqrt{d}\right)}, \qquad t\in[1, \ldots, L]. $

It was later discovered that LLMs assign disproportionately high attention to the first few tokens even though they are not semantically relevant ([14, 27]). These tokens, called the attention sinks, are tokens that transformers learn to use as repositories for any unnecessary attention. Without their inclusion in the attention mask, there is a catastrophic degradation in performance. Thus, when SWA moves past the first few (sink) tokens, performance becomes horrendous. A simple fix was later found: attend to the first $s=4$ tokens in addition to the remaining $w-4$ sliding window ([14]). This approach solves the catastrophic failure that arises after the first tokens are out of the sliding window.

SWA naturally works without any training, but additional post-training can boost its performance ([28]). There also exists learnable attention sinks ([29]), but these require additional post-training. In this work, we focus exclusively on training-free SWA with sinks.

2.3 Linear attention

Linear Attention ([12]) has been proposed as a way to alleviate the issues in Self-Attention. The main idea is to design a transformation $\phi$ such that $\exp(\mathbf{q}_t \mathbf{k}_i^\top) \approx \phi(\mathbf{q}_t) \phi(\mathbf{k}_i)^\top$. Then, the computations can be reformulated as:

$ \mathbf{x}{t} = \frac{\sum{i=1}^t \phi(\mathbf{q}_t) \phi(\mathbf{k}_i)^\top\mathbf{v}i}{\sum{i=1}^t \phi(\mathbf{q}_t) \phi(\mathbf{k}_i)^\top } = \frac{\phi(\mathbf{q}t)\sum{i=1}^t \phi(\mathbf{k}_i)^\top\mathbf{v}_i}{\phi(\mathbf{q}t)\sum{i=1}^t \phi(\mathbf{k}_i)^\top } = \frac{\phi(\mathbf{q}_t) \mathbf{s}_t}{\phi(\mathbf{q}_t) \mathbf{z}_t },\tag{2} $

Thus at inference, the equation becomes:

$ \mathbf{s}t=\mathbf{s}{t-1}+\phi(\mathbf{k}_t)^\top\mathbf{v}_t, \quad\mathbf{z}t=\mathbf{z}{t-1}+\phi(\mathbf{k}_t)^\top.\tag{3} $

This makes the inference cost $\mathrm{O}(1)$ with respect to $L$ since it does not depend on the sequence length. Thus one only need to store and update $\mathbf{s}$ and $\mathbf{v}$ over time, both of which do not grow in size over time.

This resolves the issue of increasing memory and lower speed over long context lengths. However, making Linear Attention work well in practice is extremely challenging due to the nature of having to continuously rewrite itself while trying not to forget/ignore important information.

Many variants of linear attention with varying performance have been proposed ([30, 31, 32, 33, 34, 35, 36, 37, 38, 39]). Furthermore, multiple kernels have been proposed such as ReLU or ELU ([12]). There are 3 necessary properties that are required for a good kernel: expressiveness, spikiness, and monotonicity. To obtain these properties, Hedgehog ([2]) proposed using a learnable projection followed by a dual-sided exponential transformation:

$ \phi(x) \gets (\exp(f(x)), \exp(-f(x)))\tag{4} $

where $f$ is a linear projection from $D$ to $D/2$.

2.3.1 Post-training of linear attention

Training linear attention models is expensive and most software/hardware is made for Softmax attention, which makes things more challenging. Instead of training from scratch a linear attention Transformer, one can convert existing pretrained LLMs with quadratic Self-Attention to use linear-attention instead. With some post-training, a large portion of the baseline’s performance can be recovered. Most importantly, LoLCATs ([3]) showed that one could linearize models with as little as 40M tokens using Low-Rank Adaptation (LoRA) ([40]). To make it work, they use an expressive kernel such as Hedgehog ([2]) and combine the linear attention with a small Sliding Window Attention (SWA). This idea paved the way toward extremely efficient methods for post-training linear attention models.

3. The missing comparison: Sliding Window vs Linear Attention

Section Summary: Linear attention offers one approach to handling very long inputs without the usual steep computational costs, but it often suffers from reduced effectiveness and requires extra training. Many studies compare it only against a basic sliding-window method that lacks special “sink” tokens, which makes the results misleading because that basic version breaks down quickly once key early information falls out of view. This work shows that adding attention sinks to the sliding-window approach matches or exceeds the linear methods on ordinary tasks and performs far better when contexts grow long.

Post-training linear attention is an interesting way of solving the quadratic scaling problem, although it comes with its own drawbacks (e.g., low expressivity, untractable problem of knowing what to retain and forget, requires additional training). However, most linear attention papers only compare linear-attention to regular (sink-free) SWA. This makes the comparison inadequate since we know that sink-free SWA leads to catastrophic failure once the first (sink) tokens are out of the sliding window. Furthermore, the long-context behavior of post-trained attention models is understudied.

In this work, we demonstrate that Sliding Window Attention (SWA) with attention sinks performs equally or better than post-trained linearized models on knowledge and reasoning short context tasks. SWA leads to especially large gains on long-context. See our results below.

4. Experiments

Section Summary: The experiments section evaluates various linear attention methods against sliding-window attention (SWA) and full-attention baselines across multiple models and tasks. On general knowledge and reasoning benchmarks, SWA matches or exceeds most linearized approaches in recovering the original model's performance, often without any additional training. It also shows clear advantages in long-context tasks like needle-in-haystack retrieval and BABILong, where linear methods degrade sharply, while delivering the best speed and lowest memory usage at practical window sizes.

We compare existing linearizing methods (pure linear or linear with SWA) to the base pretrained model and their sliding-window attention (SWA) analog. For simplicity and fairness, we do not compare to hybrid models containing some full-attention layers.

4.1 General Knowledge and Reasoning

We compare the different linear attention methods listed in Table 1 to Sliding Window Attention (SWA; window-size=64 with 4 sinks) on general knowledge and reasoning metrics with different base models (See Appendix A.1 for details). The summarized metrics are shown in Table 1 and the full results in Table 2.

::: {caption="Table 2: Benchmark scores (%) for distilled linearized models, teachers, and teachers with SWA (4 sinks with sliding window 64 or 128). The best non-teacher model is highlighted."}

:::

Results: SWA obtains the best average downstream performance in 9 out of 11 cases. The only exceptions are a) LoLCATs on Phi-1.5-1.3B which obtains a score barely higher (62.5 vs 62.4 for SWA), and b) QRWKV6 on Qwen2.5-32B-Instruct which performs as well the baseline model (77.3) while SWA has a slight drop of performance (76.6). On MMLU, SWA is always the best performing except for Llama2.0-7B where DiJiang is better (40.7 versus 39.8 for SWA).

Summarized results: SWA recovers the most out of MMLU baseline's performance (93.2%), followed closely by QRWKV6 (92.4%). Both SWA and QRWKV6 recover most of the average baseline performance (99.0% for SWA and 99.1% for QRWKV6). In terms of training efficiency at high performance, SWA is the winner with 0 tokens required, followed by LoLCATs which fine-tune on 40M tokens to recover 83.2% of MMLU and 97.5% of the average baseline performance.

As additional experiments, we also post-train our own linearized models with several state-of-the-art attention variants on modern architectures. Similar results were found when comparing them to SWA; see Appendix A.2 for details.

4.2 Long Context Reasoning

4.2.1 Single Needle-in-a-Haystack (S-NIAH)

We compare SWA, LoLCATs ([3]), and Liger-GLA ([4]) on Single Needle-in-a-Haystack (S-NIAH) tasks at context lengths up to 4K. The base model used by all approaches is Llama 3.1 8B ([41]). We compare at different window-sizes (128, 256, 512) Results are shown in Table 3.

::: {caption="Table 3: Accuracy on the Single Needle-in-a-Haystack (S-NIAH) across context lengths (0.5K, 1K, 2K, 4K) and window-size (128, 256, 512). The best model at each window-size is highlighted."}

:::

Results: At all window-size (128, 256, 512) and context-length, SWA obtains equal or higher score on all tasks. At 4K context-length, SWA recovers 17.2-23% of the regular full attention accuracy, while LoLCATs and Liger-GLA reach at most 5.8% and 0.8% accuracy respectively.

4.2.2 BABILong

We compare SWA and LoLCATs ([3]) on the BABILong Benchmark (average of QA1, QA2, QA3, QA4, and QA5) at context lengths up to 4K and window-size 256. The base model is Llama 3.1 8B ([41]). Results are shown in Table 4 (see Appendix A.3 for details).

::: {caption="Table 4: Accuracy on the BABILong Benchmark across context lengths (0K, 1K, 2K, 4K)."}

:::

Results: At small context tasks, LoLCATs(+SWA) obtains slightly higher scores than SWA (56% versus 55% at 0K and 22% versus 20% at 1K). At large context tasks, SWA obtains noticeably higher scores than LoLCATs (19% versus 10% at 2K and 15% versus 3% at 4K). At small context length (0K), SWA and LoLCATs(+SWA) recover 74-76% of the baseline performance. At long context length (4K), SWA recovers 25% of the baseline performance, unlike LoLCATs which recover only 5%.

4.3 Speed and Memory

We compare full attention (FA), Sliding Window Attention (SWA) with (4) attention sinks to linear attention and linear attention with SWA (LoLCATs) in terms of memory and speed. Results are shown in Figure 2. We also provide plots with additional details in Appendix A.4.

Hardware/Software: FA, and SWA both use FlashAttention ([18, 19]) as backend. For linear attention, ThunderKittens ([42]) is used as backend. For LoLCATs, the fused SWA + linear-attention kernel from ThunderKittens at window-size=256 is used. Tests are done on a 4-layer Transformer (embedding-size=1024, 16 heads with dimension 64) with batch-size=1 in float16 precision. The hardware is an NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition.

Speed: FA decreases in speed as context-length grows beyond 1K; all other methods stay with a relatively flat speed. SWA is the fastest method (window-size=64 is faster than 512, but both are faster than the other methods).

Memory: FA increases in memory cost linearly with context length. SWA increases in memory cost until it reaches its window-size, then it stays constant. SWA at window-size=64 has the lowest memory cost, followed by Linear, Linear + SWA (LoLCATs), and finally SWA at window-size=512.

Thus, SWA is faster and it has a similar or lower memory cost at window-size smaller than 512.

5. Limitations / Future Work

Section Summary: The current study examines sliding-window attention only in its basic training-free form, though further fine-tuning after initial training could boost results and should be compared against other efficient attention approaches across different data scales. The authors also note they have not explored mixtures that combine sliding-window or linear attention with full attention in selected layers or tokens, nor have they tested these ideas on very large models or demanding evaluations such as agent-based tasks. Extending the approach to multimodal and video-generation models, including three-dimensional sliding windows over space and time, appears promising, as recent evidence already shows strong performance on video diffusion tasks.

Our work is focused on training-free SWA. However, post-training can further improve the performance of SWA ([28]). Future work should investigate the effect of post-training with SWA and compare it to linearizing methods at different amounts of tokens (aka scaling laws). We did not consider hybrid models with some tokens or entire layers using full-attention (FA). It would be interesting to know the impact of different degrees of full-attention with SWA or linear attention. The analysis could also be extended to extremely large models and complex metrics, such as agentic tasks. The effect on multi-modal LLMs and video generations models with a multi-dimensional sliding window (e.g., 3D with (x, y) image position and (t) time coordinates) would also be interesting to investigate as future work.

Note that for video diffusion models ([43]), there is already strong evidence that training-free SWA is an extremely strong baseline. For example, Sliding Tile Attention ([44]) recovers 97% of baseline's VBench score ([45]) on HunyuanVideo ([46]) at 3.53 times the speed with the same number of sampling steps.

6. Conclusion

Section Summary: This study finds that sliding window attention with attention sinks outperforms most linear attention methods that require extra post-training. It delivers this edge without any additional training, while running faster and using less memory, recovering nearly all of the original model's performance on short tasks and far more on long ones. The authors therefore advise practitioners to adopt this approach when memory is limited.

In this work, we showed that Sliding Window Attention (SWA) with attention sinks achieves better performance than most linear attention post-training methods. It does so without any post-training, at higher decoding speed, and lower memory cost.

On short-context reasoning tasks, SWA recovers 99% of the average baseline’s performance, which matches the more expensive state-of-the-art linear attention post-training methods. On long-context tasks, SWA performs overwhelmingly better than linear-attention post-training. At context length 256, SWA recovers 20% and 25% of baseline performance on S-NIAH-3 and BABILong while LoLCATs only recovers 2.2% and 5% respectively.

Given our finding, we strongly recommend practitioners to use SWA with attention sinks in order to achieve the best performance at a fixed small memory cost.

Appendix

Section Summary: The appendix lists the large language models and standard benchmarks used for evaluation, ranging from smaller models like Phi-1.5 to much larger ones such as Llama and Qwen variants, with tests focused on reasoning, question answering, and commonsense tasks. It also includes additional tables with results for newly adapted linear-attention versions of models on long-context reasoning benchmarks like BABILong. Finally, it presents plots comparing the speed, memory usage, and computational costs of different attention methods across varying context lengths.

A.1 List of models and metrics

We compare methods on the following pretrained architectures: Phi-1.5-1.3B ([47]), Mistral-7B-v0.1 ([48]), Llama 2.0-7B ([49]), Llama 3.0-8B, Llama 3.0-8B-Instruct, Llama 3.1-8B, Llama 3.1-70B ([41]), Qwen 2.5-7B-Instruct, Qwen 2.5-32B-Instruct, Qwen 2.5-72B-Instruct ([50]), and QwQ ([51]). The size of the models range from 1.3B to 70B. The metrics considered are: MMLU ([52]), ARC-C, ARC-E ([53]), PIQA ([54]), WinoGrande ([55]), HellaSwag ([56]). We chose these metrics because they are the standards used by other papers, which makes comparisons easier ([3]) ([10]).

A.2 Additional results on newly linearized models

::: {caption="Table 5: Comparison of sliding-window and linear-attention methods. Teacher and SWA (64, 4) are train-free baselines; linear variants are LoLCATs-style two-stage distilled on ∼ 0.1B tokens of cleaned-Alpaca ([57]). We use different attention variants (GLA ([37]), Gated DeltaNet ([58]), QRWKV6 ([38])) on modern architectures (Qwen-3 ([59]), Phi-4-mini-reasoning ([60, 61]), and Phi-4-reasoning-plus ([62]))."}

:::

A.3 Babilong Full results

::: {caption="Table 6: BABILong Benchmark Full Results. Performance comparison on BABILong across increasing context lengths (0K ∼ 4K), evaluating long-context reasoning performance."}

:::

A.4 Speed, Memory, and FLOPs plots

References

Section Summary: This section compiles dozens of research papers, mostly from recent years, that focus on making large language models faster and more efficient. Many explore alternatives to the standard transformer architecture, such as linear attention, state-space models like Mamba, and recurrent structures, often through distillation or hybrid designs. Additional entries cover foundational work on attention mechanisms along with practical optimizations for memory use and long-context processing.

[1] Jean Mercat et al. (2024). Linearizing Large Language Models. https://arxiv.org/abs/2405.06640. arXiv:2405.06640.

[2] Michael Zhang et al. (2024). The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. https://arxiv.org/abs/2402.04347. arXiv:2402.04347.

[3] Zhang et al. (2025). Lolcats: On low-rank linearizing of large language models. In International Conference on Learning Representations. pp. 46200–46253.

[4] Lan et al. (2025). Liger: Linearizing large language models to gated recurrent structures. arXiv preprint arXiv:2503.01496.

[5] Aviv Bick et al. (2024). Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models. https://arxiv.org/abs/2408.10189. arXiv:2408.10189.

[6] Junxiong Wang et al. (2024). The Mamba in the Llama: Distilling and Accelerating Hybrid Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=uAzhODjALU.

[7] Hanting Chen et al. (2024). DiJiang: Efficient Large Language Models through Compact Kernelization. ArXiv. abs/2403.19928. https://arxiv.org/abs/2403.19928.

[8] Lin Yueyu et al. (2025). ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer. https://arxiv.org/abs/2501.15570. arXiv:2501.15570.

[9] Aviv Bick et al. (2025). Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing. https://arxiv.org/abs/2502.14458. arXiv:2502.14458.

[10] Goldstein et al. (2025). Radlads: Rapid attention distillation to linear attention decoders at scale. arXiv preprint arXiv:2505.03005.

[11] Vaswani et al. (2017). Attention is all you need. Advances in neural information processing systems. 30.

[12] Katharopoulos et al. (2020). Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning. pp. 5156–5165.

[13] Bick et al. (2024). Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models. arXiv preprint arXiv:2408.10189. doi:10.48550/arXiv.2408.10189.

[14] Xiao et al. (2024). Efficient streaming language models with attention sinks. In International Conference on Learning Representations. pp. 21875–21895.

[15] Cabannes et al. (2026). Short window attention enables long-term memorization. In International Conference on Learning Representations. pp. 79759–79774.

[16] Shazeer, Noam (2019). Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150.

[17] Ainslie et al. (2023). Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 conference on empirical methods in natural language processing. pp. 4895–4901.

[18] Dao et al. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems. 35. pp. 16344–16359.

[19] Dao, Tri (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.

[20] Kwon et al. (2023). Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. pp. 611–626.

[21] Zhang et al. (2023). H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in neural information processing systems. 36. pp. 34661–34710.

[22] Liu et al. (2023). Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems. 36. pp. 52342–52364.

[23] Li et al. (2024). Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems. 37. pp. 22947–22970.

[24] Tang et al. (2024). Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774.

[25] Beltagy et al. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

[26] Luo et al. (2016). Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems. 29.

[27] Barbero et al. (2025). Why do llms attend to the first token?.

[28] Yu et al. (2025). Swaa: Sliding window attention adaptation for efficient and quality preserving long context processing. arXiv preprint arXiv:2512.10411.

[29] Agarwal et al. (2025). gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.

[30] Gu, Albert and Dao, Tri (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.

[31] Qin et al. (2022). cosformer: Rethinking softmax in attention. arXiv preprint arXiv:2202.08791.

[32] Sun et al. (2023). Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621.

[33] Peng et al. (2023). Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048.

[34] De et al. (2024). Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427.

[35] Yang et al. (2024). Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems. 37. pp. 115491–115522.

[36] Dao, Tri and Gu, Albert (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060.

[37] Yang et al. (2023). Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635.

[38] Peng et al. (2024). Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892.

[39] Peng et al. (2025). Rwkv-7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456.

[40] Hu et al. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

[41] Grattafiori et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

[42] Spector et al. (2024). Thunderkittens: Simple, fast, and adorable ai kernels. arXiv preprint arXiv:2410.20399.

[43] Ho et al. (2022). Video diffusion models. Advances in neural information processing systems. 35. pp. 8633–8646.

[44] Zhang et al. (2025). Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507.

[45] Huang et al. (2024). Vbench: Comprehensive benchmark suite for video generative models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 21807–21818.

[46] Kong et al. (2024). Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603.

[47] Li et al. (2023). *Textbooks Are All You Need II: phi-1.5technical report*. arXiv preprint arXiv:2309.05463.

[48] Albert Q. Jiang et al. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825.

[49] Touvron et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

[50] Qwen et al. (2025). Qwen2.5 Technical Report.

[51] Qwen Team (2025). QwQ-32B: Embracing the Power of Reinforcement Learning. https://qwenlm.github.io/blog/qwq-32b/.

[52] Dan Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding. arXiv:2009.03300.

[53] Clark et al. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

[54] Bisk et al. (2020). Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence. pp. 7432–7439.

[55] Sakaguchi et al. (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM. 64(9). pp. 99–106.

[56] Rowan Zellers et al. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. https://arxiv.org/abs/1905.07830. arXiv:1905.07830.

[57] Rohan Taori et al. (2023). Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.

[58] Songlin Yang et al. (2025). Gated Delta Networks: Improving Mamba2 with Delta Rule. https://arxiv.org/abs/2412.06464. arXiv:2412.06464.

[59] Yang et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.

[60] Abdin et al. (2024). Phi-4 technical report. arXiv preprint arXiv:2412.08905.

[61] Xu et al. (2025). Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math. arXiv preprint arXiv:2504.21233.

[62] Abdin et al. (2025). Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318.