Prefix Sliding for efficient test-time scaling

Niklas Muennighoff $^{s}$ Zhengyang Wang $^{c}$ Zeyi Chen $^{w}$ Weijia Shi $^{w}$ Binyuan Hui
John Yang $^{s}$ Dapeng Jiang $^{w}$ Mika Senghaas $^{p}$ Fares Obeid $^{p}$ Johannes Hagemann $^{p}$
Sami Jaghouar $^{p}$ Ludwig Schmidt $^{s}$ Percy Liang $^{s}$ Jason Wei Andrew Y. Ng $^{s}$
Luke Zettlemoyer $^{w}$ Yejin Choi $^{s}$ Mike Lewis $^{w}$
$^{s}$ Stanford University $^{c}$ University of California at Santa Barbara $^{p}$ Prime Intellect
$^{w}$ University of Washington
[email protected]

Abstract

Test-time scaling uses extra test-time compute to improve performance, such as letting language models reason longer when solving a problem. As models keep the entire reasoning trace in memory via full attention, hard tasks that need long thinking can be prohibitively expensive. However, we find most intermediate reasoning tokens lose importance as the model continues reasoning. This calls into question whether retaining them is worth the cost. Based on this insight, we propose Prefix Sliding, which discards tokens during reasoning that are not part of the prefix or the window of the last few thousand tokens. The prefix has key instructions and tools available to the model, while the most recent tokens are the current reasoning the model is working on. This caps the total memory requirement regardless of how long the model reasons, allowing for efficient long-horizon test-time scaling. Without training, Prefix Sliding can make existing models 3x faster while maintaining performance. Training with Prefix Sliding using reinforcement learning can achieve better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens. Ablations show Prefix Sliding outperforms summarizing intermediate tokens or vanilla sliding window. Our code is at https://github.com/Muennighoff/prefix-sliding

Executive Summary: Test-time scaling improves language model performance on hard problems by allowing longer reasoning traces, yet full attention keeps every prior token in memory. This causes compute costs to grow linearly with length, plus problems such as distraction by irrelevant tokens, repetitive loops, and lost information. The work therefore asks whether most intermediate reasoning tokens can be discarded without harming results.

The authors propose Prefix Sliding, which retains only a fixed prefix of system instructions and task prompt plus a sliding window of the most recent few thousand tokens. The method requires no training to apply to existing models and can also be used inside reinforcement-learning rollouts. Experiments used the Qwen3-1.7B model on GPQA, MATH500, and AIME25, averaging 64 runs per setting, and compared against full attention as well as three bounded-cost alternatives: last-k eviction, summarization, and vanilla sliding window.

Without any training, Prefix Sliding matched full-attention accuracy while running roughly three times faster once the window was reached. Training with reinforcement learning under Prefix Sliding produced longer traces exceeding 100,000 tokens at comparable memory budgets and yielded higher final rewards. Ablations showed that Prefix Sliding delivered the best performance-efficiency trade-off; pure sliding window lost task information and flattened, while last-k and summarization incurred repeated token reprocessing and extra overhead. A custom FlashAttention kernel kept generation speed close to standard sliding-window attention.

These results indicate that constant per-token cost is achievable for sequential test-time scaling, removing the main barrier to very long reasoning. Models can therefore tackle harder problems without prohibitive latency or memory growth. The main limitations are occasional need for larger windows on tasks such as code generation, reduced speed-up on short generations, and open questions around multi-turn or agentic settings where external outputs may flood the window. Further work at larger model scales and with adaptive window or prefix-extension mechanisms would strengthen the approach before broad deployment.

1. Introduction

Section Summary: Test-time scaling lets language models tackle harder problems by devoting extra compute to longer reasoning, yet standard full attention makes this impractical because memory and computation costs grow linearly with every added token, leading to distraction, repetition, and lost information. The authors observe that intermediate reasoning steps quickly lose relevance once their results are used, while the initial prompt and most recent tokens stay important, so they introduce Prefix Sliding, which retains only the fixed prompt prefix plus a moving window of recent tokens. This keeps generation cost constant regardless of length, matches full-attention accuracy at three times the speed, and supports reasoning traces longer than 100,000 tokens without any retraining.

Test-time scaling improves the performance of language models by using extra compute for hard problems ([1]). Commonly, this compute is used by letting the model reason longer ([2, 3]). However, scaling this approach further is limited by the need to keep the entire reasoning trace in memory via full attention, as used in most language models ([4]). With full attention, the cost of each new token grows linearly with the number of already generated tokens, making long context windows prohibitively expensive. Long contexts have more issues, including distraction by old irrelevant tokens ([5]), context poisoning ([6]), repetitive loops ([7]), and lost knowledge ([8]).

We explore a simple solution based on two observations. First, intermediate reasoning tokens quickly lose importance. For example, when solving an expression like "((42 + 84) × 4) - 5", once the addition 42 + 84 is completed, the reasoning behind that step is no longer needed; only the result matters for the next operation. Second, the prefix and the most recent reasoning tokens, however, are of high importance during generation. The prefix, which includes the system instruction and prompt, contains key information about tools the model can use and the task to complete. It also serves as an "attention sink" allowing the model to allocate excess probability weight ([9]). Meanwhile, the most recent reasoning tokens capture what the model is currently working on. This has motivated prior work on letting models generate using a sliding window ([10, 11, 12, 13, 14, 15]).

We combine these two observations to propose Prefix Sliding. During reasoning, Prefix Sliding keeps only the prefix and a sliding window in memory. The prefix contains the model instructions. As the model generates, the sliding window advances, removing older intermediate tokens. For example, a 40-token system instruction and a 60-token task prompt could constitute a 100-token prefix. With a 4096-token sliding window, at most 4196 tokens are then kept in memory. The cost of generating an additional token is the same regardless of whether the model has already generated millions or billions of tokens. Such constant cost is necessary to enable very long-horizon test-time scaling.

Empirically, Prefix Sliding can match full-attention performance while running 3 $\times$ faster without training, and enables reinforcement learning rollouts beyond 100, 000 tokens.

2. Prefix Sliding

Section Summary: Prefix Sliding is a simple technique that keeps only the initial prompt tokens plus a moving window of the most recent ones while discarding the less important middle tokens during generation. This approach works immediately on existing models without retraining by reusing cached position embeddings, and it can be extended to training by using truncated backpropagation over short chunks to handle extremely long outputs without running out of memory. A custom attention kernel further improves speed by skipping unnecessary computations outside the retained prefix and window.

Motivation

In Figure 2, we depict the two observations from Section 1: (1) Intermediate tokens lack importance and (2) the prefix and recent tokens are key. For the prefix, much probability mass falls on the first four tokens as they function as attention sinks ([9]). The other tokens in the prompt also receive more attention than later intermediate reasoning tokens. The common <think> delimiter ([2]) marking the start of the reasoning trace also receives high attention, likely because of its ongoing important role in signaling to the model that it is in thinking mode. The attention probabilities then drop throughout the reasoning trace, but increase sharply toward the end, especially for the token preceding the one being generated.

{width=50%}

Prefix Sliding without training

Figure 3 shows how Prefix Sliding works by simply retaining the prefix of tokens and a sliding window. This makes it applicable to generative language models out of the box without any further training. If the model has been trained with position embeddings (PE), such as RoPE ([16]), then Figure 4 shows two options for handling them. Compared to Reset PE, Continue PE is more efficient as it does not require reapplying new position embeddings to the same token, but allows reusing cached representations with position embeddings already applied to them. While Continue PE may perform worse ([9]), we have found performance differences insignificant in Appendix D; thus, we use Continue PE. For an even simpler alternative, future work may combine Prefix Sliding with DroPE to simply remove the positional embeddings of pretrained models ([17]) or train models without position embeddings from scratch ([18]).

{width=40%}

Prefix Sliding with training

Training with Prefix Sliding enables very long RL rollouts, avoiding the common practice of truncating and discarding overlong generations ([19]). Training on completed generations can substantially improve the model. This is best accomplished in an asynchronous RL setup to avoid idle GPUs when other short generations in the same batch are already done ([20, 21]), but also works with synchronous RL. Naive backpropagation of generations that span hundreds of thousands of tokens can lead to out-of-memory errors in the trainer. Figure 5 depicts two solutions for this issue. Both rely on the limited receptive field of sliding windows. Sliding windows across multiple layers have a theoretical receptive field of $W\times L$, where $W$ is the window size and $L$ the number of layers. However, due to information bottlenecks, it is closer to 1.5 $\times W$ in practice ([22]). Thus, if we want to backpropagate a set of $W$ tokens, we may only need to pass around 1.5 $\times W$ of preceding tokens and the prefix to the trainer. Chunked backpropagation backpropagates on a reasoning chain in chunks and accumulates the gradients to ensure near-equivalence with standard full backpropagation. Truncated backpropagation involves only backpropagating on the last chunk. We use Prefix Sliding with truncated backpropagation for our training experiments, as we found its performance can match full attention with full backpropagation in Appendix E. For example, a model generated a reasoning trace of 100, 000 tokens with a sliding window size of 2048. Under truncated backpropagation, we may only send the last 8192 tokens from the sampler to the trainer for gradient computation. We then use the first 6144 tokens only as context and compute the token-level RL loss only on the final 2048 tokens. In our implementation, this is simply a loss mask: the loss for the preceding 6144 tokens is set to zero, and autograd backpropagates normally from the masked loss and only updates with respect to the last 2048 tokens. As the gradients of those 2048 tokens were computed using 4 $\times$ the sliding window size, they are very accurate relative to the full 100, 000-token generation. Like for Prefix Sliding without training, we also use Continue PE for Prefix Sliding with training. Resetting PE in the trainer is very complex due to teacher-forcing, which is key for training efficiency ([23, 24]). This is because when resetting PE, each token has seen a different combination of positions before it.

Prefix Sliding kernel implementation

We implement the Prefix Sliding attention kernel with two-level filtering:

• Intra-tile masking: For tiles that partially overlap the allowed attention region (prefix $\cup$ sliding window), we apply an elementwise mask so that only valid (q, k) pairs contribute to the softmax and output. This ensures mathematical correctness without changing the FlashAttention tiling strategy.
• Inter-tile skipping: We skip tiles that fall entirely outside the allowed region. Concretely, we restructure the producer–consumer pipeline to iterate over two disjoint block ranges (prefix blocks and window blocks). This avoids redundant loads and computations, substantially matching the efficiency of standard sliding window attention.

3. Setup

Section Summary: The setup relies on the Qwen3-1.7B model run through vLLM with custom FlashAttention kernels on Nvidia Hopper hardware to support Prefix Sliding at varying window sizes, while allowing the model itself to generate summaries when needed. Training uses the GRPO reinforcement learning method, either updating on full outputs or just the most recent sliding window of tokens, and draws on a custom dataset of math problems filtered for appropriate difficulty and solvability. Evaluation measures performance on GPQA, MATH500, and AIME25 by averaging 64 runs per problem, enforcing fixed thinking budgets, and tracking real-world speed through average seconds per sample rather than token counts.

Modeling

We use the Qwen3-1.7B model unless otherwise specified ([25]). We use vLLM ([26]) with FlashAttention ([27, 28]) for all generations. We write custom kernels for the Nvidia Hopper architecture to enable running Prefix Sliding with FlashAttention. We experiment with sliding window sizes 512, 1024, 2048, 4096, 8192, and 16384. For summary ablations, we treat the summary as a tool call and let the model itself write the summary instead of an external model.

Training

For reinforcement learning experiments, we use GRPO ([29]) via its synchronous implementation in trl ([30]), as well as its asynchronous implementation in prime-rl ([31]). We either backpropagate the entire generation or only the last sliding window of tokens, as explained in Section 2. In the latter case, we always pass four times as many of the last tokens to the trainer to ensure we compute accurate gradients for the sliding window. We do not tune other hyperparameters not directly related to Prefix Sliding (e.g., learning rate) and fix them across comparisons. We create our own dataset of math problems and filter it using guessability, verifiability, and difficulty as our three criteria. Details are in Appendix F.

Evaluation

We evaluate on standard reasoning benchmarks: GPQA ([32]), MATH500 ([33, 34]), AIME25 ([35]). We average results across 64 runs to increase confidence in our results, which we refer to as either accuracy or avg@64. We use a temperature of 0.6 and top p of 0.95 ([2]). We verify answers using a small verification library called simpleverify. We use budget forcing to keep generations to specific thinking budgets (without the use of "Wait" tokens) ([3]). We benchmark models by their average thinking time per sample, measured in seconds, as speed is what users ultimately experience, making it the most important efficiency metric. FLOPs or total generated tokens can be a good proxy, but they miss memory differences among methods.

4. Results

Section Summary: Prefix Sliding attention runs nearly as fast as standard sliding-window methods and much faster than full attention once sequences exceed the window size, since it avoids the steadily rising per-token cost of attending to every prior token. When used during reinforcement learning, it supports substantially longer reasoning traces under the same memory limits and yields higher final rewards than full attention. Ablation tests confirm that passing roughly four times the sliding-window size to the trainer keeps the mismatch in token probabilities low while remaining efficient, with only minor differences arising from separate attention kernels.

{width=50%}

Prefix Sliding without training

Figure 1 shows Prefix Sliding is more efficient even when applied to an existing model that has been trained with full attention. Figure 6 shows our FlashAttention kernel for Prefix Sliding reaches about the same speeds as a regular sliding window kernel. A slightly slower speed is expected due to the additional memory requirements of the prefix. The tokens per second for Prefix Sliding and regular sliding window drop initially before stabilizing around 5, 000. The initial drop is due to the generation spending less time in the cheap warm-up phase, where the generated tokens are still less than the sliding window. Once the sliding window size is reached, generating each new token costs the same amount. Meanwhile, full attention keeps getting slower indefinitely as the cost per token progressively increases because all prior tokens need to be in memory.

{width=50%}

{width=50%}

Prefix Sliding with training

Figure 7 shows that reinforcement learning with Prefix Sliding allows for much longer reasoning traces at near-equal memory budgets, thereby leading to higher rewards. Figure 8 ablates the choice of tokens passed to the trainer as explained in Section 2. The runs with Prefix Sliding backpropagate one final sliding window (2K tokens). As sliding windows have a limited receptive field (Section 2), only a limited number of tokens prior to this final sliding window are necessary to ensure accurate log probabilities. Only passing the sliding window itself leads to a high KL of above 0.1 as expected. Passing 2 $\times$ as much (4K) significantly lowers the mismatch. We go with 4 $\times$ (8K) for most runs as it appears to have a slightly lower KL mismatch and is around on par with 8 $\times$ (16K). Some remaining KL mismatch is expected due to tiny numerical differences between our custom Flash Attention kernel in the generator and our FlexAttention ([36]) implementation in the trainer. We find the multiplier of 4 $\times$ the window size also works well for larger windows. In Appendix E, we train a 7B model with RL using a window of 8, 192 and a multiplier of 4, and find performance comparable to full attention when controlling for sequence length.

5. Ablations

Section Summary: The section evaluates Prefix Sliding against three alternatives for managing very long AI reasoning contexts on math benchmarks. Last-k token retention and periodic summarization can preserve some information but require repeated reprocessing of tokens and introduce extra complexity plus volatile memory use, while a plain sliding window quickly forgets the original task. Overall, Prefix Sliding delivers the strongest performance-efficiency tradeoff with the fewest added hyperparameters.

{width=100%}

We compare Prefix Sliding with three key alternatives, as shown in Figure 9 and described below. In Appendix G, we provide details on their hyperparameter selection and contrast Prefix Sliding with an additional cache-eviction method.

Last k

• Explanation Text is generated until a threshold $n$ is reached, when all text except for the last k tokens is deleted. This way, the context never exceeds the length of the prompt + $n$. As long as $n$ is reasonably small, last k can use full attention without generation inevitably becoming too expensive to continue.
• Pros and Cons Last k can be very fast in terms of tokens per second. However, many of those tokens may be wasted. If k is large, the model has to reprocess a lot of tokens. This is because the last k tokens are processed twice: first upon generation and second when their context window changes due to the removal of prior tokens. If k is small, some of the more recent useful tokens may be dropped, and the model may need to regenerate parts of them (e.g., Figure 21). Last k also incurs volatile memory usage; memory usage drops drastically whenever the context is cleared, making it harder to use compute resources optimally.
• Examples Variants of last k are often used in agents: For example, after $n$ turns, earlier turns can be deleted, leaving only the most recent few turns (e.g., [37]). This approach has also been proposed as "Markovian Thinking / Delethink" in [38].

Summary

• Explanation Text is generated until a threshold $n$ is reached, when all text is summarized, either by the model itself or an external summarizer. Together with the prompt, this summary is then used to start a new context window to continue reasoning. The procedure repeats when $n$ is reached again.
• Pros and Cons A benefit of this approach is the model can draw information from anywhere in the current context window for its summary, which in theory could allow it to retain important insights over many summary steps. In practice, however, models struggle to retain important information over many turns ([39]). Summary adds complexity by introducing new hyperparameters, such as $n$, the summary length, the summarizer prompt, the summarizer model, and the placement of the summary in the new context. Further, the extra summary-generation step adds overhead, especially if the summary model is large. Like last k, the summary must be processed twice, first upon generation and second when used in the new context window. Its memory usage is also volatile like last k.
• Examples This approach has been explored via prompting ([40]), supervised finetuning ([41, 42]), or reinforcement learning ([43, 44, 45]). [46] use a hierarchical, rather than sequential, version of this approach to handle long inputs. It is also referred to as compaction and used by models like Opus 4.6 ([47]), GPT 5.4 ([48]), and Composer ([49]).

Sliding window

• Explanation This is a baseline equivalent to Prefix Sliding without prefix.
• Pros and Cons The method is very simple. However, as the prefix contains key information about the task, this method performs poorly on longer reasoning tasks. The model forgets which problem it is solving or which tools it can use.
• Examples Sliding window attention ([12]) is common in many large language models, such as gpt-neo ([50]) and gpt-oss ([51]). To compensate for the lack of information about the prefix, they interleave it with full attention layers that process the entire context.

Results

Figure 9 shows Prefix Sliding provides the best performance efficiency trade-off. Prefix Sliding also adds only one hyperparameter: the size of the sliding window. Pure sliding window attention quickly flattens out due to a lack of information about the task at long thinking times. As soon as the model reaches the sliding window size, it starts losing tokens at the beginning that contain critical information. Last k and summary approaches can reach good performance but are fundamentally constrained by their required token reprocessing and extra summary step. They also add other complexity overhead by requiring more hyperparameters and generation restarts.

6. Related Work

Section Summary: This section reviews methods for scaling computation during model inference, separating parallel approaches—which suffer from diminishing returns—from sequential ones that can extend further but are hindered by rising costs in standard transformers. It emphasizes the importance of techniques with bounded per-token costs to support very long reasoning without exploding expenses, contrasting approaches like recurrent networks or state-space models that require retraining with simpler windowing or summarization tricks that work on existing models but introduce irregular overhead. The discussion positions Prefix Sliding as a practical bounded alternative that preserves key context through a fixed prefix while integrating seamlessly with pretrained systems.

Test-time scaling

Current methods to scale compute at test-time are either sequential or parallel ([52, 3]). Parallel methods allow for infinite test-time scaling by design, e.g., majority voting ([53]) simply requires launching more parallel processes to try to solve the same question. However, they face stark diminishing returns ([54, 55, 56]). Sequential scaling can scale better than parallel ([3]). While there has been much work on improving sequential scaling and reasoning models in general ([57, 58, 59, 60, 61, 62]), long-horizon scaling remains a fundamental limitation due to the quadratic complexity of the transformer ([63]). We build a simple method that significantly improves reasoning efficiency while enabling long-horizon scaling due to its constant cost, as elaborated in the next paragraph.

{width=60%}

Context extension

We distinguish between methods that are bounded and unbounded in their cost per new token. Bounded methods are asymptotically constant; in the limit, they cost at most a certain amount per new token ([64, 65, 66]). Unbounded methods cost more for each new token in the limit, even if they may exhibit subquadratic complexity ([67, 68, 69, 70, 71, 72, 73, 74, 75, 76]). Full attention is unbounded: Every generated token gets more expensive. One can trade off space and time complexity of full attention to make one bounded, but the other stays unbounded ([77]). Crucially, to enable infinite test-time scaling, i.e., models that reason for weeks, cost must be bounded per new token. Figure 10 shows Prefix Sliding is bounded once the sequence length reaches the combined size of the prefix and the sliding window. Other bounded methods include RNNs (e.g. RWKV ([78, 79])), SSMs (e.g. Mamba ([80, 81, 82, 83])), and transformer-based approaches ([84, 85, 86, 87]). However, they do not work out of the box with existing models but require training models to adapt them. Prefix Sliding works with existing pretrained models without further training and can optionally also be used for training. Last k and summary approaches are also bounded and work out of the box, but exhibit irregular cost as shown in Figure 10 due to deleting and refilling of the context window. This makes full GPU utilization difficult. They also incur fundamental latency overhead due to duplicate token processing, which Prefix Sliding bypasses, as detailed in Section 5. One way to view Prefix Sliding is as sliding window attention with global tokens ([12]), but with many consecutive global tokens forming a prefix that preserves task instructions and other necessary context for reasoning tasks. Another related approach is StreamingLLM ([9]), which retains only a few fixed initial tokens, e.g., 4.

7. Conclusion

Section Summary: The authors introduce Prefix Sliding as a technique that lets language models carry out reasoning over extremely long sequences of text. The method works more efficiently than standard full attention even at moderate lengths, requires no extra training to apply to existing models, and can also support training with reinforcement learning while outperforming other approaches that aim for unlimited scaling. The goal is to encourage future efforts to solve harder problems by letting models think for much longer.

We propose Prefix Sliding to enable language models to reason for extremely long horizons. Even at short reasoning horizons of only thousands of tokens, Prefix Sliding is more efficient than the status quo of using full attention. Prefix Sliding is applicable to language models without further training. It can also be used during training with reinforcement learning. It outperforms alternatives that could also support infinite test-time scaling. We hope that enabling language models to think longer via Prefix Sliding inspires future work on solving ever harder problems with language models.

Limitations

Section Summary: The section highlights several constraints of the Prefix Sliding approach. Comparisons were limited to methods compatible with existing pretrained models that maintain bounded per-token costs, excluding many alternative architectures or techniques. Additional issues include potential loss of critical early information outside the sliding window during long reasoning, reduced speedups for short outputs that rarely trigger eviction, challenges in handling flooding context from system outputs or multi-turn interactions, and the need for further scaling beyond the tested model sizes and token lengths.

Limited comparisons

We restrict ourselves to empirical comparisons with alternatives that fulfill two properties: (1) they work on existing pretrained transformers out of the box and (2) they lead to a bounded cost per new token (see Section 6). This excludes many approaches, such as alternative architectures, subquadratic methods, or mixed sliding window models ([88, 89, 90, 91, 92]). Future work may consider relaxing the first criterion by comparing with alternative architectures that still exhibit a bounded cost per new token, such as RNNs. We consider it beyond the scope of this work, as it likely requires pretraining models from scratch to control for computational resources and other hyperparameters.

{width=50%}

Information loss

While intermediate tokens can lack importance for later reasoning, as we show in Section 2, sometimes this is not the case. Figure 11 shows this limitation on the example of LiveCodeBench, where a larger window size is necessary to match full attention. Inspecting samples reveals that the issue is likely that for LiveCodeBench, the model starts a function implementation during reasoning and then thinks using comments for potentially thousands of tokens (see Figure 22 for an example). By the time it continues coding, the beginning of the code may have moved outside its sliding window. This evaluation is without any training. Training with Prefix Sliding during reinforcement learning would likely teach the model to simply adapt its commenting behavior, thus enabling a shorter window size. Alternatively, a mechanism for the model to append sliding tokens to the prefix, or another knowledge store, may avoid the need for larger window sizes.

{width=50%}

Limited benefit for short generations

As is clear from Figure 6, the benefits of Prefix Sliding are larger the longer the generation of the model. For short generations, a larger proportion of the generation still uses full attention while the sliding window size has not yet been reached. We call this the sliding window warm-up phase. Only after this phase is complete does the window slide and evict old tokens, thereby offering major speed-ups over full attention. In Figure 12, we benchmark Prefix Sliding on a task that requires only 2086 tokens on average: HealthBench ([93]). As we use a sliding window of 2048 for Prefix Sliding, the model slides very rarely. It is equivalent to full attention for the many samples that require fewer than 2048 tokens. Thus, there is little room for any speed-up.

System outputs and multi-turn

In agentic tasks, a model may read the contents of a website or read files, which could flood the entire context window. This could be problematic because if the sliding window is smaller than the content the model is trying to read, then it strictly cannot read the entire content. Even worse, it may lose important content, as its sliding window is flooded with this new output. A second related issue is what to do with future user instructions in a multi-turn setup. Append them to the prefix? Let the sliding window remove them eventually? These two problems also exist with summarization or last k techniques, assuming the same window size. Extensive reinforcement learning would likely teach the model to be extra careful to avoid this behavior. Another approach could be to let the model learn to read content step by step rather than in one go (e.g., using "head" rather than "cat" commands in Unix). One can also add automatic guardrails that prevent excessive outputs in the model's context by quickly checking such outputs before and not providing them to the model beyond a prespecified threshold.

Scale

In this work, we scale up Prefix Sliding to hundreds of thousands of thinking tokens and 7 billion parameter models across training-free and reinforcement learning training setups. Future work is necessary to scale up Prefix Sliding further and study its trends.

Reproducibility Statement

Section Summary: The authors note that their core findings should be straightforward for others to recreate, since the Prefix Sliding method itself is simple and explained thoroughly in the paper. They have also released the full code in a public online repository to make verification even easier. Together these steps lower the barriers for independent checks of the work.

As Prefix Sliding is very simple and Section 2 describes it in detail, it is likely easy to reproduce our key results using only the paper. We also make our code public at https://github.com/Muennighoff/prefix-sliding.

Author Contributions

Section Summary: Niklas Muennighoff led the overall project and handled most of the core work, including running the training and evaluations, creating datasets, and writing the paper, while also contributing to technical tasks like custom software kernels and testing variations of the approach. A smaller group of collaborators assisted with specific implementation details such as kernels, experiments, evaluations, and data preparation. A larger set of researchers provided guidance and advice throughout the project.

Niklas Muennighoff ran training, evaluation, wrote the paper, led the project. Zhengyang Wang, Niklas Muennighoff worked on kernels. Zeyi Chen, Niklas Muennighoff, Dapeng Jiang implemented ablations. Niklas Muennighoff, John Yang, Weijia Shi implemented evaluation. Niklas Muennighoff, Binyuan Hui, John Yang made datasets. Mike Lewis, Yejin Choi, Luke Zettlemoyer, Weijia Shi, Andrew Y. Ng, Jason Wei, Percy Liang, Ludwig Schmidt, Sami Jaghouar, Johannes Hagemann, Fares Obeid, Mika Senghaas advised the project.

Acknowledgments

Section Summary: The authors thank the Laude Institute and the NVIDIA Academic Grant Program for supporting their research. One researcher also received a graduate fellowship from the Knight-Hennessy Scholars at Stanford University. Additional funding came from a Korean government AI research program and a gift from DSO National Laboratories.

We are extremely thankful to Laude Institute for supporting this work. Research supported by the NVIDIA Academic Grant Program. NM is supported by a graduate fellowship award from Knight-Hennessy Scholars at Stanford University. This work was supported by IITP funded by the Korean Government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project). This research was supported in part by a gift from DSO National Laboratories.

Appendix

Section Summary: The appendix supplies supplementary material that extends the paper's discussion of related work on KV cache optimization and efficient reasoning techniques, while clarifying how Prefix Sliding fits into both training and inference settings. It presents additional experimental results through figures and tables that compare sliding window sizes, positional embedding variants, and backpropagation approaches, along with details on the reinforcement learning dataset assembled from public sources and filtered by guessability, verifiability, and difficulty. The section also reports hyperparameter ablations for alternative methods such as retaining the last k tokens during context resets.

A. Extended Related Work

Key-Value (KV) Cache

The KV cache in transformers stores information from the past context, which eventually grows prohibitively expensive as generation continues. Thus, many context extensions focus specifically on handling the KV cache footprint. Methods either target the KV-cache from pre-fill, post-fill, or both ([94]). Pre-fill methods seek to reduce the cost of the KV-cache from the prompt ([95, 96]), while post-fill methods deal with the KV cache after processing the prompt ([97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110]). Many post-fill methods use recency eviction methods to discard older parts of the KV cache ([9, 111, 112, 113, 114]). Other approaches across pre-fill and post-fill optimize the KV cache and memory usage by taking hardware into consideration ([115, 116, 117, 118]) or using quantization techniques ([119, 120, 121]). Importantly, Prefix Sliding does not reduce the pre-fill cost of the KV cache, which may lead to high memory usage with extremely long prefixes. One solution could be to discard information from the prompt that is not needed for the prefix, or to introduce context management techniques ([122]). Long inputs, such as a relevant book to solve the problem, are likely better suited in a file whose path is provided to the model so it can read it step by step.

Efficient thinking

Several works have explored improving the thinking efficiency of reasoning language models. They seek to do so at train-time, test-time, or both. Train-time methods often target the RL algorithm or infrastructure-related issues ([123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133]), while test-time methods seek to work with existing models out-of-the-box that have already been trained ([134, 135, 136, 137, 138, 3]). Prefix Sliding is both a train-time and test-time method: It can be used during RL and can be applied to already-trained models to achieve better efficiency and performance.

B. Other sliding window sizes

C. Tabular results

\begin{tabular}{l|rrrrrr|rr}
\toprule
Window & \multicolumn{2}{c}{\textbf{AIME25}} & \multicolumn{2}{c}{\textbf{GPQA}} & \multicolumn{2}{c|}{\textbf{MATH500}} & \multicolumn{2}{c}{\textbf{Tok/s at}} \\
size & avg@64 & avglen & avg@64 & avglen & avg@64 & avglen & 32K & 128K \\
\midrule
2048 & 27.7 & 47643 & 35.9 & 30107 & 89.8 & 9310 & 8973 & 8737 \\
4096 & 33.9 & 29943 & 37.0 & 16707 & 91.5 & 7069 & 5479 & 5224 \\
8192 & 35.8 & 19373 & 38.0 & 13605 & 91.4 & 6229 & 3291 & 2788 \\
16384 & 35.3 & 19872 & 38.2 & 14378 & 91.5 & 6160 & 2441 & 1420 \\
Full & 34.2 & 19158 & 37.6 & 11403 & 91.7 & 6056 & 1477 & 448 \\
\bottomrule
\end{tabular}

D. Continue vs Reset PE

E. Truncated backpropagation validation

Figure 15 depicts a short experiment with DeepSeek-R1-Distill-Qwen-7B ([2]) using the prime-rl codebase for asynchronous reinforcement learning ([31]). For both models, 32768 tokens are passed to the trainer, but for Prefix Sliding only 8, 192 of them are backpropagated using truncated backpropagation (Section 2) with a multiplier of 4. The window size for Prefix Sliding is 8, 192. We find that performance is comparable, but highlight that more experiments at even larger scales are necessary in the future. Therefore, we stick with truncated backpropagation for our experiments (e.g., Figure 7), but larger-scale runs with significantly longer chains may require chunked backpropagation (Section 2).

F. Training Dataset

For our dataset for reinforcement learning training we combine public sources, specifically SkyWork ([139]) and s1 ([3]) (s1 further sources from NuminaMATH ([140]), MATH ([33]), OlympicArena ([141]), OmniMath ([142]), AGIEval ([143, 144, 33, 145, 146, 147]), OlympiadBench ([148]), TheoremQA ([149]), JEEBench ([150]), GPQA ([32]), SciEval ([151])). We also decontaminate against test data using the s1 setup.

To filter problems, we rely on three criteria: guessability, verifiability, and difficulty. For guessability, we remove samples where small models write the correct solution on any of 8 tries without thinking ([152]). For verifiability, we remove any samples that contain a set of words such as "How", "Explain", as such questions may have answers that cannot be easily objectively verified. For difficulty, we score models on all samples multiple times. We then remove those always solved by weak models, as well as those never solved by strong models, as they may be impossible to solve.

G. Other methods

G.1 Last k hyperparameters

: Table 2: Ablating last $k$. Results are avg@64 with a 2048-token context and one pass. We select $k=256$, which performs best while keeping the carried context small.

We select $k=256$ for our runs in Figure 9 based on a sweep in Table 2. This means that whenever the model runs out of context, exactly 256 tokens from the end of the generation are taken and prepended to the thinking in the next generation of the model. We do not consider proper sentence endings; thus, the tokens are likely to be cut off; see Figure 21 for an example. To keep the total thinking tokens the same after the model has received the last k tokens of the first turn, it generates $k$ fewer tokens from the second turn onward. The optimal value of last k likely depends on the context window and the number of allowed passes, so the setup in Figure 9 may have a different optimum. However, in practice one cannot predict the number of passes ahead of time, but generally needs to use the same last k value across setups.

G.2 Summary hyperparameters

We set a maximum summary length of $k=256$ based on Appendix G.1. We treat the summary as a tool call and use summary forcing: If $k$ tokens are left in the context window, and the summary tool has not yet been invoked, we force-insert the tool call into the reasoning of the model so it generates a summary. We use the model itself as the summary model. We ablate prompts in Table 3. Adding an example of how to use the tool and context (prompt 2) raises performance; possibly it leads to better summaries and their usage. Explaining the context further (prompt 3) did not help. Figure 20 shows an issue with the summary approach where the model seemingly ignores its summary, or maybe it uses it without mentioning it in its thinking ([153, 154]). Thus, we stick with prompt 2 for Figure 9. For summary, we do not subtract the summary length from the tokens the model may generate in its second turn onward; thus, it keeps slightly more tokens in memory than Prefix Sliding or last k. This may give the summary approach a slight advantage in Figure 9. We also tried inserting the summary in the model's thinking, which led to worse performance; possibly it was very out-of-distribution for the model.

\begin{tabular}{l|rr}
\toprule
\textbf{Prompt} & \multicolumn{2}{c}{\textbf{AIME25}} \\
 {} & \textbf{Accuracy} & \textbf{Coverage} \\
\midrule
  1: Tool only (Figure 16, Figure 17) & 23.2 & 33.3 \\
  2: Tool/context examples (Figure 18, Figure 17) & 26.4 & 53.3 \\
  3: Tool/context examples with info (Figure 18, Figure 19) & 25.8 & 46.7 \\
\bottomrule
\end{tabular}

G.3 H2O

$H_2O$ ([97]) retains recent tokens and "heavy hitters" that accumulate high attention scores, while evicting less influential tokens from the KV cache. Thus, prefix tokens that have not received sufficiently high attention can be discarded once they leave the recency window. $H_2O$ is not integrated with FlashAttention/vLLM (https://github.com/vllm-project/vllm/issues/3532), making a fair efficiency comparison difficult.

However, we think combining $H_2O$ with Prefix Sliding could be promising. Both retain recent tokens, but they differ in that Prefix Sliding is "forward-looking" while $H_2O$ is "backward-looking". Prefix Sliding preserves prefix tokens, which we know may become important later. For example, a tool definition may get little attention for thousands of tokens but must be available when the model eventually uses that tool. In contrast, $H_2O$ retains tokens based on their importance thus far. Preserving the prefix, heavy-hitting intermediate tokens, and a recency window may outperform Prefix Sliding alone. However, it may be challenging to make such a method work for reinforcement learning.

H. Analyzing outputs

References

Section Summary: This section is a bibliography of sources referenced in the document, consisting mainly of recent research papers, technical reports, and GitHub repositories on artificial intelligence. The listed works cover advancements in large language models, techniques for improving reasoning and handling long inputs, reinforcement learning methods, and related benchmarks. Most entries are arXiv preprints from 2020 to 2025, along with a few earlier foundational papers and implementation resources.

[1] OpenAI (2024). Learning to Reason with LLMs. https://openai.com/index/learning-to-reason-with-llms/.

[2] DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948. arXiv:2501.12948.

[3] Niklas Muennighoff et al. (2025). s1: Simple test-time scaling. https://arxiv.org/abs/2501.19393. arXiv:2501.19393.

[4] Ranajoy Sadhukhan et al. (2025). Kinetics: Rethinking Test-Time Scaling Laws. https://arxiv.org/abs/2506.05333. arXiv:2506.05333.

[5] Aryo Pradipta Gema et al. (2025). Inverse Scaling in Test-Time Compute. https://arxiv.org/abs/2507.14417. arXiv:2507.14417.

[6] Comanici et al. (2025). Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.

[7] Charilaos Pipis et al. (2025). Wait, Wait, Wait... Why Do Reasoning Models Loop?. https://arxiv.org/abs/2512.12895. arXiv:2512.12895.

[8] Nelson F. Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. https://arxiv.org/abs/2307.03172. arXiv:2307.03172.

[9] Guangxuan Xiao et al. (2024). Efficient Streaming Language Models with Attention Sinks. https://arxiv.org/abs/2309.17453. arXiv:2309.17453.

[10] Joshua Ainslie et al. (2020). ETC: Encoding Long and Structured Inputs in Transformers. https://arxiv.org/abs/2004.08483. arXiv:2004.08483.

[11] Ankit Gupta and Jonathan Berant (2020). GMAT: Global Memory Augmentation for Transformers. https://arxiv.org/abs/2006.03274. arXiv:2006.03274.

[12] Iz Beltagy et al. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150.

[13] Manzil Zaheer et al. (2021). Big Bird: Transformers for Longer Sequences. https://arxiv.org/abs/2007.14062. arXiv:2007.14062.

[14] Xuan Zhang et al. (2025). LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation. https://arxiv.org/abs/2410.13846. arXiv:2410.13846.

[15] Zichuan Fu et al. (2025). Sliding Window Attention Training for Efficient Large Language Models. https://arxiv.org/abs/2502.18845. arXiv:2502.18845.

[16] Jianlin Su et al. (2023). RoFormer: Enhanced Transformer with Rotary Position Embedding. https://arxiv.org/abs/2104.09864. arXiv:2104.09864.

[17] Yoav Gelberg et al. (2025). Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings. https://arxiv.org/abs/2512.12167. arXiv:2512.12167.

[18] Amirhossein Kazemnejad et al. (2023). The Impact of Positional Encoding on Length Generalization in Transformers. https://arxiv.org/abs/2305.19466. arXiv:2305.19466.

[19] Qiying Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476. arXiv:2503.14476.

[20] Wei Fu et al. (2025). AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. https://arxiv.org/abs/2505.24298. arXiv:2505.24298.

[21] Michael Noukhovitch et al. (2025). Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. https://arxiv.org/abs/2410.18252. arXiv:2410.18252.

[22] Guangxuan Xiao (2025). Why Stacking Sliding Windows Can't See Very Far. https://guangxuanx.com/blog/stacking-swa.html.

[23] Alex Lamb et al. (2016). Professor Forcing: A New Algorithm for Training Recurrent Networks. https://arxiv.org/abs/1610.09038. arXiv:1610.09038.

[24] Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. https://arxiv.org/abs/2005.14165. arXiv:2005.14165.

[25] An Yang et al. (2025). Qwen3 Technical Report. https://arxiv.org/abs/2505.09388. arXiv:2505.09388.

[26] Woosuk Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

[27] Tri Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135.

[28] Tri Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.08691.

[29] Zhihong Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300. arXiv:2402.03300.

[30] Leandro von Werra et al. (2020). TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl.

[31] Prime Intellect (2025). PRIME-RL. https://github.com/PrimeIntellect-ai/prime-rl.

[32] David Rein et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/abs/2311.12022. arXiv:2311.12022.

[33] Dan Hendrycks et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/abs/2103.03874. arXiv:2103.03874.

[34] Hunter Lightman et al. (2023). Let's Verify Step by Step. https://arxiv.org/abs/2305.20050. arXiv:2305.20050.

[35] Mathematical Association of America (2025). AIME. https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions/.

[36] Juechu Dong et al. (2024). Flex Attention: A Programming Model for Generating Optimized Attention Kernels. https://arxiv.org/abs/2412.05496. arXiv:2412.05496.

[37] Xingyao Wang et al. (2025). OpenHands: An Open Platform for AI Software Developers as Generalist Agents. https://arxiv.org/abs/2407.16741. arXiv:2407.16741.

[38] Milad Aghajohari et al. (2025). The Markovian Thinker. https://arxiv.org/abs/2510.06557. arXiv:2510.06557.

[39] Zhiqi Wang et al. (2026). Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction. https://arxiv.org/abs/2608.11242. arXiv:2608.11242.

[40] Vajipey et al. (2025). Simple, Scalable Reasoning via Iterated Summarization. https://openreview.net/pdf?id=uhZLKclfGB.

[41] Yuchen Yan et al. (2025). InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models. https://arxiv.org/abs/2503.06692. arXiv:2503.06692.

[42] Vasilis Kontonis et al. (2026). MEMENTO: Teaching LLMs to Manage Their Own Context. https://arxiv.org/abs/2604.09852. arXiv:2604.09852.

[43] Xixi Wu et al. (2025). ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization. https://arxiv.org/abs/2509.13313. arXiv:2509.13313.

[44] Yuchen Yan et al. (2026). InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning. https://arxiv.org/abs/2602.06960. arXiv:2602.06960.

[45] Tianjian Li et al. (2026). Self-Compacting Language Model Agents. https://arxiv.org/abs/2606.23525. arXiv:2606.23525.

[46] Jeff Wu et al. (2021). Recursively Summarizing Books with Human Feedback. https://arxiv.org/abs/2109.10862. arXiv:2109.10862.

[47] Anthropic (2026). Introducing Claude Opus 4.6. https://www.anthropic.com/news/claude-opus-4-6.

[48] OpenAI (2025). OpenAI GPT-5 System Card. https://arxiv.org/abs/2601.03267. arXiv:2601.03267.

[49] Cursor Research et al. (2026). Composer 2 Technical Report. https://arxiv.org/abs/2603.24477. arXiv:2603.24477.

[50] Black et al. (2021). GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. https://doi.org/10.5281/zenodo.5297715.

[51] OpenAI (2025). gpt-oss-120b & gpt-oss-20b Model Card. https://arxiv.org/abs/2508.10925. arXiv:2508.10925.

[52] Charlie Snell et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. https://arxiv.org/abs/2408.03314. arXiv:2408.03314.

[53] Xuezhi Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. https://arxiv.org/abs/2203.11171. arXiv:2203.11171.

[54] Bradley Brown et al. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. https://arxiv.org/abs/2407.21787. arXiv:2407.21787.

[55] Ryan Ehrlich et al. (2025). CodeMonkeys: Scaling Test-Time Compute for Software Engineering. https://arxiv.org/abs/2501.14723. arXiv:2501.14723.

[56] Rylan Schaeffer et al. (2025). How Do Large Language Monkeys Get Their Power (Laws)?. https://arxiv.org/abs/2502.17578. arXiv:2502.17578.

[57] Qiyuan Zhang et al. (2025). A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?. https://arxiv.org/abs/2503.24235. arXiv:2503.24235.

[58] Pranjal Aggarwal and Sean Welleck (2025). L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. https://arxiv.org/abs/2503.04697. arXiv:2503.04697.

[59] Yang Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. https://arxiv.org/abs/2504.13837. arXiv:2504.13837.

[60] Zheng-Xin Yong et al. (2025). Crosslingual Reasoning through Test-Time Scaling. https://arxiv.org/abs/2505.05408. arXiv:2505.05408.

[61] Ximing Lu et al. (2025). Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning. https://arxiv.org/abs/2504.04383. arXiv:2504.04383.

[62] Etash Guha et al. (2025). OpenThoughts: Data Recipes for Reasoning Models. https://arxiv.org/abs/2506.04178. arXiv:2506.04178.

[63] Ashish Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762.

[64] Hao Peng et al. (2022). ABC: Attention with Bounded-memory Control. https://arxiv.org/abs/2110.02488. arXiv:2110.02488.

[65] Tsendsuren Munkhdalai et al. (2024). Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. https://arxiv.org/abs/2404.07143. arXiv:2404.07143.

[66] Songlin Yang et al. (2024). Gated Linear Attention Transformers with Hardware-Efficient Training. https://arxiv.org/abs/2312.06635. arXiv:2312.06635.

[67] Rewon Child et al. (2019). Generating Long Sequences with Sparse Transformers. https://arxiv.org/abs/1904.10509. arXiv:1904.10509.

[68] Nikita Kitaev et al. (2020). Reformer: The Efficient Transformer. https://arxiv.org/abs/2001.04451. arXiv:2001.04451.

[69] Sinong Wang et al. (2020). Linformer: Self-Attention with Linear Complexity. https://arxiv.org/abs/2006.04768. arXiv:2006.04768.

[70] Andrew Jaegle et al. (2021). Perceiver: General Perception with Iterative Attention. arXiv:2103.03206.

[71] Yunyang Xiong et al. (2021). Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention. https://arxiv.org/abs/2102.03902. arXiv:2102.03902.

[72] Krzysztof Choromanski et al. (2022). Rethinking Attention with Performers. https://arxiv.org/abs/2009.14794. arXiv:2009.14794.

[73] Hao Liu et al. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context. https://arxiv.org/abs/2310.01889. arXiv:2310.01889.

[74] Namgyu Ho et al. (2024). Block Transformer: Global-to-Local Language Modeling for Fast Inference. https://arxiv.org/abs/2406.02657. arXiv:2406.02657.

[75] DeepSeek-AI et al. (2025). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. https://arxiv.org/abs/2512.02556. arXiv:2512.02556.

[76] Vasudev Shyam et al. (2025). Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters. https://arxiv.org/abs/2408.04093. arXiv:2408.04093.

[77] Markus N. Rabe and Charles Staats (2022). Self-attention Does Not Need $O(n^2)$ Memory. https://arxiv.org/abs/2112.05682. arXiv:2112.05682.

[78] Bo Peng et al. (2023). RWKV: Reinventing RNNs for the Transformer Era. arXiv:2305.13048.

[79] Bo Peng et al. (2024). Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence. arXiv:2404.05892.

[80] Albert Gu and Tri Dao (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. https://arxiv.org/abs/2312.00752. arXiv:2312.00752.

[81] Tri Dao and Albert Gu (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. https://arxiv.org/abs/2405.21060. arXiv:2405.21060.

[82] Albert Gu et al. (2022). Efficiently Modeling Long Sequences with Structured State Spaces. https://arxiv.org/abs/2111.00396. arXiv:2111.00396.

[83] Junxiong Wang et al. (2024). MambaByte: Token-free Selective State Space Model. https://arxiv.org/abs/2401.13660. arXiv:2401.13660.

[84] Zihang Dai et al. (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. https://arxiv.org/abs/1901.02860. arXiv:1901.02860.

[85] Jack W. Rae et al. (2019). Compressive Transformers for Long-Range Sequence Modelling. https://arxiv.org/abs/1911.05507. arXiv:1911.05507.

[86] Alexis Chevalier et al. (2023). Adapting Language Models to Compress Contexts. https://arxiv.org/abs/2305.14788. arXiv:2305.14788.

[87] Arnuv Tandon et al. (2025). End-to-End Test-Time Training for Long Context. https://arxiv.org/abs/2512.23675. arXiv:2512.23675.

[88] Tay et al. (2020). Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006.

[89] Aydar Bulatov et al. (2022). Recurrent Memory Transformer. https://arxiv.org/abs/2207.06881. arXiv:2207.06881.

[90] Dongseong Hwang et al. (2024). TransformerFAM: Feedback attention is working memory. https://arxiv.org/abs/2404.09173. arXiv:2404.09173.

[91] Zifan He et al. (2025). HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing. https://arxiv.org/abs/2405.06067. arXiv:2405.06067.

[92] Wenhao Li et al. (2025). CCF: A Context Compression Framework for Efficient Long-Sequence Language Modeling. https://arxiv.org/abs/2509.09199. arXiv:2509.09199.

[93] Rahul K. Arora et al. (2025). HealthBench: Evaluating Large Language Models Towards Improved Human Health. https://arxiv.org/abs/2505.08775. arXiv:2505.08775.

[94] Adithya Bhaskar et al. (2025). Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?. https://arxiv.org/abs/2506.17121. arXiv:2506.17121.

[95] Huiqiang Jiang et al. (2024). MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention. https://arxiv.org/abs/2407.02490. arXiv:2407.02490.

[96] Sabri Eyuboglu et al. (2025). Cartridges: Lightweight and general-purpose long context representations via self-study. https://arxiv.org/abs/2506.06266. arXiv:2506.06266.

[97] Zhenyu Zhang et al. (2023). H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. https://arxiv.org/abs/2306.14048. arXiv:2306.14048.

[98] Zichang Liu et al. (2023). Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. https://arxiv.org/abs/2305.17118. arXiv:2305.17118.

[99] Yuhong Li et al. (2024). SnapKV: LLM Knows What You are Looking for Before Generation. https://arxiv.org/abs/2404.14469. arXiv:2404.14469.

[100] Suyu Ge et al. (2024). Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. https://arxiv.org/abs/2310.01801. arXiv:2310.01801.

[101] Yilong Chen et al. (2024). NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time. https://arxiv.org/abs/2408.03675. arXiv:2408.03675.

[102] Guangtao Wang et al. (2025). LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference. https://arxiv.org/abs/2503.08879. arXiv:2503.08879.

[103] Zefan Cai et al. (2025). PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. https://arxiv.org/abs/2406.02069. arXiv:2406.02069.

[104] Junhao Hu et al. (2025). RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning. https://arxiv.org/abs/2502.11147. arXiv:2502.11147.

[105] Zefan Cai et al. (2026). R-KV: Redundancy-aware KV Cache Compression for Reasoning Models. https://arxiv.org/abs/2505.24133. arXiv:2505.24133.

[106] Michael R. Metel et al. (2026). Thinking Long, but Short: Stable Sequential Test-Time Scaling for Large Reasoning Models. https://arxiv.org/abs/2601.09855. arXiv:2601.09855.

[107] Akshat Ramachandran et al. (2026). ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models. https://arxiv.org/abs/2510.01290. arXiv:2510.01290.

[108] Zihan Wang et al. (2026). Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle. https://arxiv.org/abs/2601.16986. arXiv:2601.16986.

[109] Ting-Yun Chang et al. (2026). Value-Aware Stochastic KV Cache Eviction for Reasoning Models. https://arxiv.org/abs/2606.03928. arXiv:2606.03928.

[110] Adam Zweiger et al. (2026). Fast KV Compaction via Attention Matching. https://arxiv.org/abs/2602.16284. arXiv:2602.16284.

[111] Guangxuan Xiao et al. (2024). DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. https://arxiv.org/abs/2410.10819. arXiv:2410.10819.

[112] Tianyu Fu et al. (2025). Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths. https://arxiv.org/abs/2406.14909. arXiv:2406.14909.

[113] Shen Han et al. (2026). KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression. https://arxiv.org/abs/2607.01237. arXiv:2607.01237.

[114] Yijiong Yu et al. (2026). SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing. https://arxiv.org/abs/2512.10411. arXiv:2512.10411.

[115] Jiaming Tang et al. (2024). Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference. https://arxiv.org/abs/2406.10774. arXiv:2406.10774.

[116] Enzhe Lu et al. (2025). MoBA: Mixture of Block Attention for Long-Context LLMs. https://arxiv.org/abs/2502.13189. arXiv:2502.13189.

[117] Yash Akhauri et al. (2025). TokenButler: Token Importance is Predictable. https://arxiv.org/abs/2503.07518. arXiv:2503.07518.

[118] Jingyang Yuan et al. (2025). Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. https://arxiv.org/abs/2502.11089. arXiv:2502.11089.

[119] Ji Lin et al. (2024). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978. arXiv:2306.00978.

[120] Guangxuan Xiao et al. (2024). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. https://arxiv.org/abs/2211.10438. arXiv:2211.10438.

[121] Elias Frantar et al. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323. arXiv:2210.17323.

[122] Alex L. Zhang et al. (2026). Recursive Language Models. https://arxiv.org/abs/2512.24601. arXiv:2512.24601.

[123] Zichen Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. https://arxiv.org/abs/2503.20783. arXiv:2503.20783.

[124] Haotian Luo et al. (2025). O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning. https://arxiv.org/abs/2501.12570. arXiv:2501.12570.

[125] Muzhi Dai et al. (2025). S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models. https://arxiv.org/abs/2505.07686. arXiv:2505.07686.

[126] Haoran Zhao et al. (2025). Let LRMs Break Free from Overthinking via Self-Braking Tuning. https://arxiv.org/abs/2505.14604. arXiv:2505.14604.

[127] Yi Shen et al. (2026). DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models. https://arxiv.org/abs/2503.04472. arXiv:2503.04472.

[128] Chen Li et al. (2025). Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning. https://arxiv.org/abs/2503.15952. arXiv:2503.15952.

[129] Bairu Hou et al. (2025). ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning. https://arxiv.org/abs/2504.01296. arXiv:2504.01296.

[130] Violet Xiang et al. (2025). Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning. https://arxiv.org/abs/2506.05256. arXiv:2506.05256.

[131] Haizhong Zheng et al. (2025). Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts. https://arxiv.org/abs/2506.02177. arXiv:2506.02177.

[132] Yang Zhou et al. (2026). Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models. https://arxiv.org/abs/2606.08446. arXiv:2606.08446.

[133] Yongji Wu et al. (2026). RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs. https://arxiv.org/abs/2510.19225. arXiv:2510.19225.

[134] Mehul Damani et al. (2024). Learning How Hard to Think: Input-Adaptive Allocation of LM Computation. https://arxiv.org/abs/2410.04707. arXiv:2410.04707.

[135] Junyan Li et al. (2025). Steering LLM Thinking with Budget Guidance. https://arxiv.org/abs/2506.13752. arXiv:2506.13752.

[136] Menghua Wu et al. (2025). Thought calibration: Efficient and confident test-time scaling. https://arxiv.org/abs/2505.18404. arXiv:2505.18404.

[137] Yesheng Liang et al. (2026). ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference. https://arxiv.org/abs/2511.10645. arXiv:2511.10645.

[138] Harry Dong et al. (2026). Scalable LLM Reasoning Acceleration with Low-rank Distillation. https://arxiv.org/abs/2505.07861. arXiv:2505.07861.

[139] Jujie He et al. (2025). Skywork Open Reasoner 1 Technical Report. https://arxiv.org/abs/2505.22312. arXiv:2505.22312.

[140] Jia Li et al. (2024). NuminaMath. https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf.

[141] Zhen Huang et al. (2024). OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI. https://arxiv.org/abs/2406.12753. arXiv:2406.12753.

[142] Bofei Gao et al. (2024). Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models. https://arxiv.org/abs/2410.07985. arXiv:2410.07985.

[143] Wanjun Zhong et al. (2023). AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. https://arxiv.org/abs/2304.06364. arXiv:2304.06364.

[144] Wang Ling et al. (2017). Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems. https://arxiv.org/abs/1705.04146. arXiv:1705.04146.

[145] Jian Liu et al. (2020). LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning. https://arxiv.org/abs/2007.08124. arXiv:2007.08124.

[146] Haoxi Zhong et al. (2019). JEC-QA: A Legal-Domain Question Answering Dataset. https://arxiv.org/abs/1911.12011. arXiv:1911.12011.

[147] Siyuan Wang et al. (2021). From LSAT: The Progress and Challenges of Complex Reasoning. https://arxiv.org/abs/2108.00648. arXiv:2108.00648.

[148] Chaoqun He et al. (2024). OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. https://arxiv.org/abs/2402.14008. arXiv:2402.14008.

[149] Wenhu Chen et al. (2023). TheoremQA: A Theorem-driven Question Answering dataset. https://arxiv.org/abs/2305.12524. arXiv:2305.12524.

[150] Daman Arora et al. (2023). Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models. https://arxiv.org/abs/2305.15074. arXiv:2305.15074.

[151] Liangtai Sun et al. (2024). SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research. https://arxiv.org/abs/2308.13149. arXiv:2308.13149.

[152] Kimi et al. (2025). Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599.

[153] Tamera Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. https://arxiv.org/abs/2307.13702. arXiv:2307.13702.

[154] Yanda Chen et al. (2025). Reasoning Models Don't Always Say What They Think. https://arxiv.org/abs/2505.05410. arXiv:2505.05410.