Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning cover

📄 Executive Summary

Modern artificial intelligence reasoning models generate extensive step-by-step thinking traces that can span tens of thousands of tokens, creating a critical memory bottleneck in hardware during deployment. To manage memory footprints, prevailing techniques evict stored past tokens from the key-value cache by scoring and ranking them according to heuristics such as accumulated attention, value magnitude, or position.

The article investigates whether complex scoring and ranking signals genuinely improve accuracy when compressing reasoning traces. It introduces Random Attention—a straightforward policy that permanently retains the original input prompt and discards remaining intermediate reasoning tokens uniformly at random across attention heads—to establish a rigorous baseline and test the true value of learned selection signals.

The researchers evaluated Random Attention across four language models (Qwen3-4B, 14B, 32B, and Phi-4-reasoning) on six demanding benchmarks covering mathematics, science, and coding under three-to-fourfold memory compression. Controlled ablation studies, mechanistic planted-fact probing experiments, and production-style serving benchmarks using standard deployment runtimes were conducted to measure exact task accuracy and hardware throughput.

The core findings reveal that complex scoring signals contribute almost no meaningful benefit over uniform random eviction. In 31 of 60 task comparisons, Random Attention significantly outperformed existing baselines, matching the top-performing baseline across math and science benchmarks. The performance variance previously attributed to superior scoring was primarily driven by whether an evictor protected the initial prompt: once all baselines pinned the prompt, the performance gaps between them largely vanished. Intermediate reasoning traces inherently protect themselves because the model repeatedly restates active concepts in the text and redundantly distributes information across multiple attention heads. Serving benchmarks showed that eliminating the scoring pass allowed Random Attention to achieve 32% to 43% higher serving throughput than the strongest baseline in production-style paged environments.

These results imply that system architects and engineers can substantially lower serving costs and latency without sacrificing reasoning accuracy by replacing compute-heavy scoring algorithms with simple prompt pinning and random retention. Prior research on cache eviction over-indexed on ranking algorithms because it failed to account for prompt loss and inherent trace redundancy. Complex scoring signals are only necessary in edge cases where an isolated fact is stated once, never repeated, and recalled much later—a pattern rarely found in natural reasoning chains.

Organizations serving long-context reasoning models should adopt prompt protection paired with random eviction as a practical, tuning-free default for memory-constrained deployments. Research and engineering teams developing new eviction methods should use Random Attention under matched prompt protection as their reference baseline and redirect optimization efforts toward efficiently budgeting long system prompts rather than ranking reasoning tokens.

The conclusions are robust across multiple model scales and benchmarks using paired statistical testing. However, the evaluation assumes tasks with relatively short, fixed prompts; applications featuring massive prompt scaffolding, such as extensive codebase contexts, may require dedicated prompt compression to avoid disproportionate cache consumption.

📚 What to read before this

Efficient Memory Management for Large Language Model Serving with PagedAttention, Woosuk Kwon et al. (2023). Introduces PagedAttention and the vLLM serving framework, establishing the foundational memory-management architecture and serving baseline on which Random Attention's deployment is benchmarked.
Efficient Streaming Language Models with Attention Sinks, Guangxuan Xiao et al. (2023). Identifies the critical role of initial prompt tokens as attention sinks in streaming generation, providing the essential insight behind why protecting the prompt is central to effective KV cache eviction.
Prefix Sliding for efficient test-time scaling, Niklas Muennighoff et al. (2026). Demonstrates that preserving the prefix while discarding intermediate reasoning tokens retains full reasoning accuracy during test-time scaling, establishing direct empirical precedent for prompt-preserving eviction.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Tri Dao et al. (2022). Presents the foundational IO-aware exact attention algorithm that defines hardware-level GPU memory traffic and performance constraints for large language model attention.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Joshua Ainslie et al. (2023). Analyzes KV cache bottlenecks across multi-head and grouped-query attention architectures, motivating memory-efficient attention designs during decoding.

📚 What to read after this

A Probabilistic Interpretation of KV Cache Eviction, Renato Geh et al. (2026). Provides a formal probabilistic foundation showing why deterministic scoring heuristics fail and how stochastic sampling offers an unbiased alternative for KV cache eviction.
Language Models Can Control Their Own Attention, Namgyu Ho et al. (2026). Explores how models can actively declare their own attention scope during reasoning traces to bypass external scoring heuristics entirely in vLLM serving.
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, DeepSeek-AI (2026). Builds upon advanced KV cache reduction strategies to scale multi-turn reasoning and agentic workloads across million-token contexts.
Efficient Reasoning on the Edge, Yelysei Bondarenko et al. (2026). Applies KV cache reuse and reasoning-trace compression techniques to enable efficient multi-step reasoning under strict edge hardware constraints.

Executive Summary: Modern artificial intelligence reasoning models generate extensive step-by-step thinking traces that can span tens of thousands of tokens, creating a critical memory bottleneck in hardware during deployment. To manage memory footprints, prevailing techniques evict stored past tokens from the key-value cache by scoring and ranking them according to heuristics such as accumulated attention, value magnitude, or position.

The article investigates whether complex scoring and ranking signals genuinely improve accuracy when compressing reasoning traces. It introduces Random Attention—a straightforward policy that permanently retains the original input prompt and discards remaining intermediate reasoning tokens uniformly at random across attention heads—to establish a rigorous baseline and test the true value of learned selection signals.

The researchers evaluated Random Attention across four language models (Qwen3-4B, 14B, 32B, and Phi-4-reasoning) on six demanding benchmarks covering mathematics, science, and coding under three-to-fourfold memory compression. Controlled ablation studies, mechanistic planted-fact probing experiments, and production-style serving benchmarks using standard deployment runtimes were conducted to measure exact task accuracy and hardware throughput.

The core findings reveal that complex scoring signals contribute almost no meaningful benefit over uniform random eviction. In 31 of 60 task comparisons, Random Attention significantly outperformed existing baselines, matching the top-performing baseline across math and science benchmarks. The performance variance previously attributed to superior scoring was primarily driven by whether an evictor protected the initial prompt: once all baselines pinned the prompt, the performance gaps between them largely vanished. Intermediate reasoning traces inherently protect themselves because the model repeatedly restates active concepts in the text and redundantly distributes information across multiple attention heads. Serving benchmarks showed that eliminating the scoring pass allowed Random Attention to achieve 32% to 43% higher serving throughput than the strongest baseline in production-style paged environments.

These results imply that system architects and engineers can substantially lower serving costs and latency without sacrificing reasoning accuracy by replacing compute-heavy scoring algorithms with simple prompt pinning and random retention. Prior research on cache eviction over-indexed on ranking algorithms because it failed to account for prompt loss and inherent trace redundancy. Complex scoring signals are only necessary in edge cases where an isolated fact is stated once, never repeated, and recalled much later—a pattern rarely found in natural reasoning chains.

Organizations serving long-context reasoning models should adopt prompt protection paired with random eviction as a practical, tuning-free default for memory-constrained deployments. Research and engineering teams developing new eviction methods should use Random Attention under matched prompt protection as their reference baseline and redirect optimization efforts toward efficiently budgeting long system prompts rather than ranking reasoning tokens.

The conclusions are robust across multiple model scales and benchmarks using paired statistical testing. However, the evaluation assumes tasks with relatively short, fixed prompts; applications featuring massive prompt scaffolding, such as extensive codebase contexts, may require dedicated prompt compression to avoid disproportionate cache consumption.

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

Heng Wang $^{1,2}$, Jielin Qiu $^{1}$, Wenting Zhao $^{1}$, Cheng Qian $^{1,2}$, Liangwei Yang $^{1}$, Jiawei Han $^{2}$, Heng Ji $^{2}$, Silvio Savarese $^{1}$, Shelby Heinecke $^{1}$, Huan Wang $^{1}$
$^{1}$ Salesforce AI Research, $^{2}$ University of Illinois Urbana-Champaign

1. Introduction

Section Summary: Reasoning models generate long chains of thought that quickly exhaust memory through their growing key-value caches, prompting various eviction methods that try to retain only the most important tokens based on attention scores or other heuristics. The paper demonstrates that these scoring approaches add almost no benefit: a simple method called Random Attention, which discards cached tokens uniformly at random while always preserving the original prompt, matches or exceeds the accuracy of leading eviction techniques across multiple models and reasoning tasks while running substantially faster. The results indicate that prompt protection and the natural redundancy in reasoning traces matter far more than precise selection, shifting focus toward better handling of long inputs and rare facts.

$**Figure 1:** **(a)** Mean accuracy over the six reasoning tasks of Table 1 and Table 5 at ${\sim}4\times$ compression: Random Attention matches the strongest prior evictor on every model (the small gaps to TriAttention at 14B and 32B are mostly driven by code reasoning, where long prompts consume the budget, § 4.2). **(b)** vLLM serving throughput at $32$ k-token generations (Table 4); labels give Random Attention's multiple of full attention and its margin over TriAttention: with no scoring pass it serves $32$ – $43\%$ higher throughput.$

Reasoning models ([1, 2, 3]) solve hard problems by generating chains of thought that run to tens of thousands of tokens. The key-value (KV) cache grows linearly with the length of generation, creating a severe memory bottleneck. KV cache eviction methods have been developed to address this by keeping a fixed budget of KV cache entries and discarding the rest as decoding proceeds. Existing KV cache eviction methods follow the same paradigm: score each cached token by an estimate of how much it will matter later and keep the top-scoring ones. This line of work is a sequence of better scores, from accumulated attention ([4]), attention from a recent window ([5]), and attention combined with redundancy ([6]), to value magnitude ([7]) and position-dependent key statistics ([8]). Each new score is motivated by heuristics observed to be correlated with task accuracy, and the premise behind all of them is that the score decides accuracy under compression.

We test that premise directly and show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all. Across four models (Qwen3-4B, 14B and 32B, and Phi-4-reasoning) and six reasoning tasks spanning math, science and code, it is comparable to the strongest baseline (Figure 1 a) and even significantly ahead in 31 of the 60 baseline comparisons in the main result table. Served through vLLM integration, it delivers $32$–$43%$ more tokens per second at $32$k-token generations than the strongest baseline, since it never runs a scoring pass (Figure 1 b). The only cell in that table where a baseline is significantly ahead is code reasoning on Qwen3-32B, traced in § 4.2 to prompt length rather than the selection signal. The selection signal, in other words, contributes almost nothing beyond uniform random selection.

Two controlled experiments explain why. First, the prompt is the fragile part of the cache. Prior evictors differ in whether the prompt survives, some pinning it by rule and others leaving it to the score, so once every method is given the same rule (keep the prompt), most of the gap between them disappears, and each method gains exactly as much as its score had been losing of the question (§ 5.1). Second, the reasoning trace protects itself. It is stored redundantly at two levels, in the text, because the model restates what it is still using, and across attention heads, because every head holds its own copy of every token and eviction decides per head which copies die. A planted-fact probe shows the model reading a value from whichever heads still hold it: a fact kept in one head is almost never retrieved, kept in several it almost always is, and the shape of the surviving copies does not matter (§ 5.2). Once the prompt is safe, a random draw keeps enough copies of what the model still needs; what a signal still buys is the rare fact stated once and never restated, which reasoning traces seldom produce (§ 5.3).

Our findings have two practical implications. First, Random Attention is a deployable method in its own right. It needs no calibration, no tuning and no scoring pass, and it is the fastest evictor we measured at equal accuracy, so it is a reasonable default for serving reasoning models under a memory budget, and the baseline that any new selection signal has to beat at matched budget and matched prompt protection. Second, the findings redirect what eviction research should optimise. The accuracy of an evictor is decided by what it protects, not by how it ranks the rest, which moves the open questions to where protection still matters: how to budget long prompts, especially in code tasks where protecting the entire prompt consumes a substantial fraction of the cache budget, and how to recover rare once-stated facts that only a content-dependent signal can preserve (§ 5.3).

2. Preliminaries

Section Summary: This section describes the challenge of permanently discarding key-value cache entries during long model decoding runs to stay within a fixed memory budget, as opposed to methods that merely subsample attention while retaining everything. It sets up the relevant notation for attention weights and cache contents at each decoding step, then details a periodic eviction process that scores older entries whenever a small recent buffer fills and retains only the top-scoring ones. Finally, it expresses several existing eviction heuristics as alternative choices for the scoring function, ranging from simple attention accumulation to adjustments for redundancy or value magnitude.

Setting.

We study KV cache eviction during decoding. This is the regime reasoning models create: a model answering a short (e.g. $200$-token) math question may generate a very long chain of thought (e.g. more than $10{,}000$ tokens). Eviction permanently discards key-value pairs once the cache reaches a budget, which bounds memory but risks irrecoverable loss; it is therefore distinct from sparse-attention selection, which attends to a subset but keeps every pair in memory and so still grows linearly with sequence length. Everything below concerns eviction.

Notation.

Let $t$ index decode steps. At step $t$ an attention head holds $N$ cached key-value pairs $(k_i, v_i)$, $i = 1,\dots,N$, with $k_i, v_i \in \mathbb{R}^{d}$, where $i$ is the position of the token that produced the pair; positions $1,\dots, \ell_{\mathrm{p}}$ are the prompt. The head forms its output from the current query $q_t \in \mathbb{R}^{d}$ as

$ o_t = \sum_{i \le N} \alpha^{(t)}_i v_i , \qquad \alpha^{(t)}i = \frac{\exp!\big(q_t^{\top} k_i / \sqrt{d}\big)} {\sum{j \le N} \exp!\big(q_t^{\top} k_j / \sqrt{d}\big)} , $

where $\alpha^{(t)}i$ is the attention weight the query at step $t$ places on position $i$ ($\sum{i \le N} \alpha^{(t)}i = 1$). We write $v{ij}$ for the $j$-th coordinate of $v_i$ and $|k_i|$ for the norm of $k_i$. Eviction decisions are made independently in every layer and KV head, as is standard; we drop both indices throughout.

Periodic eviction with budget $K$ and buffer $r$.

We adopt the decode-phase framework of [6] and [7]. The cache keeps a persistent budget of $K$ pairs plus a buffer of the $r \ll K$ most recent pairs, which is never scored. Each decode step appends one pair, so the buffer fills every $r$ steps and triggers an eviction: every candidate $i$ in the candidate set $\mathcal{C}_t$, the cached positions outside the buffer, receives a real-valued score $s_i$ from a policy-specific rule, which may depend on the whole cache and on the past queries, and the $K$ highest-scoring candidates are kept,

$ \mathcal{S}t ;=; \operatorname{top-}! K{,i \in \mathcal{C}_t}; s_i , $

returning the cache to $K + r$ entries per head. Eviction is monotonic: a discarded pair is gone for good, as in a memory-bounded deployment. Eviction methods differ mostly in $s$.

Baselines as choices of $s$.

We write each prior method's $s$ in the notation above, with every score evaluated at the eviction step $t$. StreamingLLM ([9]) has no score: it keeps the first few (attention-sink) positions and the recent buffer. H2O ([4]) keeps the positions that have received the most attention since they entered the cache, $s_i=\sum_{t'=i}^{t}\alpha^{(t')}i$. SnapKV ([5]) uses only the last $w$ queries, $s_i=\sum{t'=t-w+1}^{t}\alpha^{(t')}i$, max-pooled over neighbouring positions. R-KV ([6]) mixes the SnapKV score with a redundancy term, $s_i = \lambda, s^{\mathrm{Snap}}i + (1-\lambda),(1-\bar{c}i)$ with $\lambda \in [0,1]$ and $\bar{c}i = \frac{1}{N-1}\sum{j \ne i}\cos(k_i, k_j)$ the mean cosine similarity of $k_i$ to the other cached keys, so that a restated fact is not kept twice. VaSE ([7]) scores values rather than keys: it keeps the $n{\mathrm{large}} < K$ positions with the largest value range $\max{j} v{ij}-\min_{j} v_{ij}$ and fills the remaining $K - n_{\mathrm{large}}$ slots by sampling positions with probability proportional to their SnapKV score. TriAttention ([8]) scores a position by its distance from the current query, $s_i=f(t-i)$, where $f$ is a trigonometric series whose coefficients are calibrated per head from the concentration of that head's queries and keys, combined with $|k_i|$.

3. Random Attention

Section Summary: Random Attention is an eviction method for managing the limited key-value cache during language model generation that relies on no attention signals or importance scores. It permanently shields the initial prompt and question from removal, since that text cannot be recovered if lost, while assigning fresh random scores to every generated token and keeping only the top slice of them independently for each attention head. The result is a lightweight, evenly distributed retention pattern across the output history that serves as a basic baseline for testing whether other, more elaborate eviction rules actually deliver gains.

Random Attention is a signal-free eviction policy defined by two structural choices. The intuition is to separate the irreplaceable input from the model-generated trace: the question is stated once and cannot be recovered if evicted, whereas the trace revisits and restates its intermediates as generation proceeds. We protect the former and use random sampling for the latter:

Protect the question. Positions $1,\dots, \ell_{\mathrm{p}}$, the entire prefill (the system prompt, chat template, and question), are never evicted.
Scatter the rest, per head. Every remaining cached position receives an i.i.d. uniform random score, and each KV head keeps its top-$K$ independently. Sampling without a signal spreads the retained budget evenly over the whole trace, and differently in every head.

In the notation of § 2, the entire method is

$ s_i ;=; \begin{cases} +\infty, & i \le \ell_{\mathrm{p}} \quad \text{(the prompt)},\ u_i \sim \mathrm{Uniform}(0,1), & \text{otherwise}, \end{cases} $

drawn independently per KV head at every eviction; Equation 2 does the rest in four lines:

Require: batch size $B$; number of KV heads $H_{\mathrm{kv}}$; number of
cached positions $S$; prompt length $\ell_{\mathrm{p}}$; budget $K$
Ensure: $\mathrm{keep}$, the $K$ positions each KV head retains,
of shape $(B,\ H_{\mathrm{kv}},\ K)$
$s \gets \mathrm{rand}(B,\ H_{\mathrm{kv}},\ S)$
// i.i.d. uniform score per cached position, per KV head
$s[\,:,\ :,\ 0\!:\! \ell_{\mathrm{p}}\,] \gets +\infty$
// force-keep the question
$\mathrm{keep} \gets \operatorname{topk}(s,\ K)$
// independent top- $K$ per KV head
return $\mathrm{keep}$

The per-eviction cost is one $\mathrm{rand}$ and one $\operatorname{topk}$. This is the weakest selection signal we can write down: Random Attention is both a deployable method and a null hypothesis. Any signal-based selector that cannot beat it at matched budget is not extracting usable information from its signal.

4. Experiments

Section Summary: The experiments evaluate several KV cache eviction strategies, including specialized selectors like SnapKV and VaSE, against a simple random baseline and full attention, using Qwen3 and Phi-4 models on math, science, and code reasoning tasks at roughly 4x compression. Across repeated runs with statistical validation, random selection matches or significantly outperforms the other methods on most tasks, with no selector delivering consistent gains; the gap widens at higher compression ratios, though some methods struggle more on code due to long prompts. Overall, the results indicate that elaborate selection rules add little value over randomness for preserving accuracy in these settings.

4.1 Setup

Baselines.

We compare against the four eviction methods of § 2, SnapKV, R-KV, VaSE and TriAttention, each run as released at the same budget (detailed configurations in Appendix F). Full attention (no eviction) is the ceiling. The diagnostics of § 5.1 additionally use recency+prompt (keep the KV caches corresponding to the prompt, fill with the contiguous recent window, StreamingLLM-style ([9])). All methods are evaluated with FlashAttention-2 kernels ([10]), without PagedAttention ([11]).

Models and tasks.

We evaluate Qwen3-4B, Qwen3-14B, Qwen3-32B ([12]), and Phi-4-reasoning (14B) ([3]) across six reasoning tasks spanning math, science, and code: MATH500 ([13, 14]) ($500$ problems), GPQA-Diamond (GPQA-D) ([15]) ($198$ problems), AIME 2025 and 2026 ($30$ problems each, reported as one pooled AIME column) and HMMT ($60$ problems) via MathArena ([16]), and LiveCodeBench-v6 medium ([17]) ($383$ problems; pass@1 by real test execution). Generation uses each model's released sampling settings (temperature $0.6$ for the Qwen3 models, $0.8$ for Phi-4-reasoning; nucleus $p{=}0.95$ for all), and each result is repeated as independently sampled runs: $2$ on MATH500, $4$ on GPQA-D and LiveCodeBench, and $16$ on AIME and HMMT. Every accuracy in the paper is the average over those repeated runs. The main grid fixes each task's budget at ${\sim}4\times$ compression of its typical trace (${\sim}3\times$ for LiveCodeBench); the per-head budget $K$ for each task appears in Table 1's header and is detailed in Appendix F. We set the maximum generation length to 32,768 (32k) tokens.

Metrics and statistics.

The primary metric is accuracy, judged by whether the final boxed answer is correct (following [7] and [18]). Every claimed margin is gated by a paired, problem-clustered percentile bootstrap ($95%$ CI) plus an exact sign test; grid cells significantly below Random Attention are grayed.

4.2 Main Results

Table 1 presents the performance on Qwen3-4B, Phi-4-reasoning, and Qwen3-32B; Qwen3-14B replicates the pattern at an intermediate scale in Appendix A. Paired tests put Random Attention significantly ahead in $31$ of the table's $60$ baseline cells and significantly behind in one.

::: {caption="Table 1: Accuracy under KV cache eviction at each task's ${\sim}4\times$ compression (LiveCodeBench: ${\sim}3\times$); the header gives each task's per-head KV budget $K$ (§ 2). Bold: best eviction method per column; grayed: significantly below Random Attention (paired clustered bootstrap + sign test, 95%)."}

:::

A selection signal buys nothing on math and science reasoning.

On MATH500 and GPQA-D, Random Attention beats VaSE and SnapKV significantly on every model, and R-KV significantly on Qwen3-4B. No selector beats it significantly on these tasks: the leads that do appear (TriAttention by $0.3$–$0.6$ points on two GPQA-D cells) sit inside the noise.

On competition math no selector pulls ahead.

Competition math tasks including AIME and HMMT make the comparison ride on a smaller and harder sample. With $16$ sampled runs per problem, SnapKV still trails Random Attention significantly on every model, as do R-KV on the Qwen3 models and VaSE on Phi-4-reasoning; no selector in Table 1 is significantly above Random Attention on either task, and the nominal leads run both ways: VaSE edges it on Qwen3-32B by $1.7$ and $1.5$ points, against a run-to-run standard deviation of $\pm 5$ points for both methods on a $30$-problem set. These tasks separate only once the budget tightens (§ 4.3), and then in Random Attention's favour.

On code reasoning most signal-based selectors fall apart due to much longer prompts.

LiveCodeBench is the one task with large gaps: SnapKV loses $20$–$35$ points to Random Attention on every model, VaSE collapses on Phi-4-reasoning ($0.373$, $29$ points behind) and is grayed on two of the three models, as is R-KV. TriAttention and Random Attention tie on Qwen3-4B and Phi-4-reasoning, while TriAttention leads on Qwen3-32B by about three points, the single significant baseline win in the main grid. The prompt is what sets code apart. LiveCodeBench prompts average $557$ tokens, which is six times MATH500's under the same tokenizer, and the longest can consume up to half of the $K{=}3072$ budget. Therefore a selector that fails to capture the prompt loses more here than anywhere else; we show in the following section that protecting it closes the SnapKV and VaSE gaps (§ 5.1). Random Attention itself pins every prompt token, so on code reasoning a large, variable share of its budget is spent before selection begins; much of a code prompt is scaffolding (I/O formats, harness instructions) that a smarter rule might compress rather than pin whole, which we leave to future work since Random Attention's value as a null lies in having nothing to tune.

$**Figure 2:** Accuracy from $2\times$ to $16\times$ compression on Qwen3-4B and Phi-4-reasoning, on the four math and science tasks; dashed lines mark full attention.$

4.3 Compression Pressure Widens the Gap, in Every Family

Figure 2 presents the performance from $2\times$ to $16\times$ compression on Qwen3-4B and Phi-4-reasoning, on the four math and science tasks, and both families tell the same story: at $2\times$ every method sits near full attention; as the budget tightens, Random Attention stays tied with TriAttention, while the gap from both to VaSE opens. LiveCodeBench is left out of the sweep since its prompts alone would not fit the small budgets.

5. Why the Selection Signal Buys So Little

Section Summary: The section explains that KV caches in reasoning models split into a fragile prompt, which methods often discard unevenly and which causes most reported performance gaps when lost, and a working trace of reasoning steps that proves highly redundant. Once the prompt is protected equally across methods, learned selection scores add almost nothing and often trail a policy that keeps the prompt while evicting the trace at random. The trace itself survives random drops because its useful facts are restated in the text and duplicated across attention heads, so enough copies remain for the model to retrieve what it needs.

The content of the KV cache in reasoning can be divided into two types. The prompt is stated once and never stated again. The working state (i.e., the intermediate reasoning steps a solution builds on) is written and rewritten continually as the model reasons. We show that the first is fragile, and methods differ in how they treat it; the second is redundant enough that a random draw over it keeps what the model still needs.

::: {caption="Table 2: Performance before and after protecting the prompt. R-KV never gains more than $1.9$ points, SnapKV gains everywhere, VaSE gains materially only on Phi-4-reasoning."}

:::

5.1 The Prompt Is the Fragile Part

Methods disagree about the prompt. TriAttention keeps the whole input by default, while VaSE, R-KV, and SnapKV only keep the sink tokens ([9, 19]) by default and leave every slot to the score, so a comparison across papers also compares protection regimes ([20]). Giving every method the same rule separates the score from the protection (Table 2), and one pattern orders the outcome: the rule pays each method according to how much of the question its score was losing, measured for every selector by logging, round by round, how much of the prompt it keeps (Appendix D). SnapKV, whose score retains the least of the prompt, always gains, up to $22.5$ points on Phi-4-reasoning GPQA-D; VaSE gains only where its retention fails, little on Qwen3-4B but $+4.2$ and $+10.2$ on Phi-4-reasoning, whose prompts are two to three times longer; R-KV, which retains the most, never gains more than $1.9$ points (survival fractions in Appendix D). Once every method keeps the prompt, the three baselines land within $2.2$ points of one another in every setting. On Phi-4-reasoning they also land within about two points of Random Attention, which ranks nothing; on Qwen3-4B a residual of $4$–$6$ points below Random Attention remains for all three, and R-KV and VaSE, which already kept most of the prompt there, gain almost nothing from the rule. Code reasoning on both models, competition math on Qwen3-4B and GPQA-D at 32B (Appendix C) show the same pattern: the rule closes every gap that was large and method-specific, and what survives it is smaller and runs in Random Attention's favour. Most of the difference between the baselines was therefore the prompt. Whatever their scores add beyond it is small, and where a residual remains it is a deficit: with the prompt protected, every learned score still trails a policy that ranks nothing.

The two signal-free rows make the same point from the other side. Without the rule, a recency window allocates all the budget to the recent trace and none to the prompt and scores as low as $0.09$, and Random Attention, which keeps the prompt only at the uniform rate, falls to $0.23$–$0.76$. With the rule, the same two policies lose nothing that matters: Random Attention is the best policy in every setting and a plain recency window comes within two points of the best baseline. Losing the prompt is catastrophic and cutting the trace at random is not, which is what makes the prompt the fragile part of the cache. The same confound explains the large gaps others report between random retention and signal-based selection in previous works ([21, 22]): their random baselines perform poorly since the prompt is lost.

5.2 The Working State Protects Itself

The rest of the cache is the model's own reasoning trace (working state), which is stored redundantly at two levels. The first redundancy is in the text and is already shown by [6]: reasoning traces restate what they are still using, so a value that matters rarely lives at one position only. The second is across heads: each of the KV heads caches its own copy of every token, and eviction decides per head which copies die. A token is only lost when all KV heads happen to drop it.

We show how the model makes use of the second, cross-head redundancy with a planted-fact probing experiment. A synthetic fact (e.g. Let zq = 4729; a fresh variable and value each time) is inserted into real model-generated MATH500 reasoning traces, with a question needing the value appended at the end. The fact is planted $1{,}536$ tokens before the question, so the cache is evicted $15$ times between the two (other distances in Appendix B); the question itself is always kept. What we control is which key-value heads keep the fact: a condition is a chosen set of heads in which the fact's tokens are pinned, with the fact evicted from every other head and the standard per-head uniform eviction running on everything else. Two metrics measure what survives. Retrieval is the fraction of traces whose greedy decode reproduces the value at the question. Because retrieval falls to zero in the hardest conditions, a graded recall carries the comparison there:

$ R ;=; \frac{\sum_i \bigl(\mathrm{LP}_i - \mathrm{LP}_i^{\mathrm{del}}\bigr)} {\sum_i \bigl(\mathrm{LP}_i^{\mathrm{kept}} - \mathrm{LP}_i^{\mathrm{del}}\bigr)}, $

where $\mathrm{LP}_i$ is the log-probability the model assigns to trace $i

#39;s correct value under the condition being tested, and $\mathrm{LP}_i^{\mathrm{kept}}$ and $\mathrm{LP}_i^{\mathrm{del}}$ are the same quantity with the fact kept in every cache and deleted from every cache. $R{=}1$ therefore means the surviving copies are worth as much as never evicting the fact, and $R{=}0$ that they are worth nothing. Every condition is scored on the same $250$–$500$ planted traces. The probe shows the model exploiting the redundancy in two ways.

Copies pool across heads.

Attention heads specialise: only three of Qwen3-4B's eight key-value heads retain a usable trace of the fact on their own, consistent with the retrieval-head specialisation of [23], and even those three are weak alone: the best single head yields the value in $3%$ of trials, the next in $1%$. But the readout does not depend on any one head: the same two heads together yield the fact in $60%$ of trials, three heads in $83%$, and all eight in $99%$ (Figure 3 a). Pooling is thus strongly superadditive, a pair being worth many times the sum of its singles, and it even crosses facts: two values held in different heads, both needed by the answer, give $R{=}0.31$ together against $0.10$ and $0.16$ alone (Figure 3 b). For an eviction policy the consequence is direct: a value stays usable as long as some heads keep a copy, which is exactly what independent per-head draws maximise. On real MATH500 traces this extra coverage is not even needed: a shared draw, the same random positions in every head, scores within $0.3$ points of Random Attention at $4\times$ and $8\times$ (Appendix D), because the text-level redundancy already keeps a restated copy; the cross-head level carries what the text does not restate, which is the probe's regime.

The shape of the copies does not matter.

We deal the fact out token by token across heads, so that no two consecutive tokens share a head and no head holds a readable span; retrieval barely moves ($0.33$, against $0.39$ for an intact sentence at the same retained mass) and recall is essentially unchanged ($R{=}0.75$ vs. $0.76$). The same insensitivity appears on real MATH500 traces at full scale: keeping the history in contiguous blocks rather than scattered tokens costs nothing as blocks grow from $1$ to $64$ tokens; accuracy drops only at block size $256$, where the budget leaves a head just four or two blocks, and drops more with two, so what matters is blocks per head, not block length (Figure 3 c). Together the two findings say the answer depends on whether some usable copy of a needed value survives somewhere, not on which copy, in which head, or in what shape.

**Figure 3:** **(a)** A fact held in one head is almost never retrieved; held in several it is. **(b)** Two facts in *different* heads are worth more together than the sum of each alone (dashed). **(c)** Real MATH500: contiguous blocks cost nothing up to size $64$; accuracy drops only once a head is left with $4$ ($K{=}1024$) or $2$ ($K{=}512$) blocks.

5.3 What Is Left for a Selection Signal

\begin{tabular}{lcc}
\toprule
Policy & Retr. & $\log p$ \\
\midrule
Random Attention (ours) & $0.000$ & $-18.35$ \\
VaSE & $0.344$ & $-3.88$ \\
SnapKV & $0.004$ & $-11.11$ \\
R-KV & $0.836$ & $-0.71$ \\
TriAttention & $0.016$ & $-11.11$ \\
\bottomrule
\end{tabular}

The one case a signal-free policy cannot cover is a fact stated once, never restated, and needed much later. We announce a passcode once, $57$ compression rounds before the question, let each policy retain as it sees fit, and report two numbers (Table 3): the fraction of traces in which the model reproduces the passcode (Retr.), and the log-probability it assigns to the correct passcode at the question, averaged over traces ($\log p$). A $\log p$ of $0$ means the model would produce the passcode with certainty; a number like $-18$ means the passcode is effectively gone from the cache. Random Attention never reproduces it, and retrieval tracks the attention statistic each selection signal scores by: R-KV, whose importance accumulates attention over the whole history, finds the passcode $84%$ of the time; VaSE's sampled attention a third of the time; the recent-window signals of SnapKV and TriAttention almost never. [24] prove that random caches must lose at pointer-chasing when nothing is redundant. Needle-finding is thus real selection skill, but it neither implies nor follows from aggregate strength: R-KV, the best needle-finder, leads only one column of Table 1, while TriAttention, the strongest baseline there, recovers almost nothing here. On real traces the case is rare, because the model keeps restating what it is still using.

6. Efficiency Evaluation

Section Summary: In evaluations using both the vLLM serving system with paged attention and standard batched decoding, Random Attention delivered substantially higher throughput than alternatives like TriAttention by avoiding any scoring step during cache compression. This allowed it to support larger numbers of concurrent requests on a single GPU, achieving 1.6–2.7 times the speed of full attention and 32–43 percent more throughput than comparable methods under realistic serving loads. The advantage stems from eliminating repeated scoring computations that accumulate across many parallel requests, where even small per-compression delays create noticeable waiting time for the entire batch.

We evaluate efficiency in two settings: 1) vLLM ([11]) with PagedAttention, following the protocol of TriAttention, and 2) batched decoding in HuggingFace Transformers with FlashAttention-2 and no paging (the setup of § 4.1), where the throughput is measured at the largest batch that fits on one GPU. The first setting compares Random Attention with TriAttention only, because only TriAttention and R-KV have vLLM ports, and [8] have already shown that TriAttention is more efficient than R-KV at matched budget. The second setting compares all methods. Details of the two protocols are given in Appendix G.

::: {caption="Table 4: Serving throughput (output tok/s, and the multiple of full attention) under vLLM with PagedAttention on one H200 ($K{=}2048$, $1$ k-token prompts, $32$ k-token generations)."}

:::

Under paged serving, Random Attention is $32$–$43%$ faster than TriAttention.

Both methods run on one H200 at $K{=}2048$ with $1$k-token prompts, $32$k-token generations and $128$ requests. vLLM preempts requests when the KV cache demand exceeds its cache pool, and a preempted request can no longer be compressed, so on Qwen3-32B, whose larger weights leave a smaller pool, the compressed runs are capped at $96$ concurrent requests; full attention runs under no such limit.

At $32$k tokens per request the KV cache, not the compute, limits how many requests a GPU can hold at once, so a smaller cache means more requests in flight and higher throughput. Random Attention serves $1.6$–$2.7\times$ the full-attention throughput across the four models, $32$–$43%$ more than TriAttention on the same kernels (Table 4).[^1] The margin is not specific to this operating point: it holds at short generations, where compression itself does not yet pay, and at lighter loads.

[^1]: These runs sit within $7%$ of the capacity plateau: offering $512$ requests raises throughput by only $7%$ on Qwen3-4B and moves it by under $1%$ on Qwen3-14B; the margin over TriAttention holds at capacity ($+41%$ and $+42%$).

At equal memory every evictor gains, and the scoring pass decides the rest.

A second comparison gives each method the largest batch it can fit on one $143$ GB H200 at $K{=}3072$ with $32$k generations (Figure 5 in Appendix G). Full attention fits $28$ (Qwen3-4B) and $20$ (Qwen3-14B) concurrent sequences; the compressed caches fit $109$–$200$, and that capacity, which every method shares, is where the $3$–$10\times$ speedups come from. The residual ordering is the scoring pass: Random Attention computes no score, fits the largest batch at the smallest peak footprint ($101$ and $89$ GB), and reaches $10.0\times$ and $8.8\times$ full-attention throughput; the ordering is unchanged at $16$k generations, and at the tighter $K{=}1024$ of MATH500 the same protocol reaches $28.8\times$ (Appendix G). The $32$–$43%$ of Table 4 is the margin we claim, while the threefold gap to TriAttention here reflects an unfused re-implementation of its scorer, not the method itself.

Why skipping the scoring pass is worth $32$–$43%$ in serving.

Timed alone, the pass is cheap. One eviction round costs $0.30$ ms under Random Attention, which only compacts the cache, and $1.47$–$1.64$ ms under TriAttention, which scores it first; in a single stream the extra scoring amounts to a few percent of decoding time. Serving multiplies that small cost in two ways. First, the compressions pile up: with $128$ concurrent requests, each compressed every $64$ of its own tokens, vLLM compresses some request at nearly every decoding step (about $62$k times per workload), and it performs each compression at a synchronisation point between batched steps, so all $128$ requests wait while one of them is compressed. Second, content-dependent scoring is more expensive under paged serving than in plain batched decoding. Cache-statistic selectors need an extra pass over the paged KV state, and attention-weight selectors must recompute or explicitly expose attention statistics, because the fused kernels do not materialise them; Random Attention needs neither and only compacts. In our vLLM comparison, TriAttention makes an additional pass over the candidate cached keys, through vLLM's block tables and layer by layer, to compute its score, whereas Random Attention performs only the shared compaction step. On Qwen3-14B, the extra $910$ s that TriAttention's $32$k run takes over Random Attention's amounts to about $15$ ms of whole-batch waiting per compression, against well under a millisecond for Random Attention.

7. Related Work

Section Summary: Research on KV cache compression has largely focused on long-input, short-output scenarios, where techniques such as low-precision storage, attention-based token eviction, and selective querying reduce memory while preserving performance on document-based tasks. In contrast, work on long reasoning chains, which fill the cache during extended generation from short prompts, relies more on eviction policies that score token importance, though results vary depending on which elements like the original prompt are protected. The present setting differs by emphasizing long model-generated traces and showing that simple random eviction can remain competitive when the prompt is always retained.

KV cache for long-context understanding.

A major line of work on the KV cache targets long inputs and short outputs: one or many documents fill the cache at prefill and a short answer follows. Three families reduce its memory. Quantization keeps every token at lower precision ([25, 26]). Eviction discards tokens under a budget ([27, 28, 29]), scored by cumulative attention ([4]), its persistence across steps ([30]), or attention from a window of recent queries ([5]), with budgets adapted per head or per layer ([31, 32]), structural rules that keep the attention sinks and a recent window ([9]), or a policy chosen per head ([33]). Query-aware selection keeps every token and attends to a subset per query ([34]), which saves compute but not memory. Randomness has also been studied in KV cache management. At the serving-system level, [35] randomly evict unmarked leaf tokens from a prefix-sharing cache, making eviction more robust to dynamic or adversarial query arrivals. For long-context compression, [36] sample for coverage of the input, and [37] show, for long-context question answering under a global cache cap, that once the prompt-boundary tokens are guarded the choice of score is second-order and a random policy shares in the recovery. Our setting differs from all three: a short prompt is followed by a long generation, so what fills the cache is the model's own trace, and what must be protected is the whole question rather than its boundaries.

KV cache for long reasoning.

Chains of thought invert the ratio: the prompt is a few hundred tokens and the cache is filled during decoding. Sparse-attention selection has been adapted to this regime ([18, 38]), but it keeps the full cache in memory and attends to part of it, so peak GPU memory still grows with the length of the trace; eviction is the only route that bounds it. Decode-time evictors generally estimate KV importance and use these scores to determine retention: R-KV uses a SnapKV-style score with a redundancy penalty so that restated content is not kept twice ([6]), VaSE scores by value magnitude and fills the remaining budget stochastically ([7]), following earlier score-guided randomized eviction ([39]), and TriAttention scores keys by position through calibrated trigonometric statistics; LazyEviction estimates future importance from recurring attention patterns ([40]). SpeContext ([41]) uses a lightweight distilled-model retrieval head to predict important KV entries, enabling their asynchronous prefetching before the corresponding LLM computation. Concurrently with this work, Prefix Sliding ([42]) keeps only the prompt and a window of recent tokens, the recency+prompt policy of § 5.1, and also explores incorporating KV cache eviction into training, as [43] do. Evaluations of KV compression on reasoning report that eviction hurts chains of thought more than long-context tasks and that random or recency baselines fall far behind scored selection ([22, 21]), and [20] show that protocols differing in what they protect make results across papers hard to compare. We run baselines at matched budget and show that when the prompt is kept, even evicting at random can be comparable to the strongest baseline while being much more efficient.

8. Conclusion

Section Summary: Researchers discovered that trying to rank and selectively keep the most important tokens in a language model's memory cache during reasoning adds almost no value. A basic approach of always retaining the original prompt while randomly discarding other tokens performs just as well as sophisticated methods across multiple models and tasks, while also allowing much higher processing speeds. This works because the prompt is the most vulnerable element and the model's step-by-step thinking tends to repeat key information in redundant ways, so random selection usually preserves what is needed.

KV cache eviction for reasoning has been treated as a ranking problem: estimate which cached tokens will matter and keep those. We find that the ranking contributes almost nothing. A policy that keeps the prompt and evicts uniformly at random within each head matches the strongest baselines across four models and six tasks, and serves $32$–$43%$ higher throughput in vLLM deployment. We explain this by showing that 1) the prompt is the fragile part of the cache, and once every method keeps it, most of the gap between methods disappears; 2) the reasoning trace protects itself through redundancy, in the text and across heads, so once the prompt is safe a random draw keeps enough copies of what the model still needs. What remains for a selection signal is the rare fact stated once and never restated, which reasoning seldom produces.

AI use statement

In this work, we used generative AI tools (LLM-based coding and writing assistants) to assist with drafting and editing the manuscript, implementing analysis and plotting code, and typesetting tables. We have not used generative AI tools to design the experiments, to select or register the hypotheses and thresholds, or to produce any reported measurement. All AI-assisted code and text were reviewed by the authors; reported statistics were recomputed from the released per-instance logs, and every registered prediction and its verdict is recorded in the paper independently of any AI-generated draft. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI.

Acknowledgements

The authors would like to thank Zhiyi Shi and Haodong Wen for helpful discussion.

Appendix

Section Summary: The appendix extends the main experiments by testing an intermediate-scale Qwen3-14B model, where Random Attention again outperforms most baselines on math and science tasks while a few methods show isolated advantages on code or competition problems. It then details a controlled “planted-fact” probe that inserts specific numerical facts into model-generated reasoning traces and tracks their survival under different eviction rules, along with validation steps confirming the setup’s reliability. Finally, it applies prompt-protection controls across additional model–task combinations, demonstrating that safeguarding the original context largely closes performance gaps for methods like SnapKV but leaves some residual differences intact.

A. Generality: additional models

Table 5 repeats the main grid on Qwen3-14B, an intermediate scale within the headline family. The picture from Table 1 replicates: Random Attention beats VaSE and SnapKV on MATH500 and GPQA-D (both significant). Three baseline cells are significantly ahead at this scale: TriAttention on LiveCodeBench ($+2.6$ points, $p{=}.007$), matching its code win on Qwen3-32B and the prompt-length account of § 4.2, TriAttention on MATH500 ($+2.1$ points, $p{=}.02$), and VaSE on AIME ($+2.6$ points, $p{=}.007$), consistent with its nominal AIME edge on Qwen3-32B. Every cell here uses the same rollout count as the corresponding cell of Table 1 ($R{=}2$ on MATH500, $4$ on GPQA-D and LiveCodeBench, $16$ on HMMT; the Random Attention AIME cell pools $32$) for every method.

::: {caption="Table 5: Performance of Qwen3-14B with the same setting as in Table 1."}

:::

B. The planted-fact probe: instrument and metric

Instrument.

Each instance is a short prefilled stub (chat template plus think-opener, 23 tokens) followed by a fully scripted body fed through the decode path token by token: real model-generated MATH500 reasoning as filler (screened so it never contains the planted variable or value), the fact inside a fixed 16-token box, a second fact 256 tokens later (inert unless the cell uses it), and a terminal query whose value tokens are the measurement points. Values are 4-digit numbers, distinct across instances, and exactly four tokens under Qwen3's digit tokenizer. Because every scripted token enters through the true decode path, eviction treats it exactly as generated reasoning, and the model cannot restate the fact on its own. Pinning gives the fact's span a $+\infty$ score in the designated (layer, head) sites and $-\infty$ everywhere else (evicted at first eligibility); background eviction is the standard per-head uniform draw with no protection rule anywhere, and a per-event audit log records the surviving set. Distances are eviction-event counts $E \in {3,15,39,57}$ with per-copy survival $(1024/1088)^{E}$. Four checks validate the instrument: (i) token-by-token forcing matches one-shot prefill within kernel tolerance on every probe tested; (ii) the audit confirms the pinned span survives in exactly the specified sites at every eviction event; (iii) pinning is non-perturbative: keep-everywhere under eviction scores at least as high as no-eviction ($+0.22$ nats); and (iv) the endpoints are far apart (mean value logprob $-0.19$ nats with the fact kept everywhere vs. $-13.59$ deleted everywhere; exact match $0.986$ vs. $0.000$).

Recall metric.

Recall $R$ is defined in Equation 4: the average log-probability gain of the condition over the average available gain, with both endpoints measured per needle. Retrieval leads wherever it separates conditions; $R$ carries the comparison where retrieval floors, as it does on the two-needle probe.

Statistics.

Confidence intervals are instance-clustered bootstrap percentile intervals with $4{,}000$ replicates; every condition is scored on the same instances, so contrasts are paired. Re-running under fresh eviction randomness moves the reported values by one to two points.

C. Matched protection in the remaining settings

::: {caption="Table 6: Matched protection in the settings Table 2 does not cover (LiveCodeBench: pass@1; others: accuracy; AIME pools 2025+2026). Small numbers give the gain from the rule in points, red when at least $2$; bold marks the best protected method in each setting. TriAttention and Random Attention keep the prompt by construction and appear only in the protected column (values as in Table 1)."}

:::

Table 2 gives every method the prompt-protection rule on Qwen3-4B and Phi-4-reasoning for MATH500 and GPQA-D. Table 6 extends the same control to the settings where Table 1 shows the largest remaining deficits: code on both models, competition math on Qwen3-4B, and GPQA-D at 32B. Two facts hold across all fifteen protected model–task settings.

The payoff follows the retention deficit.

Every gain from the rule is ordered by how much of the prompt the score was losing (Appendix D). SnapKV, which retains the least, gains $+35.2$, $+22.1$, $+16.5$, $+13.9$ and $+1.3$ points; VaSE gains $+27.3$, $+3.2$ and $+2.9$ where its retention falls short, and moves by $-2.1$ and $-1.0$ on competition math, where it does not; R-KV, which already keeps most of the prompt, moves by $-1.4$ to $+0.9$ points, essentially zero, in all five settings. The smallest SnapKV gain is on HMMT, where its unprotected deficit is itself small at $32$k-token generations, so there is little to close.

Protection closes most of the gap, not all of it.

Where the deficit was prompt-driven the rule closes it completely: SnapKV and VaSE become statistical ties with Random Attention on code for both models, and SnapKV and VaSE on HMMT. Elsewhere a residual survives protection (paired tests as in § 4.1): all three protected baselines remain $4$–$6$ points below Random Attention on GPQA-D at 32B, where TriAttention ties it exactly; SnapKV, R-KV and VaSE remain $5.4$, $11.1$ and $3.5$ points below on AIME, and R-KV $6.6$ below on HMMT; and R-KV remains $3.9$ and $2.6$ points below on code for both models even though it never lost the prompt, a negative selection effect that mirrors TriAttention's positive one on Qwen3-32B code. Since TriAttention and Random Attention both keep the prompt by construction, every TriAttention-versus-Random Attention cell of Table 1 is already a matched-protection comparison; the confound applies only to the other baselines' deficits. The strongest reading of the 32B GPQA-D setting is the simplest: a uniform draw plus the rule beats every protected learned score there. We have not run the protected grid on Qwen3-14B.

D. Keep-log measurements

The numbers in § 5.1–§ 5.3 that describe what a policy retains come from logging every eviction round of real runs: nineteen policy–cell logs, $16$ traces each, of order $10^{4}$–$10^{5}$ rounds per log, recording for each round the keep-set of every (layer, key-value head) pair together with the age of each retained position. Three quantities are used. Slot coverage is the fraction of candidate positions retained by at least one head immediately after a round; it is $0.999$–$1.000$ for Random Attention and VaSE, $0.993$ for TriAttention, and $0.938$ for the shared-draw control, which by construction has no cross-head diversity. Prompt survival is the fraction of prompt positions still held, by any head and by a given head: $0.994$–$0.999$ and the same per head for Random Attention, against $0.55$–$0.91$ (union) and $0.26$–$0.67$ (per head) for R-KV, $0.56$–$0.70$ and $0.20$–$0.29$ for VaSE, and $0.32$–$0.42$ and $0.11$–$0.22$ for SnapKV, the lowest on every one of the four model–task settings. Survival by age is the fraction of positions in an age band a head still holds (Figure 4); at $1$–$2$k it is $0.188$ for Random Attention, $0.161$ for the shared draw, $0.145$ for SnapKV, $0.105$ for R-KV and $0.086$ for VaSE, and the cross-head union in that band is $0.776$ for Random Attention against $0.368$ for VaSE and $0.161$ for the shared draw, whose union cannot exceed its per-head survival. TriAttention holds $0.127$ per head in that band with a cross-head union of only $0.199$: it pins the prompt in full, and its heads keep nearly the same positions, the least cross-head diversity of any per-head policy measured here.

Implicit age bias.

Although no score is computed, the policy is not age-blind. A position that survives one eviction faces a fresh draw at the next, so its probability of still being cached after $n$ evictions is $\big(\tfrac{K-\ell_{\mathrm{p}}}{K+r-\ell_{\mathrm{p}}}\big)^{n}$, about $0.94^{n}$ at $K{=}1024$ and $r{=}64$. Random Attention is therefore a soft recency window: recent positions are almost always present, and old ones survive as a thin tail that differs from head to head (Figure 4). § 5.1 separates the two ingredients, a hard recency window with the prompt kept and the tail that the scatter adds.

Accuracy of the shared draw.

The most direct task-level test of cross-head diversity is to remove it: the shared-draw control keeps the prompt and draws one random keep-set that every head uses, so its cross-head union equals its per-head survival ($0.161$ in the $1$–$2$k band against $0.776$ for Random Attention). On Qwen3-4B MATH500 it scores $0.871$ at $K{=}1024$ and $0.788$ at $K{=}512$, against $0.874$ and $0.789$ for Random Attention. Removing cross-head diversity therefore costs nothing on real traces at these budgets: the text-level redundancy already keeps a restated copy of what the model still needs, so the second level is not load-bearing there. It is load-bearing where the first level is absent (a fact stated once and never restated), which is the regime of the planted-fact probe (§ 5.2) and of the boundary in § 5.3. The two redundancies are substitutes, and Random Attention preserves both.

**Figure 4:** Fraction of positions of a given age that a head still holds (log scale; Qwen3-4B MATH500, $K{=}1024$). Random Attention decays geometrically with age; VaSE concentrates and freezes a tail of old favourites; TriAttention spends almost uniformly across ages, keeping less of the recent middle than Random Attention but several times more of the very old tail.

E. Generation lengths and run-to-run variability

Table 7 reports the mean number of generated tokens for every cell of Table 1 and Table 5, and Table 8 the standard deviation of per-run accuracy across each cell's independently sampled runs. Two observations are worth noting. First, the budgets of § 4.1 track the full-attention lengths: MATH500 traces are the shortest and competition-math traces the longest on every model, and eviction generally lengthens generation relative to full attention, most for the weakest selectors. Averaged over the five tasks, Random Attention is the shortest-generating evictor on Qwen3-4B and Qwen3-14B and within about $5%$ of the shortest on the other two models, so its accuracy parity is not bought with longer generations. Second, the variability matches the significance treatment in the main text: run-to-run standard deviation is at or under about one point on MATH500, one to three points on GPQA-D and LiveCodeBench, and two to five points on competition math, which is why every claim is decided by the paired, problem-clustered tests of § 4.1 rather than by raw cell differences. MATH500 cells have $R{=}2$, so their standard deviation is a two-sample estimate.

::: {caption="Table 7: Mean generated tokens (thousands) per cell of Table 1 and Table 5, measured over every run of the cell; Avg = unweighted mean over the five tasks."}

:::

::: {caption="Table 8: Run-to-run variability: standard deviation of per-run accuracy (points) across each cell's independent sampled runs, including LiveCodeBench, whose runs are graded individually by test execution."}

:::

F. Engine details

We implement per-KV-head physical eviction on the VaSE engine: every $r{=}64$ decode steps, once the cache exceeds $K{+}r$, each KV head scores its candidate slots (all cached positions except the $r$-token recent buffer), keeps $\operatorname{top-}K$, and the KV tensors are compacted with a gather. Keys are stored post-RoPE with a cumulative-length counter; keep-sets are sorted chronologically before compaction for every method; grouped-query models evict per KV head (query groups share their head's keep-set). Eviction is monotonic. Sampling and prompting follow the reference repository defaults for every method; the model decodes until end-of-sequence, up to a uniform $32$k-token limit.

Baseline configurations.

Every baseline runs in this engine at the same budget, trigger and buffer. VaSE uses $n_{\mathrm{large}}= K/4$, the setting in its released run scripts (a fixed $n_{\mathrm{large}}{=}256$ at larger budgets degrades it toward a recency policy). R-KV uses $\lambda{=}0.5$, the setting behind the R-KV rows in [7], which our cells reproduce to within $2.3$ points on MATH500 and $0.8$ on GPQA-D; the recommended $\lambda{=}0.1$ scores $7$–$9$ points lower on Qwen3-4B in both our port and a line-by-line re-implementation of the official repository. TriAttention is ported verbatim from the official implementation in its stronger per-head variant, with per-model calibration statistics that reproduce the reference ranking (mean reciprocal rank $0.99$); its released harness evaluates reasoning models with the chat template disabled, which is why we do not compare against the paper's reported numbers. Our full-attention path reproduces the full-attention accuracies reported by [7] on Qwen3-4B to within $0.5$ points on all four shared tasks. All cells are graded at the $32$k-token limit by the final boxed answer (LiveCodeBench: by test execution), and no partially completed cell enters any table.

Hardware.

Accuracy generation ran on a mixed fleet of NVIDIA H200 nodes. Every efficiency measurement in § 6 and Appendix G is H200-only, with a single job per GPU and nothing else on the node.

Per-task budgets.

The main grid sets MATH500 $K{=}1024$, GPQA-D $K{=}2048$, AIME/HMMT $K{=}4096$, and LiveCodeBench $K{=}3072$, ${\sim}3\times$ compression of its ${\sim}9$k-token full-attention traces.

G. Efficiency: protocols and additional measurements

The vLLM setting (Table 4).

Every run is bf16 on one H200, vLLM v0.19.0 with PagedAttention (CUDA graphs and prefix caching disabled), budget $2048$ and $1$k-token prompts, inside the compression integration released with TriAttention, which we run unmodified except for the selector: Random Attention is added as a scoring function inside it, so both methods share the same attention kernels, paging, scheduler and compression trigger and differ only in which positions are kept, with a request compressed every $64$ generated tokens. Compression is active throughout every compressed run ($55{,}296$ eviction events at the $8$k point and $61{,}568$ at the $32$k point, identical for the two methods, with no skipped round and no prefix mismatch; $59{,}008$ at the $32$k point for Phi-4-reasoning), and the same integration reproduces our accuracy at the same cadence ($0.864$ on Qwen3-4B MATH500 at $K{=}1024$). The compressed cells are single runs; repeating the benchmark with fresh seeds, two to three per cell in earlier rounds at these load points, moved every arm by at most $1.1%$ (Qwen3-14B Random Attention by $0.05%$), and the full-attention cells, which the plugin never touches, are means of two runs with three repetitions agreeing within $0.4%$; run-to-run variability is therefore about one percent, two orders below the margins we report. Qwen3-32B runs the same protocol; its $64$ GB of weights shrink the free KV pool, which lowers the preemption-safe cap (below) and, at $32$k generations, where a full-length full-attention sequence carries over $8$ GB of KV, drops full-attention throughput to $346$ tok/s. Phi-4-reasoning also runs the same protocol, with $31.5$k-token generations, the most its $32$k context allows after the $1$k-token prompt. The $128$-request point is close to the paged capacity plateau, the regime in which TriAttention report their headline multiple: offering $512$ requests on Qwen3-4B with Random Attention at its preemption-safe ceiling of $224$ resident sequences (verified, zero preemptions) raises steady-state throughput from $2046$ to $2188$ tok/s, $+7%$; the decoding step there fits $9.5$ ms $+ 0.415$ ms per resident sequence, so throughput is already $85%$ batch-proportional at $128$ and cannot rise by more than $18%$ at any batch. On Qwen3-14B the same probe leaves throughput within $1%$ of the $128$-request row ($1817$ vs. $1819$ tok/s). With both methods at the ceiling the margin holds: $2117$ vs. $1501$ tok/s on Qwen3-4B ($+41%$) and $1817$ vs.\ $1276$ on Qwen3-14B ($+42%$), against $+37%$ and $+40%$ at $128$ requests. On Qwen3-32B the $128$-request run already operates at its $96$-sequence ceiling, so no separate probe is needed.

Short generations, where compression does not pay.

A second operating point offers $512$ requests of $1$k in, $8$k out, with the compressed runs at their preemption-safe caps ($224$ on Qwen3-4B and 14B, $96$ on Qwen3-32B) and full attention uncapped. There the arithmetic, not the cache, bounds throughput, and a compressed cache buys nothing: Random Attention serves $0.52\times$, $0.70\times$ and $0.96\times$ the full-attention throughput on Qwen3-4B, 14B and 32B, and $0.76\times$ on Phi-4-reasoning; TriAttention $0.38\times$, $0.49\times$, $0.70\times$ and $0.57\times$. The deficit shrinks with model size because per-token KV grows with the model while the weights shrink the cache pool, so the same workload becomes memory-bound as the model grows, reaching near parity on Qwen3-32B. Phi-4-reasoning, with the parameter count of Qwen3-14B but a quarter more KV per token ($40$ layers of $10$ KV heads against $40$ of $8$), sits between Qwen3-14B and Qwen3-32B, as this account predicts. The margin between the two compressed methods is unchanged there: Random Attention leads TriAttention by $+39%$, $+42%$ and $+36%$ on the Qwen3 models and $+35%$ on Phi-4-reasoning. The $64$-request and single-request measurements keep the same request shape with the offered load reduced to $64$ and $1$. At the $64$-request load Random Attention leads TriAttention by $+35%$ on Qwen3-4B and $+30%$ on Qwen3-14B; at a single request the two are within about one percent of each other ($115.8$ vs. $117.0$ s on Qwen3-4B, $125.8$ vs. $126.9$ s on Qwen3-14B), consistent with the sub-$2$ ms rounds of Table 9: the serving margin is the barrier-multiplied cost of reading paged cache state, not a kernel-time gap, and an integration that scored asynchronously, off the barrier, could shrink it; none is released, and Random Attention needs none. Two pitfalls in the released tooling silently make it measure the wrong thing and are worth recording: vLLM's own throughput benchmark ignores the requested output length (its default $128$-token outputs never reach the compression threshold, so it times full-attention decoding), and the integration's deduplication guard can disable all later compaction after one benign under-budget round. We corrected both with bookkeeping changes that leave selection and kernel semantics untouched; every reported run is verified by its applied-event counters.

TriAttention report ${\sim}2.5\times$ over full attention from a capacity measurement taken at a different budget and decode length, on a different GPU; our $1.2$–$2.0\times$ for their method sits below it, and the comparison we draw is between the two methods we measured rather than against their published figure.

The paged table runs at $K{=}2048$ and the equal-memory table (Table 10) at $3072$; the $32$k point rerun at $K{=}3072$ preserves the ordering (Random Attention $2011$ and $1696$ tok/s against TriAttention's $1437$ and $1223$ on Qwen3-4B and 14B, $+40%$ and $+39%$), so the budget difference between the two tables drives neither comparison.

The concurrency cap.

When the KV pool is oversubscribed, vLLM V1 preempts requests silently, and the integration's compression state does not survive preemption: on resume it compacts inside block IDs the request no longer owns, and the scheduler's consistency check then kills the engine. Capacity measurements therefore need a cap that keeps the pool undersubscribed; ours is $224$ requests on Qwen3-4B and 14B and $96$ on Qwen3-32B at the $8$k capacity point, whose $512$ offered requests keep the pool under sustained admission pressure. The $32$k point offers $128$ requests in total and runs preemption-free at that concurrency on Qwen3-4B and 14B; on Qwen3-32B it is capped at $96$. An uncapped compressed run under sustained oversubscription is not possible in this integration. Both compressed methods run under the identical cap, so the comparison between them is unaffected. Full attention tolerates preemption (vLLM recomputes the evicted request) and runs uncapped, which favours it on the two smaller models; on Qwen3-32B the $512$ offered requests oversubscribe its pool, so part of its deficit there may be preemption overhead rather than capacity alone, and the comparison we rest on at that scale is the cap-matched one against TriAttention. The $32$k full-attention baselines are oversubscribed in the same way on every model, so the full-attention multiples of Table 4 bundle capacity with full attention's preemption overhead; the margins over TriAttention, measured under identical load, are unaffected.

::: {caption="Table 9: Cost of one eviction round (scoring $+$ compaction), measured with CUDA events on an otherwise idle H200: $K{=}1024$, $4096$ decode steps, single stream. Random Attention performs no scoring, so its round time is the compaction floor every evictor pays; the excess over it is the price of the selection signal. The ordering, and the per-call costs to within 12%, are unchanged across a 3.5× change in model size."}

:::

Eviction-round timing (Table 9).

The full round, scoring and compaction, is timed with CUDA events on an otherwise idle node, single stream, $K{=}1024$, $4096$ decode steps, $1872$ ($4$B) and $2080$ ($14$B) eviction calls.

$**Figure 5:** Equal-memory serving: decode throughput relative to full attention at each method's largest batch on one H200 ($K{=}3072$, $32$ k generations). $^*$TriAttention here is an unfused re-implementation of its scorer, far slower than the vLLM version.$

::: {caption="Table 10: Serving throughput when each method runs at the largest batch that fits one $143$ GB H200, at $K{=}3072$ with $32$ k generations. The small cache is what buys the batch, so every evictor collects most of the win; the ordering among them follows the cost of their scoring pass. The TriAttention row is an unfused re-implementation of its scorer, far slower than the authors' kernels, on which the gap to Random Attention on these two models is 1.4×, not $2.7$--3.0× (4). Every other row runs on one shared code path."}

:::

The equal-memory comparison (Table 10, i.e. Figure 5 with exact throughput).

Every method runs on the same engine and code path, so the comparison among them isolates the scoring pass; only the batch differs, found per method by bisecting for the largest that fits a $143$ GB H200. Full attention needs its own search at each decode length (at $14$B and $32$k, batches of $40$, $38$ and $34$ all run out of memory and the largest feasible is $20$, at $129.2$ GB peak). One caveat applies to TriAttention and to no other selector: its row runs our PyTorch port, and their paper states that no optimised kernel exists yet, so the row reflects an unfused implementation rather than the speed TriAttention can attain, which is why the method-level comparison in § 6 is the vLLM one, run on the kernels they release. No throughput number in this paper comes from the batched accuracy runs, where several workers share a GPU.

::: {caption="Table 11: Equal-memory serving at $K{=}1024$ (Qwen3-4B, $32$ k generations, one $143$ GB H200): the tighter budget fits a batch of $584$ against $28$ for full attention, and Random Attention reaches 28.8× full-attention throughput. R-KV and TriAttention were not measured at this budget."}

:::

Equal memory at a tighter budget.

The multiple over full attention is set by the batch the cache admits, so it grows as the budget shrinks. Table 11 repeats the protocol on Qwen3-4B at $K{=}1024$, the MATH500 budget of Table 1: the compressed caches now fit $544$–$584$ sequences against $28$, and Random Attention serves $28.8\times$ the full-attention throughput, $16%$ more than SnapKV and $20%$ more than VaSE at their own largest batches.

Quoting TriAttention's R-KV comparison.

We cite their matched-budget row rather than their headline. Their reported throughput varies almost entirely with the budget and hardly at all with the selector ($1405$, $760$, $564$ and $414$ tok/s at $K{=}1024$, $2048$, $3072$ and $4096$), and the headline $+85%$ over R-KV places the two methods at different budgets ($1024$ for TriAttention, $2048$ for R-KV), so it measures the budget, not the selection signal. At equal budget their table reports $1405.2$ against $1345.5$, which is the $+4.4%$ we quote and the conservative figure for our purposes. That row was measured on an A100 at $16$k decode and maximum batch, neither our hardware nor our operating point, so it combines with our own measurement only as an indication, not as a controlled comparison.

What porting an evictor to a paged runtime involves.

Two settings are necessary because no released runtime hosts every method. TriAttention ships a vLLM v0.19.0 plugin, which is what Table 4 uses. R-KV ships ports as well, but against a different pinned vLLM release, so the two cannot serve in one comparison without re-porting one of them; SnapKV and VaSE are released as HuggingFace-side implementations. The wiring is the larger part of the work and is independent of the score: physical eviction under paging has to rewrite each request's surviving KV into its blocks, free the tail, keep rotary positions logical while the physical slots shrink, and stay correct across preemption; R-KV's released port spans ${\sim}849$ lines of wiring across $13$ upstream files and requires vLLM's V1 model runner, and the integration we serve on mishandles preemption (above). The score then decides what else is needed. A score read from the cache alone (VaSE's value range, TriAttention's calibrated key statistics) costs gathers across the block table. A score read from attention weights (SnapKV, R-KV's importance term, VaSE's attention-proportional fill) cannot be read at all from a fused paged kernel, which never materialises the weights, so it must either be recomputed as an explicit window-query product against every candidate key (chunked, since the transient is $\text{heads} \times \text{window} \times \text{cache}$ per request) or be extracted by modifying the attention kernel itself. Random Attention needs neither: its keep-set is a random permutation of slot indices, so it reads nothing and the runtime's existing compaction path is the whole integration, which is why adding it to TriAttention's plugin took a single function.

Why the two settings report such different multiples.

On Qwen3-14B at $32$k generations, Random Attention runs at $1436$ tok/s here and $1819$ in vLLM, while full attention runs at $164$ here and $925$ there: the compressed method gains ${\sim}1.3\times$ from the better engine and the full-attention baseline gains ${\sim}5.6\times$. Paging is the reason for the asymmetry: reserving a full cache per sequence is exactly the constraint compression relieves, and a paged allocator relieves it too, admitting requests as memory frees rather than capping the batch at $20$. The compressed methods have little left to gain, since their caches are small under either allocator. The two settings also differ in budget ($3072$ here, $2048$ there) and in request count, so this attribution is indicative rather than a controlled decomposition; what it explains is why an unpaged capacity multiple of $8.8\times$ and a paged one of $1.97\times$ are consistent measurements of the same effect.

References

Section Summary: This section compiles a list of academic papers, technical reports, and conference proceedings focused on large language models, with emphasis on reasoning capabilities, reinforcement learning techniques, and efficiency optimizations like KV cache compression. Many entries cover benchmarks for math, code, and graduate-level question answering, alongside methods for handling long contexts and memory constraints in model inference. The works range from arXiv preprints to publications at venues such as NeurIPS, ICLR, and ACL, primarily dated between 2023 and 2026.

[1] DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948.

[2] OpenAI (2024). Learning to Reason with LLMs. https://openai.com/index/learning-to-reason-with-llms.

[3] Marah Abdin et al. (2025). Phi-4-reasoning Technical Report. arXiv preprint arXiv:2504.21318.

[4] Zhang et al. (2023). H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Advances in Neural Information Processing Systems.

[5] Yuhong Li et al. (2024). SnapKV: LLM Knows What You are Looking for Before Generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=poE54GOq2l.

[6] Zefan Cai et al. (2025). R-KV: Redundancy-aware KV Cache Compression for Reasoning Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=2jwAjomEDB.

[7] Ting-Yun Chang et al. (2026). Value-Aware Stochastic KV Cache Eviction for Reasoning Models. arXiv preprint arXiv:2606.03928.

[8] Weian Mao et al. (2026). TriAttention: Efficient Long Reasoning with Trigonometric KV Compression. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=0tgzJK50Jz.

[9] Guangxuan Xiao et al. (2024). Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=NG7sS51zVF.

[10] Tri Dao (2024). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=mZn2Xyh9Ec.

[11] Kwon et al. (2023). Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. pp. 611–626.

[12] An Yang et al. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.

[13] Dan Hendrycks et al. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=7Bywt2mQsCe.

[14] Hunter Lightman et al. (2024). Let's Verify Step by Step. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=v8L0pN6EOi.

[15] David Rein et al. (2024). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling. https://openreview.net/forum?id=Ti67584b98.

[16] Mislav Balunovic et al. (2026). MathArena: Evaluating LLMs on Uncontaminated Math Competitions. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=y0zL9IZxZ7.

[17] Naman Jain et al. (2025). LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=chfJJYC3iL.

[18] Yizhao Gao et al. (2026). Sparse Attention Adaptation for Long Reasoning. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=c5BOcHM6J8.

[19] Han et al. (2024). LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 3991–4008. doi:10.18653/v1/2024.naacl-long.222. https://aclanthology.org/2024.naacl-long.222/.

[20] Chen et al. (2026). The Pitfalls of KV Cache Compression. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 41530–41553. doi:10.18653/v1/2026.acl-long.1926. https://aclanthology.org/2026.acl-long.1926/.

[21] Aojie Yuan et al. (2026). Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning. arXiv preprint arXiv:2605.09490.

[22] Minghui Liu et al. (2025). Hold Onto That Thought: Assessing KV Cache Compression on Reasoning. arXiv preprint arXiv:2512.12008.

[23] Wenhao Wu et al. (2025). Retrieval Head Mechanistically Explains Long-Context Factuality. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=EytBpUGB1Z.

[24] Xiao Wang (2026). How Much Cache Does Reasoning Need? Depth-Cache Tradeoffs in KV-Compressed Transformers. arXiv preprint arXiv:2604.17935.

[25] Zirui Liu et al. (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In Forty-first International Conference on Machine Learning. https://openreview.net/forum?id=L057s2Rq8O.

[26] Coleman Richard Charles Hooper et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=0LXotew9Du.

[27] Xu et al. (2025). RefreshKV: Updating Small KV Cache During Long-form Generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 24878–24893. doi:10.18653/v1/2025.acl-long.1211. https://aclanthology.org/2025.acl-long.1211/.

[28] Junyoung Park et al. (2025). KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=uBaFH7aQnC.

[29] Li et al. (2026). REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 39035–39052. doi:10.18653/v1/2026.acl-long.1811. https://aclanthology.org/2026.acl-long.1811/.

[30] Liu et al. (2023). Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. In Advances in Neural Information Processing Systems.

[31] Yuan Feng et al. (2025). Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=tcisuhGsQZ.

[32] Cai et al. (2024). PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. arXiv preprint arXiv:2406.02069.

[33] Ge et al. (2024). Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. In International Conference on Learning Representations.

[34] Tang et al. (2024). Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference. In International Conference on Machine Learning.

[35] Fangzhou Wu et al. (2026). Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=R7fv5NWfMm.

[36] Shuvendu Roy et al. (2026). Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM. arXiv preprint arXiv:2606.29563.

[37] Gabriel Garcia (2026). Protection Is (Nearly) All You Need: Structural Protection Dominates Scoring in Globally Capped KV Eviction. arXiv preprint arXiv:2605.18053.

[38] Lijie Yang et al. (2026). Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention. In Forty-third International Conference on Machine Learning. https://openreview.net/forum?id=trSWJ99WzS.

[39] Chen et al. (2024). NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7913–7926. doi:10.18653/v1/2024.acl-long.428. https://aclanthology.org/2024.acl-long.428/.

[40] Zhang et al. (2026). LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 36335–36352. doi:10.18653/v1/2026.acl-long.1683. https://aclanthology.org/2026.acl-long.1683/.

[41] Xu et al. (2026). SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs. In Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. pp. 1832–1847. doi:10.1145/3779212.3790224. https://doi.org/10.1145/3779212.3790224.

[42] Muennighoff et al. (2026). Prefix Sliding for Efficient Test-Time Scaling. arXiv preprint arXiv:2608.26070.

[43] Vasilis Kontonis et al. (2026). MEMENTO: Teaching LLMs to Manage Their Context. In Third Conference on Language Modeling. https://openreview.net/forum?id=YaYiQDVsi0.