Unlocking Lossless Speedups in LLMs via Discrete Diffusion cover

📄 Executive Summary

The paper addresses a core inefficiency in large language models: next-token prediction produces high-quality outputs but forces slow, sequential generation that wastes accelerator capacity and inflates latency and training cost, especially for long traces and concurrent agentic workloads.

The work set out to deliver lossless speedups that preserve the exact output distribution of a strong autoregressive model while generating multiple tokens per step.

The authors introduce diffusion-augmented LLMs, called Uno. They keep a full set of autoregressive weights trained with standard next-token prediction and add lightweight LoRA adapters that are trained in a short Diffusion Distillation phase to propose blocks of tokens in parallel. At inference, a new Ψ-Spec sampler draws parallel drafts from the diffusion pathway and verifies them against the frozen autoregressive distribution, accepting the longest valid prefix. The method can be applied either by training both pathways from scratch or by augmenting existing open-weight models such as Qwen3-8B. Throughput is measured on a fixed 1 K / 8 K input-output test at multiple batch sizes, and quality is assessed on agentic tool-use, coding, and long-context reasoning benchmarks.

Across evaluations, Uno achieves up to 3× higher throughput than the base autoregressive model and retains at least 2× speedup at the largest batch size the base model can support. It exceeds the speed-quality frontier of EAGLE-3, DFlash, DiffusionGemma, Nemotron-Labs-Diffusion, and Mercury 2 while matching or surpassing their accuracy. When the same adapters are used during reinforcement-learning post-training, end-to-end training time drops by up to 40 percent. The adapters remain effective after RL updates, with only a 6 percent drop in tokens accepted per step.

These results indicate that practical serving capacity and post-training throughput can be increased materially without sacrificing output quality or requiring separate draft models. The approach therefore offers a direct substitute for speculative decoding in production pipelines and a way to shorten RL rollouts.

Further work should test larger block sizes and tree configurations under production traffic, explore additional denoising steps for inference-time quality gains without lengthening context, and combine the adapters with multi-token prediction heads. The main limitations are that reported speedups depend on chosen sampler hyperparameters and that the distillation phase still requires several billion tokens of data; results at extreme context lengths or on highly specialized domains remain untested. Overall confidence in the core claims is high because the method is shown to be lossless on multiple model scales and the throughput measurements follow standard fixed-length protocols.

📚 What to read before this

DiffusionGemma Technical Report, DiffusionGemma Team et al.. DiffusionGemma provides the specific open-weight discrete diffusion architecture and distillation approach that Uno directly builds upon and extends for lossless acceleration.
Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, Aaron Lou et al. (2024). SEDD establishes the foundational discrete diffusion score-entropy framework that underpins the transition from standard autoregressive LLMs to parallel diffusion-augmented models.

📚 What to read after this

How Transparent is DiffusionGemma?, Joshua Engels et al. (2026). This report investigates the transparency and interpretability implications of text diffusion models like DiffusionGemma and Uno.
Learning from the Self-future: On-policy Self-distillation for dLLMs, Yifu Luo et al. (2026). This work extends diffusion LLMs by introducing on-policy self-distillation to further boost reasoning performance during post-training.

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

Subham Sekhar Sahoo$^{*,\dagger,1}$, Lingjie Chen$^{\dagger,1,2}$, Khiem Pham$^{\dagger,1,3}$, Jonathan Geuter$^{\dagger,1,4}$, Chaitanya Dwivedi$^{1}$, Varad Pimpalkhute$^{1}$, Yash Akhauri$^{1}$, Alexander Moreno$^{1}$, Mikhail Yurochkin$^{1}$, Zhenting Wang$^{1}$, Mostafa Elhoushi$^{5}$, Nolan Dey$^{5}$, Shane Bergsma$^{5}$, Joel Hestness$^{5}$, John Thickstun$^{3}$, Eric Xing$^{1}$, Zhengzhong Liu$^{1}$

$^{1}$Institue of Foundation Models, $^{2}$University of Illinois Urbana-Champaign, $^{3}$Cornell Tech
$^{4}$Harvard University $^{5}$Cerebras Systems
$^{\dagger}$ Core Contributors $^{*}$ Correspondence to [email protected]

Abstract

Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation. To overcome this bottleneck, we introduce diffusion-augmented LLMs, a new class of models that defines an AR model distribution while using diffusion to draw multiple tokens in parallel from that distribution. We decouple the parameters of these models into two sets: AR weights, trained using the standard NTP objective, and lightweight diffusion weights, trained to generate multiple tokens simultaneously. The diffusion weights are learned through a simple Diffusion Distillation phase that adds negligible overhead to existing LLM training pipelines. We also introduce $\Psi$-Spec, a family of samplers that enables lossless acceleration and inference-time scaling at a fixed context length. Unlike speculative decoding, our method requires no separate draft model. Unlike diffusion LLMs (d-LLMs), it accelerates generation without sacrificing the quality of the underlying AR model. The resulting models, called Uno, can be trained from scratch or built by augmenting existing open-weight AR LLMs. Uno achieves higher throughput than leading speculative-decoding methods at every evaluated batch size and delivers up to $3\times$ speedups over the base AR model, including at the largest batch size supported by the device. Notably, our 8B Uno model outperforms the leading open d-LLM, the 26B DiffusionGemma, and the proprietary Mercury 2 across all evaluated benchmarks in agentic tool use, coding, and long-context reasoning. We release code and checkpoints on: https://s-sahoo.com/uno

Executive Summary: The paper addresses a core inefficiency in large language models: next-token prediction produces high-quality outputs but forces slow, sequential generation that wastes accelerator capacity and inflates latency and training cost, especially for long traces and concurrent agentic workloads.

The work set out to deliver lossless speedups that preserve the exact output distribution of a strong autoregressive model while generating multiple tokens per step.

The authors introduce diffusion-augmented LLMs, called Uno. They keep a full set of autoregressive weights trained with standard next-token prediction and add lightweight LoRA adapters that are trained in a short Diffusion Distillation phase to propose blocks of tokens in parallel. At inference, a new Ψ-Spec sampler draws parallel drafts from the diffusion pathway and verifies them against the frozen autoregressive distribution, accepting the longest valid prefix. The method can be applied either by training both pathways from scratch or by augmenting existing open-weight models such as Qwen3-8B. Throughput is measured on a fixed 1 K / 8 K input-output test at multiple batch sizes, and quality is assessed on agentic tool-use, coding, and long-context reasoning benchmarks.

Across evaluations, Uno achieves up to 3× higher throughput than the base autoregressive model and retains at least 2× speedup at the largest batch size the base model can support. It exceeds the speed-quality frontier of EAGLE-3, DFlash, DiffusionGemma, Nemotron-Labs-Diffusion, and Mercury 2 while matching or surpassing their accuracy. When the same adapters are used during reinforcement-learning post-training, end-to-end training time drops by up to 40 percent. The adapters remain effective after RL updates, with only a 6 percent drop in tokens accepted per step.

These results indicate that practical serving capacity and post-training throughput can be increased materially without sacrificing output quality or requiring separate draft models. The approach therefore offers a direct substitute for speculative decoding in production pipelines and a way to shorten RL rollouts.

Further work should test larger block sizes and tree configurations under production traffic, explore additional denoising steps for inference-time quality gains without lengthening context, and combine the adapters with multi-token prediction heads. The main limitations are that reported speedups depend on chosen sampler hyperparameters and that the distillation phase still requires several billion tokens of data; results at extreme context lengths or on highly specialized domains remain untested. Overall confidence in the core claims is high because the method is shown to be lossless on multiple model scales and the throughput measurements follow standard fixed-length protocols.

1. Introduction

Section Summary: Large language models generate text one token at a time, which makes them slow and inefficient for long outputs even though much of language is predictable and could be produced in blocks. Current fixes such as speculative decoding or diffusion-based models either require extra models, lose quality, or lose their speed advantage when many requests run together, which is how most real systems operate. The paper introduces Uno, a single model that keeps its original high-quality weights but adds small diffusion weights so it can safely predict multiple tokens in parallel, delivering faster generation and training that holds up at practical batch sizes.

Large language models (LLMs) are increasingly showing human-level capabilities across coding, mathematics, and complex reasoning ([1, 2]). These capabilities arise from a simple yet highly effective training objective: next-token prediction (NTP) over massive text corpora ([3]). A key limitation of this objective is that inference is inefficient. By training models to predict only the next token, it produces autoregressive (AR) systems that generate just one token per decoding step. This sequential constraint becomes increasingly costly as reasoning traces grow longer, increasing serving latency and slowing reinforcement-learning (RL) post-training, where rollout generation dominates runtime ([4]).

Sequential decoding is poorly suited to both natural language and modern accelerators. Language contains predictable collocations and formulaic sequences that could be generated together in blocks ([5, 6]), but standard LLMs cannot exploit this redundancy. Moreover, decoding is often memory bound, especially at long context lengths, because moving model weights and key-value states limits inference speed and leaves GPUs underutilized ([7]). Predicting multiple tokens per step can amortize these memory transfers and better use parallel compute.

Existing approaches only partially address this opportunity. Speculative decoding accelerates generation by verifying tokens proposed by a smaller draft model ([8, 9]), but its gains depend on an efficient, well-aligned drafter and require maintaining a separate model. Discrete diffusion models ([10, 11]) support parallel generation natively and have succeeded in biological domains ([12, 13, 14, 15]), yet leading d-LLMs (d-LLMs) still face a quality-speed tradeoff relative to AR models ([16, 17, 18, 19]). Their speedups also vanish at large inference batch sizes ([20, 21, 19]). The (lossy) speedups offered by d-LLMs at low batch sizes can therefore be insufficient in practice, as agentic workloads now predominate in contemporary LLM applications ([22, 23]). In these workloads, a single request can trigger parallel agents, branching trajectories, tool calls, and retries, while serving systems batch calls across requests to improve accelerator utilization. Batch-size-one latency therefore captures a narrow operating regime and may overstate speedups that diminish under concurrency. Thus, practical acceleration should be evaluated at realistic batch sizes.

Our core idea is simple: define a high-quality AR distribution, then learn to sample multiple tokens in parallel from that same distribution. We realize this idea through diffusion-augmented LLMs (Section 3), which use a single architecture with two decoupled sets of weights: one governing response quality and the other optimized for generation speed. Unlike multi-token prediction (MTP) approaches ([24, 25]), which modify the LLM architecture by adding prediction heads for future tokens, our method augments each layer in the LLM with lightweight diffusion weights alongside the standard AR weights. Our training pipeline first trains the AR weights through NTP loss, then freezes them and learns the diffusion weights through Diffusion Distillation (Section 3) to generate token blocks in parallel with negligible training overhead. Our $\Psi$-Spec sampler (Section 4) performs parallel prediction from the AR distribution. The resulting model is a drop-in alternative to speculative decoding and self-speculative decoding, requiring neither a separate draft model nor lossy AR-to-diffusion conversion.

We call our models Uno because they unify AR and diffusion weights within one architecture. Across our evaluations, Uno improves the speed-quality frontier over the speculative-decoding methods EAGLE-3 ([26]) and DFlash ([27]), the open-weight d-LLMs Nemotron-Labs-Diffusion ([21]) and DiffusionGemma ([19]), and the proprietary d-LLM Mercury 2 ([28]). Its speedup over the base AR model persists at every evaluated batch size, including the largest batch that fits on the device with the AR weights, supporting both low-latency generation and high-throughput serving. Our contributions are:

We introduce diffusion-augmented LLMs, which decouple the parameters that determine response quality from lightweight parameters optimized for generation speed, together with a corresponding training pipeline (Section 3).
We propose $\Psi$-Spec samplers to sample from diffusion-augmented LLMs. These samplers enable both lossless, AR-verified acceleration and inference-time scaling through additional denoising at a fixed context length; see Section 4.
We show how to train diffusion-augmented LLMs from scratch (Section 5.1) or create them by augmenting an existing open-weight AR LLM (Section 5.2).
We show that Uno achieves up to a $2\times$ speedup at the largest batch size supported by the base AR model, accelerating both inference and end-to-end RL training (Section 5.1).

2. Background

Section Summary: The background section first introduces mathematical notation for representing sequences of discrete tokens and probability distributions over them. It then describes autoregressive language models, which generate text sequentially one token at a time according to the chain rule, along with speculative decoding, a technique that accelerates inference by using a smaller draft model to propose candidate tokens that a larger model verifies in parallel. The section also covers discrete diffusion models, which gradually corrupt clean data into noise and then learn to reverse the process through iterative denoising, including uniform-state variants, specialized samplers that incorporate correction steps, and consistency distillation methods that compress the multi-step process into faster few-step generation.

Notation

We denote scalar discrete random variables with $K$ categories as 'one-hot' column vectors and define $\mathcal{V} = {\mathbf{v} \in {0, 1}^K: \sum_{i=1}^K \mathbf{v}i = 1}$ as the set of all such vectors. Define $\text{Cat}(\cdot\ ;\ \boldsymbol{\mathit{\pi}})$ as the categorical distribution over $K$ classes with probabilities given by $\boldsymbol{\mathit{\pi}} \in \Delta$, where $\Delta$ denotes the $K-1$-simplex. We also assume that the $K$-th category corresponds to a special [MASK] token and let ${\mathbf m} \in \mathcal{V}$ be the one-hot vector for this mask, i.e., ${\mathbf m}{K} = 1$. Additionally, let ${\bm{1}} = {1}^{K}$, and $\langle \mathbf{a}, \mathbf{b} \rangle$ and $\mathbf{a} \odot \mathbf{b}$ respectively denote the dot and Hadamard products between two vectors $\mathbf{a}$ and $\mathbf{b}$. We use $\times \in \mathcal{V}^L$ to represent clean data which is a length- $L$ sequence containing no mask tokens, and let $\times^{{\ell}}$ denote its $\ell^{\text{th}}$ element where $\ell$ refers to the token index. Under our notation, each $\times^{{\ell}}$ is a one-hot vector. The notation $\times^{m:n}$ denotes the subsequence from index $m$ to index $n$, inclusive, while $\times^{<\ell} \coloneqq \times^{1:\ell-1}$ denotes the prefix consisting of the first $\ell-1$ tokens. Finally, $[\cdot, \cdot]: \mathcal{V}^m \times \mathcal{V}^n \to \mathcal{V}^{m+n}$ denotes the operator that concatenates sequences of lengths $m$ and $n$. Colors are used throughout the paper solely for emphasis and carry no mathematical meaning.

2.1 Autoregressive Models

Consider a length- $L$ sequence $\times \in \mathcal{V}^L \sim q_{\text{data}}$ drawn from the data distribution $q_{\text{data}}$. Autoregressive (AR) language models factorize the joint distribution using the chain rule:

$ \log p_\theta(\times) ;=; \sum_{\ell=1}^{L} \log p_\theta(\times^\ell \mid \times^{<\ell}) .\tag{1} $

Here, $p_\theta: \mathcal{V}^{L} \to \Delta^{L}$ is typically implemented with a causal Transformer ([29]) with parameters $\theta$. This factorization leads to strong likelihood modeling and enables efficient inference primitives such as KV caching. However, AR generation is intrinsically sequential: $\ell^\text{th}$ token can only be generated after the prefix $\times^{<\ell}$ has been generated. As a result, producing an output of length $L$ requires $L$ causal decoding decisions.

Speculative Decoding

Speculative decoding ([8, 9]) uses a smaller draft model to autoregressively generate a block of candidate tokens, while a larger base model verifies them. The key observation is that although sampling from the base model is inherently sequential and expensive, evaluating the likelihood of an entire candidate sequence can be performed efficiently in parallel. Speculative decoding exploits this asymmetry by generating candidate tokens with the draft model that can be sampled more efficiently, and verifying them via rejection sampling in a single forward pass, accepting the longest valid prefix. This accelerates inference while exactly preserving the target distribution.

2.2 Discrete Diffusion Language Models

Discrete diffusion corrupts a clean sequence $\times \in \mathcal{V}^L \sim q_{\mathrm{data}}$ into a simple prior $\boldsymbol{\mathit{\pi}}^L$ and learns to reverse the corruption. We use the interpolating forward process from [30]:

$ {\mathbf z}t^{{ {\ell}}}\sim q_t^{{ {\ell}}}(\cdot\mid \times;{\boldsymbol{\mathit{\pi}}}) := \text{Cat}!\left(\cdot\ ;\ {\alpha{t}}\times^{ {\ell}}+(1-{\alpha_{t}})\boldsymbol{\mathit{\pi}}\right),\tag{2} $

where ${\mathbf z}t$ denotes the noisy sequence at time step $t\in[0, 1]$, and the noise schedule ${\alpha{t}}$ decreases monotonically from $\alpha_0=1$ at clean data to $\alpha_1=0$ at the prior. A denoising model $\mathbf {x}_\theta: \mathcal{V}^L\times[0, 1]\rightarrow\Delta^L$, parameterized by $\theta$, predicts the clean tokens from ${\mathbf z}_t$ and $t$. Masked diffusion uses $\boldsymbol{\mathit{\pi}}= {\mathbf m}$, while uniform-state diffusion uses $\boldsymbol{\mathit{\pi}}= {\bm{1}}/K$. We use the latter because it natively supports self-correction, few-step generation ([31]), and better inference-time scaling than masked diffusion models ([32]).

Sampling

For $0\leq s<t$, let $q_{s\mid t}$ denote the reverse posterior that maps ${\mathbf z}_t$ to a less noisy state ${\mathbf z}_s$. The $\Psi$-samplers proposed by [32] form a family of discrete-diffusion samplers that generalize posterior (ancestral) sampling by incorporating predictor-corrector capabilities, thereby producing higher-quality samples. Their practical token-wise transition kernel is given by

$ [\Psi^\theta_{s\mid t}(\cdot\mid {\mathbf z}t; \mathbf {x}\theta({\mathbf z}t), {\boldsymbol{\mathit{\pi}}})]^{{\ell}} =\kappa_t q{s\mid t}^{{\ell}}(\cdot\mid {\mathbf z}t, \mathbf {x}\theta({\mathbf z}t);{\boldsymbol{\mathit{\pi}}}) +(1-\kappa_t)\left[\alpha_s q{0\mid t}^{{\ell}}(\cdot\mid {\mathbf z}t, \mathbf {x}\theta({\mathbf z}_t);{\boldsymbol{\mathit{\pi}}}) +(1-\alpha_s){\boldsymbol{\mathit{\pi}}} \right],\tag{3} $

where $\kappa_t\in[0, 1]$ controls the strength of the correction. The exact functional form of Equation (3) for MDMs and USDMs, see Suppl. Appendix A.4.

Discrete Consistency Distillation

Discrete Consistency Distillation (DCD) compresses a multistep uniform-state diffusion process into a few-step generator ([31]). It constructs a deterministic discrete trajectory from ${\mathbf z}_1 \sim {\bm{1}}^L / K$ to $\times$ using the Gaussian diffusion process underlying the uniform-state discrete diffusion process, . For adjacent states ${\mathbf z}_t$ and ${\mathbf z}s$, with $s<t$, the student distribution $\times{\bm \theta}^\ell({\mathbf z}t, t)$ is trained to match the teacher distribution $\times{\bm \theta_0}^\ell({\mathbf z}_s, s); \forall \ell \in [L]$:

$ \mathcal{L}{\mathrm{DCD}}(\bm \theta; \bm \theta_0) =\sum{\ell=1}^{L} \text{D}{\text{KL}}!\left(\times{\bm \theta}^\ell({\mathbf z}t, t), |, \times{\bm \theta_0}^\ell({\mathbf z}_s, s)\right).\tag{4} $

Increasing the gap between $s$ and $t$ across distillation rounds teaches the student to take larger denoising steps. Refer Suppl. Appendix A.5 for further details.

3. Diffusion-augmented LLMs

Section Summary: The framework augments a standard autoregressive LLM by adding a separate set of diffusion weights to each layer, allowing parallel token drafting while the original autoregressive weights continue to control output quality and verification. The diffusion weights are trained in a dedicated distillation phase, often as lightweight LoRA adapters on frozen base parameters, so the model can propose blocks of tokens at once and then check them against the autoregressive distribution. This separation preserves the original output exactly, enabling faster inference on existing models without any loss in response quality.

We propose a new framework for designing LLMs that decouples generation quality from generation speed by augmenting each layer of a standard AR model with a separate set of diffusion weights dedicated to parallel token generation. We call the resulting model a diffusion-augmented LLM. The key distinction is that each layer contains two sets of weights: AR weights, trained to generate tokens autoregressively and determine response quality, and diffusion weights, trained to generate tokens in parallel. This separation preserves the established AR training pipeline while introducing a dedicated phase for training the diffusion weights to accelerate inference. At generation time, both sets of weights draft tokens in parallel, after which the AR weights alone verify the drafts. Our sampler provably preserves the AR model's output distribution (Section 4), enabling lossless acceleration without sacrificing response quality. The framework can be applied to any causal autoregressive neural network, including causal Transformers ([29]) and State-Space Models (SSMs; [33], [33]; [34], [34]). Because of this separation, the framework can also augment existing open-weight LLMs with parallel generation capabilities, providing lossless speedups without retraining their base parameters (Section 5.2).

The remainder of this section develops the framework in three stages. We first describe the roles of the AR and diffusion weights in a Diffusion-augmented LLM (Section 3.1). We then present the Diffusion Distillation Phase (Section 3.2), which trains the diffusion weights using a one-step block-denoising formulation and associated training objectives. Finally, we discuss the training curriculum for two settings: accelerating inference alone and accelerating both RL rollouts and inference (Section 3.3).

3.1 Autoregressive and Diffusion Pathways

Each layer of a diffusion-augmented LLM contains a set of autoregressive (AR) weights and a set of diffusion weights. These two sets of weights follow different training procedures, as described below.

Autoregressive Weights

We denote the base AR weights by ${\theta_\text{AR}}$. These parameters are trained using the standard LLM recipe of Pre-training ([3, 35, 36]), supervised fine-tuning (SFT; [37], [37]; [38], [38]), and RL post-training ([39, 40, 41]). Given a length- $L$ sequence $\times \in \mathcal{V}^L$, the model $\times_{{\theta_\text{AR}}}: \mathcal{V}^L \to \Delta^L$ is causally masked, so its output at position $\ell$ observes only the prefix $\times^{< {\ell}}$, the distribution of the $\ell^\text{th}$ token is modeled as:

$ p_{{\theta_\text{AR}}}(\times^{{\ell}} \mid \times^{< {\ell}}) = \times^{\ell - 1}{{\theta\text{AR}}}(\times), \qquad 1 < \ell \leq L.\tag{5} $

We assume that the first token in $\times$ is a special $\texttt{\textless{}BOS\textgreater{}}$ token that requires no modeling.

Diffusion Weights

The diffusion weights ${\theta_{\Delta}}$ are used solely to accelerate generation. As described in Section 4, our sampler uses the diffusion pathway to draft tokens and performs rejection sampling against the AR distribution defined by ${\theta_\text{AR}}$. We parameterize the diffusion weights as LoRA (Low RAnk) adapters ([42]): each AR weight matrix has an associated LoRA weight. Thus, the diffusion pathway uses ${\theta_\text{AR}}+{\theta_{\Delta}}$ to generate draft tokens, whereas the verification pathway uses ${\theta_\text{AR}}$, with the two distributions coupled through their shared base parameters. We train only ${\theta_{\Delta}}$ during the Diffusion Distillation Phase while keeping ${\theta_\text{AR}}$ frozen; see Section 3.2.

For effective rejection sampling, the draft distribution must remain closely coupled to the verification distribution. Parameterizing the draft pathway as a LoRA adaptation of the base model promotes this coupling while adding minimal inference-time memory overhead compared with a separate set of full-sized diffusion weights. This design retains the lossless verification of standard speculative decoding without requiring a separate draft model. At the same time, like self-speculative decoding, it tightly couples the drafting and verification pathways, but does so while remaining lossless..

Diffusion training teaches the LLM to refine an entire sequence in parallel. Specifically, given a noisy sequence $({\mathbf z}t \in \mathcal{V}^L){t \in [0, 1]}$ obtained by corrupting a clean sequence $\times \in \mathcal{V}^L$ according to (2), the model is trained to recover $\times$. Throughout this work, the LLM retains the NTP parameterization of standard autoregressive LLMs: the logits at each position predict the subsequent clean token. This parameterization differs from that of conventional diffusion language models ([10, 43, 12]), which typically predict the clean token at the same position. Accordingly, the distribution of the clean token $\times^{{\ell}}$ is:

$ p_{{\theta_\text{AR}}, {\theta_{\Delta}}}(\times^{{\ell}} \mid {\mathbf z}t) = \times^{\ell - 1}{{\theta_\text{AR}}, {\theta_{\Delta}}}({\mathbf z}_t), \qquad 1 < \ell \leq L. $

3.2 Diffusion Distillation Phase

We train the diffusion parameters ${\theta_{\Delta}}$ to draft a block of tokens in parallel that the autoregressive (AR) model with parameters ${\theta_\text{AR}}$ would generate sequentially. Specifically, our goal is to match the single-step diffusion distribution $p_{{\theta_\text{AR}}, {\theta_{\Delta}}}(\cdot)$ over a token block to the AR distribution $p_{{\theta_\text{AR}}}(\cdot)$ over the same block. Discrete Consistency Distillation (DCD; Section 2.2) is well suited to this task because it can distill a multistep diffusion process into a few-step generator while closely approximating the distribution induced by the original process.

In what follows, we first describe how we adapt DCD to approximate the AR distribution through single-step diffusion. We then extend the DCD training objective, namely the single-step distillation loss $\mathcal{L}{\text{DCD}}$, with another auxiliary loss term: $\mathcal{L}{\text{TV}}$ to encourage longer diffusion drafts to be accepted by the AR verifier. The resulting objective is:

$ \begin{aligned} & {\mathcal{L}}({\theta_{\Delta}}; {\theta_\text{AR}}, \alpha, \beta) = \mathbb{E}{\times \sim {\mathcal{D}}, {\mathbf z}1 \sim \boldsymbol{\mathit{\pi}}^L} \left[\alpha {\mathcal{L}{\text{DCD}}({\theta{\Delta}}; {\theta_\text{AR}}, \times, {\mathbf z}_1)}

\beta {\mathcal{L}{\text{TV}}({\theta{\Delta}}; {\theta_\text{AR}}, \times, {\mathbf z}_1)}\right], \end{aligned}\tag{6} $

where ${\mathbf z}_1$ is the fully corrupt sequence. Although $\alpha=0, \beta=1$ yields the largest speedups, training first with $\alpha=\beta=1$ and then switching to $\alpha=0, \beta=1$ accelerates convergence. We ablate both loss terms in Section 5.2.4. We describe each term below.

One-step Distillation

Standard DCD constructs a multistep denoising trajectory by simulating intermediate states of the underlying PF-ODE. At LLM scale, repeatedly constructing and storing these intermediate Gaussian latents is prohibitively expensive. We therefore distill the entire trajectory into a single denoising step that maps a fully corrupted input ${\mathbf z}1 \sim \boldsymbol{\mathit{\pi}}^L$ directly to the clean sequence $\times$. The frozen base model defines the autoregressive teacher distribution $\times{{\theta_\text{AR}}}$, and the adapted model defines the student distribution $\times_{{\theta_\text{AR}}, {\theta_{\Delta}}}$. We train the student to match the teacher directly, eliminating the need to simulate or materialize any intermediate PF-ODE states.

Block Diffusion

Recovering an entire long sequence from a fully corrupted input in a single denoising step is prohibitively difficult; we therefore perform one-step denoising blockwise. We partition the clean sequence $\times$ and the fully corrupt sequence ${\mathbf z}_1$ into $N$ blocks of size $B$. We use $\times^{(b)}$ and ${\mathbf z}_1^{(b)}$ to denote the $b^\text{th}$ blocks of $\times$ and ${\mathbf z}1$, respectively. For each block, we train ${\theta{\Delta}}$ to match the student diffusion distribution over the clean block $\times^{(b)}$, conditioned on the noisy block ${\mathbf z}_1^{(b)}$ and the preceding clean context $\times^{(<b)}$, to the teacher AR distribution over the same block, conditioned on the preceding clean context. We compute the teacher and student predictions in a single forward pass over the concatenated sequence $[\times, {\mathbf z}_1]$. Specifically, we use a block-causal attention mask that permits causal attention within $\times$ and within each noisy block ${\mathbf z}_1^{(b)}$. Tokens in ${\mathbf z}_1^{(b)}$ additionally attend to all preceding clean blocks $\times^{(<b)}$.

The main challenge is to compute the teacher logits at positions in $\times$ using only ${\theta_\text{AR}}$, while computing the student logits at positions in ${\mathbf z}1$ using both ${\theta\text{AR}}$ and ${\theta_{\Delta}}$. We achieve this using gated LoRA ([44]), which disables the adapters at clean-sequence positions and enables them at noisy-sequence positions. Consequently, ${\theta_\text{AR}}$ produces the teacher logits, whereas ${\theta_\text{AR}}$ and ${\theta_{\Delta}}$ jointly produce the student logits. The resulting blockwise DCD objective is:

$ \mathcal{L}{\text{DCD}}({\theta{\Delta}}; {\theta_\text{AR}}, \times, {\mathbf z}1) = \sum{b = 1}^N \sum_{\ell = 1}^{B} \text{D}{\text{KL}} \left(\times^{(N + b, \ell)}{{{\theta_{\Delta}}, {\theta_\text{AR}}}} ([\times, {\mathbf z}1]) | \times^{(b, \ell)}{{{\theta_\text{AR}}}} ([\times, {\mathbf z}_1]) \right).\tag{7} $

The base parameters ${\theta_\text{AR}}$ remain frozen throughout fine-tuning, and only the LoRA parameters ${\theta_{\Delta}}$ are updated.

Total Variation Loss

Our sampler (Section 4) performs rejection sampling against the AR distribution and retains only the longest consecutive prefix of the diffusion prediction. Sampling efficiency therefore depends on the length of this retained prefix. To increase its expected length, we minimize the blockwise Total Variation (TV) distance between the diffusion and the AR distributions following Corollary 3.6 of [8]:

$ {\mathcal{L}{\text{TV}}({\theta{\Delta}}; {\theta_\text{AR}}, \times, {\mathbf z}1)} = \sum{b = 1}^N \sum_{\ell = 1}^{B} \left| \times^{(N + b, \ell)}{{{\theta{\Delta}}, {\theta_\text{AR}}}} ([\times, {\mathbf z}1]) - \times^{(b, \ell)}{{{\theta_\text{AR}}}} ([\times, {\mathbf z}_1]) \right|.\tag{8} $

Minimizing this objective increases the probability of consecutive tokens being accepted, and, consequently, increases the expected length of the accepted draft prefix.

3.3 Practical Considerations

The AR weights ${\theta_\text{AR}}$ and diffusion adapters ${\theta_{\Delta}}$ serve complementary roles and are optimized using separate objectives, making their training order an important practical consideration. In particular, Diffusion Distillation can be performed either after all AR training is complete or before RL post-training so that the resulting adapters can accelerate rollout generation. We therefore consider two training regimes, depending on whether diffusion-based generation is used only during inference or during both RL post-training and inference.

Faster Inference Only

If the sole objective is faster inference, we first complete autoregressive pre-training and post-training, or simply initialize them with an Open-weights LLM. We then freeze the AR weights ${\theta_\text{AR}}$ and train ${\theta_{\Delta}}$ using Diffusion Distillation (Section 3.2).

Faster RL Training & Faster Inference

We demonstrate in Section 5 that our method provides significant generation speedups wrt the base AR model across all batch sizes, and can therefore accelerate RL post-training due to faster rollouts. In this setting, we first apply diffusion distillation after supervised fine-tuning and before RL, as illustrated in Figure 1 (top). The resulting adapters can then accelerate rollouts during RL. Standard RL policy-optimization recipes, including PPO and GRPO-based methods ([39, 40, 41, 45]), update ${\theta_\text{AR}}$ but not the ${\theta_{\Delta}}$. One might therefore expect that, as ${\theta_\text{AR}}$ changes, the draft distribution will drift away from the verifier distribution, reducing the acceptance rate and eroding the speedup.

Surprisingly, we show in Section 5.1 that the speedup is retained even as the AR weights are trained using RL.

4. Psi-Speculative Sampler

Section Summary: The Ψ-Speculative sampler generates blocks of multiple tokens in parallel by using a diffusion process to quickly propose candidate sequences from a partially generated output. These proposals are then verified against the base autoregressive model through rejection sampling, which accepts the longest valid prefix and thereby preserves the original model's distribution exactly. The method supports both single-candidate linear sampling and multi-candidate tree sampling to balance speed and verification cost depending on batch size and available compute.

Our goal is to sample from the AR distribution defined by ${\theta_\text{AR}}$ while drawing multiple tokens in parallel. We therefore introduce the $\Psi$-Speculative sampler ($\Psi$-Spec), which uses the diffusion pathway to propose a block of tokens in parallel (Section 4.1) and performs rejection sampling against the base AR distribution to accept the longest valid prefix (Section 4.2).

4.1 Diffusion Sampler

Let $\times \in \mathcal{V}^{L}$ denote a partially generated sequence of length $L$. To generate a block of $B$ tokens, we first append $B-1$ random tokens sampled from the prior, ${\mathbf z}_1 \sim \boldsymbol{\mathit{\pi}}^{B-1}$. Denoising then proceeds from $t=1$ to $t=0$, with ${\mathbf z}t$ denoting the partially denoised sequence at time $t$. At each step, the denoiser $\times{\theta}(\cdot\ ;\ \times^{<L}): \mathcal{V}^{B} \to \Delta^{B}$ operates on the block $[\times^{L}, {\mathbf z}_t] \in \mathcal{V}^B$, while the activations for the preceding sequence $\times^{<L}$ remain in the KV cache. In Section 4.1.1, we define the proposal distribution over a block of tokens, and in Section 4.1.2, we describe how candidates are sampled from this distribution.

4.1.1 Diffusion Proposal Distribution

Given a noisy block ${\mathbf z}t$, we use the $\Psi$-Spec transition in Equation (3) to sample a less noisy block ${\mathbf z}s$, where $s<t$. Because the denoiser uses the NTP parameterization, the clean-token distributions used to construct ${\mathbf z}s$ are given by $\times^{1:B-1}{{\theta\text{AR}}, {\theta{\Delta}}}([\times^{L}, {\mathbf z}_t])$. The input contains both clean and corrupted tokens, while the diffusion weights are trained only on corrupted tokens. To avoid distribution shift, we compute the logits for the first, clean position using only the base AR weights, and those for the noisy positions using both the AR and diffusion weights. We achieve this in a single forward pass using the gated LoRA technique of [44].

Let $\Psi_0$ denote the distribution induced by the $\Psi$-sampler at $t=0$. At the final step, we sample the first token using only the base AR weights:

$ {\mathbf z}0^1 \sim \Psi^{\ell=1}{0} \left(\cdot \mid {\mathbf z}t ; \times^{{ 1:B-1}}{{\theta_\text{AR}}}(\cdot), { \boldsymbol{\mathit{\pi}}} \right).\tag{9} $

We sample the remaining $B-2$ tokens from the following joint distribution, discussed in Section 4.1.2:

$ {\mathbf z}0^{2:B} \sim \prod{\ell = 2}^B \Psi^{\ell}{0} \left(\cdot \mid {\mathbf z}t ; \times^{{ 1:B-1}}{{\theta\text{AR}}, {\theta_{\Delta}}}(\cdot), { \boldsymbol{\mathit{\pi}}} \right).\tag{10} $

4.1.2 Sampling Candidates

Given the proposal distribution in Equation (10), we construct a candidate set $\mathcal{C}={{\mathbf c}: {\mathbf c}\sim \prod_{\ell = 2}^B \Psi^{\ell}_{0} }$. The number of draft candidates determine the tradeoff between acceptance length and verification cost. Sampling more candidates increases the probability of accepting a longer prefix, but also increases the cost of verification. We therefore select the largest candidate set that can be verified without reducing inference throughput, subject to the available serving batch size.

Linear Sampler (System Throughput Optimized)

The simplest approach samples each draft token directly from its marginal distribution in Equation (10), producing a single candidate sequence. At high batch sizes, inference can become compute-bound, leaving little spare compute for verifying additional candidates. The linear sampler is therefore well suited to maximizing aggregate system throughput in this regime.

Tree Sampler (Single-user Throughput Optimized)

At low batch sizes, inference is typically memory-bound, leaving substantial compute capacity underutilized. We exploit this spare compute by sampling multiple candidates and verifying them in parallel. Specifically, we use the tree-based sampling procedure of [24], which selects the top $K$ tokens at each position within a block. Rather than evaluating all $K^{B-1}$ candidate sequences, we adopt the candidate-pruning strategy of [26] which ranks candidates by their log-probabilities and retains the top $V\in\mathbb{Z}^{+}$ prefixes. Thus, $(B, K, V)$ are the hyperparameters of the tree sampler, corresponding to the block size, branching factor, and prefix budget, respectively.

4.2 One-step Diffusion w/ AR Verification

The central goal of this work is to accelerate generation. We therefore draft an entire block of $B$ tokens using a single diffusion forward pass. In Suppl. Appendix B.2, we show that, for single-step diffusion generation, Eqns. (9, Equation 10) reduce to

$ \begin{aligned} & {\mathbf z}0^1 \sim \times^{1}{{\theta_\text{AR}}}(\cdot), \quad\text{(a)} \ & {\mathbf z}0^{2:B} \sim p\text{draft} = \prod_{\ell = 2}^B \times^{\ell}{{\theta\text{AR}}, {\theta_{\Delta}}}(\cdot). \quad\text{(b)} \end{aligned}\tag{11} $

From $p_\text{draft}$, we sample candidates $\mathcal{C}={{\mathbf c}: {\mathbf c}\sim p_\text{draft}}$ and apply the standard speculative-decoding rejection correction ([8]), retaining the longest prefix accepted by the base AR model. This preserves the base model's target distribution. For $|\mathcal{C}|>1$, candidates are verified concurrently as a prefix tree using tree attention ([24]). Because ${\theta_\text{AR}}$ remains frozen and only the diffusion LoRA adapters are trained, the base AR model provides an unchanged verifier and enables lossless speculative speedups. The complete sampling algorithm is provided in Algo. Algorithm 1.

Tokens-Per-Forward-pass (TPF)

During drafting, the first token is generated using the base AR weights and therefore matches the verifier distribution exactly, so it is always accepted. If a subsequent draft token is rejected, the verifier samples a replacement from the renormalized residual distribution, as in standard speculative decoding. Consequently, even an immediate rejection produces two output tokens: the always-accepted first token and the verifier-sampled replacement token. If all $B$ draft tokens are accepted, the verifier additionally samples one token from the logits following the final draft token, producing $B+1$ output tokens. Because each iteration requires two forward passes, one for drafting and one for verification, the TPF is bounded by $1 \leq \mathrm{TPF} \leq \frac{B+1}{2}$.

We provide support for both the Linear and Tree samplers in Nano-vLLM ([46]) and SGLang ([47]). All experiments reported in this paper were conducted using our Nano-vLLM implementation.

4.3 Inference-Time Scaling

$\Psi$-Spec introduces an additional axis for inference-time scaling in LLMs by allowing the number of denoising steps $T$ to exceed the number of drafted tokens $B$. [32] show that, for $0 \leq \kappa_t < 1$, increasing $T$ consistently improves sample quality. The central question is whether this improvement can eventually surpass the quality of AR generation. If so, AR verification should be disabled, since it would constrain the final output to the lower quality level of the AR model and prevent the gains from additional denoising from being realized.

Unlike conventional inference-time scaling methods, this approach allocates additional computation without increasing the context length. We leave a systematic exploration of this quality-compute tradeoff to future work.

5. Experiments

Section Summary: The experiments section describes training diffusion-augmented LLMs in two setups—one building both autoregressive and diffusion components from scratch on matching proprietary data, and another adding diffusion weights via adapters to an existing open model like Qwen3-8B using a different dataset—to demonstrate the approach's flexibility. It outlines a standardized throughput evaluation using fixed input and output lengths to enable fair comparisons across methods, distinguishing per-request performance from overall system throughput at large batch sizes on hardware like the H200 GPU. The section then details the model architecture, training process on trillions of tokens, benchmarks covering agentic and reasoning tasks, and comparisons against baselines such as Nemotron and DiffusionGemma.

We train diffusion-augmented LLMs in two settings. In the first, we train the AR weights from scratch on proprietary data and train the diffusion weights on data drawn from the same distribution. In the second, we augment an open-weight AR model, Qwen3-8B ([48]), without access to its original training data. We instead train the diffusion weights on a different data distribution using the open-source OpenThoughts dataset ([49]). Together, these experiments demonstrate the versatility of our method and show that the diffusion and AR weights need not be trained on the same data distribution.

Throughput Analysis

Throughput depends strongly on context length. However, existing evaluations ([20, 21]) measure throughput on downstream tasks whose reasoning traces vary in length. This can misleadingly favor models that generate shorter traces. Following standard LLM evaluation practice^1, we instead evaluate every method using $m$ random input tokens and a fixed output length of $n$ tokens. For diffusion and speculative decoding methods, which draft a block of $B$ tokens per forward pass, we first measure the average number of accepted tokens per forward pass (TPF) across all benchmarks. We then run $\left \lceil \frac{n}{\mathrm{TPF}}\right \rceil $ decoding steps, constraining each method to accept $\mathrm{TPF}$ tokens per step on average. This ensures that all methods are evaluated at the same input and effective output lengths. Throughout the paper, we use $m=1024$ and $n=8192$ and refer to this setting as the "1K/8K throughput test".

Per-request Throughput vs. System Throughput

We report throughput at batch size 1 and at the largest batch size that fits on a single H200 GPU. Batch-size-1 throughput is less representative of practical serving because agentic workloads often generate concurrent requests even while serving a single user as described in Section 1. We therefore also report system throughput, which is the aggregate token generation rate at the maximum feasible batch size, which better reflects serving capacity and cost efficiency. For each regime, we select the TPF setting that maximizes throughput and use the 1K/8K test described above.

5.1 End to End Diffusion Augmented LLM Training

5.1.1 Setup

Benchmarks

We evaluate on both agentic and non-agentic benchmarks. The agentic suite comprises the Telecom, Airline, and Retail domains of $\tau^3$-Bench ([50]), $\tau^2$-Bench ([51]), Terminal-Bench v2.1 ([52]), and SWE-bench Verified ([53]). The non-agentic suite includes AA-Omniscience ([54]), AA-LCR ([55]), Humanity's Last Exam (HLE; [56], [56]), GPQA-Diamond ([57]), GSM8K ([58]), MATH500 ([59]), AIME 2024 and 2025 ([60]), AIME 2026 ([61]), and MBPP ([62]). We use a context window of $262, 144$ with $131, 072$ maximum generation tokens, allowing for long prompts on agentic benchmarks, and sampling parameters $\text{temp}=1$, top- $p=0.95$, and top- $k=50$.

Metrics

For each method, we report average pass@1 over multiple generations and average TPF; see Table 4 for details. We compute throughput using the "1k/8k throughput test, " averaging TPF across benchmarks. System throughput is measured using the largest batch size that fits on a single H200 GPU, whereas per-request throughput is measured at batch size 1.

Baselines

We compare against Nemotron-Labs-Diffusion-14B ([21]), DiffusionGemma-26B-A4B ([19]), and Mercury 2 ([28]). Nemotron-Labs-Diffusion is pretrained autoregressively on 1T tokens and then fine-tuned as a diffusion model on 300B tokens. It also uses masked diffusion with bidirectional attention within draft blocks. We evaluate the largest Nemotron-Labs-Diffusion variant because it performs better on benchmarks than the smaller variants. DiffusionGemma is a sparse 26B mixture-of-experts model with 4B active parameters, fine-tuned from Gemma 4 ([19]). Since both Nemotron-Labs-Diffusion and DiffusionGemma modify the base AR parameters, they are lossy unlike Uno. We use the default baseline configurations: Nemotron-Labs-Diffusion uses Linear Self-Speculation with a block size of 32, a maximum generation length of 8192, a thinking-token budget of 6000, and temperatures of 0 for drafting and verification. DiffusionGemma uses a block size of 256, temperature 1, a maximum generation length of 131072, a context length of 162144, and an entropy-bounded diffusion sampler with a bound of 0.1 and thinking enabled. We also compare against Inception Labs' closed-source d-LLM, Mercury 2, whose parameter count is undisclosed.

5.1.2 Uno: Diffusion Augmented LLM (Ours)

We first define the architecture of our model. The weights corresponding to every layer are the the AR weights ${\theta_\text{AR}}$. Corresponding to every weight matrix, we introduce rank-128 LoRA adapters with LoRA- $\alpha=256$ which serve as the diffusion weights ${\theta_{\Delta}}$. We share more details subsequently.

LLM Architecture

We use a dense decoder-only causal Transformer with 36 layers, hidden width $4, 096$, SwiGLU-style MLP width $12, 288$, $32$ query heads, $8$ KV heads, head dimension $128$, grouped RMSNorm, RoPE ($\theta=10^7$), a $250, 624$-token vocabulary, and a maximum configured context length of $524, 288$ tokens. The model contains $6.95$ B transformer-body parameters plus approximately $2.05$ B parameters from the large untied input/output vocabulary matrices.

Training AR weights

The AR weights are trained on approximately 23T tokens on internal quality data with a staged context-length extension schedule. Pretraining uses 21.9T tokens at an 8K-token context length, with a peak learning rate of 3 x 10^-4 under a warmup-stable-decay (WSD) configuration and a 3, 000-step warmup. We then extend the context length to 32K, training on an additional 1.1T tokens while linearly decaying the learning rate from 3 x 10^-4 to 3 x 10^-5, following a 1, 250-step warmup. Next, the context length is increased to 128K and the model is trained on 0.5T tokens at a constant learning rate of 3 x 10^-5, with a 500-step warmup. Finally, the context length is extended to 512K and trained for 0.3T tokens with a constant learning rate of 3 x 10^-5 and a 200-step warmup.

Training Diffusion Weights

Diffusion weights are implemented as rank-128 LoRA adapters with LoRA- $\alpha=256$. We train them on 7B tokens sampled randomly from the SFT training data while keeping the AR weights frozen. We employ curricula for context length and diffusion block size: 1.8B tokens at context length $16, 384$, divided into increasing block sizes of two, four, and eight with 600M tokens each; followed by 5.2B tokens at context length $65, 536$ and block size 8. We train on a global batch size of $128$ with a WSD learning rate scheduler with 200 warmup steps and a peak learning rate of 5 x 10^-5. In Equation (6), we set $\alpha=0.01, \beta=1$. Training takes approximately 60 hours on 8 H200 nodes with 8 GPUs each.

To sample from Uno, we use the Linear Sampler with $B=4$ to maximize system throughput and the Tree Sampler with $(B, K, V)=(16, 32, 32)$ to maximize single-request throughput at batch size 1.

5.1.3 Results

$\Psi$-Spec Sampler Configurations

We examine how TPF and throughput vary across Linear and Tree sampler configurations in Table 6. For the Linear sampler, TPF increases from $1.8$ at $B=4$ to $2.2$ at $B=8$, but reaches only $2.3$ at $B=16$; $B=4$ achieves the highest system throughput. For the Tree sampler, with $B=16$ and $K=32$, $V=32$ provides higher per-request throughput than $V=64$, likely because the larger verification cost reduces throughput.

Uno vs. Base AR

Uno qualitatively matches the base AR model while achieving higher throughput at every batch size. In the 1k/8k throughput test, the largest batch size supported by the base AR model is 64, at which Uno is $1.5\times$ faster (see Table 7). This speedup benefits multi-user serving and RL post-training. At batch size 1, Uno is approximately $2.2\times$ faster.

Comparison with Open Weights Diffusion Language Models

As shown in Table 1, Uno outperforms the open-weight DiffusionGemma and Nemotron-Labs-Diffusion models on all tasks, with larger gains on agentic tasks. Using the TPF values in Table 1, we measure maximum system throughput at the largest feasible batch size and per-request throughput at batch size 1. Despite using full attention in every layer, Uno achieves higher system throughput (with Tree Sampler; $(B, K, V)=(16, 32, 32)$) than both baselines. Notably, DiffusionGemma uses strided attention, with five local sliding-window self-attention layers for every global self-attention layer, yet remains slower at large batch sizes. DiffusionGemma is faster at batch size 1 but has substantially lower accuracy, while Nemotron-Labs-Diffusion trails Uno in both throughput and output quality. Although smaller blocks may improve throughput, further tuning offers no practical benefit given the baselines' lower quality. Also, note that both baselines are slower than their respective base AR models at large batch sizes ([21, 19]).

Comparison with Proprietary Diffusion Language Models

As shown in Table 1, the 8B-parameter Uno outperforms Mercury 2 on all Agentic Tool Use, Agentic Coding, and Long-Context Reasoning benchmarks, trailing only on one Science and Knowledge benchmark, potentially due to differences in model size and training data. More importantly, Mercury 2^2 reports a system throughput of 1, 154 tokens/s for a 1K-token input at batch size 10. Uno achieves a maximum system throughput $\sim4.6\times$ higher, despite Mercury 2 running on substantially faster Blackwell GPUs. Moreover, Mercury 2 does not disclose its quantization, so it may also benefit from lower precision, whereas Uno and the other baselines use bfloat16. This result highlights the strength of diffusion-augmented LLMs.

Faster RL Training

From the supervised fine-tuning (SFT) checkpoint, we trained four experts in mathematics, code generation, tool use, and web search using the DAPO reinforcement learning algorithm. During this stage, we updated only the base AR weights; the diffusion weights trained on the SFT checkpoint remained frozen and were used only to speed up RL rollouts. This yielded up to a $40%$ end-to-end training speedup, primarily for the mathematics and code experts. Gains were smaller for the tool-use and search experts because tool calls dominated their runtime. We will provide detailed results in the next revision. We then consolidated the four experts into a single model using ISO-Merger (RAM; [63], [63]), a data-free method that combines specialists trained from a shared base checkpoint without additional rollouts. As shown in Table 8, the diffusion adapters trained on the SFT checkpoint retain their speedups after RL post-training, with only a nominal 6% decrease in TPFs.

::: {caption="Table 1: Accuracy and TPF of Uno, Nemotron-Labs-Diffusion, Mercury 2, and DiffusionGemma on agentic and non-agentic benchmarks. Mercury 2 results are from Artificial Analysis. Results for Nemotron-Labs-Diffusion and DiffusionGemma are computed from their open-source checkpoints (Table 5). For Uno, subscripts report 'TPF_1 / TPF_2, ' where TPF_1 uses the system-throughput-optimal Linear sampler with B=4, and TPF_2 uses (B, K, V)=(16, 32, 32), optimized for per-request throughput. ^*Reported by Artificial Analysis's live tracker on August 30, 2026."}

:::

5.2 Open Weights LLMs

We show that Diffusion-augmented LLMs can initialize their autoregressive (AR) weights from the open-weight Qwen3-8B model while retaining the speedups enabled by diffusion weights trained on a different data distribution, specifically, the open-source OpenThoughts dataset. Although training Qwen3-8B on this dataset degrades its quality (see Table 9), the diffusion weights trained on the same nevertheless enable lossless speedups on the same benchmarks.

5.2.1 Setup

Benchmarks

We evaluate on mathematical reasoning (GSM8K, MATH500, AIME-24, AIME-25, and AIME-26), code generation: HumanEval ([64]), MBPP ([62]), and LiveCodeBench v6 ([65])); Science reasoning: GPQA and GPQA-Diamond ([57]); instruction following: IFEval with prompt-level strict accuracy ([66]), and General Knowledge: MMLU-Pro ([67]).

Lossless Baselines (Speculative Decoding Methods)

Our primary lossless baselines are EAGLE-3 ([26]), which uses an AR drafter, and DFlash ([27]), which uses a diffusion drafter. Both draft models are designed for Qwen3, and we evaluate their open-source checkpoints. EAGLE-3 adds a lightweight $0.40\mathrm{B}$-parameter drafter that sequentially predicts tokens from fused intermediate features of the target model. DFlash instead adds a $1.05\mathrm{B}$-parameter diffusion drafter that predicts multiple tokens in parallel, conditioned on target-model features. Thus, DFlash introduces roughly three times as many parameters as our method. Its training is also substantially more expensive. For block size $B$ and sequence length $L$, DFlash requires a training context length of $B\cdot L$, whereas our method always uses $2\cdot L$, independent of $B$.

This makes DFlash significantly more expensive to train than our method. We evaluate the open-source checkpoints for EAGLE-3 and DFlash; see Table 5. Refer Suppl. Appendix C.1.2 for sampler configurations.

Lossy Baselines (Diffusion Methods)

We also compare against lossy parallel-generation methods. Jacobi Forcing ([68]), SDAR ([69]), OPDLM ([70]), and I-DLM ([71]) are fine-tuned from Qwen3; Fast-dLLM v2 ([20]) is fine-tuned from Qwen2.5; FLARE ([72]) is fine-tuned from Qwen3.5; and LLaDA2.1-Flash ([18]) is trained from scratch. We evaluate the open-source checkpoints for Fast-dLLM v2 and SDAR; see Table 5. OPDLM, I-DLM, and FLARE are concurrent works.

Metrics

We report average pass@1 accuracy over multiple generations for each benchmark; refer Table 4 for details. For speculative decoding methods, one generation step consists of a draft forward pass followed by a verifier forward pass. We report the number of tokens decoded per step, $\tau$, which comprises one draft and one verify step. However, this metric does not account for differences in drafter size and can therefore be misleading, particularly because the baseline drafters are smaller than ours. We consequently measure throughput using the 1K/8K throughput test described earlier. Following standard practice, we do not report accuracy for lossless methods because any differences arise from sampling randomness and numerical nondeterminism. We report $\tau$ under sampler configurations optimized for (1) system throughput and (2) per-request throughput. For each method, we select these configurations through a grid search over sampling hyperparameters; see Table 18. We use thinking mode for all methods. Although DFlash recommends disabling it for greater speedups ([73]), doing so reduces average accuracy from $76.36%$ to $55.40%$; see Suppl. Appendix C.7. We therefore retain thinking mode for a fair comparison. When comparing with lossy methods, we instead report tokens per forward pass (TPF). For our method, $\mathrm{TPF}=\tau/2$ because the drafter and verifier are similar in size. For both drafting and verification, we use temperature $\text{temp}=1$, top- $p=0.95$, and top- $k=50$ unless otherwise specified. We use a context length of $32{,}768$, the native context length of Qwen3-8B, for all benchmarks and methods.

5.2.2 Uno Qwen: Qwen-based Diffusion Augmented LLM (Ours)

Autoregressive Weights

We initialize the AR weights from the open-source Qwen3-8B checkpoint ([48]) and keep them frozen during training.

Training Diffusion Weights

For each AR weight matrix, we add a rank-128 LoRA adapter with $\alpha_{\text{LoRA}}=256$. These adapters introduce $0.35\mathrm{B}$ trainable parameters. We train them for three epochs, corresponding to $14.7\mathrm{B}$ tokens, on OpenThoughts3-1.2M ([49]), using a maximum sequence length of $4{,}096$. We use a block-size curriculum with $B\in{2, 4, 6, 8, 12, 16}$, increasing the block size every half epoch. Training uses a global batch size of $64$ and a constant learning-rate of $10^{-5}$ with 2% warmup steps. We set the loss coefficients $\alpha=0$ and $\beta=1$. Training takes approximately 32 hours on 4 nodes, each with 8 H200 GPUs. For ablations, we train all models for one epoch with $B=8$, denoted Uno $_\text{Qwen}^{\text{1ep}}$ . Unless stated otherwise, we evaluate this variant using the Linear sampler with $B=16$.

5.2.3 Results

Comparison with Lossless Speculative Decoding Methods

$**Figure 2:** System versus per-request throughput across batch sizes (concurrency $C$). Uno Pareto-dominates speculative decoding and achieves up to $2.5\times$ speedup over the base AR model. Parentheses indicate the sampler configuration yielding the highest throughput for Uno. Exact throughputs are in Table 18.$

Table 2 compares the average number of tokens per step, $\tau$, for Uno $\text{Qwen}$, EAGLE-3, and DFlash at $\text{temp}=1$. For each method, we identify the configurations that maximize system and per-request throughput through a grid search over sampling parameters, as detailed in Table 18. At the largest supported batch size, with Linear sampler $B=4$, all methods achieve their highest system throughput. Uno $\text{Qwen}$ exceeds $5700$ tokens per second, outperforming DFlash and EAGLE-3 and achieving a $1.6\times$ speedup over the base AR model. This improvement enables faster RL training and offers practical speedups in LLM inference. For the best per-request throughput (batch size of 1), Uno $\text{Qwen}$ and EAGLE-3 perform best with tree-sampler configurations $(B, K, V)=(16, 32, 32)$ and $(8, 32, 60)$, respectively, while DFlash performs best with $B=16$. Uno $\text{Qwen}$ substantially outperforms both baselines, achieving a $2.5\times$ speedup over the base AR model. It is also Pareto-dominant in throughput across all batch sizes Figure 2. Unlike our method, which shares a single KV cache, EAGLE-3 and DFlash maintain separate caches for drafting and verification, resulting in higher peak memory usage.

Comparison with (Lossy) Diffusion Methods

Table 3 compares Uno $\text{Qwen}$ with the leading lossy diffusion-based acceleration methods for completeness. Unlike our method Uno $\text{Qwen}$, I-DLM, TiDAR, Jacobi Forcing, FLARE, Fast-dLLM-v2, and SDAR achieve lossy speedups relative to their respective parent models. Despite optimizing for quality first, our method achieves a higher TPF than most methods across numerous benchmarks. Benchmarks highlighted in red indicate accuracy degradation. FLARE supports both lossy and target-lossless sampling, but the paper does not state which produced the results.

::: {caption="Table 2: Acceptance lengths (tau), throughput (1K/8K test), peak memory usage, and additional parameter counts for Uno _Qwen, EAGLE-3, and DFlash at sampling temp=1. We report the tau values that maximize system and per-request throughput."}

:::

5.2.4 Ablation

We ablate the main training choices in Table 12 using Uno $_\text{Qwen}^{\text{1ep}}$ with the Linear sampler and $B=16$.

Ablation 1: Loss Terms

The diffusion loss in Equation (6) combines the distillation loss, $\mathcal{L}{\text{DCD}}$, and the total variation loss, $\mathcal{L}{\text{TV}}$. Training with $\mathcal{L}{\text{TV}}$ alone achieves a TPF of $2.39$, outperforming both $\mathcal{L}{\text{DCD}}+\mathcal{L}{\text{TV}}$ ($2.23$) and $\mathcal{L}{\text{DCD}}$ alone ($2.23$). Further analysis revealed that the $\mathcal{L}{\text{DCD}}$ loss was an order of magnitude larger than the $\mathcal{L}{\text{TV}}$ loss. Consequently, reducing the $\mathcal{L}_{\text{DCD}}$ weight to $0.01$ yielded a marginal improvement, increasing the TPF to $2.40$.

Ablation 2: Training Curriculum

In Suppl. Appendix C.6.2, we show that increasing the training block size from 4 to 16 every half epoch improves TPF from $2.65$ to $2.71$, compared with using a fixed block size of 16 for two epochs.

Ablation 3: Diffusion Weights Configuration

In Suppl. Appendix C.6.3, we show that distributing diffusion adapters across all layers is more effective than placing the same number of parameters in only a subset of layers. We also ablate the LoRA rank $r_{\text{LoRA}}$ and scale $\alpha_{\text{LoRA}}$. Increasing $r_{\text{LoRA}}$ from $128$ to $256$ improves TPF, as shown in Table 12, but also increases inference cost. Across the tested values of $\alpha_{\text{LoRA}}/r_{\text{LoRA}}$, a ratio of $64$ performs best.

::: {caption="Table 3: Benchmark accuracy (ACC) and TPF for our lossless method, Uno _Qwen, and color[RGB] 250, 142, 142 lossy diffusion methods. Entries are reported as ACC_TPF. For Uno _Qwen, TPF uses temp=0 and the tree sampler with (B, K, V)=(16, 32, 60). Lossy speedups are taken from the respective papers, except for Fast-dLLM v2 and SDAR, which we evaluated using the provided checkpoints. Accuracy drops relative to the corresponding parent AR model are shaded in red by severity. Jacobi Forcing ^J uses separate math and coding models trained from Qwen2.5-Math-7B-Instruct and Qwen2.5-Coder-7B-Instruct, respectively. ^F Fast-dLLM v2 uses Qwen2.5-7B-Instruct."}

:::

6. Related Work

Section Summary: The section reviews various techniques for speeding up text generation in large language models, including speculative decoding with separate draft models, self-speculative methods that reuse the same model in different modes, diffusion-based approaches, and multi-token prediction architectures. These alternatives often demand extra training, add memory overhead, alter the original model's behavior, or lose effectiveness at larger scales. Uno is positioned as a simpler alternative that avoids these drawbacks while delivering reliable gains without changing the model's outputs.

Speculative Decoding

Speculative decoding methods such as EAGLE-3 and DFlash also provide lossless speedups but require training a separate draft model that is smaller than the target model. Designing a draft model requires numerous choices about its depth, hidden dimension, MLP expansion, attention heads, parameter sharing, and KV projections for target features. [74, 75] on the other hand use large pretrained dLLMs for drafting and separate AR models for verification. These large drafters incur substantial memory and latency overhead, making these methods slower than DFlash. In contrast, Uno uses a single architecture with distinct drafting and verification pathways. Consequently, Uno maintains a single KV cache, introduces fewer additional parameters, and requires less peak inference memory than conventional speculative decoding. These advantages yield larger speedups at high batch sizes, making Uno a drop-in replacement for methods that use separate draft and target models.

Self-Speculative Decoding

[16] introduced a hybrid AR-diffusion framework in which a single denoising model performs either multi-token diffusion generation or AR generation, depending on the available compute budget. This approach substantially outperforms block diffusion ([76]). Building on self-speculative decoding ([77]), [78] extended this framework by using the same denoising model to draft sequences in diffusion mode and verify them via rejection sampling in AR mode, yielding significant qualitative improvements over standalone diffusion generation. However, because this method modifies the base AR model's weights to support diffusion generation, it does not preserve the model's original distribution. Moreover, its speedups are limited to small batch sizes ([21]). In contrast, Uno preserves the base AR model's distribution while providing lossless speedups, making it a drop-in alternative to these methods.

Diffusion Models

Large-scale d-LLMs ([79, 80, 18, 21, 19]) can generate faster than similarly sized AR models at small batch sizes but generally lag behind them in quality. Their lossy speedups also diminish at larger batch sizes, limiting their applicability. Post-training these models is more expensive because rollouts become slower at large batch sizes and existing RL recipes and objectives require nontrivial modifications ([81, 82]). In contrast, Uno provides lossless speedups across batch sizes, uses standard AR post-training algorithms without modification, and substantially accelerates post-training.

Multi-Token Prediction (MTP)

Methods such as Medusa ([24]) and the approach of [25] provide lossless speedups but modify the transformer architecture to predict future tokens. [26] showed that speculative decoding with a separately trained draft model is faster than these approaches. We further show that Uno outperforms speculative decoding methods and, consequently, these MTP methods. Nevertheless, architectural changes for predicting future tokens are complementary to Uno and could improve its draft acceptance rate. We leave this combination to future work.

Quadratic Samplers

Each generation step in our sampler uses two forward passes: one for drafting and one for verification. Quadratic sampling, proposed by [44] and used in TiDAR and Nemotron-Labs-Diffusion, can combine these operations into a single pass. For a draft of $k$ tokens, it inserts $k$ masked placeholders after each draft position, yielding $k^2$ masked positions that represent possible future continuations. This structure verifies the current draft while generating candidates for the next step. Practical speedups require efficient kernels, which we leave to future work.

Comparison with Concurrent work

Concurrent methods such as OPDLM, FLARE, and I-DLM provide lossy speedups relative to the base AR model because they fine-tune its weights; see Table 3. Although I-DLM is lossy, its I-DLM (R-ISD) variant uses gated LoRA. Despite the authors' theoretical argument that its sampler is lossless, we could not reproduce lossless speedups using the released LoRA adapters and sampler implementation. Because this method is compatible with our proposed $\Psi$-spec sampler using a mask prior, we evaluate the combination and show that it achieves lossless speedups, although it remains substantially slower than our method. We provide a detailed comparison with I-DLM, including an evaluation of its official LoRA checkpoint, in Suppl.Appendix C.5.

7. Conclusion

Section Summary: This work introduces diffusion-augmented large language models that combine an autoregressive generation path with a parallel diffusion path inside one unified model. The added diffusion components need far less training data than the original model and can be attached to existing systems to produce faster output without separate draft models or major quality loss. The resulting Uno approach delivers substantial speedups for both everyday inference and reinforcement-learning training while preserving the strengths of standard autoregressive generation.

In this work, we introduced diffusion-augmented LLMs, which unify two distinct generation pathways within a single architecture: an autoregressive pathway parameterized by ${\theta_\text{AR}}$ and a diffusion pathway parameterized by ${\theta_\text{AR}} + {\theta_{\Delta}}$. We showed that training the diffusion weights requires orders of magnitude fewer tokens than the AR weights. This formulation allows Uno either to be trained from scratch or to be created by augmenting an existing AR model with lightweight diffusion weights for faster generation.

Uno serves as a drop-in alternative to speculative diffusion methods such as EAGLE-3 and DFlash, without requiring a separate draft model, and to self-speculative methods such as TiDAR, without introducing lossy speedups. Across the evaluated batch sizes, Uno achieves up to a $3\times$ speedup over the base AR model, while retaining up to a $2\times$ speedup at the largest batch size supported by that model. These gains accelerate both inference and RL post-training, where rollout generation is a major computational bottleneck. More broadly, diffusion-augmented LLMs expand the design space of AR language models by combining the quality of AR generation with the parallelism of diffusion in a unified architecture.

Acknowledgement

We thank A. Feder Cooper, Willis Guo, Rupesh Srivastava, and Matthew Yang for their helpful feedback and insightful discussions.

Appendix

Section Summary: This appendix reviews two families of diffusion models for discrete sequence data. Masked diffusion models gradually replace tokens with a special mask token and derive a reverse process plus a simple weighted denoising loss that cannot revisit or correct earlier choices. Uniform-state diffusion models instead use a uniform noise distribution that lets every token be updated at every step, yielding a more involved NELBO that supports self-correction, and the section briefly notes how self-speculative decoding can exploit a single network for both proposal and verification of future tokens.

A. Background

A.1 Masked Diffusion

MDMs use a masked prior, where $\boldsymbol{\mathit{\pi}} = {\mathbf m} \in \mathcal{V}$ is the one-hot representation of a special [$\textsc{mask}$] token. During the forward process Equation (2), tokens either remain unchanged or transition to the masked state ${\mathbf m}$, after which they stay masked. This behavior carries over to the reverse process. The posterior of the reverse process $q^{\text{MDM}}_{s|t}$ for $0 \leq s < t < 1$ can be derived using Bayes' Rule, and is given by:

$ \begin{aligned} [q^\ell_{s|t}(\cdot\mid {\mathbf z}_t, \times)]^{\text{MDM}} = \begin{cases} \text{Cat}\left(\ \cdot\ ;\ \frac{\alpha_s - \alpha_t}{1 - \alpha_t} \times^{{\ell}} + \frac{1 - \alpha_s}{1 - \alpha_t}{\mathbf z}_t^{{\ell}} \right) & \text{if }{\mathbf z}_t^{{\ell}} = {\mathbf m}\text{,} \ \text{Cat}(\ \cdot\ ;\ \times^{{\ell}}) & \text{otherwise.} \end{cases} \end{aligned}\tag{12} $

The learned reverse posterior is

$ [{p^\theta_{s|t}}({\mathbf z}_s \mid {\mathbf z}t)]^{\text{MDM}} = q^{\text{MDM}}{s|t}({\mathbf z}_s \mid {\mathbf z}t, \times = \times\theta({\mathbf z}_t, t)), $

where $\mathbf {x}\theta: \mathcal{V} ^ L \times [0, 1]\to \Delta^{L}$ is the denoising model. A key limitation is that once unmasked, tokens cannot be remasked. This can create compounding errors during inference, as the denoising model $\mathbf {x}\theta$ imperfectly models the clean data.

NELBO

The likelihood of a sequence under the learned reverse diffusion process is generally intractable to evaluate exactly, as it requires marginalizing over all possible unmasking trajectories. Therefore, typically the negative evidence lower bound (NELBO) is minimized, which is a tractable upper bound on the negative log-likelihood. For continuous-time masked diffusion, the NELBO simplifies to the weighted denoising objective ([30, 83, 84])

$ \mathcal{L}{\text{NELBO}}^{\text{MDM}} = \mathbb{E}{t\sim \mathcal{U}[0, 1], q_t} \frac{{\alpha'{t}}}{1 - {\alpha{t}}} \log \langle \times^\ell_\theta({\mathbf z}_t, t), \times^\ell\rangle.\tag{13} $

A.2 Uniform-State Diffusion Models

We now focus on Uniform-State Diffusion Models (USDMs), which use a uniform prior $\boldsymbol{\mathit{\pi}}= {\bm{1}}/K$ ([10, 31]). Unlike masked diffusion, where tokens are fixed once unmasked, USDMs allow continuous updates of all token positions, which naturally supports self-correction at inference. This capability makes them particularly effective for few-step generation ([31]), inference-time scaling ([32]) and guided sampling ([13]). Given a noisy sequence ${\mathbf z}t$ at time $t$, the Markov property defines the reverse posterior $q{s|t}$ for obtaining a less noisy sequence ${\mathbf z}_s$ ($0 \leq s < t$) by sampling from the following distribution:

$ [{q^{{{\ell}}}{s|t}}(\cdot \mid {\mathbf z}t, \times)]^{\text{USDM}} = \text{Cat}\left(\cdot\ ;\ \frac{K {\alpha{t}} {\mathbf z}t^{{\ell}} \odot \times^{{\ell}} + ({ \alpha{t|s}} - {\alpha{t}}){\mathbf z}t^{{\ell}} + (\alpha_s - \alpha_t)\times^{{\ell}} + (1 - {\alpha{t|s}})(1- \alpha_{s}){\bm{1}} / K}{K {\alpha_{t}}\langle {\mathbf z}t^{{\ell}}, \times^{{\ell}}\rangle + 1 - {\alpha{t}}}\right).\tag{14} $

Similar to the posterior for masked diffusion models above, the true reverse posterior Equation (14) depends on the unknown clean sequence $\times$, hence it is approximated as:

$ [{p^\theta_{s|t}}({\mathbf z}_s \mid {\mathbf z}t)]^{\text{USDM}} = q^{\text{USDM}}{s|t}({\mathbf z}_s \mid {\mathbf z}t, \times = \times\theta({\mathbf z}_t, t)),\tag{15} $

where $\mathbf {x}_\theta: \mathcal{V}^L\times [0, 1] \rightarrow \Delta^L$ is the denoising network parameterized by $\theta$.

NELBO

This posterior induces the following NELBO ([31]):

$ \begin{aligned} \mathcal{L}{\text{NELBO}}^{\text{USDM}}\left({ q}, p\theta; \times \right) & = -\mathbb{E}{t \sim \mathcal{U}[0, 1], q_t} ;\sum{\ell \in [L]}\textit{f}({\mathbf z}t^{{\ell}}, \mathbf {x}\theta^{{\ell}} ({\mathbf z}t^{{\ell}}, t), {\alpha{t}}; \times^{{\ell}}), \end{aligned}\tag{16} $

where

$ \begin{aligned} \textit{f}({\mathbf z}t^{{\ell}}, \mathbf {x}\theta^{{\ell}}({\mathbf z}t^{{\ell}}, t), & {\alpha{t}}; \times^{{\ell}}) = \frac{{\alpha_{t}}'}{K {\alpha_{t}}}\Bigg[\frac{K}{\bar{\mathbf{x}}^{{\ell}}r} - \frac{K}{(\bar{\mathbf{x}}\theta^{{\ell}})r} \nonumber - \left(\zeta_t \mathbf{1}{{\mathbf z}t^{{\ell}} = \times^{{\ell}}} + \mathbf{1}{{\mathbf z}t^{{\ell}} \neq \times^{{\ell}}}\right)\sum{j} \log\frac{(\bar{\mathbf{x}}^{{\ell}}\theta)r}{(\bar{\mathbf{x}}^{{\ell}}\theta)j} \ & - K\frac{{\alpha{t}}}{1 - {\alpha{t}}} \log \frac{(\bar{\mathbf{x}}^{{\ell}}_\theta)r}{(\bar{\mathbf{x}}^{{\ell}}\theta)i}\mathbf{1}{{\mathbf z}t^{{\ell}} \neq \times^{{\ell}}} - \left((K - 1)\zeta_t \mathbf{1}{{\mathbf z}t^{{\ell}} = \times^{{\ell}}} - \frac{1}{\zeta_t}\mathbf{1}{{\mathbf z}_t^{{\ell}} \neq \times^{{\ell}}}\right)\log \zeta_t \Bigg]. \end{aligned} $

Here, $\bar{\mathbf{x}}^{{\ell}} = K\alpha_t \times^{{\ell}} + (1 - \alpha_t){\bm{1}}$, $\bar{\mathbf{x}}^{{\ell}}\theta = K\alpha_t \mathbf {x}\theta^{{\ell}}({\mathbf z}t, t) + (1 - \alpha_t){\bm{1}}$, $\alpha_t'$ denotes the time derivative of $\alpha_t$, $r = \arg\max{j\in [K]} ({\mathbf z}^{{\ell}}_t)_j$ is the nonzero entry of ${\mathbf z}_t$, $\zeta_t = \frac{1 - \alpha_t}{K \alpha_t + 1 - \alpha_t}$, and $i$ denotes the index in $\times$ corresponding to 1, that is, $\times_i = 1$.

A.3 Self-speculative Decoding

Self-speculative decoding eliminates the need for a separate draft model by using a single network for both proposal and verification. [77] first proposed augmenting the base model with auxiliary prediction heads to predict a block of future tokens in parallel, then using the same model to verify these predictions and commit the longest validated prefix. Their exact-match verification rule guarantees equivalence to greedy decoding, rather than exact sampling. [78] instead convert an AR model into a diffusion model: the model drafts tokens in diffusion mode and verifies them in AR mode, with the attention pattern changing between the two modes while the model weights remain the same in both modes. They combine this approach with the rejection-sampling acceptance rule of [8] and [9] to recover exact sampling from the base model's distribution. Concurrent work ([72, 71]) builds on the same idea.

Crucially, however, this approach requires modifying the base AR model's weights, so it is not lossless.

A.4 Psi-Samplers

$\Psi$-Samplers ([32]) are predictor-corrector samplers for discrete diffusion models that share the marginals of the ancestral sampler. They interpolate between the reverse posterior and the forward process Equation (2):

$ \Psi_{s\mid t}(\cdot\mid \times, {\mathbf z}t; \boldsymbol{\mathit{\pi}}) = \kappa_t {q}{s\mid t}(\cdot\mid {\mathbf z}_t, \times; \boldsymbol{\mathit{\pi}})

(1-\kappa_t)q_s(\cdot\mid \times; \boldsymbol{\mathit{\pi}}),\tag{17} $

where $\boldsymbol{\mathit{\pi}}$ is the noise prior (e.g., the masked prior $\boldsymbol{\mathit{\pi}}= {\mathbf m}$ for MDMs or the uniform prior $\boldsymbol{\mathit{\pi}}= {\bm{1}}/K$ for USDMs), ${q}_{s\mid t}(\cdot \mid \cdot\ ;\ \boldsymbol{\mathit{\pi}})$ is the reverse posterior for MDMs Equation (12) or USDMs Equation (14), $q_s$ is the forward noising process, and $\kappa_t\in[0, 1]$ sets the correction strength. The first term predicts by denoising the current state; the second corrects by re-injecting noise, so earlier decisions can be revised. Setting $\kappa_t=1$ recovers ancestral sampling.

Since $\times$ is unavailable at generation time, the practical sampler substitutes the denoiser prediction $\mathbf {x}_\theta({\mathbf z}_t, t)$, giving the token-wise transition

$ \begin{aligned} & [\Psi^\theta_{s | t}(\cdot\mid {\mathbf z}t; \mathbf {x}\theta({\mathbf z}t, t), \boldsymbol{\mathit{\pi}})]^{{\ell}} \nonumber \ & = \kappa_t {q{s|t}^{{{\ell}}}}(\cdot\mid {\mathbf z}t, \mathbf {x}\theta({\mathbf z}t, t); \boldsymbol{\mathit{\pi}}) + (1 - \kappa_t) \left[\alpha_s {q{0|t}^{{{\ell}}}}(\cdot\mid {\mathbf z}t, \mathbf {x}\theta({\mathbf z}_t, t); \boldsymbol{\mathit{\pi}}) + (1 - \alpha_s) \boldsymbol{\mathit{\pi}} \right]. \end{aligned}\tag{3} $

Ancestral samplers can lock in incorrect early predictions, whereas $\Psi$-samplers correct errors explicitly: for MDMs the corrector can remask decoded tokens, and for USDMs it assigns every token non-zero probability, permitting transitions into states that the denoiser incorrectly scores as low-probability. Sample quality therefore keeps improving with additional steps.

For MDMs, the transition Equation (3) simplifies to

$ \begin{aligned} &[\Psi^\theta_{s\mid t}(\cdot\mid {\mathbf z}t; \mathbf {x}\theta({\mathbf z}t, t), \boldsymbol{\mathit{\pi}} = {\mathbf m})]^{{\ell}} \nonumber\ &= \begin{cases} \operatorname{Cat}!\left(\cdot, ;, [\alpha_s+\kappa_t(1-\alpha_s)] \times\theta^{\ell}({\mathbf z}_t, t) + (1-\kappa_t)(1-\alpha_s){\mathbf m} \right), & {\mathbf z}t^{{\ell}}\neq {\mathbf m}, \ \operatorname{Cat}!\left(\cdot, ;, \left[\alpha_s-\frac{\kappa_t\alpha_t(1-\alpha_s)}{1-\alpha_t} \right] \times^{{\ell}}\theta({\mathbf z}_t, t) + \left[1-\alpha_s+\frac{\kappa_t\alpha_t(1-\alpha_s)}{1-\alpha_t} \right]{\mathbf m} \right), & {\mathbf z}_t^{{\ell}}= {\mathbf m}. \end{cases} \end{aligned}\tag{18} $

To obtain the marginals for single-step generation, we simply plug in $s=0$ and $t=1$ into Equation (18) which gives:

$ \begin{aligned} &[\Psi^\theta_{s=0\mid t=1}(\cdot\mid {\mathbf z}t; \mathbf {x}\theta({\mathbf z}t, t), \boldsymbol{\mathit{\pi}} = {\mathbf m})]^{{\ell}} \nonumber\ &= \begin{cases} \operatorname{Cat}!\left(\cdot, ;, \left[1+\kappa_t(1-1)\right] \times^{{\ell}}\theta({\mathbf z}_1, 1) + (1-\kappa_t)(1-1){\mathbf m} \right), & {\mathbf z}1^{{\ell}}\neq {\mathbf m}, \ \operatorname{Cat}!\left(\cdot, ;, \left[\alpha_s-\frac{\kappa_t\cdot 0 \cdot(1-1)}{1-0} \right] \times^{{\ell}}\theta({\mathbf z}_1, 1) + \left[1-1+\frac{\kappa_t\cdot 0 \cdot (1-1)}{1-0} \right]{\mathbf m} \right), & {\mathbf z}1^{{\ell}}= {\mathbf m}. \end{cases} \ &= \begin{cases} \text{Cat}(\cdot; \times\theta^\ell({\mathbf z}_1)), & {\mathbf z}1^{{\ell}}\neq {\mathbf m}, \ \text{Cat}(\cdot; \times\theta^\ell({\mathbf z}_1)), & {\mathbf z}1^{{\ell}}= {\mathbf m}. \end{cases} \ & = \text{Cat}(\cdot; \times\theta^\ell({\mathbf z}_1)) \end{aligned}\tag{19} $

For USDM, the transition Equation (3) simplifies to:

$ \begin{aligned} &[\Psi^\theta_{s\mid t} (\cdot\mid {\mathbf z}t; \mathbf {x}\theta({\mathbf z}_t, t), \boldsymbol{\mathit{\pi}} = {\bm{1}}/K)]^{{\ell}} \nonumber\ &= \operatorname{Cat}\Bigg(\cdot, ;, \frac{1}{ K\alpha_t \left\langle {\mathbf z}t^{{\ell}}, [\mathbf {x}\theta({\mathbf z}_t, t)]^{{\ell}} \right\rangle +1-\alpha_t } \Bigg[K\alpha_t\left[\alpha_s+\kappa_t(1-\alpha_s)\right] {\mathbf z}t^{{\ell}}\odot[\mathbf {x}\theta({\mathbf z}t, t)]^{{\ell}} \nonumber\ &\qquad\qquad +\kappa_t(1-\alpha_s)\alpha{t\mid s}{\mathbf z}t^{{\ell}} + \left[\alpha_s(1-\alpha_t) -\kappa_t\alpha_t(1-\alpha_s) \right] [\mathbf {x}\theta({\mathbf z}t, t)]^{{\ell}} \nonumber\ &\qquad\qquad +(1-\alpha_s) \left[\kappa_t(1-\alpha{t\mid s}) + (1-\kappa_t) \left(K\alpha_t \left\langle {\mathbf z}t^{{\ell}}, [\mathbf {x}\theta({\mathbf z}_t, t)]^{{\ell}} \right\rangle +1-\alpha_t \right) \right] \frac{{\bm{1}}}{K} \Bigg] \Bigg). \end{aligned}\tag{20} $

To obtain the marginals for single-step generation, we simply plug in $s=0$ and $t=1$ into Equation (14) which gives

$ [\Psi^\theta_{s=0\mid t=1} (\cdot\mid {\mathbf z}t; \mathbf {x}\theta({\mathbf z}t), \boldsymbol{\mathit{\pi}} = {\bm{1}}/K)]^{{\ell}} = \text{Cat} (\cdot; \times^{{\ell}}\theta({\mathbf z}_1, 1)).\tag{21} $

A.5 Discrete Consistency Distillation

[31] show that Uniform-state discrete diffusion emerges from an underlying Gaussian diffusion process ([85, 86, 87]) defined on the one-hot representation $\times^{{\ell}} \in \mathcal{V}$: applying an $\operatorname*{arg, max}$ projection to the Gaussian latents recovers the discrete process. This correspondence lets us transfer techniques from the continuous to the discrete domain, most notably distillation for few-step generation. Distillation requires deterministic trajectories, namely the probability-flow ODE, which exist for Gaussian but not for discrete diffusion. Since discrete latents are simply $\operatorname*{arg, max}$ projections of Gaussian ones, [31] construct the trajectory in continuous space and map it back to the discrete domain.

Discrete Consistency Distillation (DCD) trains few-step generators on these trajectories. First, the Gaussian probability-flow ODE is constructed on the one-hot vectors and the resulting continuous states are mapped back to tokens via $\operatorname*{arg, max}$, yielding a deterministic sequence of discrete latents. The teacher $\times_{\bm \theta_0}: \mathcal{V}^L\times [0, 1] \rightarrow \Delta^L$ is a neural network with parameters $\theta_0$ that are held fixed, while the student $\times_{\bm \theta}: \mathcal{V}^L\times [0, 1] \rightarrow \Delta^L$ is a neural network whose parameters $\theta$ are trained. Given two adjacent states along this path, a noisier ${\mathbf z}_t$ at time $t$ and a less noisy ${\mathbf z}_s$ at time $s = t - T$, the student takes the noisier state as input and is trained to match the teacher's clean-token distribution at the less noisy one, in the spirit of consistency distillation ([88, 31]):

$ \mathcal{L}_{\mathrm{DCD}}(\bm \theta; \bm \theta_0)

\sum_{\ell=1}^{L} D_{\mathrm{KL}}!\left(\times_{\bm \theta}^\ell({\mathbf z}t^{1:L}, t), \middle|, \times{\bm \theta_0}^\ell({\mathbf z}_t^{1:L}, \ s) \right), $

Note that the teacher's parameters $\bm \theta_0$ are not updated during training. Distillation proceeds over numerous rounds, with the step size $\Delta$ increased at every round; at the end of each round the trained student parameters $\theta$ are copied into the teacher, which then serves as the target for the next round. By taking progressively larger denoising steps while remaining consistent along a shared trajectory, DCD compresses a many-step USDM sampler into a few-step one, making Uniform-state diffusion particularly effective in the low-step regime.

Limitation:

DCD is trained on deterministic probability-flow trajectories, which differ from the stochastic denoising trajectories followed at sampling time. This train-test mismatch limits its effectiveness.

B. Psi-Spec samplers

B.1 Diffusion Proposal Distribution Extended

Given a noisy block ${\mathbf z}t$, we use the $\Psi$-Spec transition in Equation (3) to sample a less noisy block ${\mathbf z}s$, where $s<t$. Because the denoiser uses next-token-prediction (NTP) parameterization, the clean-token distributions used to construct ${\mathbf z}s$ are given by $\times^{1:B-1}{{\theta\text{AR}}, {\theta{\Delta}}}([\times^{L}, {\mathbf z}_t])$. The resulting transition is obtained by drawing each token in parallel from its corresponding marginal distribution:

$ {\mathbf z}s^{\ell} \sim \begin{cases} \Psi^{\ell}{s|t} \left(\cdot \mid {\mathbf z}t ; \times^{{ 1:B-1}}{{\theta_\text{AR}}, {\theta_{\Delta}}}(\cdot), { \boldsymbol{\mathit{\pi}}} \right), ; & \ell > 1 \quad\text{(a)}\ \Psi^{\ell=1}{s|t} \left(\cdot \mid {\mathbf z}t ; \times^{{ 1:B-1}}{{\theta\text{AR}}}(\cdot), { \boldsymbol{\mathit{\pi}}} \right). ; & \ell = 1 \quad\text{(b)} \end{cases}\tag{22} $

Notice that the logits for the first, clean position use only the base AR parameters ${\theta_\text{AR}}$, as shown in Equation (22a), whereas the logits at the noisy positions use both the AR weights and the diffusion adapters. This separation prevents distribution shift because the diffusion adapters are trained only to denoise noisy tokens. We compute Equation (22a) and (22b) in a single forward pass using the gated LoRA technique of [44].

B.2 Single-Step Generation

After removing time conditioning from the denoising model and shifting the logits one position to the left to account for its next-token prediction (NTP) formulation in Equation (19) for MDMs and (21) for USDMs, we show that the resulting marginal distribution has the following form. Let $\Psi_0$ denote the distribution induced by the $\Psi$-sampler at $t=0$:

$ \begin{aligned} \Psi_0^{\ell} = \begin{cases} \times^{\ell}{{\theta\text{AR}}, {\theta_{\Delta}}}([\times^{L}, {\mathbf z}t]), & \ell > 1 \ \times^{\ell}{{\theta_\text{AR}}}([\times^{L}, {\mathbf z}_t]). & \ell = 1 \end{cases} \end{aligned} $

B.3 Sampling Algorithm

Algo. Algorithm 1 shows Uno's complete sampling algorithm for the single-sequence case. Differences to speculative decoding ([8]) are written in orange.

Require: Prefix $\times$ of length $L$, target model $\times_{{\theta_\text{AR}}, {\theta_{\Delta}}}$, block size $B$
{Sample noise
${\mathbf z} \overset{\mathrm{i.i.d.}}{\sim} \prod_1^{B-1}\mathcal{U}[\mathcal{V}]$} // Initialize a block of $B-1$ random tokens
$[\mathbf{q}_0,\mathbf{q}] \gets \times_{{\theta_\text{AR}}, {\theta_{\Delta}}} ([\times^L, {\mathbf z}]; \times^{<L})$
// ${\theta_\text{AR}}$ apply on $\times^L$; ${\theta_\text{AR}}+{\theta_{\Delta}}$ on ${\mathbf z}$
Sample $\times^{L+1}\sim \mathbf{q}_0$ // free clean token from draft step
Sample $\tilde{\times} \sim \mathbf{q}$ // $B-1$ tokens sampled in parallel
$\mathbf{p}\gets \times_{\theta_\text{AR}}([\times^{L + 1}, \tilde{\times}]; \times)$
// AR verifier distribution defined by ${\theta_\text{AR}}$
sample
$\mathbf{r} \overset{\mathrm{i.i.d.}}{\sim} \prod_{1}^{B-1}\mathcal{U}[0,1]$
$n\gets \min\!\left( \left\{ i \in [B-1]: r_i> \min\!\left( 1,\frac{\mathbf{p}_i}{\mathbf{q}_i} \right) \right\} \cup\{B\} \right)$
if $n=B$ then
  sample $\tilde{{\mathbf z}} \sim \mathbf{p}_B$
else
  sample
  $\tilde{{\mathbf z}} \sim \operatorname{Norm}\!\left( [\mathbf{p}_n-\mathbf{q}_n]_+ \right)$
end if
return $[\times, \times^{L+1}, \tilde{\times}^{1:n-1}, \tilde{{\mathbf z}}]$

C. Additional Experiments

C.1 Experiment Configs

C.1.1 Benchmarks

We provide additional details about the set of benchmarks we evaluate Uno on in Table 4.

::: {caption="Table 4: Benchmarks used in our Uno evaluation. 'Generations' denotes the number of responses sampled per example when computing average pass@1."}

:::

C.1.2 Uno $_\text{Qwen}$, EAGLE-3, DFlash Sampler Configs for Qwen3-8B

At $\text{temp}=1$, we use top- $p=0.95$ and top- $k=50$; $\text{temp}=0$ denotes greedy decoding. Uno $_\text{Qwen}$ uses linear blocks of size $B\in{4, 8, 16}$ or, for tree verification, $B=16$, tree size $V=60$, and candidate top- $K=32$. DFlash uses blocks $B\in{4, 8, 16}$ and we evaluate both thinking-enabled and thinking-disabled variants (see Suppl.Appendix C.7). Linear EAGLE-3 $B\in{4, 8, 16}$ draft tokens; its tree configuration uses seven draft steps, top- $k=10$, and at most $V=60$ draft tokens.

: Table 5: Open-source model checkpoints used in our evaluations.

Model	Checkpoint
DiffusionGemma	https://huggingface.co/google/diffusiongemma-26B-A4B-it
Nemotron-Labs-Diffusion	https://huggingface.co/nvidia/Nemotron-Labs-Diffusion-14B
EAGLE-3	https://huggingface.co/AngelSlim/Qwen3-8B_eagle3
DFlash	https://huggingface.co/z-lab/Qwen3-8B-DFlash-b16
Fast-dLLM v2	https://huggingface.co/Efficient-Large-Model/Fast_dLLM_v2_7B
SDAR	https://huggingface.co/JetLM/SDAR-8B-Chat-b16

C.2 Uno RL

Starting from the supervised fine-tuning (SFT) checkpoint, four specialized experts were trained for mathematics, code generation, tool use, and browse search using the DAPO reinforcement learning algorithm ([45]). The Math expert was trained for 2, 560 steps and generated 28.9B response tokens, while the Code expert was trained in two stages of 600 and 1, 500 steps, producing 6B and 11.94B response tokens, respectively. Both experts used a learning rate of (5 $\times 10^{-7}$ , 16 groups, and 32 prompts per step, yielding 512 sequences per step across 32 nodes per expert. Math training used maximum context lengths of 32, 768 and 65, 536 tokens, while Code training supported contexts up to 65, 536 and 128K tokens. The Tool-use expert completed 39 steps with 128 prompts per step and generated 1.4B tokens, including tool-call tokens. The Browse Search expert completed 59 steps with 64 prompts per step and generated 8.4B tokens, including tool-call tokens. For both Tool-use and Browse Search, the maximum trajectory length was 128K tokens, covering model-generated tokens as well as tool outputs. Finally, the four experts were consolidated into a single model using ISO-Merger (RAM; [63], [63]), a data-free approach that combines specialists trained from a shared base checkpoint without requiring additional rollouts, gradient updates, or distillation.

C.3 Uno Evaluations

::: {caption="Table 6: We report TPFs for a subset of the evaluations using different sampler configurations for the Uno model. Among all configurations, the Linear sampler with B=4 achieves the highest system throughput at a batch size of 64, while the Tree sampler with (B, K, V)=(16, 32, 32) achieves the highest per-user throughput (batch size 1)."}

:::

\begin{tabular}{c|cc}
  \toprule
  Method & Max. Sys. Throughput (toks / sec) & Max. Per-req. Throughput (toks / sec) \\
  \midrule
  Uno (Ours) & 5255 & 383 \\
  AR (Ours) & 3577 & 176\\
  \text{DiffusionGemma} & 1136 & 836\\
  \text{Nemotron-Labs-Diffusion} & 2794 & 290 \\
  Mercury-2 & 1197$^*$ & 769$^*$ \\
  \bottomrule
  \end{tabular}

\begin{tabular}{lcc}
\toprule
 {} & {SFT}
 {} & {Post-Trained} \\
\midrule

\multicolumn{3}{l}{\textit{Math}} \\
\quad GSM8K & $2.66$ & $1.95$ \\ \quad MATH500 & $2.27$ & $1.93$ \\ \quad AIME-24 & $2.17$ & $1.93$ \\ \quad AIME-25 & $2.26$ & $1.97$ \\ \quad AIME-26 & $2.22$ & $1.83$ \\
\midrule

\multicolumn{3}{l}{\textit{Coding}} \\
\quad HumanEval & $1.97$ & $2.34$ \\ \quad MBPP & $2.35$ & $2.19$ \\
\midrule

\multicolumn{3}{l}{\textit{Science and Knowledge}} \\
\quad GPQA-Diamond & $2.03$ & $1.97$ \\
\midrule

\multicolumn{3}{l}{\textit{Instruction Following}} \\
\quad IFEval & $2.12$ & $2.50$ \\
\midrule

\multicolumn{3}{l}{\textit{Other Benchmarks}} \\
\quad HLE & $2.14$ & $2.15$ \\ \quad AA-Omniscience & $2.57$ & $2.34$ \\
\midrule

{TPF}
 {} & ${2.25}$ & ${2.10}$ \\
\bottomrule
\end{tabular}

C.4 Qwen Finetuned on OpenThoughts

Table 9 compares Qwen3-8B with a variant fine-tuned on OpenThoughts across a subset of benchmarks. Fine-tuning on OpenThoughts substantially degrades performance, reducing accuracy by up to $15%$ points, depending on the benchmark. Nevertheless, diffusion weights trained on OpenThoughts successfully accelerate generation despite the distribution mismatch between OpenThoughts and the data used to train the AR weights.

\begin{tabular}{lcc}
\toprule
 {} & Qwen3-8B & Qwen-finetuned \\
\midrule
\textit{Math} & & \\
\quad GSM8K & $96_{\pm 0.2}$ & $95.2_{\pm 1.1}$ \\ \quad MATH500 & $96.2_{\pm 0.3}$ & $94.8_{\pm 0.0}$ \\ \quad AIME-24 & $77.7_{\pm 3.0}$ & $68.3_{\pm 17.6}$ \\ \quad AIME-25 & $70.7_{\pm 4.2}$ & $53.3_{\pm 0.0}$ \\ \quad AIME-26 & $67.7_{\pm 3.9}$ & $61.7_{\pm 5.9}$ \\ \midrule \textit{Coding} & & \\ \quad HumanEval & $94.4_{\pm 1.0}$ & $77.8_{\pm 4.6}$ \\ \quad MBPP & $88.7_{\pm 0.4}$ & $78.1_{\pm 0.4}$ \\ \quad LCBv6 & $50.9_{\pm 1.2}$ & $42.5_{\pm 0.8}$ \\
\bottomrule
\end{tabular}

C.5 Uno-Qwen vs I-DLM (R-ISD) Comparison

In this section, we compare I-DLM (R-ISD), the lossless variant of I-DLM, with our method, Uno $_\text{Qwen}$ .

In the first experiment, we train I-DLM (R-ISD) with a rank-128 LoRA adapter for one epoch on the OpenThoughts dataset, using exactly the same training configuration as Uno $\text{Qwen}^{\text{1ep}}$ . For inference, we use the sampler provided in the official I-DLM repository^3. All results for I-DLM (R-ISD) in Table 10 are evaluated with a block size of $B=16$. The results show that the I-DLM sampler degrades accuracy on numerous benchmarks relative to the Qwen AR model, as highlighted in red, whereas Uno $\text{Qwen}^{\text{1ep}}$ does not. Upon inspecting the I-DLM codebase, we find that its sampler performs greedy drafting^4, but its rejection-sampling verification procedure is not adjusted accordingly. As a result, the sampler does not preserve the claimed losslessness property.

I-DLM (R-ISD) uses LoRA adapters and retains the transformer's causal attention pattern. It is therefore compatible with our $\Psi$-Spec sampler using the masked prior $\boldsymbol{\mathit{\pi}} = {\mathbf m}$. In the second set of experiments, we evaluate I-DLM (R-ISD) with $\Psi$-Spec. As shown in Table 10, this combination remains lossless. We observe the same result when evaluating the I-DLM (R-ISD) adapters released in the official repository. However, both variants provide considerably lower speedups than our method.

::: {caption="Table 10: I-DLM and Uno _Qwen^1ep evaluation with the linear sampler at B=16 and temp=1. We compare the I-DLM checkpoints (Ours and Official) under Ψ-spec samplers with π = m; accuracy is reported with 95% confidence intervals across seeds, using two seeds for MMLU-Pro and ten seeds for all other benchmarks."}

:::

Additionally, Table 11 provides TPF for the I-DLM LoRA checkpoint evaluated on block sizes $B=4$, $B=8$, and $B=16$. The results show a significant increase in TPF from $B=4$ to $B=8$, while TPF seems to saturate beyond $B=8$.

\begin{tabular}{lccc}
  \toprule
   {} & $B$ = $4$ & $B$ = $8$ & $B$ = $16$ \\
  \midrule
  \textit{Math} & & & \\
  \quad GSM8K & 1.87 & 2.20 & 2.23 \\
  \quad MATH500 & 1.80 & 2.04 & 2.06 \\
  \quad AIME-24 & 1.68 & 1.80 & 1.80 \\
  \quad AIME-25 & 1.66 & 1.81 & 1.79 \\
  \quad AIME-26 & 1.67 & 1.78 & 1.78 \\
  \midrule
  \textit{Coding} & & & \\
  \quad HumanEval & 1.75 & 1.96 & 1.98 \\
  \quad MBPP & 1.73 & 1.93 & 1.93 \\
  \quad LCBv6 & 1.54 & 1.60 & 1.60 \\
  \midrule
  \textit{Science and Knowledge} & & & \\
  \quad GPQA & 1.61 & 1.71 & 1.71 \\
  \quad GPQA-Diamond & 1.59 & 1.69 & 1.69 \\
  \quad MMLU-Pro & 1.69 & 1.83 & 1.84 \\
  \midrule
  \textit{Instruction Following} & & & \\
  \quad IFEval & 1.73 & 1.86 & 1.85 \\
  \midrule
  Average & 1.69 & 1.85 & 1.86 \\
  \bottomrule
  \end{tabular}

C.6 Uno-Qwen Ablations

C.6.1 LoRA Rank and Loss Ablations

In Table 12, we ablate the LoRA rank of Uno $_\text{Qwen}^{\text{1ep}}$, as well as various weightings of KL and TV loss terms. We see that average TPF increases from 2.39 to 2.47 when increasing the LoRA rank from 128 to 256, while the number of trainable parameters increases from 349M to 698M. It turns out that a loss weighting of $0.01\times$ KL plus $1\times$ TV slightly outperforms a pure TV loss objective. Note that a factor of $0.01$ for the KL term amounts to a KL loss term only about $10\times$ smaller in magnitude than the TV term, as the KL loss is naturally around $10\times$ larger than the TV loss.

::: {caption="Table 12: TPF results for one-epoch Uno _Qwen^1ep checkpoints trained with TV-only, KL-only, and weighted KL+TV objectives. The TV-only objective is evaluated using rank-128 LoRA adapters (the default Uno _Qwen configuration, highlighted in blue) and rank-256 adapters, while all other objectives use rank-128 adapters. All adapters are applied to every projection and use alpha_LoRA/r=2. All checkpoints are evaluated using the linear sampler with B=16 and temp=1."}

:::

C.6.2 Training Block Size Curriculum

Here we ablate two block-size curricula for Uno $\text{Qwen}$ training. Table 13 shows the TPF attained by two different three-epoch block-size curricula, both initialized from a model trained on block size 2 for 0.5 epochs, and block size 4 for another 0.5 epochs (written as $0.5@2, \ 0.5@4$). The first curriculum is our default curriculum used for Uno $\text{Qwen}$, which continues at $0.5@6, \ 0.5@8$, $0.5@12$, and $0.5@16$; the second one continues at $2@16$. Average TPF drops from 2.71 to 2.65 when employing the second curriculum, which shows that incrementally increasing block size during training is beneficial over consistently training at a large block size.

::: {caption="Table 13: TPF for two three-epoch TV-only block curricula. The notation e@B denotes e epochs of training at block size B. Both models first train on 0.5@2, 0.5@4. The standard curriculum then progressively increases the block size, whereas the fixed- B=16 curriculum trains for the remaining two epochs entirely at B=16. Models are evaluated with the linear sampler at B=16 and temp=1."}

:::

C.6.3 LoRA Adapter Ablations

We ablate which projection matrices in the transformer layer to apply LoRA weights to in Table 14. By default, we apply LoRA to all projections (this includes the Q, K, V, O matrices from the attention layer, as well as the MLP gate, up, and down projections); in addition, we ablate only applying LoRA to the attention layers (i.e., Q, K, V, and O projections); only to the Q and K projections of the attention layer; only to the Q projections; and only to the O projections, which is the setup recommended in [21]. In each setting, we adapted the rank to keep parameter parity at 349M LoRA parameters. All models for this ablation were trained on one epoch at block size eight. Table 14 shows that at fixed $\alpha_\text{LoRA}\ / \ \text{rank}=2$, applying LoRA to all projections work best. Consequently, we apply LoRA all to projections for our Uno models.

::: {caption="Table 14: TPF for Uno _Qwen^1ep LoRA target-projection ablations with alpha_LoRA/r_LoRA=2. Ranks are chosen to approximately maintain LoRA parameter parity. All checkpoints are evaluated with the linear sampler at B = 16 and temp=1."}

:::

C.6.4 Varying LoRA $\alpha / r$

For LoRA with rank $r_{\text{LoRA}}$ and a projection $W_\text{base}\in\mathbb{R}^{n\times m}$ of the base model, both LoRA training and inference use

$ W = W_{\text{base}}+\frac{\alpha_{\text{LoRA}}}{r_{\text{LoRA}}}BA,\tag{23} $

where $B\in\mathbb{R}^{n\times r}$, $A\in\mathbb{R}^{r\times m}$, and $\alpha_\text{LoRA}>0$ is a hyperparameter that controls the overall contribution of the LoRA adapter. In Table 15 we ablate the ratio $\alpha_\text{LoRA}/r_{\text{LoRA}}$ across values between 2 and 256 or both one-epoch and three-epoch models. All three-epoch models in this ablation were trained on the block size curriculum described in Suppl. Appendix C.6.2.

The table shows that the optimal ratio differs depending on the training horizon, and $\alpha_{\text{LoRA}}/r_{\text{LoRA}}=16$ is optimal at three epochs, while $\alpha_{\text{LoRA}}/r_{\text{LoRA}}=64$ is optimal at one epoch. Consequently, we pick $\alpha_{\text{LoRA}}/r_{\text{LoRA}}=16$ for Uno $\text{Qwen}$, while for Uno, we set $\alpha{\text{LoRA}}/r_{\text{LoRA}}=64$ which we observed to be preferable.

::: {caption="Table 15: TPF for one-epoch and three-epoch, TV-only Uno _Qwen^1ep models with rank-128 LoRA adapters applied to all projections, across LoRA scaling values. All checkpoints are evaluated with the linear sampler at B=16 and temp=1; averages are unweighted across the 12 benchmarks."}

:::

C.7 DFlash Thinking Mode Ablation

In Table 16 we evaluate DFlash ([27]) with thinking mode enabled and disabled, at $\text{temp}=1$. Our results show that while disabling thinking mode significantly increases tokens per step, it deteriorates accuracy from around $76%$ to roughly $55%$. This quality difference does not contradict losslessness: enabling thinking changes the chat template and hence the target-model distribution that the speculative decoder preserves. In light of these results, we subsequently enable thinking mode for DFlash ([73]).

::: {caption="Table 16: DFlash thinking ablation at temp=1, top- p=0.95, and top- k=50. Accuracy in percent. AL denotes average acceptance length, i.e., tokens per verification step."}

:::

C.8 Extended Comparison vs EAGLE-3 and DFlash

By default, we evaluate DFlash with block sizes $B \in {8, 16}$. For EAGLE-3, we use a linear block size of $B=8$, as well as its standard tree configuration with depth $D=7$ (corresponding to block size $B=8$), top- $k=10$, and a verification budget of $V=60$ tokens. These EAGLE-3 tree parameters follow the official settings provided in the paper and repository.

We provide extended evaluation results for Uno $_\text{Qwen}$, EAGLE-3, and DFlash across various sampler settings in Table 17.

::: {caption="Table 17: Tokens per step across samplers for Uno _Qwen, EAGLE-3, and thinking-enabled DFlash. Values at both temp=0 and temp=1 are unfiltered."}

:::

::: {caption="Table 18: Median per-request throughput (tok/s/stream) for the 1K-input/8K-output Qwen3-8B workload at temp=1. Each value is the median of three fresh repetitions; '–' denotes a resident-capacity-infeasible point."}

:::

References

Section Summary: The references section compiles a list of academic papers, preprints, and technical reports centered on recent progress in large language models and related AI techniques. Many entries address model architectures, inference speedups such as speculative decoding, and diffusion-based approaches for generating text or other discrete data. The works draw from conferences, arXiv submissions, and earlier studies on language patterns, with dates mostly clustered in the mid-2020s.

[1] Kimi KimiTeam, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.

[2] GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Zhong, Mingdao Liu, Mingming Zhao, Pengfan Du, Qian Dong, Rui Lu, Shuang-Li, Shulin Cao, Song Liu, Ting Jiang, Xiaodong Chen, Xiaohan Zhang, Xuancheng Huang, Xuezhen Dong, Yabo Xu, Yao Wei, Yifan An, Yilin Niu, Yitong Zhu, Yuanhao Wen, Yukuo Cen, Yushi Bai, Zhongpei Qiao, Zihan Wang, Zikang Wang, Zilin Zhu, Ziqiang Liu, Zixuan Li, Bojie Wang, Bosi Wen, Can Huang, Changpeng Cai, Chao Yu, Chen Li, Chengwei Hu, Chenhui Zhang, Dan Zhang, Daoyan Lin, Dayong Yang, Di Wang, Ding Ai, Erle Zhu, Fangzhou Yi, Feiyu Chen, Guohong Wen, Hailong Sun, Haisha Zhao, Haiyi Hu, Hanchen Zhang, Hanrui Liu, Hanyu Zhang, Hao Peng, Hao Tai, Haobo Zhang, He Liu, Hongwei Wang, Hongxi Yan, Hongyu Ge, Huan Liu, Huanpeng Chu, Jia'ni Zhao, Jiachen Wang, Jiajing Zhao, Jiamin Ren, Jiapeng Wang, Jiaxin Zhang, Jiayi Gui, Jiayue Zhao, Jijie Li, Jing An, Jing Li, Jingwei Yuan, Jinhua Du, Jinxin Liu, Junkai Zhi, Junwen Duan, Kaiyue Zhou, Kangjian Wei, Ke Wang, Keyun Luo, Laiqiang Zhang, Leigang Sha, Liang Xu, Lindong Wu, Lintao Ding, Lu Chen, Minghao Li, Nianyi Lin, Pan Ta, Qiang Zou, Rongjun Song, Ruiqi Yang, Shangqing Tu, Shangtong Yang, Shaoxiang Wu, Shengyan Zhang, Shijie Li, Shuang Li, Shuyi Fan, Wei Qin, Wei Tian, Weining Zhang, Wenbo Yu, Wenjie Liang, Xiang Kuang, Xiangmeng Cheng, Xiangyang Li, Xiaoquan Yan, Xiaowei Hu, Xiaoying Ling, Xing Fan, Xingye Xia, Xinyuan Zhang, Xinze Zhang, Xirui Pan, Xu Zou, Xunkai Zhang, Yadi Liu, Yandong Wu, Yanfu Li, Yidong Wang, Yifan Zhu, Yijun Tan, Yilin Zhou, Yiming Pan, Ying Zhang, Yinpei Su, Yipeng Geng, Yong Yan, Yonglin Tan, Yuean Bi, Yuhan Shen, Yuhao Yang, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yurong Wu, Yutao Zhang, Yuxi Duan, Yuxuan Zhang, Zezhen Liu, Zhengtao Jiang, Zhenhe Yan, Zheyu Zhang, Zhixiang Wei, Zhuo Chen, Zhuoer Feng, Zijun Yao, Ziwei Chai, Ziyuan Wang, Zuzhou Zhang, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-5: from vibe coding to agentic engineering, 2026. URL https://arxiv.org/abs/2602.15763.

[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

[4] Minseo Kim, Minjae Lee, Seunghyuk Oh, Kevin Galim, Donghoon Kim, Coleman Hooper, Harman Singh, Amir Gholami, Hyung Il Koo, and Wonjun Kang. Efficientrollout: System-aware self-speculative decoding for rl rollouts. arXiv preprint arXiv:2606.18967, 2026.

[5] Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29, 1990. URL https://aclanthology.org/J90-1003/.

[6] Douglas Biber. A corpus-driven approach to formulaic language in English: Multi-word patterns in speech and writing. International Journal of Corpus Linguistics, 14(3):275–311, 2009. doi:10.1075/ijcl.14.3.08bib. URL https://doi.org/10.1075/ijcl.14.3.08bib.

[7] Shashank Verma and Neal Vaidya. Mastering LLM techniques: Inference optimization. NVIDIA Technical Blog, November 2023. URL https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/. Accessed 2026-08-24.

[8] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19286. PMLR, 2023.

[9] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.

[10] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021a.

[11] Subham S. Sahoo. Foundations of Diffusion Language Models. PhD thesis, Cornell University, 2026. URL https://www.proquest.com/dissertations-theses/foundations-diffusion-language-models/docview/3355013570/se-2. Copyright - Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated - 2026-06-23.

[12] Subham Sekhar Sahoo, Aaron Gokaslan, Christopher De Sa, and Volodymyr Kuleshov. Diffusion models with learned adaptive noise. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://openreview.net/forum?id=loMa99A4p8.

[13] Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander M Rush, Thomas PIERROT, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=i5MrJ6g5G1.

[14] Sophia Tang, Yinuo Zhang, and Pranam Chatterjee. Peptune: De novo generation of therapeutic peptides with multi-objective-guided discrete diffusion. ArXiv, pp. arXiv–2412, 2025.

[15] Seul Lee, Karsten Kreis, Srimukh Prasad Veccham, Meng Liu, Danny Reidenbach, Yuxing Peng, Saee Paliwal, Weili Nie, and Arash Vahdat. Genmol: A drug discovery generalist with discrete diffusion. arXiv preprint arXiv:2501.06158, 2025.

[16] Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models: A family of any-order diffusion llms. arXiv preprint arXiv:2506.01928, 2025b.

[17] Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu, John Thickstun, and Ante Jukic. Scaling beyond masked diffusion language models. arXiv preprint arXiv:2602.15014, 2026.

[18] Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Daozhuo Feng, Haibo Feng, Mingliang Gong, Zhuocheng Gong, Yanmei Gu, Jian Guan, Kaiyuan Guan, Hongliang He, Zenan Huang, Juyong Jiang, Zhonghui Jiang, Zhenzhong Lan, Chengxi Li, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Yuan Lu, Yuxin Ma, Xingyu Mou, Zhenxuan Pan, Kaida Qiu, Yuji Ren, Jianfeng Tan, Yiding Tian, Zian Wang, Lanning Wei, Tao Wu, Yipeng Xing, Wentao Ye, Liangyu Zha, Tianze Zhang, Xiaolu Zhang, Junbo Zhao, Da Zheng, Hao Zhong, Wanli Zhong, Jun Zhou, Junlin Zhou, Liwang Zhu, Muzhi Zhu, and Yihong Zhuang. Llada2.1: Speeding up text diffusion via token editing, 2026. URL https://arxiv.org/abs/2602.08676.

[19] DiffusionGemmaTeam, Adrien Ali Taïga, James Assiene, Daniele Calandriello, Rahma Chaabouni, João Gante, Tamara von Glehn, Nate Keating, Chris Knutsen, Martin Kukla, Tianlin Liu, Ivan Lobov, Ofir Nabati, João Gabriel Oliveira, Nicolas Perez-Nieves, Nastasia Prutianova, Bobak Shahriari, Jean Tarbouriech, Pavel Tyletski, Çağlar Ünlü, Cindy Wu, Glenn Cameron, Jerome Connor, Sertan Girgin, Maarten Grootendorst, Alon Levkovitch, Eliya Nachmani, Omar Sanseviero, Piotr Stanczyk, Quentin Berthet, Andrew Campbell, Clément Crepy, Valentin De Bortoli, Arnaud Doucet, Romuald Elie, Alexandre Galashov, Klaus Greff, Alexis Jacq, David Ruhe, Yu-Han Wu, Sebastian Flennerhag, Brendan O'Donoghue, George Scrivener, and Shantanu Thakoor. Diffusiongemma technical report, 2026. URL https://arxiv.org/abs/2608.00146.

[20] Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. Fast-dllm v2: Efficient block-diffusion llm, 2025. URL https://arxiv.org/abs/2509.26328.

[21] Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, Weijia Chen, Sajad Norouzi, Jingyu Liu, Shiyi Lan, Ligeng Zhu, Jin Wang, Jindong Jiang, Morteza Mardani, Mehran Maghoumi, Song Han, Ante Jukic, Nima Tajbakhsh, Jan Kautz, and Pavlo Molchanov. Nemotron-labs-diffusion: A tri-mode language model unifying autoregressive, diffusion, and self-speculation decoding. Technical report, NVIDIA, 2026. Technical report.

[22] Google Cloud. The roi of ai 2025: Measuring the impact of ai and ai agents. Technical report, Google Cloud, 2025. URL https://cloud.google.com/resources/content/roi-of-ai-2025.

[23] Amazon Web Services. AGENTPERF01-BP01: Define performance-aligned success criteria for agent workloads. AWS Well-Architected Framework, Agentic AI Lens, 2026. URL https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentperf01-bp01.html.

[24] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=PEpbUobfJv.

[25] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737, 2024.

[26] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. Advances in Neural Information Processing Systems, 38:136737–136756, 2026b.

[27] Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block diffusion for flash speculative decoding, 2026. URL https://arxiv.org/abs/2602.06036.

[28] Stefano Ermon. Introducing Mercury 2. Inception, 2026. URL https://www.inceptionlabs.ai/blog/introducing-mercury-2.

[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

[30] Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum?id=L4uaAR4ArM.

[31] Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T Chiu, and Volodymyr Kuleshov. The diffusion duality. In Forty-second International Conference on Machine Learning, 2025a. URL https://openreview.net/forum?id=9P9Y8FOSOk.

[32] Justin Deschenaux, Caglar Gulcehre, and Subham Sekhar Sahoo. The diffusion duality, chapter II: ${\backslash}psi$-samplers and efficient curriculum. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=RSIoYWIzaP.

[33] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474–1487, 2020.

[34] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.

[35] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.

[36] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

[37] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.

[38] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of machine learning research, 25(70):1–53, 2024.

[39] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.

[40] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.

[41] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.

[42] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022.

[43] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2024.

[44] Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajtabar. Your llm knows the future: Uncovering its multi-token prediction potential. arXiv preprint arXiv:2507.11851, 2025.

[45] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026a.

[46] Xingkai Yu. Nano-vLLM: A lightweight vLLM implementation built from scratch. https://github.com/GeeeekExplorer/nano-vllm, 2025. Software repository.

[47] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs, 2024. URL https://arxiv.org/abs/2312.07104.

[48] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.

[49] Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https://arxiv.org/abs/2506.04178.

[50] Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, and Victor Barres. $\tau$-knowledge: Evaluating conversational agents over unstructured knowledge. arXiv preprint arXiv:2603.04370, 2026.

[51] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. $\tau^2$-bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506.07982.

[52] The Terminal-Bench Team. Terminal-bench 2.1, may 2026. URL https://www.tbench.ai/news/terminal-bench-2-1.

[53] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.06770.

[54] Declan Jackson, William Keating, George Cameron, and Micah Hill-Smith. AA-Omniscience: Evaluating cross-domain knowledge reliability in large language models, 2025. URL https://arxiv.org/abs/2511.13029.

[55] Artificial Analysis. Announcing artificial analysis long context reasoning (AA-LCR), 2025. URL https://artificialanalysis.ai/articles/announcing-aa-lcr.

[56] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025.

[57] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98.

[58] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.

[59] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2305.20050.

[60] Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating llms on uncontaminated math competitions. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2505.23281.

[61] Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvalddson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: MathArena as an evaluation platform for mathematics with llms, 2026. URL https://arxiv.org/abs/2605.00674.

[62] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models, 2021b. URL https://arxiv.org/abs/2108.07732.

[63] Xiangchi Yuan, Dachuan Shi, Chunhui Zhang, Zheyuan Liu, Shenglong Yao, Soroush Vosoughi, and Wenke Lee. Behavior knowledge merge in reinforced agentic models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 33007–33028, 2026.

[64] Mark Chen et al. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374.

[65] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chfJJYC3iL.

[66] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911.

[67] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2406.01574.

[68] Lanxiang Hu, Siqi Kou, Yichao Fu, Samyam Rajbhandari, Tajana Rosing, Yuxiong He, Zhijie Deng, and Hao Zhang. Fast and accurate causal parallel decoding using jacobi forcing. arXiv preprint arXiv:2512.14681, 2025.

[69] Shuang Cheng, Yihan Bian, Dawei Liu, Yuhua Jiang, Yihao Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, et al. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 22058–22075, 2026.

[70] Xingyu Su, Jacob Helwig, Shubham Parashar, Atharv Chagi, Lakshmi Jotsna, Degui Zhi, James Caverlee, Dileep Kalathil, and Shuiwang Ji. Data-efficient autoregressive-to-diffusion language models via on-policy distillation. arXiv preprint arXiv:2606.06712, 2026.

[71] Yifan Yu, Yuqing Jian, Junxiong Wang, Zhongzhu Zhou, Donglin Zhuang, Xinyu Fang, Sri Yanamandra, Xiaoxia Wu, Qingyang Wu, Shuaiwen Leon Song, et al. Introspective diffusion language models. arXiv preprint arXiv:2604.11035, 2026b.

[72] Yuchen Zhu, Jing Shi, Chongjian Ge, Hao Tan, Yiran Xu, Wanrong Zhu, Jason Kuen, Koustava Goswami, Rajiv Jain, Yongxin Chen, et al. Flare: Diffusion for hybrid language model. arXiv preprint arXiv:2606.01774, 2026.

[73] Z Lab. Qwen3-8B-DFlash-b16 model card. https://huggingface.co/z-lab/Qwen3-8B-DFlash-b16, 2026. Accessed 2026-08-03.

[74] Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, and Jun Wang. Diffuspec: Unlocking diffusion language models for speculative decoding. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 20896–20910, 2026a.

[75] Jameson Sandler, Jacob Christopher, Tom Hartvigsen, and Ferdinando Fioretto. Specdiff-2: Scaling diffusion drafter alignment for faster speculative decoding. Proceedings of Machine Learning and Systems, 8:1128–1147, 2026.

[76] Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tyEyYT267x.

[77] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In Neural Information Processing Systems, 2018. URL https://api.semanticscholar.org/CorpusID:53208380.

[78] Jingyu Liu, Xin Dong, Zhifan Ye, Rishabh Mehta, Yonggan Fu, Vartika Singh, Jan Kautz, Ce Zhang, and Pavlo Molchanov. Tidar: Think in diffusion, talk in autoregression. arXiv preprint arXiv:2511.08923, 2025.

[79] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.

[80] Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haziza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer, and Yaron Lipman. Set block decoding is a language model inference accelerator. arXiv preprint arXiv:2509.04185, 2025.

[81] Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. Advances in Neural Information Processing Systems, 38:56729–56762, 2026.

[82] Guanghan Wang, Gilad Turok, Yair Schiff, Marianne Arriola, and Volodymyr Kuleshov. d2: Improving reasoning in diffusion language models via trajectory likelihood estimation. arXiv preprint arXiv:2509.21474, 2025.

[83] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131–103167, 2024.

[84] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=sMyXP8Tanm.

[85] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256–2265. PMLR, 2015.

[86] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.

[87] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696–21707, 2021.

[88] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 32211–32252. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/song23a.html.