Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

Ying Fan$^{1}$ $^{*}$
$^{1}$Microsoft Research

Anej Svete$^{2}$
$^{2}$ETH Zürich

Kangwook Lee$^{3,4}$
$^{3}$KRAFTON $^{4}$Ludo Robotics

$^{*}$ Preprint. Work partially done at UW-Madison.

Abstract

Language models typically reason via explicit chain-of-thought (CoT), generating intermediate steps token-by-token. Latent CoT offers an alternative: it performs multi-step reasoning in the model's hidden states, replacing decoded tokens with continuous representations for greater efficiency. However, existing latent CoT methods underperform explicit CoT beyond 1B parameters, and the gap widens with scale. Looped, or recurrent-depth, Transformers, which reuse their weights to increase computation depth without adding parameters, are a natural fit for latent reasoning. We therefore ask whether looped Transformers can bridge this gap. We answer affirmatively with a simple recipe: a looped padded Transformer that processes $K$ latent blocks in parallel for $R$ iterations, with a cross-entropy loss on each latent position's gold CoT-step token, similar to explicit CoT supervision. We instantiate it as LOTUS ($\underline{\text{Lo}}$oped $\underline{\text{T}}$ransformers with parallel s$\underline{\text{u}}$pervision on latent$\underline{\text{s}}$). LOTUS is, to our knowledge, the first latent-CoT method to bridge the gap to explicit CoT at the 3B scale, while cutting thought-phase latency by $2.5\times$–$6.9\times$ from compact math expressions to natural language. Projecting LOTUS's post-loop latents through the base LM head recovers the gold reasoning steps and even surfaces alternative valid intermediate steps, evidence that its latent space is interpretable and CoT-aligned. Ablations confirm that both the looped backbone and the parallel supervision on gold CoT tokens are essential. Code is available at https://github.com/yingfan-bot/lotus.

Executive Summary: Large language models increasingly rely on explicit chain-of-thought (CoT) reasoning—emitting intermediate steps token-by-token—to solve complex problems such as math word problems. While accurate, this approach incurs high latency because each token is generated sequentially. Latent CoT methods attempt to perform the same multi-step reasoning inside the model’s continuous hidden states rather than decoded tokens, promising major speed-ups, but they have consistently fallen short of explicit CoT performance beyond the 1 B parameter scale, with the gap widening as models grow.

This work set out to test whether a simple, scalable architecture—looped (recurrent-depth) Transformers combined with direct parallel supervision on gold CoT tokens—could close that performance gap while retaining the latency advantage of latent reasoning. The authors introduce LOTUS, which inserts a fixed grid of learnable latent blocks between the question and answer, refines them over a small number of parallel iterations using the same weights, and aligns each block to its corresponding gold CoT step via cross-entropy loss through the base model’s language-model head.

Experiments were run on Llama-3.2 backbones from 124 M to 3 B parameters, training on GSM8K-Aug and evaluating on GSM8K plus three out-of-domain math benchmarks. At the 3 B scale, LOTUS reaches within 1–1.5 points of explicit CoT accuracy on the in-domain test set and surpasses explicit CoT on the out-of-domain average. Thought-phase latency drops by a factor of 2.5× on compact math expressions and 6.9× on longer natural-language rationales. Ablations confirm that both the looped backbone and the parallel gold-token supervision are required; removing either sharply degrades results. Direct readout of the post-loop latent states through the base LM head recovers most gold intermediate tokens and assigns nontrivial probability to valid but unseen reasoning steps, indicating that the latent space is structured and CoT-aligned.

These findings matter because they demonstrate, for the first time at production-relevant scale, that latent reasoning need not trade accuracy for speed. The approach therefore offers a practical route to lower inference cost and higher throughput for reasoning workloads without sacrificing capability. The main limitations are that the method has so far been tested only on math problems, that block count, width, and loop depth are fixed hyperparameters chosen to cover typical step counts, and that very long chains still require fallback autoregressive completion.

For decision makers, the immediate next step is to pilot LOTUS-style latent reasoning on internal math or code-generation workloads where latency or throughput is a bottleneck. Longer-term priorities include making the latent budget adaptive at inference time, extending evaluation to non-mathematical domains, and measuring end-to-end cost and quality on larger models. With these caveats, the evidence indicates that LOTUS-style looped latent reasoning is now a credible, high-performance alternative to explicit CoT.

1. Introduction

Section Summary: The introduction discusses how chain-of-thought reasoning improves language models by generating step-by-step thoughts but remains expensive due to sequential token generation. It notes that prior latent reasoning methods, which try to perform those steps in hidden states instead, fall short of matching explicit chain-of-thought accuracy at larger scales because they still process thoughts sequentially and lack direct ties to correct reasoning tokens. The paper presents LOTUS, a looped transformer approach that uses parallel supervision on gold chain-of-thought tokens to close this accuracy gap at the 3B scale while cutting thinking time by several times.

Scaling inference compute, i.e., letting a model "think" before it answers, has become a dominant lever for increasing language model capabilities, with stronger performance now coming from longer reasoning chains rather than from model size alone ([1, 2]). Chain-of-thought (CoT) reasoning ([3]), where the model emits intermediate reasoning steps, drives this trend. However, since each token must be decoded sequentially, generating a CoT of length ${N}$ takes ${N}$ sequential model evaluations, making reasoning costly.

Latent reasoning aims to achieve the same at a fraction of the cost: it carries out the intermediate computation in continuous hidden states rather than decoded tokens, condensing many steps into fewer model evaluations. On small backbones such as $\textsc{GPT-2}$ ([4]), latent methods ([5, 6, 7]) already match CoT accuracy. Yet at the scales where CoT matters most, the promise breaks down: beyond $1$ B parameters, no existing latent method matches explicit CoT on math reasoning, and the gap widens with model size ([7]).

We identify two common issues in latent methods. (P1) Sequential generation: methods like Coconut ([5]), CODI ([6]), and SIM-CoT ([7]) produce latent thoughts in a pure autoregressive way, so the number of sequential forward passes still scales linearly with the number of latent tokens, keeping CoT's sequential generation bottleneck while giving up the readable intermediate steps that explicit CoT provides. (P2) Lack of CoT grounding: Explicit CoT gives every reasoning step a direct, position-aligned token target. Without similarly grounded supervision, latent traces can drift from meaningful computation and destabilize training at scale ([8, 9]). Methods like PCCoT ([10]) and KaVa ([11]) mitigate the sequential bottleneck with Jacobi-style iteration but still ground their latents through indirect signals that fall short of CoT-aligned target, such as hidden-state distillation from a teacher CoT model or a compressed teacher key–value cache.

An architecture that resolves both issues at once would refine its latents in a few parallel passes rather than one autoregressive step per token, and would ground them in a target as direct as explicit CoT's. Looped, or recurrent-depth, Transformers are a natural fit for the first requirement: they reuse the same weights across iterations to add computation depth without extra parameters, a design that recent work scales to billion-parameter pretraining ([12, 13, 14]). On parallelizable problems, they can provably reach a solution in fewer model iterations CoT ([15]); for example, LPTs solve graph reachability in a logarithmic number of iterations ([16])—an exponential improvement over sequential latent methods. This raises the question we study: can a looped Transformer bridge the gap to explicit CoT while reasoning in fewer sequential steps?

We find that pairing a looped padded backbone with simple supervision on the gold CoT tokens is surprisingly effective. We present LOTUS ($\underline{Lo}$ oped $\underline{T}$ ransformers with parallel s $\underline{u}$ pervision on latent $\underline{s})$. LOTUS places ${K}$ learnable padded latent blocks of ${c}$ tokens each between the question and answer, and processes the sequence with the base model ${R}$ times. An explicit CoT of length ${N}$ is thus processed in ${R} !\ll! N$ iterations of dense parallel computation rather than ${N}$ sequential per-token generations, which addresses P1. To address P2, LOTUS supervises the latents directly through the base model's own LM head: a cross-entropy loss aligns each latent position to its corresponding gold CoT-step token, all in parallel. The supervision target can also be routed through an auxiliary decoder that is conditioned on all latents and scores the gold CoT tokens under teacher forcing, which we instantiate on the identical looped backbone as LOTUS-aux. Both routings reach near-CoT accuracy at the 3B scale. LOTUS is, to our knowledge, the first latent reasoning method at the $\textsc{Llama-3.2-3B-Instruct}$ scale to bridge the in-domain gap to explicit CoT on GSM8k (Figure 1), while surpassing CoT on the out-of-domain average and cutting thinking time by $\mathbf{2.5\times}$. On a more verbose natural-language CoT stress test, LOTUS is on par with explicit CoT in accuracy while reducing thought-phase latency by $\mathbf{6.9\times}$. LOTUS's latent space is also interpretable: reading the post-loop latents through the base LM head recovers the gold reasoning steps, and even surfaces alternative valid intermediate steps the model was never trained on, evidence that the latents are genuinely CoT-aligned rather than opaque. We summarize our contributions as follows:

• We propose the recipe of looped padded Transformers with parallel cross-entropy supervision on gold CoT tokens. The method uses a padded latent prefix (Section 3.1), refines it with a looped backbone (Section 3.2), and supervises the post-loop latents through the base LM head (Section 3.3). We also instantiate LOTUS-aux, which routes the same latent supervision through an auxiliary decoder used only during training (Section 3.4).
• We show that LOTUS is, to our knowledge, the first latent reasoning method to bridge the latent–explicit CoT gap on math reasoning at the 3B scale: on $\textsc{Llama-3.2-3B-Instruct}$, it brings the GSM8K test accuracy to within $1$ point of explicit CoT and surpasses explicit CoT on the out-of-domain math average. It also cuts thought-phase latency by $2.5\times$ in the math-expression setting and by $6.9\times$ in the natural-language CoT stress test (Section 4.2).
• Ablations show that both the parallel supervision design (Section 4.3.1) and the looped architecture design (with enough block width and loop depth, Section 4.3.2) are important. The direct LM-head routing is robust across scale, whereas the auxiliary decoder routing matches it at 3B but degrades on smaller backbones (Section 4.2).
• LOTUS yields a transparent, CoT-aligned latent space. Reading the post-loop latents recovers most of the gold intermediate tokens (Section 5.1) while assigning nontrivial mass to unseen-but-valid alternatives (Section 5.2). A loss ablation shows that the CoT supervision anchors the readout to the gold chain, while answer supervision helps select the coherent joint (Section 5.3).

2. Preliminaries

Section Summary: This section introduces chain-of-thought (CoT) reasoning in language models, where explicit methods generate visible intermediate reasoning tokens step by step during inference, creating extra latency even though training uses parallel computation. It then describes latent reasoning approaches that perform similar multi-step thinking inside the model's hidden states rather than producing actual tokens, noting that these hidden representations can carry more information than discrete words. The text also defines looped padded transformers, which reuse the same layers repeatedly and insert extra trainable tokens to expand computation without adding parameters, and briefly contrasts several recent latent-reasoning techniques that vary in how they generate, align, or distill these internal states.

A glossary of all notation is provided in Table 11 in Appendix B.

Explicit CoT.

A language model (LM) ${f_{{{{\boldsymbol{\theta}}}}}}$ can map a question ${{Q}}$ to an answer ${{A}}$ via multi-step reasoning: generating ${S}$ intermediate steps ${T_{1}}, \ldots, {T_{{S}}}$ before the answer, where ${T_{{i}}}$ (${i}\in{1, \dots, S}$) may span multiple tokens. We write ${N}$ for the total number of CoT tokens across all ${S}$ steps. Standard CoT supervision trains all CoT tokens in parallel via teacher forcing, but at inference time the tokens must be generated sequentially, increasing latency. We refer to this setup as Explicit CoT.

Latent reasoning

replaces decoded CoT tokens with multi-step inference in the model's hidden states, which are commonly referred to as latent thoughts. [17] show latent thoughts carry strictly more information than discrete tokens: linearly many latent steps in a Transformer can solve graph reachability that would otherwise require quadratically many explicit CoT steps.

Looped padded Transformers.

A looped Transformer ([18]) applies ${f_{{{{\boldsymbol{\theta}}}}}}$ 's backbone multiple times, gaining depth without adding parameters. Padding the input with extra learnable tokens ([19, 20]) adds parallel computation to each iteration. The combination of the two creates looped padded Transformers (LPTs), which are more capable than either component alone.

Prior latent reasoning methods.

We contrast LOTUS against the most closely related latent reasoning methods, which we summarize here (details in Appendix C). Coconut ([5]) replaces each CoT step with a ${c}$ latent tokens generated autoregressively from the previous one, under a curriculum, and supervises only the answer. CODI ([6]) keeps this autoregressive latent decoding but adds a distillation target, aligning one designated latent's hidden state with the corresponding hidden state of a teacher CoT model. SIM-CoT ([7]) instead trains an auxiliary autoregressive decoder that aligns each latent with its CoT token. PCCoT ([10]) and KaVa ([11]) drop the autoregressive nature, refining a fixed budget of latent tokens in parallel over a few Jacobi iterations; they ground the latents indirectly, through CODI-style distillation and a compressed teacher key–value cache, respectively. We refer the reader to Appendix A for an extended discussion of related work on latent reasoning and looped Transformers, and to Appendix C for a detailed comparison with the related latent reasoning methods.

3. Method

Section Summary: LOTUS converts a standard language model into a latent reasoner by inserting a fixed grid of learnable latent tokens between special delimiters after the input question. These tokens are refined over multiple iterations of the model’s forward pass in a looped fashion that reuses a cached prefix, allowing the hidden states to perform parallel internal computation. After the loops, each latent position is directly supervised with the corresponding gold chain-of-thought token through the model’s own language-model head, while a final forward pass supervises the answer tokens, thereby grounding the latent trajectory in explicit reasoning steps.

Although LPTs are expressive enough to carry out efficient parallel reasoning, without structured supervision they often fail to generalize out-of-distribution ([21]). This motivates our central design choice: grounding the latent computation directly in the gold CoT tokens. We turn a standard LM into a latent reasoner with a recipe built from two ingredients:

• (i) a padded latent prefix processed by an LPT, and
• (ii) parallel cross-entropy supervision on exact gold CoT tokens.

We instantiate this recipe as LOTUS ($\underline{Lo}$ oped $\underline{T}$ ransformers with parallel s $\underline{u}$ pervision on latent $\underline{s}, [$ @fig:dslt]), which reads the supervision directly through the base model's LM head. We also study LOTUS-aux (Figure 3), which instead routes the supervision through an auxiliary decoder. We detail the padded latent prefix in Section 3.1, the looped computation in Section 3.2, and LOTUS's objective—together with an analysis of why it works (Section 3.3.2)—in Section 3.3, then introduce the LOTUS-aux variant in Section 3.4.

3.1 Padded Latent Prefix

For a question ${{Q}}$, ${S}$ chain-of-thought (CoT) steps ${T_{1}}, \dots, {T_{{S}}}$, and answer ${{A}} = (A_1, \dots, A_{| {{A}}|})$ (with $A_{| {{A}}|} = \langle\text{EoS}\rangle$), we construct an input

$ {\boldsymbol{x}} ;=; \big[, {{Q}}, ; {\langle\texttt{BoT}\rangle}, ; \underbrace{{\langle\texttt{lat}\rangle}\cdot s {\langle\texttt{lat}\rangle}}{\text{block } 1;(c \text{ tokens})}, ; \cdot s, ; \underbrace{{\langle\texttt{lat}\rangle}\cdot s {\langle\texttt{lat}\rangle}}{\text{block } K;(c \text{ tokens})}, ; {\langle\texttt{EoT}\rangle}, ; {{A}}, \big],\tag{1} $

where ${\langle\texttt{lat}\rangle}$ is a learnable latent token shared across positions, and ${\langle\texttt{BoT}\rangle}, {\langle\texttt{EoT}\rangle}$ are learnable special tokens that delimit the latent region. The block budget ${K}$ and per-block width ${c}$ are fixed hyperparameters, so the latent region contains ${K}{c}$ latent tokens plus the two delimiters. The latent prefix follows prior work on padding or pause tokens ([19, 20]), with each block ${i}$ intended to encode the latent representation of CoT step ${T_{{i}}}$, ${i}\in{1, \dots, K}$. Unlike the example-dependent step count ${S}$ of Section 2, the block budget ${K}$ is fixed across examples; we choose it to cover the step counts in our data, so that ${K}\ge S$ on almost all examples and each block can align to one CoT step (Appendix D).

3.2 Looped Latent Computation

Let ${f_{{{{\boldsymbol{\theta}}}}}}$ be the base causal LM and ${{{{\bm{E}}}}}\in {{\mathbb{R}}}^{{K}{c} \times d}$ the learnable latent embeddings, indexed as ${{{{\bm{E}}}}}{{i}, j}$ for block ${i}\in{1, \dots, K}$ and intra-block position $j\in{1, \dots, c}$. LOTUS first runs a single forward pass over the prefix $[{{Q}}, {\langle\texttt{BoT}\rangle}]$ to populate a KV cache ${\mathcal{C}}\text{pre}$, which is reused throughout training and inference without recomputation. It then iterates ${f_{{{{\boldsymbol{\theta}}}}}}$ for ${R}$ iterations, each refining the latent embeddings while attending to ${\mathcal{C}}\text{pre}$ (the trailing ${\langle\texttt{EoT}\rangle}$ and the answer suffix ${{A}}$ do not enter the loop). Writing ${{{\bm{h}}}^{(t)}}\in {{\mathbb{R}}}^{{K}{c} \times d}$ for the hidden states at the latent positions after iteration ${t}$, with entries ${{{\bm{h}}}^{(t)}{{i}, j}}$:

$ \begin{aligned} {{{\bm{h}}}^{(0)}} ;&=; {f_{{{{\boldsymbol{\theta}}}}}}!\left({{{{\bm{E}}}}} ;\middle|; {\mathcal{C}}\text{pre}\right), \ {{{\bm{h}}}^{(t)}} ;&=; {f{{{{\boldsymbol{\theta}}}}}}!\left({{{{\bm{E}}}}} + {{{\bm{h}}}^{(t-1)}};\middle|; {\mathcal{C}}_\text{pre}\right), \quad t = 1, \dots, R. \end{aligned}\tag{2} $

where the brackets denote subsequence concatenation. This is a finite-unroll, input-injected recurrence over a looped Transformer ([18, 22]), with a fixed unroll depth ${R}$ and gradients propagated through all ${R}$ iterations.

3.3 Direct Step-Aligned Supervision

3.3.1 Objective

Step CoT supervision loss.

After ${R}$ iterations of looping (Section 3.2), we supervise each latent position $(i, j)$ in the grid with the corresponding CoT step token. Let ${T_{{i}}} = ({T_{{i}, 1}}, \dots, {T_{{i}, c}})$ be the tokenized CoT step ${i}$ padded or truncated to ${c}$ tokens. We apply a single batched cross-entropy that directly aligns each block to its target step:

$ {\mathcal{L}{\mathrm{step}}} ;=; \frac{1}{{N{\mathrm{step}}}}\sum_{{i}=1}^{{K}}\sum_{j=1}^{{c}} {\mathrm{CE}}!\left({f_{\mathrm{head}}}({{{\bm{h}}}^{(R)}{{i}, j}}), , {T{{i}, j}}\right),\tag{3} $

where ${f_{\mathrm{head}}}$ is the LM head and ${N_{\mathrm{step}}} = \sum_{{i}} |{T_{{i}}}|$ is the total number of supervised (non-padding) CoT tokens.[^1]

[^1]: Padding tokens used to pad CoT steps to length ${c}$ are ignored by the cross-entropy (and excluded from ${N_{\mathrm{step}}}$).

Answer supervision loss.

${\mathcal{L}{\mathrm{ans}}}$ is computed in a separate final forward of ${f{{{{\boldsymbol{\theta}}}}}}$ that reuses the prefix cache ${\mathcal{C}}_\text{pre}$ and inserts the post-loop latent hidden states ${{{\bm{h}}}^{(R)}}$ at the latent rows:

$ {{{\bm{z}}}} ;=; {f_{{{{\boldsymbol{\theta}}}}}}!\left([{\langle\texttt{EoT}\rangle}, , {{A}}] ;\middle|; [{\mathcal{C}}_\text{pre}, , {{{\bm{h}}}^{(R)}}]\right),\tag{4} $

where ${{{\bm{z}}}}m\in {{\mathbb{R}}}^{{d}}$ collects the resulting hidden state at each answer-suffix position $m\in{0, 1, \dots, | {{A}}|-1}$ ($m=0$ being the trailing ${\langle\texttt{EoT}\rangle}$ position, which predicts $A_1$). We denote hidden states here by ${{{\bm{z}}}}$, distinct from the looped forward latents ${{{\bm{h}}}^{(t)}}$, to mark that they come from different sources. Applying the LM head ${f{\mathrm{head}}}$, standard next-token cross-entropy on the answer suffix gives:

$ {\mathcal{L}{\mathrm{ans}}} ;=; \frac{1}{| {{A}}|}\sum{m=0}^{| {{A}}|-1} {\mathrm{CE}}!\left({f_{\mathrm{head}}}({{{\bm{z}}}}m), , A{m+1}\right).\tag{5} $

Given a step supervision weight ${\lambda_{\mathrm{step}}}$, the full objective is

$ {\mathcal{L}} ;=; {\mathcal{L}{\mathrm{ans}}} + {\lambda{\mathrm{step}}}, {\mathcal{L}_{\mathrm{step}}}.\tag{6} $

Key properties of the objective.

The supervision in Equation 6 has three properties:

• (i) it is direct: the per-block target ${T_{{i}}}$ is scored through the same LM head used to produce the answer;
• (ii) it is parallel: all ${K}$ blocks are supervised simultaneously; and
• (iii) it is post-loop: the readout reads ${{\bm{h}}}^{(R)}$ after the final iteration, rather than at every iteration.

We ablate the design choice in Section 4.3, and compare against prior latent reasoning methods in Appendix C.

Inference.

At inference, LOTUS runs the looped forward and then decodes the answer. We compute the KV cache of the question ${{Q}}$ once, iterate the loop ${R}$ times to obtain the post-loop latents ${{{\bm{h}}}^{(R)}}$, and then decode the answer ${{A}}$ autoregressively through the base LM head while conditioning on those latents (same as Figure 2 only without computing the losses). The reasoning is carried entirely by the parallel latent blocks, so the only sequential decoding is over the short answer suffix, which is the source of the latency gains reported in Section 4.2.

No latent decoding for answer generation.

Note that the latent reasoning steps are never read out during answer generation. They are projected to the token space through the LM head only to align them with CoT reasoning during training with ${\mathcal{L}{\mathrm{step}}}$. The answer is always generated from the latent hidden states, both in the final-forward ${\mathcal{L}{\mathrm{ans}}}$ pass Equation (4) and at inference.

3.3.2 Roles of the step and answer losses

A natural worry about the LOTUS objective Equation (6) is that it supervises each latent position independently: Equation 3 scores every gold CoT token in parallel, with no autoregressive factorization of the chain. Why should independent per-position targets train a model that produces a globally coherent answer? We give a lens that resolves this, distinguishing the role of the two losses and explaining why both are needed.

Parallel chain likelihood.

The step loss ${\mathcal{L}{\mathrm{step}}}$ supervises a parallel chain likelihood (PCL) over the readout positions. Maximizing the per-position probability $p{{{{\boldsymbol{\theta}}}}}({T_{{i}, j}} \mid Q)$ of each gold token at its latent position induces the chain likelihood

$ p_{{{{\boldsymbol{\theta}}}}}^\mathrm{PCL}(T\mid Q) =\prod_{{i}=1}^{{K}}\prod_{j=1}^{{c}} p_{{{{\boldsymbol{\theta}}}}}({T_{{i}, j}}\mid Q),\tag{7} $

rather than the autoregressive conditionals $p_{{{{\boldsymbol{\theta}}}}}({T_{{i}, j}} \mid Q, T_{< i}, {T_{{i}, <j}})$. The factorization treats the readout as conditionally independent across positions, but the latent states themselves are not independent: the looped Transformer computes them jointly over the padded workspace, so the dependence the factorization drops is carried by the shared latent computation rather than by the loss.

Complementary roles.

This view makes the division of labor between the two losses precise. First, the support inclusion

$ \mathrm{supp}!\left(q(T \mid Q)\right) ;\subseteq; \prod_{{i}=1}^{{K}}\prod_{j=1}^{{c}} \mathrm{supp}!\left(q({T_{{i}, j}} \mid Q)\right)\tag{8} $

shows that every gold chain lies inside the Cartesian product of per-position gold token supports.[^2] So ${\mathcal{L}{\mathrm{step}}}$ plays a support-coverage role: it makes each position place mass on the right gold tokens, without needing to reproduce the joint distribution—which LOTUS never samples, since the answer is decoded directly from the jointly computed latent states. Second, ${\mathcal{L}{\mathrm{ans}}}$ supplies the global selection pressure that PCL alone lacks: because the answer is trained while conditioning on the entire latent configuration, gradients favor jointly computed hidden states that can actually support the correct answer. The two losses are thus complementary by construction—coverage from ${\mathcal{L}{\mathrm{step}}}$, joint selection from ${\mathcal{L}{\mathrm{ans}}}$ —which is exactly the behavior we verify empirically in Section 5.3, where dropping either loss degrades the gold chain likelihood.

[^2]: Minimizing ${\mathcal{L}{\mathrm{step}}}$, the cross-entropy to these marginals, pulls the model's PCL toward that product, i.e. minimizes $\mathrm{KL}\big(\prod{{i}, j}{q}({T_{{i}, j}}\mid Q), \big|, p_{{{{\boldsymbol{\theta}}}}}^\mathrm{PCL}(T\mid Q)\big)$.

To further probe how LOTUS compares to modeling the autoregressive chain likelihood, we introduce an autoregressive decoder variant in Section 3.4 and compare the results in Section 4.2.

3.4 LOTUS-aux: An Auxiliary Decoder Variant

An alternative route for CoT supervision explored in existing work is autoregressive chain likelihood: an auxiliary autoregressive decoder conditioned on the latent blocks scores the CoT tokens under teacher forcing. This creates a trade-off: in exchange for modeling the chain autoregressively, CoT supervision no longer flows directly through the answer-generating base LM head, and teacher forcing introduces an extra train/inference mismatch.

We instantiate this routing on our looped padded backbone as LOTUS-aux. LOTUS-aux reuses the same looped forward and final forward as LOTUS (Figure 2). The only difference is the supervision in the looped forward, which is routed through an auxiliary decoder rather than read directly through the base LM head. The auxiliary decoder mirrors SIM-CoT ([7]), but instead of generating latents autoregressively and supervising a single latent per CoT step, LOTUS-aux produces all ${K}$ latent blocks in parallel through the looped backbone and supervises a full ${c}$-token block per step. We describe this routing below.

Latent supervision via the auxiliary decoder.

As shown in Figure 3, the auxiliary decoder ${g_\phi}$ is an extra decoder model that, at each loop iteration $t$, reads latent block ${{{\bm{h}}}^{(t)}}t$ (we select block $i=t$ at iteration $t$, the ${c}$ post-loop latents of the $t$-th block) as a ${c}$-token prefix and predicts the gold CoT step ${T{t}}=(T_{t, 1}, \dots, T_{t, \lvert{T_{t}}\rvert})$ under teacher forcing, where $\lvert{T_{t}}\rvert$ is the step's token length. Let ${\widetilde{{{\bm{z}}}}^{(t)}}=({\widetilde{{{\bm{z}}}}^{(t)}{0}}, \dots, {\widetilde{{{\bm{z}}}}^{(t)}{\lvert{T_{t}}\rvert-1}})$ denote the output hidden states at each input CoT position $m\in{0, 1, \dots, \lvert{T_{t}}\rvert-1}$ ($m{=}0$ being the last latent position ${{{\bm{h}}}^{(t)}{t, c}}$, which predicts $T{t, 1}$, and $m{>}0$ the gold token $T_{t, m}$). Mirroring the final-forward pass of Equation 4, we generate ${\widetilde{{{\bm{z}}}}^{(t)}}$ with

$ {\widetilde{{{\bm{z}}}}^{(t)}} ;=; {g_\phi}!\left([, {{{\bm{h}}}^{(t)}{t, c}}, ; T{t, 1}, \dots, T_{t, \lvert{T_{t}}\rvert-1},] ;\middle|; {{{\bm{h}}}^{(t)}{t, 1}}, \dots, {{{\bm{h}}}^{(t)}{t, c-1}} \right),\tag{9} $

where the first ${c}-1$ latent positions ${{{\bm{h}}}^{(t)}{t, 1}}, \dots, {{{\bm{h}}}^{(t)}{t, c-1}}$ are the conditioning prefix. The teacher-forced next-token cross-entropy on the gold CoT gives:

$ {\mathcal{L}{\mathrm{step}}^{\mathrm{aux}}} ;=; \frac{1}{N\text{step}} \sum_{t=1}^{{K}} \sum_{m=0}^{\lvert{T_{t}}\rvert-1} {\mathrm{CE}}!\left({g_{\mathrm{head}}}({\widetilde{{{\bm{z}}}}^{(t)}{m}}), ; T{t, m+1}\right),\tag{10} $

where $N_\text{step}=\sum_{t=1}^{{K}}\lvert{T_{t}}\rvert$ is the total number of supervised CoT tokens. The supervision itself remains parallel: under teacher forcing the gold prefix is fed at every position, so all CoT positions are scored in a single forward pass with no sequential generation. In other words, the autoregressive factorization affects only the loss, not the computation.

Per-iteration supervision and the full objective.

Equation 10 reads block $i=t$ at iteration $t$ (per-iteration, rather than at the final iteration $R$). We validate this choice in Section 4.3.1, where per-iteration readouts outperform reading all blocks at the final iteration ${R}$ for the auxiliary decoder routing. The answer loss ${\mathcal{L}{\mathrm{ans}}}$ from Section 3.3 is unchanged, so the full LOTUS-aux objective is ${\mathcal{L}^{\mathrm{aux}}} = {\mathcal{L}{\mathrm{ans}}} + {\lambda_{\mathrm{step}}}, {\mathcal{L}{\mathrm{step}}^{\mathrm{aux}}}$. In our experiments ${g\phi}$ is initialized with a copy of the base LM and trained together with the base LM. Training and architectural details are in Appendix D.

Inference.

The auxiliary decoder ${g_\phi}$ is used only at training time. At inference, LOTUS-aux generates the final answer autoregressively from the post-loop latents exactly as LOTUS does (Figure 2(b)), so the inference efficiency gain of LOTUS also applies to LOTUS-aux.

4. Experiments

Section Summary: The experiments evaluate LOTUS's accuracy, inference efficiency, component contributions, and latent-space properties across GSM8K and related math-reasoning benchmarks using GPT-2 and Llama backbones of varying sizes. They compare against explicit chain-of-thought, no-CoT baselines, and prior latent-reasoning methods such as CODI and KaVa, testing different supervision strategies, block widths, and iteration counts. Results show that LOTUS approaches explicit CoT accuracy at the 3B scale, outperforming earlier latent approaches whose gaps widen with model size, while direct language-model supervision proves more robust than auxiliary-decoder variants at smaller scales.

Our experiments investigate four questions: how accurate LOTUS is, how efficient it is at inference, how much each design choice contributes, and whether its latent space is interpretable and CoT-aligned. Section 4.1 introduces the experimental setup, Section 4.2 presents accuracy and latency results against explicit CoT and prior latent baselines, Section 4.3 ablates the supervision design, latent block width ${c}$, loop depth ${R}$, and inference-time robustness to changing ${c}$ and ${R}$ without retraining, and Section 5 analyzes the learned latent representations. Full training and evaluation details are in Appendix D.

4.1 Setup

Datasets and backbones.

Following SIM-CoT ([7]), we train on GSM8k-Aug ([23]) with 385k training samples and report in-domain accuracy on the GSM8K test set ([24]), together with three out-of-domain benchmarks: GSM-Hard, MultiArith, and SVAMP. We additionally train on a natural-language version of GSM8K-Aug ([10]) as an efficiency stress test, where each reasoning step is a full-sentence rationale rather than a compact math expression. We use three backbones in increasing size: $\textsc{GPT-2}$ (124M) ([4]), $\textsc{Llama-3.2-1B-Instruct}$, and $\textsc{Llama-3.2-3B-Instruct}$ ([25]).

Methods.

We report LOTUS (${K}{=}6$ blocks of ${c}{=}25$ tokens on the $\textsc{Llama}$ backbones, ${c}{=}13$ on $\textsc{GPT-2}$, and ${R}{=}6$ looped iterations)[^3] and LOTUS + CODI, which adds CODI's single-position distillation loss. The auxiliary decoder variants LOTUS-aux and LOTUS-aux + CODI (Section 3.4) share the same configuration. We compare with Explicit CoT and No-CoT as reference, and with CODI ([6]) and CODI + SIM-CoT ([7]) for each backbone, which share the sequential latent reasoning budget (${R}{=}6$) with LOTUS and differ only in the number of latent tokens per step. Latent methods that use a different sequential compute budget: Coconut ([5]), Coconut + SIM-CoT (with 10 sequentially generated latent thought in total), and the parallel-latent methods PCCoT ([10]) and KaVa ([11]) (with 3 sequential steps and 24 latent tokens in total) are compared separately in Table 12 and Table 13 (Appendix C).

[^3]: ${K}$ is chosen to cover the target maximum CoT step count: GSM8K solutions have at most six steps for $99%$ of examples, so ${K}{=}6$ assigns one block per step and falls back to autoregressive completion otherwise.

4.2 Main Results

LOTUS matches explicit CoT at scale, where prior latent methods fall behind.

LOTUS scales well from $\textsc{GPT-2}$ to $\textsc{Llama-3.2-3B-Instruct}$ (Table 1). Prior latent methods match explicit CoT at small scale but fall behind as the backbone grows: CODI + SIM-CoT is level with explicit CoT at $\textsc{GPT-2}$ ($42.6$ vs. $42.7$) yet trails it by $9.2$ points at 3B, and KaVa ([11]) widens similarly ($1.9$ to $5.8$ points, Table 12). LOTUS instead stays within ${\sim}1.5$ points of explicit CoT in-domain at each scale, and its gap does not widen. While the smaller-scale results are rather clustered, the decisive comparison is at $\textsc{Llama-3.2-3B-Instruct}$: LOTUS surpasses explicit CoT on the out-of-domain average and brings the in-domain GSM8K gap to within $1.5$ points. LOTUS + CODI further tightens it to within $1$ point.

The two routings match at 3B, but only direct LOTUS stays robust at smaller scales.

At 3B, LOTUS-aux performs comparably to LOTUS and well above the CODI + SIM-CoT baseline, which uses an auxiliary decoder of the same size as the base LM. The gains therefore come from the looped padded backbone with parallel CoT supervision, regardless of the routing. This parity holds only at 3B: at smaller scales LOTUS-aux degrades, whereas direct LM-head supervision (LOTUS) stays robust across scale.

\begin{tabular}{l l c c c c c}
  \toprule
   {} & & \textbf{In-domain} & \multicolumn{4}{c}{\textbf{Out-of-domain}} \\
  \cmidrule(lr){3-3} \cmidrule(lr){4-7}
  Backbone & Method
   {} & GSM8K $\uparrow$
   {} & GSM-Hard $\uparrow$
   {} & MultiArith $\uparrow$
   {} & SVAMP $\uparrow$
   {} & Average $\uparrow$ \\
  \midrule
  \multirow{8}{*}{\textsc{GPT-2}}
   {} & Explicit CoT & $42.7$ & $9.0$ & $85.0$ & $41.6$ & $45.2$ \\
   {} & No-CoT & $19.1$ & $4.3$ & $41.1$ & $16.4$ & $20.6$ \\
  \cmidrule(l){2-7}
   {} & CODI & $42.0$ & $9.4$ & $\mathbf{93.0}$ & $41.7$ & $\underline{48.0}$ \\
   {} & CODI + SIM-CoT
   {} & $42.6$ & $9.4$ & $\underline{92.8}$ & $\mathbf{42.6}$ & $\mathbf{48.3}$ \\
   {} & \textbf{{LOTUS}}
   {} & $\mathbf{44.1{\scriptstyle\, \pm\, 0.7}}$
   {} & $\underline{9.5{\scriptstyle\, \pm\, 0.2}}$
   {} & $92.4{\scriptstyle\, \pm\, 1.4}$
   {} & $\underline{41.8{\scriptstyle\, \pm\, 0.9}}$
   {} & $47.9$ \\
   {} & \textbf{{LOTUS} + CODI}
   {} & $\underline{43.1{\scriptstyle\, \pm\, 1.2}}$
   {} & $\mathbf{9.8{\scriptstyle\, \pm\, 0.5}}$
   {} & $90.0{\scriptstyle\, \pm\, 4.3}$
   {} & $41.6{\scriptstyle\, \pm\, 0.4}$
   {} & $47.1$ \\
   {} & \textbf{{LOTUS}-aux}
   {} & $35.5{\scriptstyle\, \pm\, 4.5}$
   {} & $8.2{\scriptstyle\, \pm\, 1.2}$
   {} & $82.0{\scriptstyle\, \pm\, 2.8}$
   {} & $35.5{\scriptstyle\, \pm\, 1.5}$
   {} & $41.9$ \\
   {} & \textbf{{LOTUS}-aux + CODI}
   {} & $38.1{\scriptstyle\, \pm\, 3.7}$
   {} & $8.6{\scriptstyle\, \pm\, 0.5}$
   {} & $85.7{\scriptstyle\, \pm\, 4.1}$
   {} & $36.1{\scriptstyle\, \pm\, 2.6}$
   {} & $43.5$ \\
  \midrule
  \multirow{8}{*}{\shortstack{\textsc{Llama}\\ \textsc{3.2-1B}}}
   {} & Explicit CoT & $58.4$ & $13.9$ & $96.7$ & $65.7$ & $58.8$ \\
   {} & No-CoT & $28.8$ & $ 6.3$ & $50.3$ & $26.7$ & $27.8$ \\
  \cmidrule(l){2-7}
   {} & CODI & $52.7$ & $11.9$ & $95.0$ & $60.6$ & $55.8$ \\
   {} & CODI + SIM-CoT
   {} & $56.1$ & $\mathbf{12.7}$ & $96.2$ & $\mathbf{61.5}$ & $56.8$ \\
   {} & \textbf{{LOTUS}}
   {} & $\underline{57.3{\scriptstyle\, \pm\, 0.2}}$
   {} & $\mathbf{12.7{\scriptstyle\, \pm\, 0.5}}$
   {} & $\underline{98.3{\scriptstyle\, \pm\, 2.0}}$
   {} & $60.9{\scriptstyle\, \pm\, 0.6}$
   {} & $\underline{57.3}$ \\
   {} & \textbf{{LOTUS} + CODI}
   {} & $\mathbf{58.3{\scriptstyle\, \pm\, 0.8}}$
   {} & $\mathbf{12.7{\scriptstyle\, \pm\, 0.2}}$
   {} & $\mathbf{98.9{\scriptstyle\, \pm\, 0.6}}$
   {} & $\underline{61.1{\scriptstyle\, \pm\, 0.9}}$
   {} & $\mathbf{57.6}$ \\
   {} & \textbf{{LOTUS}-aux}
   {} & $55.4{\scriptstyle\, \pm\, 0.2}$
   {} & $12.6{\scriptstyle\, \pm\, 0.2}$
   {} & $97.8{\scriptstyle\, \pm\, 0.0}$
   {} & $58.0{\scriptstyle\, \pm\, 0.6}$
   {} & $56.1$ \\
   {} & \textbf{{LOTUS}-aux + CODI}
   {} & $50.6{\scriptstyle\, \pm\, 4.7}$
   {} & $11.4{\scriptstyle\, \pm\, 0.8}$
   {} & $95.7{\scriptstyle\, \pm\, 1.2}$
   {} & $55.0{\scriptstyle\, \pm\, 3.6}$
   {} & $54.0$ \\
  \midrule
  \multirow{8}{*}{\shortstack{\textsc{Llama}\\ \textsc{3.2-3B}}}
   {} & Explicit CoT & $71.5$ & $17.0$ & $98.3$ & $71.0$ & $62.1$ \\
   {} & No-CoT & $38.3$ & $ 9.5$ & $88.7$ & $52.9$ & $50.4$ \\
  \cmidrule(l){2-7}
   {} & CODI & $60.8$ & $14.3$ & $98.7$ & $73.3$ & $62.1$ \\
   {} & CODI + SIM-CoT
   {} & $62.3$ & $14.6$ & $98.8$ & $\underline{74.9}$ & $62.8$ \\
   {} & \textbf{{LOTUS}}
   {} & $70.0{\scriptstyle\, \pm\, 0.9}$
   {} & $16.0{\scriptstyle\, \pm\, 0.5}$
   {} & $\mathbf{99.9{\scriptstyle\, \pm\, 0.3}}$
   {} & $\mathbf{75.7{\scriptstyle\, \pm\, 0.8}}$
   {} & $\mathbf{63.9{\scriptstyle\, \pm\, 0.3}}$ \\
   {} & \textbf{{LOTUS} + CODI}
   {} & $\mathbf{70.6{\scriptstyle\, \pm\, 0.2}}$
   {} & $16.3{\scriptstyle\, \pm\, 0.3}$
   {} & $99.6{\scriptstyle\, \pm\, 0.3}$
   {} & $74.4{\scriptstyle\, \pm\, 1.4}$
   {} & $63.4{\scriptstyle\, \pm\, 0.3}$ \\
   {} & \textbf{{LOTUS}-aux}
   {} & $69.9{\scriptstyle\, \pm\, 0.9}$
   {} & $\mathbf{16.6{\scriptstyle\, \pm\, 0.5}}$
   {} & $\underline{99.8{\scriptstyle\, \pm\, 0.3}}$
   {} & $74.6{\scriptstyle\, \pm\, 2.2}$
   {} & $\underline{63.7{\scriptstyle\, \pm\, 0.8}}$ \\
   {} & \textbf{{LOTUS}-aux + CODI}
   {} & $\underline{70.5{\scriptstyle\, \pm\, 0.7}}$
   {} & $\underline{16.5{\scriptstyle\, \pm\, 0.4}}$
   {} & $99.4{\scriptstyle\, \pm\, 0.6}$
   {} & $73.9{\scriptstyle\, \pm\, 0.1}$
   {} & $63.3{\scriptstyle\, \pm\, 0.4}$ \\
  \bottomrule
  \end{tabular}

LOTUS reasons $2.5\times$ faster than explicit CoT.

The thought phase dominates the difference between methods (Table 2): CoT decodes its chain autoregressively, while LOTUS compresses the same "thinking" into ${R}$ parallel latent iterations, making its thought $\mathbf{2.5\times}$ faster than CoT and $\mathbf{1.2\times}$ faster than SIM-CoT[^4] ($\mathbf{2.1\times}$ faster than CoT in total[^5]). CODI is faster than LOTUS in the thought phase because it decodes only a single latent per step, rather than LOTUS's ${K}{c}!=!150$ parallel positions, but comes at the cost of accuracy. LOTUS is nonetheless only modestly slower ($133$ vs. $88$ ms) because it consumes all positions in parallel. The inference-time width sweep in Table 7 further separates sequential depth from parallel width: with ${R}$ fixed, increasing the latent prefix from $6$ to $300$ positions changes thought latency by only $30$ ms.

[^4]: We measure latency on a single NVIDIA H100 NVL at batch size $1$ with greedy decoding, split into query-prefill, thought, and answer phases. SIM-CoT and CODI both replace each explicit CoT step with a single latent token decoded sequentially. SIM-CoT additionally adds LoRA adapters (rank $128$) on top of CODI, which inflates every phase.

[^5]: LOTUS's prefill and answer phases are just the standard LM operations (Figure 2), so they match explicit CoT in Table 2 ($16.7$ vs. $15.9$ ms and $31.5$ vs. $29.5$ ms); the looped latent thought is the only phase that differs.

\begin{tabular}{l c c c c}
  \toprule
  Phase & \textbf{{LOTUS}} & Explicit CoT & SIM-CoT & CODI \\
  \midrule
  Query prefill & $16.7$ & $15.9$ & $29.5$ & $16.5$ \\
  Thought & $133.0$ & $338.8$ & $162.7$ & $88.2$ \\
  Answer & $31.5$ & $29.5$ & $59.2$ & $30.4$ \\
  \midrule
  Sum & $181.2$ & $384.2$ & $251.4$ & $135.1$ \\
  \bottomrule
  \end{tabular}

Speedup grows to $6.9 \times$ on natural-language CoT traces.

The main GSM8K-Aug setting uses compact math-expression steps, while natural-language CoT requires explicit models to decode much longer reasoning in natural language. In the 3B natural-language CoT stress test (Table 3), LOTUS is on par with explicit CoT in accuracy ($68.13$ vs. $68.41$) while cutting thought-phase latency from $963.6$ ms to $140.8$ ms with a $6.9\times$ speedup. On this natural-language setting, LOTUS ($68.13%$ GSM8K) far exceeds the latent baselines PCCoT ($47.6%$), CODI ($55.9%$), and KaVa ($60.0%$) reported by [11] (Table 13), staying within variance of explicit CoT (Table 3).

\begin{tabular}{l c c}
  \toprule
  Method & GSM8K acc. (\%) $\uparrow$ & Thought ms/example $\downarrow$ \\
  \midrule
  Explicit CoT & $68.41 \pm 0.59$ & $963.6$ ($1.0\times$) \\
  \textbf{{LOTUS}} & $68.13 \pm 0.77$ & $\mathbf{140.8}$ ($\mathbf{6.9\times}$) \\
  \bottomrule
  \end{tabular}

4.3 Ablation Studies

We ablate each component of LOTUS to isolate its contribution. Unless noted, all ablations use $\textsc{Llama-3.2-3B-Instruct}$, evaluate on GSM8K test set ([24]).

4.3.1 Latent Supervision Design

LOTUS supervises the latent blocks via a direct readout (${\mathcal{L}{\mathrm{step}}}$) of the post-loop latents ${{{\bm{h}}}^{(R)}}$ through the main LM head ${f{\mathrm{head}}}$. We compare it against five alternatives, holding the rest fixed (${K}{=}{R}{=}6$, ${c}{=}25$, also keeping ${\mathcal{L}{\mathrm{ans}}}$): (i) no latent supervision (only ${\mathcal{L}{\mathrm{ans}}}$); (ii) CODI ([6]), distilling a single pre-answer latent onto the teacher CoT model's hidden state; (iii) LOTUS (per-iter), reading ${{{\bm{h}}}^{(t)}}$ through ${f_{\mathrm{head}}}$ at each iteration $t$ rather than the post-loop ${{{\bm{h}}}^{(R)}}$; (iv) LOTUS-aux, reading ${{{\bm{h}}}^{(t)}}$ through a separate auxiliary decoder ${g_\phi}$ per iteration (default, Section 3.4); and (v) LOTUS-aux (post-loop), reading ${{{\bm{h}}}^{(R)}}$ through ${g_\phi}$ instead.

Both the looped backbone and latent supervision are beneficial.

Without any latent supervision, the looped backbone alone ($63.3%$) already exceeds CODI + SIM-CoT ($62.3%$, Table 1), which uses a single latent per step; the looped padded backbone is thus already beneficial on its own. Every supervised variant exceeds both no latent supervision and CODI-only, confirming that the gold CoT supervision contributes to the gain. LOTUS (post-loop direct, $70.0%$) matches the best LOTUS-aux schedule (per-iter, $69.9%$). The direct (LOTUS) and auxiliary decoder (LOTUS-aux) routings thus perform comparably at their best schedule.

The best supervision schedule differs between LOTUS and LOTUS-aux.

LOTUS performs best with post-loop readout ($70.0%$ vs. $68.2%$ per-iter), which lets the early latent blocks refine fully until the end of the loop. LOTUS-aux instead works best with per-iteration readout ($69.9%$ vs. $68.4%$ post-loop), which shortens the gradient path—possibly helpful because its extra decoder already lengthens it.

\begin{tabular}{l l c}
  \toprule
  Latent supervision & Routed via & Test acc. (\%) $\uparrow$ \\
  \midrule
  None (only ${\mathcal{L}_{\mathrm{ans}}}$) & --- & $63.3$ \\
  CODI only & --- & $64.4$ \\
  \textbf{{LOTUS}} (per-iter supervision of ${{{\bm{h}}}^{(t)}}$) & main LM head & $68.2$ \\
  \textbf{{LOTUS}} (post-loop supervision of ${{{\bm{h}}}^{(R)}}$, default) & main LM head & $\mathbf{70.0}$ \\
  \textbf{{LOTUS}-aux} (per-iter supervision of ${{{\bm{h}}}^{(t)}}$, default) & auxiliary decoder & $69.9$ \\
  \textbf{{LOTUS}-aux} (post-loop supervision of ${{{\bm{h}}}^{(R)}}$) & auxiliary decoder & $68.4$ \\
  \bottomrule
  \end{tabular}

4.3.2 Looped Architecture Design

::: {caption="Table 5: Training-time looped iterations R (separate models, latent prefix fixed at Kc=150)."}

:::

Latent token budget per block c.

We train separate models at $c\in{1, 5, 10, 25, 30}$ to measure how the per-block budget affects accuracy, holding $K{=}6$ fixed (Table 6). Accuracy rises sharply from $c{=}1$ ($49.7%$) to $c{=}5$ ($67.5%$), then climbs only marginally before plateauing with $c{=}25$ and $c{=}30$ tied at $70.0%$. A single token per CoT step is too narrow for the direct-readout supervision, but moderate widths suffice.

Inference-time c.

We sweep the inference-time budget $c\in{1, 5, 10, 25, 30, 50}$ on a single checkpoint trained at $c{=}25$ (Table 7). Reducing $c$ below the trained value hurts accuracy ($-19$ points at $c{=}1$, $-11$ points at $c{=}5$, $-6$ points at $c{=}10$), while exceeding it slightly helps before plateauing ($70.5%$ at both $c{=}30$ and $c{=}50$). Thought-phase latency increases only ${\sim}30$ ms ($111$ to $141$ ms) as $c$ varies by $50\times$, since the latents are processed in parallel within the fixed ${R}{=}6$ sequential steps. This contrasts with autoregressive CoT, whose latency grows linearly in the number of thought tokens.

: Table 6: Trained variants of LOTUS with different latent token budgets per block $c$.

$c$ (tokens / block)	Total latent positions ($Kc$)	Test acc. (%)
$1$	$6$	$49.7$
$5$	$30$	$67.5$
$10$	$60$	$68.4$
$25$	$150$	$70.0$
$30$	$180$	$70.0$

\begin{tabular}{l c c c}
  \toprule
  Inference-time $c$ & $Kc$ & GSM8K acc. (\%) $\uparrow$ & Avg. thought (ms / example) $\downarrow$ \\
  \midrule
  $1$ & $6$ & $51.4$ & $110.9$ \\
  $5$ & $30$ & $59.4$ & $120.3$ \\
  $10$ & $60$ & $64.2$ & $127.0$ \\
  $25$ & $150$ & $70.0$ & $133.0$ \\
  $30$ & $180$ & $70.5$ & $136.0$ \\
  $50$ & $300$ & $70.5$ & $141.2$ \\
  \bottomrule
  \end{tabular}

Looped iterations R.

We train separate models for ${R}\in{2, \dots, 6}$, fixing $K{=}6$ and $c{=}25$ so that the latent prefix has $Kc=150$ positions and only the depth of sequential refinement changes (Table 5). Larger ${R}$ gives the model more refinement iterations before the readout. As a result, accuracy rises steeply with ${R}$ (from $14.6%$ at ${R}{=}2$ to $70.0%$ at ${R}{=}6$), with gains continuing through ${R}{=}5$ ($68.1%$) before nearly saturating at ${R}{=}6$.

Inference-time R

Reusing the model trained at ${R}{=}6$, we vary ${R}\in{1, \dots, 7}$ at inference with the latent prefix fixed at $150$ positions (Table 5). Accuracy climbs with ${R}$ ($22.7%$ at ${R}{=}1$ to $70.0%$ at ${R}{=}6$), peaks at the trained ${R}{=}6$, and dips slightly at ${R}{=}7$ ($69.3%$). Running fewer iterations leaves the model less opportunity to refine, while running more than at training time brings no further gain.

5. Latent Representation Analysis

Section Summary: The analyses of LOTUS's learned latent states reveal that its performance improvements stem from organized internal computation rather than rote memorization. Post-loop representations directly encode chain-of-thought signals that can be read out as token probabilities, assign meaningful likelihood to valid but unseen reasoning steps, and depend on both step-by-step and final-answer supervision to form coherent intermediate states. Together these results indicate the model performs genuine latent reasoning, with the auxiliary decoder confirming that the structure is learned even when direct token supervision is limited.

We analyze the learned latent representations along three axes. Together, these analyses show that LOTUS's accuracy gains are accompanied by structured latent computation: the post-loop latents carry readable CoT token signal, assign nontrivial mass to unseen valid alternatives, and rely on both gold CoT supervision and answer supervision to form coherent reasoning states. Section 5.1 measures the likelihood assigned to the gold CoT under different readouts, Section 5.2 tests whether the latents place mass on unseen but valid reasoning chains, and Section 5.3 ablates the roles of gold CoT and answer supervision in shaping the representation. Extended block-to-step decode examples are deferred to Appendix F.1.

5.1 Likelihood of the Gold Chain under Different Readout Options

Recall that ${{{\bm{h}}}^{(R)}}$ denotes the post-loop latents and ${\widetilde{{{\bm{z}}}}^{(t)}}$ the aux decoder's output latents for $t\in{1, \dots, R}$. We evaluate four readouts in Table 8: (i) LOTUS, reading ${{{\bm{h}}}^{(R)}}$ through the base LM head; (ii) LOTUS-aux, reading ${{{\bm{h}}}^{(R)}}$ through ${g_{\mathrm{head}}}$; (iii) LOTUS-aux, reading ${\widetilde{{{\bm{z}}}}^{(t)}}$ through ${g_{\mathrm{head}}}$ under teacher forcing (TF) that matches training (each block's ${c}$ latents paired with that block's gold step tokens); and (iv) LOTUS-aux, reading ${\widetilde{{{\bm{z}}}}^{(t)}}$ through ${g_{\mathrm{head}}}$ but under free-running (FR), [^6] feeding the aux decoder's own generated token in place of the gold tokens.[^7]

[^6]: LOTUS-aux (FR) gives a fairer reference to LOTUS than LOTUS-aux (TF): at inference the gold CoT is unavailable, so LOTUS-aux must feed its decoder its own generated tokens, exactly the FR setting. The TF row instead feeds the gold prefix the model would not have at test time.

[^7]: Here we report greedy decoding results. Replacing them with temperature- $T$ sampling ($T \in {0.5, 0.7, 1.0, 1.5}$) moves FR NLL by $\leq 0.05$ and top- $1$ by $\leq 1$ pp.

Metrics.

We report the negative log-likelihood (NLL) of the gold CoT token sequence ${T}$ at the latent positions.[^8] We also report average ${P(\text{gold}\in\text{top-1})}$ and ${P(\text{gold}\in\text{top-5})}$ against the full $128\text{K}$-vocabulary softmax. All metrics are averaged over the $1{,}319$ GSM8K test questions.

[^8]: Note that the ${{{\bm{h}}}^{(R)}}$ readout scores the gold CoT under the PCL (Section 3.3.2) factorization and ${\widetilde{{{\bm{z}}}}^{(t)}}$ under an autoregressive one, so NLL is only directly comparable under the same factorization and only for reference across factorizations. Mathematically, PCL NLL has a higher lower bound than the autoregressive NLL (details in Appendix E).

LOTUS's post-loop latent ${{{\bm{h}}}^{(R)}}$ contains the gold CoT via a direct readout.

Reading LOTUS's post-loop latent ${{{\bm{h}}}^{(R)}}$ through ${f_{\mathrm{head}}}$ assigns the gold CoT a low $3.07$ NLL and $70.9%$ top- $1$ accuracy (Table 8), edging out LOTUS-aux (FR) at $64.5%$. The teacher-forced LOTUS-aux reader does better still ($1.56$ NLL, $84.3%$ top- $1$), but with the caveat that it uses the gold prefix as input, so we read it as an upper bound rather than a direct comparison. Still, the results confirm that the latents carry enough information to reconstruct the gold CoT chain.

LOTUS-aux has CoT signal in both ${{{\bm{h}}}^{(R)}}$ and ${\widetilde{{{\bm{z}}}}^{(t)}}$.

Surprisingly, even though ${{{\bm{h}}}^{(R)}}$ is never directly supervised to predict the gold CoT through ${g_{\mathrm{head}}}$ in LOTUS-aux, we can still read ${{{\bm{h}}}^{(R)}}$ through ${g_{\mathrm{head}}}$ with a $25.8%$ top- $5$ probability. Greedy decoding also surfaces some meaningful gold tokens, indicating that the CoT content is carried by the latents themselves. In fact, despite being supervised through different heads, LOTUS and LOTUS-aux have a similar effect of making the post-loop latents ${{{\bm{h}}}^{(R)}}$ put mass on the CoT steps, with the caveat that LOTUS-aux readouts are less formatted (which is expected since it is not supervised directly). We provide side-by-side ${{{\bm{h}}}^{(R)}}$ readout examples for both models in Appendix F.

::: {caption="Table 8: Likelihood of the gold CoT from different readout configurations."}

:::

5.2 Mass on Unseen Valid Chains

Beyond the gold chain in the test set, we test whether LOTUS and LOTUS-aux place mass on unseen valid reasoning chains rather than only memorizing the seen trace in the training set. We use the same four readout configurations as in Section 5.1.

Setup.

Given each question, we compare its ground-truth reasoning chain against an unseen-but-valid alternative. Let $G$ and $U$ denote the sets of intermediate numbers appearing uniquely in the ground-truth and the unseen reasoning chain, respectively. A random control set serves as a baseline. We report each set's average per-token negative log-likelihood (NLL) at the position where it surfaces most strongly (lower means more confident). We also report $P(U\in\text{top-}k)$, the fraction of $U$ numbers that surface within the top- $k$ readout at any latent position. See Appendix F.2 for the full construction and metric details.

Findings.

Across all four readouts the ordering $G !\ll! U !\ll!$ Random holds (Table 9, lower NLL $=$ surfaces more): each reader assigns much lower NLL to a ground-truth-only number than to an unseen-but-valid one, which in turn sits far below the random control. Crucially, the $U$ signal is concentrated rather than diffuse: $P(U\in\text{top-}k)$ columns confirm the unseen-valid intermediates genuinely surface in the top- $k$ (e.g., for LOTUS through the base LM head, $15.3%$ are top- $1$ and $64.0%$ fall within top- $5$, despite never being trained on). Appendix F.3 and Figure 4 give a path-level example and visual grid: two valid chains reach the same answer ($108$) through disjoint intermediates, and the unseen-path numbers ($24$, $18$) surface in the post-loop readout despite never appearing in the trained chain, question, or answer.

5.3 Disentangling the Two Losses in LOTUS

We now ask what each of LOTUS's two supervision losses contributes to the latent representation observed above. The theory in Section 3.3.2 ascribes complementary roles to ${\mathcal{L}{\mathrm{step}}}$ and ${\mathcal{L}{\mathrm{ans}}}$. We compare three models that differ in supervision: LOTUS (both losses), ${\mathcal{L}{\mathrm{step}}}$ only, and ${\mathcal{L}{\mathrm{ans}}}$ only.

Gold-chain NLL.

Reading the ${{{\bm{h}}}^{(R)}}$ through the base LM head ${f_{\mathrm{head}}}$, Table 10 reports NLL, $P(\text{gold}\in \text{top-1})$, and $P(\text{gold}\in \text{top-5})$ on the GSM8K test set, using the same protocol as Table 8.

::: {caption="Table 9: Evaluation on both seen and unseen valid CoT chains."}

:::

::: {caption="Table 10: Disentangling on the two loss components; NLL and per-position gold CoT accuracy on the GSM8K test."}

:::

${\mathcal{L}{\mathrm{step}}}$ and ${\mathcal{L}{\mathrm{ans}}}$ play complementary roles.

LOTUS beats both ablations (Table 10). ${\mathcal{L}{\mathrm{step}}}$-only, lacking the cross-block coupling and the answer grounding that ${\mathcal{L}{\mathrm{ans}}}$ supplies, is worst on every column. ${\mathcal{L}{\mathrm{ans}}}$-only, without directly tying ${{{\bm{h}}}^{(R)}}$ to the gold CoT, lands nearer the gold tokens (NLL $5.97$, top- $5$ $26.5%$) yet has a top- $1$ rate similar to ${\mathcal{L}{\mathrm{step}}}$-only and stays far short of LOTUS (NLL $3.07$, top- $5$ $85.8%$). The answer loss alone settles ${{{\bm{h}}}^{(R)}}$ into an answer-consistent neighborhood without aligning to the gold chain, matching the complementary roles posited in Section 3.3.2.

6. Conclusion

Section Summary: LOTUS introduces a latent reasoning approach that trains a looped transformer model to match explicit chain-of-thought performance by supervising it directly on gold reasoning tokens. On a Llama-3.2-3B model for math tasks, it matches or exceeds standard methods while cutting computation time by 2.5 times, with analysis confirming the hidden states capture meaningful and flexible reasoning structures. The work notes limitations, including evaluation only on math benchmarks and fixed settings that may struggle with unusually long reasoning chains.

We introduce LOTUS, showing that latent reasoning can approach the performance of explicit CoT by supervising a looped padded Transformer in parallel against the gold CoT tokens under the simple cross-entropy objective. On $\textsc{Llama-3.2-3B-Instruct}$, LOTUS bridges the in-domain gap to explicit CoT on GSM8K, surpasses CoT on the out-of-domain average, and cuts thought-phase latency by $2.5\times$. Ablations show the looped backbone, parallel gold CoT supervision, and sufficient block width and loop depth are each necessary. The latent representation analysis further shows the latents are transparent: the gold CoT is recoverable from them by a direct readout, they place graded probability on unseen but valid reasoning chains rather than a single memorized trace, and the step and answer losses contribute complementary structure.

Limitations.

We follow prior latent reasoning work ([5, 6, 7]) and evaluate on math benchmarks. Whether the recipe transfers to other domains remains an open direction for future work. The block budget ${K}$, per-block width ${c}$, and loop depth ${R}$ are fixed hyperparameters that must be set to cover the expected step count, so chains longer than ${K}$ steps fall back to autoregressive completion of the tail. Making ${K}$, ${c}$, and ${R}$ adaptive is a natural and promising direction for future work.

Acknowledgments

Anej Svete is supported by the ETH Zürich AI Center doctoral fellowship.

Appendix

Section Summary: The appendix provides an overview of supplementary materials that cover prior research on looped transformers, latent reasoning, and related methods, along with summaries of notation, model comparisons, experimental details, theoretical connections between certain bounds, and analyses of learned latent representations. It then surveys adjacent work in depth, highlighting how recurrent-depth transformers achieve efficient computation by reusing layers, how diffusion models and padding symbols support parallel or non-autoregressive reasoning, and how various approaches compress or replace explicit step-by-step chains with continuous representations. These discussions position the paper's contributions relative to existing techniques for scaling test-time compute and hidden reasoning.

Appendix A discusses related works including looped transformers, latent reasoning, and speculative decoding methods. Appendix B summarizes the notation used in the paper. Appendix C compares LOTUS and LOTUS-aux with the closest latent reasoning baselines. Appendix D includes the experimental and implementation details. Appendix E explains the relation between PCL and autoregressive NLL lower bounds. Finally, Appendix F provides the latent representation analysis details, including greedy readout examples and the multi-path setup and examples.

A. Related Work

We refer the reader to [26, 27, 28, 29, 30], and [31] for broad surveys of the latent reasoning and looped transformers landscapes. Below, we discuss the work most directly adjacent to ours.

Looped and recurrent-depth transformers.

Building on Universal Transformers ([18]), a line of work has established the theoretical foundations of looped architectures: [32] show Turing completeness via program simulation, placing looped transformers alongside CoT in expressivity ([33, 34, 35]). [15, 16, 36, 37] study the efficient reasoning abilities of looping, establishing their ability to solve problems with fewer model evaluations than CoT. [15] show that a shallow Transformer looped multiple times nearly matches a non-looped Transformer with the equivalent total depth on arithmetic and reasoning tasks, with looping implicitly simulating hidden reasoning steps. [38, 39] compare looped Transformers to CoT, and [40] draw connections to masked diffusion models. The latter connection also inspires approaches to improve looped transformers with time modulation ([41, 42]) and efficient sampling ([43]). An active line of work scales looped models to large-scale pretraining: [12] train a 3.5B recurrent-depth Transformer whose loop depth can be increased at test time, [13] pretrain LoopLM with token-level recurrence and learned depth allocation, and [14, 44] pretrain models that loop by superpositioning symbols in continuous space based on the logits produced after each iteration. LoopLM shows that looped computation can scale in pretraining, but its per-token loop at inference time does not by itself create the parallel padded latent workspace that LOTUS uses. Complementary work addresses different ways of implementing adaptive compute with looped architectures: Depth can be modulated per-symbol via gating ([45, 46, 47]), mixture-of-depths routing ([48]), or RL-trained stopping criteria ([49]). Depth-recurrent attention mixtures augment recurrence with attention mechanisms that can attend to latent representations of past iterations ([50, 51]). Parameter efficiency and flexibility are addressed by per-iteration LoRA adapters ([52]), symbol-level recursion routing ([53]), inner thinking mechanisms ([54]), and Mixture-of-Experts compatibility ([55]). Further work explores retrofitting recurrence into pretrained models ([56, 57]), compositional and length generalization ([22, 58]), reinforcement pretraining ([59]), and whether pretrained models use their depth efficiently ([60]).

Diffusion language models.

Diffusion language models are another route to parallel, non-autoregressive reasoning. Discrete-token variants face a parallel-decoding trap, where tokens denoised in parallel are conditionally independent, so recovering quality forces a partial return to sequential decoding ([61]). Notably, [40] show masked diffusion models are equivalent to padded looped Transformers, motivating supervising such a backbone directly, as we do. Continuous diffusion variants such as LaDiR ([62]) instead refine latent thoughts with a VAE and a separate diffusion model over many denoising steps for diversity and adaptive compute, while showing limited efficiency gain when reaching CoT-level performance.

Pause and padding symbols.

A parallel strand of work studies the affordances of discrete padding symbols. [19] and [20] show that they improve performance on natural language and formal tasks; [63] use discrete symbols in the continuous representation space as planning signals; and [64] introduce span-level pause symbols for efficient and interpretable latent reasoning. These discrete-symbol approaches motivate our use of continuous latent vectors as a richer computation medium, while our looped architecture lets each iteration refine the same latent rather than extending the sequence.

Compressing and replacing discrete CoT.

Latent CoT methods replace or compress explicit reasoning symbols with continuous representations. [65] supervise hidden-state representations of salient reasoning steps, [66] dynamically compress CoT chains into latent thoughts, [67] autoregressively encode CoT steps into hidden states decoded back to ground-truth symbols (analogously to our approach), and [68] interleave supervised latent states with generated text. MUX ([69]) studies continuous reasoning with multiplexed vocabulary-space targets. It uses KL regression to multiplexed targets. Each target is a position-weighted superposition over aligned reasoning subwords, so several subwords are mixed into one sequential latent. In contrast, LOTUS keeps step and token positions explicit in a parallel padded workspace and applies per-position cross-entropy on gold CoT traces. Latent CoT also enables parallel exploration of reasoning paths ([70, 71]), and test-time compute can be scaled in latent space via policy optimization ([72]), instance-level policy gradient ([73]), sampling conditioned on perturbed prompts ([74]), or parallel search ([75, 76]). Latent reasoning can be integrated with generation by mixing latent and text symbols ([77, 78]), updating the KV cache with an additional model ([79]), learning meta-symbols that guide generation ([80]), and semantically aligning padding symbols to spans of CoT symbols ([81]). Further work studies latent reasoning with variational autoencoder features ([82, 62]) and energy-based guidance ([83]). GRAM ([84]) guides thinking with recursive latent-variable modeling through shared transitions. It uses stochastic high-level updates and learns a target-conditioned posterior and prior through amortized variational inference. GRAM samples distinct stochastic trajectories, whereas LOTUS represents them within a jointly refined continuous workspace. Interpretability studies probe whether latents actually contribute to reasoning ([85, 86]), and empirical comparisons examine the effects of different supervision regimes ([87, 88]).

Speculative decoding.

Speculative decoding uses draft models to propose token continuations ([89, 90]). Related approaches include Lookahead ([91]) with Jacobi-style updates, Medusa ([92]) with auxiliary prediction heads, and EAGLE ([93]) with future hidden features. These methods parallelize proposals within an explicit token trace, whereas LOTUS parallelizes latent reasoning and decodes only the answer suffix.

B. Notation

Table 11 summarizes the notation used throughout the paper.

::: {caption="Table 11: Notation used in this paper."}

:::

C. Closely Related Latent Reasoning Methods

We position LOTUS and its auxiliary-decoder variant LOTUS-aux against the most closely related latent reasoning methods.

Coconut ([5])

introduced a curriculum-based approach that progressively replaces explicit CoT steps with latent tokens conditioned autoregressively on the previous latent. Both LOTUS and LOTUS-aux depart from Coconut in two ways:

• (i) they use a padded latent prefix processed by a looped Transformer, so all ${K}$ steps are decoded in parallel, not autoregressively, giving the ${{{{\mathcal{O}}}(R)}}$-iteration thought-phase cost, and
• (ii) they add step-aligned supervision on the latent blocks, where Coconut supervises only the answer.

Our ablation without latent supervision (Table 4) is the Coconut-style regime within our looped architecture, and it underperforms both variants.

CODI ([6]) and SIM-CoT ([7])

retain Coconut's autoregressive latent decoding but add an auxiliary supervision target: CODI aligns the hidden state of a single designated token with the corresponding teacher hidden state from a CoT model, and SIM-CoT trains an auxiliary autoregressive decoder that aligns each latent token with the corresponding CoT token. Our LOTUS removes the auxiliary path entirely, supervising the model's own post-loop hidden states through the answer LM head at block granularity. LOTUS-aux keeps an auxiliary autoregressive decoder, but produces all blocks in parallel through the looped backbone and supervises a full block per step rather than one latent per step.

Parallel Continuous CoT (PCCoT) ([10]) and KaVa ([11])

form a single Jacobi-iteration family: PCCoT introduces the backbone and KaVa builds on it. They both refine a fixed budget of $24$ latent tokens over $3$ Jacobi iterations and differ only in supervision. Like LOTUS, they process latent symbols in parallel and refine them over passes, but differ from our methods in three ways:

• (i) Architecture: PCCoT and KaVa use right-shifted Jacobi iterations to simulate sequential continuous CoT in parallel ([5]), so their latents are continuous thought vectors with no fixed alignment to individual gold CoT steps. LOTUS instead pins block ${i}$ to fixed positions across all loops, so it always maps to gold CoT step ${i}$ and is naturally supervised against that step's tokens;
• (ii) Supervision: PCCoT uses CODI ([6]), and KaVa supervises indirectly by distilling a compressed teacher KV-cache, whereas both our variants supervise each latent block against its exact gold CoT step; and
• (iii) Depth: PCCoT reports accuracy peaking at about $3$ Jacobi iterations and degrading beyond that, which is why KaVa inherits the $3$-iteration setting, whereas LOTUS improves monotonically with loop depth up to 6 (Table 5).

\begin{tabular}{l c c c}
  \toprule
  Method & \textsc{GPT-2} & \textsc{Llama-3.2-1B} & \textsc{Llama-3.2-3B} \\
  \midrule
  Explicit CoT & $42.7$ & $58.4$ & $71.5$ \\
  \midrule
  Coconut ([5]) & $36.6$ & $33.2$ & $-$ \\
  Coconut + SIM-CoT ([7]) & $44.8$ & $42.2$ & $-$ \\
  \midrule
  PCCoT ([10]) & $\mathbf{49.5}$ & $54.2$ & $54.7$ \\
  KaVa ([11]) & $-$ & $56.5$ & $65.7$ \\
  \textbf{{LOTUS}} & $44.1$ & $\mathbf{57.3}$ & $\mathbf{70.0}$ \\
  \bottomrule
  \end{tabular}

Comparison with latent methods at other compute budgets.

The main results (Table 1, Section 4.2) compare LOTUS against latent methods at the same sequential step budget (CODI and CODI + SIM-CoT, both ${R}{=}6$). Here we provide a more comprehensive accuracy comparison in Table 12 against latent methods whose sequential compute budget instead differs from LOTUS's ${R}{=}6$ sequential steps. Coconut and Coconut + SIM-CoT decode latents autoregressively under the curriculum of [5] ($10$ autoregressive latent tokens), while PCCoT and KaVa decode in parallel over a shared Jacobi-iteration backbone ($24$ latent tokens in total, refined over $3$ iterations). We take accuracies from [7], [11], and [10] for each method. LOTUS attains higher in-domain GSM8K accuracy than KaVa at both shared scales ($57.3$ vs. $56.5$ at $1$ B, $70.0$ vs. $65.7$ at $3$ B), staying within $1.5$ points of explicit CoT while PCCoT and KaVa fall further behind as the backbone grows. The missing entries follow what each source reports: [7] evaluate Coconut only up to $\textsc{Llama-3.2-1B}$, attributing Coconut's failure at larger scale to a latent-instability issue in which the latent representations become homogeneous and lose semantic diversity, causing training to collapse as the latent budget grows. [11] do not run KaVa on $\textsc{GPT-2}$.

\begin{tabular}{l c c c}
  \toprule
  Method & GSM8K acc. (\%) $\uparrow$ & GSM-Hard acc. (\%) $\uparrow$ & SVAMP acc. (\%) $\uparrow$ \\
  \midrule
  Explicit CoT & $68.41 \pm 0.59$ & $18.27 \pm 0.85$ & $71.93 \pm 1.62$ \\
  \midrule
  PCCoT & $47.6$ & $11.0$ & $65.2$ \\
  {{CODI}} & $55.9$ & $13.6$ & $70.1$ \\
  KaVa & $60.0$ & $14.8$ & $66.1$ \\
  \textbf{{LOTUS}}& $\mathbf{68.13 \pm 0.77}$ & $\mathbf{16.27 \pm 0.19}$ & $\mathbf{73.40 \pm 0.35}$ \\
  \bottomrule
  \end{tabular}

Natural-language CoT stress-test.

For the natural-language CoT stress test in Section 4.2, we use the $\textsc{Llama-3.2-3B-Instruct}$ backbone. The explicit CoT run reaches $68.41%$ GSM8K accuracy with $963.6$ ms thought latency. The looped run reaches $68.13%$ accuracy with $140.8$ ms thought latency. The latent baselines trail well behind on GSM8K (PCCoT $47.6$, CODI $55.9$, KaVa $60.0$). Out of domain, LOTUS leads all methods on SVAMP ($73.40$, above explicit CoT's $71.93$) and stays close to explicit CoT on GSM-Hard ($16.27$ vs. $18.27$), ahead of every latent baseline (Table 13). We exclude SIM-CoT from this comparison because [69] report weak SIM-CoT accuracy on the natural-language subset.

D. Experimental Details

This appendix includes the training and evaluation details in Section 4.1.

Latent prefix budget.

Each of the ${K}$ blocks holds ${c}$ latent positions, so the prefix spans ${K}{c}$ positions: ${c}{=}25$ ($150$ positions) for the $\textsc{Llama}$ backbones and ${c}{=}13$ ($78$ positions) for $\textsc{GPT-2}$. We use ${K}{=}6$, which covers the reference CoT step count for $99%$ of GSM8k-Aug examples. Longer problems fall back to autoregressive completion of the unsupervised tail.

Training curriculum.

We follow the staged curriculum of [5], converting one additional CoT step into a latent block every $E_\text{stage}$ epochs. At epoch $e$, we use

$ K_e ;=; \min!\left(\lfloor e / E_\text{stage}\rfloor, ; K\right), $

latent blocks: up to the first ${K}_e$ CoT steps become latent blocks, and the rest remain visible tokens. Thus $e{=}0$ is standard CoT fine-tuning and the curriculum saturates at ${K}_e{=}{K}$. We set the loop count at epoch $e$ to ${R}_e = r, K_e$, where $r$ is the loops-per-block ratio. Since each pass refines all blocks in parallel, ${K}_e$ controls the latent workspace width and ${R}e$ controls the depth. Unless stated otherwise, $E\text{stage}{=}1$ and $r{=}1$, so the saturated model uses ${R}{=}{K}{=}6$. The loop-depth ablation (Section 4.3.2) fixes ${K}{=}6$ and varies $r$ to decouple depth from width.

Optimization details.

We initialize from each backbone's standard explicit CoT checkpoint, and fine-tune all parameters with AdamW and gradient-norm clipping at $1.0$, in bf16 for the Llama backbones and fp32 for $\textsc{GPT-2}$. The learning rate is constant: $1!\times!10^{-4}$ for $\textsc{GPT-2}$ and $\textsc{Llama-3.2-1B-Instruct}$, and $5!\times!10^{-5}$ for $\textsc{Llama-3.2-3B-Instruct}$. The training objective Equation (6) weights the step loss at ${\lambda_{\mathrm{step}}}{=}0.05$ (for both LOTUS and LOTUS-aux). Adding CODI weights its loss at $1.0$. We train for $30$ epochs with an overall batch size of $128$, and select the best checkpoint by GSM8k validation accuracy, though the final epoch checkpoints perform similarly.

Natural-language training setup.

For the natural-language stress-test on LOTUS, we keep the same 3B looped backbone but switch to the natural-language GSM8k-Aug training set ([10]), increase latent width to $c{=}50$ and set ${\lambda_{\mathrm{step}}}$ to $0.033$. We initialize from the CoT checkpoint trained on the natural-language GSM8k-Aug. Other settings are identical to those used for the original GSM8k-Aug training.

LOTUS-aux auxiliary decoder.

For LOTUS-aux, the auxiliary decoder is a full-size deep copy of the base model, initialized from its weights but trained together with the main model. This full-size choice matches SIM-CoT ([7]), whose auxiliary decoder is architecturally identical to the base model. We also tried lightweight auxiliary decoders but found them unstable in training.

Evaluation.

Inference uses greedy decoding with batch size $1$. This isolates per-example latency from batching effects and calculates the average latency per sample accurately. Latency is measured on a single NVIDIA H100 and decomposed into query, thought, and answer phases (Table 2).

Batched KV-cache sharing.

For training, we left-pad shorter examples so the ${\langle\texttt{lat}\rangle}$ positions align across the batch. This lets us compute each example's question-prefix KV cache once and reuse it across all ${R}$ looped iterations.

E. PCL Versus Autoregressive NLL Lower Bounds

Here we provide the explanation mentioned in Section 5.1. PCL (Section 3.3.2) scores each gold token in parallel, $-\sum_{i, j}\log {{p_{{{{\boldsymbol{\theta}}}}}}}({T_{{i}, j}}\mid{{{\bm{h}}}^{(R)}})$, whereas the autoregressive factorization also conditions on the preceding gold tokens, $-\sum_{i, j}\log {{p_{{{{\boldsymbol{\theta}}}}}}}({T_{{i}, j}}\mid{{{\bm{h}}}^{(R)}}, T_{<(i, j)})$, where ${T}_{<(i, j)}$ denotes all gold tokens before position $(i, j)$ in reading order. The lowest expected NLL each can reach is the corresponding conditional entropy, and since conditioning never increases entropy,

$ H!\big({T_{{i}, j}} \mid {{{\bm{h}}}^{(R)}}\big) ;\geq; H!\big({T_{{i}, j}} \mid {{{\bm{h}}}^{(R)}}, T_{<(i, j)}\big),\tag{11} $

with equality only when the gold tokens are conditionally independent given the latents. So PCL has a strictly higher NLL floor whenever cross-token dependence exists.

F. Details for Latent Representation Analysis

F.1 Greedy Readout Examples

Here we give examples of two correctly answered and two incorrectly answered GSM8K test problems. We show three readouts of the post-loop latents: (A) the per-block top- $1$ token sequence reading ${{{\bm{h}}}^{(R)}}$ through LOTUS's base LM head ${f_{\mathrm{head}}}$; (B) the same readout applied to LOTUS-aux's post-loop latents ${{{\bm{h}}}^{(R)}}$ through its auxiliary head ${g_{\mathrm{head}}}$; and (C) the autoregressive CoT produced by LOTUS-aux's trained auxiliary decoder from ${\widetilde{{{\bm{z}}}}^{(t)}}$, conditioned on the same latents and decoded greedily from the start-of-CoT token. (A) and (B) show the top- $1$ token stream per block verbatim (newlines rendered as n), truncated at $60$ characters per block with ... marking the cut. (C) shows the raw aux-decoder output. We additionally report the final answer the model generates autoregressively from ${{{\bm{h}}}^{(R)}}$ after all ${R}{=}6$ loops. In each (A)/(B) readout we list the ${K}{=}6$ latent blocks top to bottom, labeled B0--B5.

For readability, blue marks the readable gold content: well-formed <<…>> CoT steps and exact gold chain numbers (step operands and intermediate results), including a number stuttered back-to-back such as 540540 for $540$. Wrong digit runs (e.g. 120120 or 3636) are left unmarked, and within each readout line we highlight only the first surfacing of each gold value.

Example 1 — Janet's ducks.

Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? Reference: <<16-3-4=9>>, <<9*2=18>>. Gold answer $18$.

(A) LOTUS LM head (${{{\bm{h}}}^{(R)}}$ through ${f_{\mathrm{head}}}$).

]
B0: @rdhl[<<3+4=7>>]n00>>n7>>n>>n>>n@rdhl[16]16–@rdhl[9]999999<<
B1: @rdhl[<<16-7=9>>]n00>>n>>n>>n>>n>>n*2==@rdhl[18]18>>n>>n>>n1818<<
B2: @rdhl[<<9*2=18>>]n00>>n>>n>>n18>>n>>n>>n= 1818>>n18>>n>>n...
B3: @rdhl[<<18*2=18>>]n>>n>>n>>n>>n>>n>>n>>n>>n=@rdhl[9]9>>n>>n18>>...
B4: ###134\*@rdhl[7]=>>n@rdhl[18]>>n>>n>>n>>n>>n>>n>>n2=2 .>>n181818>>...
B5: >>n@rdhl[18]=21818>>n>>n>>n===.###2== ,>>n1818222

(B) LOTUS-aux LM head (${{{\bm{h}}}^{(R)}}$ through ${g_{\mathrm{head}}}$).

]
B0: <<'+)@rdhl[7]))<<@rdhl[16]16–7= 1313nn###22*22
B1: =2189)#########2222= @rdhl[18]18#########22*@rdhl[9]=
B2: @rdhl[18]18 per#########2922 =1818 fresh######@rdhl[9]9\* no@rdhl[16]= 18
B3: @rdhl[18] fresh<|eot_id|>#########@rdhl[9]\* no no nothing=1818 Clean<|eot_...
B4: @rdhl[18]ly<|eot_id|>###### no nothing no nothing 1818<|eot_id|>###...
B5: ############ nothing nothing nothing=@rdhl[18]1818############### n...

(C) LOTUS-aux auxiliary decoder (${\widetilde{{{\bm{z}}}}^{(t)}}$ through ${g_{\mathrm{head}}}$, FR).

<<3+4=7>>\n<<16-7=9>>\n<<9*2=18>>\n<<18=18>>\n<<9*2=18>>\n<<18+18=18>>\n<<18*2=18>>\n<<18=18>>

Final answer (autoregressive, after $6$ loops). LOTUS: ### 18. LOTUS-aux: ### 18. Gold: $18$. Both correct.

Reading the chains. (A): the gold intermediates $\mathbf{9}$ and $\mathbf{18}$ surface from B1 onward and stabilize by B5. (B): the same numerics ($9, 18, 16, 7$) are present but interleaved with <|eot_id|> runs and filler. (C): the aux decoder produces the gold first two steps <<3+4=7>>, <<16-7=9>>, <<9*2=18>> and then loops on $18$.

Example 2 — James's sprints.

Question: James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week? Reference: <<3*3=9>>, <<9*60=540>>. Gold answer $540$.

(A) LOTUS LM head (${{{\bm{h}}}^{(R)}}$ through ${f_{\mathrm{head}}}$).

]
B0: @rdhl[<<3*60=180>>]n00>>n180180>>n>>n3==@rdhl[540]540>>n>>n>>n>>n>...
B1: @rdhl[<<180*3=540>>]n00>>n>>n>>n540>>n### 540>>n>>n >>n...
B2: @rdhl[<<540*540=540>>]n00>>n>>n>>n>>n>>n>>n### 540>>n>>n ...
B3: @rdhl[<<540=180=540>>]n540>>n>>n>>n>>n>>n>>n### 540>>n>>n...
B4: ###@rdhl[540]= =>>n###>>n>>n>>n 2 ###### 540540>>n>>n ###
B5: ### == >>n######### >>n### >>n>>n >>n

(B) LOTUS-aux LM head (${{{\bm{h}}}^{(R)}}$ through ${g_{\mathrm{head}}}$).

]
B0: @rdhl[3]3*3= @rdhl[180]180180\*###9993== @rdhl[540]540)######@rdhl[9]
B1: @rdhl[9]*@rdhl[60]= @rdhl[540]540))#########9*@rdhl[3] nothing = 540540)############
B2: /\* nothing= @rdhl[540]540)#########540+### nothing 540############...
B3: nothing is @rdhl[540]540###############540,### also 540540540#####...
B4: is@rdhl[540]540540############540,### also 540540#################...
B5: @rdhl[540]540###############540### definitely 540540540############...

(C) LOTUS-aux auxiliary decoder (${\widetilde{{{\bm{z}}}}^{(t)}}$ through ${g_{\mathrm{head}}}$, FR).

<<3*60=180>>\n<<180*3=540>>\n<<540=540>>\n<<540/60=9>>\n<<540=540>>\n<<540/10=540>>\n<<540>>

Final answer (autoregressive, after $6$ loops). LOTUS: ### 540. LOTUS-aux: ### 540. Gold: $540$. Both correct.

Reading the chains. (A): the gold step <<3\*60=180>> appears in B0 already (a valid alternative decomposition: meters per day first), and $\mathbf{540}$ stabilizes from B1. (B): $180, 540, 9, 3$ are all present but with heavy ### runs around them. (C): the aux decoder reproduces $\mathbf{180} \to \mathbf{540}$ cleanly, then loops.

Example 3 — Carla's download (failure).

Question: Carla is downloading a 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces a restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How long does it take to download the file? Reference: <<200*40*.01=80>>, <<80/2=40>>, <<200/2=100>>, <<40+100+20=160>>. Gold answer $160$.

(A) LOTUS LM head (${{{\bm{h}}}^{(R)}}$ through ${f_{\mathrm{head}}}$).

]
B0: @rdhl[<<200/2=6=120>>]n6>>n>>n>>n>>n>>n2>>n505050>>n605050<...
B1: @rdhl[<<40\*.2=40>>]n120>>n666>>n>>n>>n>>n>>n60>>n@rdhl[80]80>>n>>...
B2: @rdhl[<<60+40=120>>]n8>>n>>n>>n>>n>>n>>n>>n6050>>n>>n>>n...
B3: @rdhl[<<140+20=120>>]n8>>n33>>n>>n>>n>>n>>n2020=120>>n>>n>...
B4: @rdhl[<<140+20=120>>]n120>>n33>>n>>n>>n>>n###+60+/1>>n>>n>>...
B5: @rdhl[160]160+@rdhl[20]=120>>n180>>n>>n>>n>>n180>>n###+600+100120>>\...

(B) LOTUS-aux LM head (${{{\bm{h}}}^{(R)}}$ through ${g_{\mathrm{head}}}$).

]
B0: 100200/) @rdhl[100]100))<<100100**@rdhl[40] percentage,404040,<<100-+
B1: @rdhl[20]20= @rdhl[80]+ and###@rdhl[100]100+20++ 120120)###100100+220+
B2: 120120 plus###@rdhl[100]100*10020 140 plus###12080+2100= 120240)n...
B3: ###120120+@rdhl[20]20+120140140######120+202020 140140)n######120+
B4: +2= 140140)n)n######120120+@rdhl[20]2020 140100)n Ris######
B5: ###120+@rdhl[20]20= @rdhl[100]100 Ris#########120+20= 100100 Ris##########...

(C) LOTUS-aux auxiliary decoder (${\widetilde{{{\bm{z}}}}^{(t)}}$ through ${g_{\mathrm{head}}}$, FR).

<<100+20=120>>\n<<120+20=140>>\n<<100+20=120>>\n### 140

Final answer (autoregressive, after $6$ loops). LOTUS: ### 180. LOTUS-aux: ### 120. Gold: $160$.

Reading the chains. The latents recover most of the right intermediates ($\mathbf{80, 100, 40, 20}$) but fail to combine them: (A) flickers between $120, 140, 160, 180$ in the late blocks; (B) commits to $120 + 20$ chains; (C) outputs <<100+20=120>>, <<120+20=140>>, ... and emits $140$. Both methods land on plausible-but-wrong totals.

Example 4 — Melanie's vacuums (failure).

Question: Melanie is a door-to-door saleswoman. She sold a third of her vacuum cleaners at the green house, 2 more to the red house, and half of what was left at the orange house. If Melanie has 5 vacuum cleaners left, how many did she start with? Reference: <<5*2=10>>, <<10+2=12>>. Gold answer $18$. One extra step is omitted from the reference; the true chain is $5!\to!10!\to!12!\to!18$ ($12 \times 3/2 = 18$).

(A) LOTUS LM head (${{{\bm{h}}}^{(R)}}$ through ${f_{\mathrm{head}}}$).

]
B0: @rdhl[<<5*2=10>>]n33>>n@rdhl[12]>>n12>>n22=1312>>n1212121313<<
B1: @rdhl[<<10+2=17>>]n@rdhl[5]>>n5>>n17>>n>>n1//1181818.55<<
B2: @rdhl[<<9/(1/2/5=20.5>>]n20>>n15.18>>n>>n>>n>>n5>>n<<
B3: @rdhl[<<15*1=30>>]n15>>n@rdhl[5]>>n5>>n20>>n5=151818>>n>>n55<<
B4: @rdhl[<<3=15>>]n30>>n15>>n@rdhl[5]>>n5>>n>>n>>n55515>>n15>>n5>>n...
B5: @rdhl[<<15+5=15>>]n15>>n5>>n>>n>>n17>>n55515>>n@rdhl[18]>>n18>>n>...

(B) LOTUS-aux LM head (${{{\bm{h}}}^{(R)}}$ through ${g_{\mathrm{head}}}$).

]
B0: 105\* half @rdhl[10]10)+101010+2)= @rdhl[12]12)<<1212*3
B1: 3)1512..<<@rdhl[12]1212*3 third = 3636.###121212*3)
B2: 3636)######@rdhl[12]+33= 3636)######36+125= 
B3: 3636/#########36+@rdhl[12]++ 4836 Egg#########36+12+= 
B4: 483636#########+@rdhl[5]== 4836############36+5 3636###
B5: ############15+3 3636###############15###1 = 363615###

(C) LOTUS-aux auxiliary decoder (${\widetilde{{{\bm{z}}}}^{(t)}}$ through ${g_{\mathrm{head}}}$, FR).

<<10+2=12>>\n<<10+2=12>>\n<<10+2=12>>\n<<12+2=14>>\n...

Final answer (autoregressive, after $6$ loops). LOTUS: ### 30. LOTUS-aux: ### 36. Gold: $18$.

Reading the chains. (A): LOTUS's latents flicker between $15, 18, 30$ across blocks. The valid sub-chain $\mathbf{5!\to!10!\to!12!\to!18}$ surfaces in B5, but the final readout commits to $30$. (B): LOTUS-aux locks onto a wrong $\times 3$ branch (presumably "a third" read as multiply-by-three), with $36$ saturating B4 to B5. (C): the aux decoder loops on <<10+2=12>>.

What these readouts are (and are not). The readouts (A), (B), (C) are just diagnostic projections of the post-loop continuous latents back into the discrete token space. They are not part of the inference pipeline. At inference, the final answer is generated autoregressively from the continuous post-loop latents (the "Final answer" line in each example), conditioning on the latent representation itself rather than on any discretized chain. The intermediate streams are intended to give intuition about what numbers the latents place mass on, not as a quality measure of the model's reasoning: a noisy or repetitive (A)/(B) stream is compatible with a correct final answer, and a clean (C) chain is not required for the model to be correct.

F.2 Details for Multi-Path Analysis

Datasets.

Note that GSM8K-Aug ([23]) augments the original GSM8K ([24]) training split (about $7.5$ K problems) into the $385{,}620$ training examples we use. GSM8K is thus a subset of GSM8K-Aug, and the multi-path bank below is built from the original GSM8K problems, which are also in the training set.

Path bank.

For each question we identify its trained path (its gold chain in the training data) and an unseen-but-valid alternative drawn from a bank of up to ten correct paths obtained by rejection-sampling $\textsc{Llama-3.1-8B}$ ([25]) on the original GSM8K training questions ($7{,}444$ questions with $58{,}155$ verified paths in total), with each alternative verified absent from training.

Candidate sets and filtering.

We partition intermediate numbers into three disjoint sets: $G$ (trained-path only), $U$ (unseen-path only), and a random control from other questions matched to $U$ in both cardinality and single/multi-token composition. Numbers shared by both paths or appearing in the question or answer are excluded, since they cannot indicate which chain a readout recovered. The random control is fixed and shared across all four readouts and verified disjoint from the trained path, unseen path, question, and answer numbers. Intersecting the bank with our training set and keeping questions with $\geq 2$ disjoint intermediates per side yields $653$ candidate questions. We then apply a token-clean filter, requiring every only- $U$ number to share no sub-token with any trained-path, question, or answer number, leaving the final $341$ questions, spanning $871$ $G$, $974$ $U$, and $974$ Random numbers. Random stays matched to $U$ in cardinality and composition under every criterion.

Metrics.

We score a target number by its best-position, per-token NLL. A number spans one or more consecutive sub-tokens, and at every latent position the readout gives a softmax distribution over the vocabulary. We slide the number's sub-tokens across the latent positions. At each placement we average the sub-tokens' NLLs at their aligned positions, and keep the lowest such per-token average over all placements (for a single-token number, simply the negative log of its highest readout probability across positions). Per-token normalization puts single and multi-token numbers on the same scale, and scoring only the most probable span avoids double counting overlaps. We average this over the numbers in sets $G$, $U$, and Random within each question, then over the $341$ questions. The $P(U\in\text{top-}k)$ columns instead report a hit rate: the fraction of $U$ numbers recovered within the top- $k$ candidates (top- $k$ membership at any position for single-token numbers, and a consecutive top- $k$ match for multi-token ones), counting each number once however many positions it surfaces at, averaged within each question and then over the $341$ questions.

F.3 Multi-Path Readout Example

This example (referenced from Section 5.2) shows a question that admits two valid reasoning chains with the same answer $108$, read out by LOTUS (${{{\bm{h}}}^{(R)}}$ through the base LM head ${f_{\mathrm{head}}}$, i.e. the first row of Table 9).

Q: The bus driver drives an average of 2 hours each day, 5 days a week. From Monday to Wednesday he drove at an average speed of 12 kilometers per hour, and from Thursday to Friday at an average speed of 9 kilometers per hour. How many kilometers did the driver travel during these 5 days?

Trained path $G$: 3*2=6 -> 6*12=72 -> 2*2=4 -> 4*9=36 -> 72+36=108.

Unseen path $U$: 2*12=24 -> 24+24+24=72 -> 2\*9=18 -> 18+18=36 -> 72+36=108.

$\text{only}_G = {3, 4, 6}$, $\text{only}_U = {24, 18}$ (question and final answer excluded).

Figure 4 visualizes the same example, aligning the trained path, the unseen valid path, and the latent readout positions where the only- $U$ intermediates surface. The only- $U$ intermediates $24$ and $18$ are the per-day distances in the alternative decomposition, which forms daily distances before summing rather than aggregating hours first. Although neither appears in the trained reference path, both surface in the post-loop latent readout through the base LM head: $P(24){=}0.77$ (top- $1$, position $142$) and $P(18){=}0.14$ (top- $2$, position $116$). Every number here is a single token, so no only- $U$ number shares a sub-token with any trained-path, question, or answer number, confirming the surfacing is genuine unseen-chain content rather than a digit-overlap artifact.

References

Section Summary: This references section lists dozens of academic papers and preprints on advanced reasoning techniques in large language models. The works explore topics such as chain-of-thought prompting, latent or continuous internal reasoning spaces, looped transformers, and methods for compressing or scaling model thinking processes. Most entries come from recent conferences, arXiv uploads, and journals, spanning foundational studies from 2019 through newer 2025–2026 analyses and surveys.

[1] DeepSeek-AI (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature. 645(8081). pp. 633-638. doi:10.1038/s41586-025-09422-z. https://doi.org/10.1038/s41586-025-09422-z.

[2] OpenAI (2026). OpenAI o1 System Card. arXiv preprint arXiv:2412.16720. https://arxiv.org/abs/2412.16720. arXiv:2412.16720.

[3] Wei et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems.

[4] Radford et al.. Language models are unsupervised multitask learners.

[5] Shibo Hao et al. (2025). Training Large Language Models to Reason in a Continuous Latent Space. In Second Conference on Language Modeling. https://openreview.net/forum?id=Itxz7S4Ip3.

[6] Shen et al. (2025). CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 677–693. doi:10.18653/v1/2025.emnlp-main.36. https://aclanthology.org/2025.emnlp-main.36/.

[7] Xilin Wei et al. (2025). SIM-CoT: Supervised Implicit Chain-of-Thought. arXiv preprint arXiv:2509.20317. https://arxiv.org/abs/2509.20317. arXiv:2509.20317.

[8] Juncai Li et al. (2026). Chain Of Thought Compression: A Theoritical Analysis. arXiv preprint arXiv:2601.21576. https://arxiv.org/abs/2601.21576. arXiv:2601.21576.

[9] Jiaxuan Zou et al. (2026). Capabilities and Fundamental Limits of Latent Chain-of-Thought. arXiv preprint arXiv:2602.01148. https://arxiv.org/abs/2602.01148. arXiv:2602.01148.

[10] Wu et al. (2025). Parallel Continuous Chain-of-Thought with Jacobi Iteration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 914–926. doi:10.18653/v1/2025.emnlp-main.47. https://aclanthology.org/2025.emnlp-main.47/.

[11] Anna Kuzina et al. (2026). KaVa: Latent Reasoning via Compressed KV-Cache Distillation. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=ePrhcLbtGv.

[12] Jonas Geiping et al. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv preprint arXiv:2502.05171. https://arxiv.org/abs/2502.05171. arXiv:2502.05171.

[13] Rui-Jie Zhu et al. (2025). Scaling Latent Reasoning via Looped Language Models. arXiv preprint arXiv:2510.25741. https://arxiv.org/abs/2510.25741. arXiv:2510.25741.

[14] Boyi Zeng et al. (2026). PonderLM: Pretraining Language Models to Ponder in Continuous Space. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=UrM4MNRYZm.

[15] Nikunj Saunshi et al. (2025). Reasoning with Latent Thoughts: On the Power of Looped Transformers. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=din0lGfZFd.

[16] William Merrill and Ashish Sabharwal (2025). A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=5pHfYe10iX.

[17] Hanlin Zhu et al. (2025). Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought. In ICML 2025 Workshop on Methods and Opportunities at Small Scale. https://openreview.net/forum?id=1cD9iO5Isv.

[18] Mostafa Dehghani et al. (2019). Universal Transformers. arXiv preprint arXiv:1807.03819. https://arxiv.org/abs/1807.03819. arXiv:1807.03819.

[19] Sachin Goyal et al. (2024). Think before you speak: Training Language Models With Pause Tokens. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=ph04CRkPdC.

[20] Jacob Pfau et al. (2024). Lettextquoterights Think Dot by Dot: Hidden computation in transformer language models. In COLM. https://openreview.net/forum?id=NikbrdtYvG.

[21] Awni Altabaa et al. (2025). Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning. arXiv preprint arXiv:2510.14095. https://arxiv.org/abs/2510.14095. arXiv:2510.14095.

[22] Ying Fan et al. (2025). Looped Transformers for Length Generalization. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=2edigk8yoU.

[23] Deng et al. (2023). Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460.

[24] Karl Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. https://arxiv.org/abs/2110.14168. arXiv:2110.14168.

[25] Grattafiori et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

[26] Xinghao Chen et al. (2025). Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning. arXiv preprint arXiv:2505.16782. https://arxiv.org/abs/2505.16782. arXiv:2505.16782.

[27] Xinlei Yu et al. (2026). The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook. arXiv preprint arXiv:2604.02029. https://arxiv.org/abs/2604.02029. arXiv:2604.02029.

[28] Rui-Jie Zhu et al. (2025). A Survey on Latent Reasoning. arXiv preprint arXiv:2507.06203. https://arxiv.org/abs/2507.06203. arXiv:2507.06203.

[29] Jindong Li et al. (2025). Implicit Reasoning in Large Language Models: A Comprehensive Survey. arXiv preprint arXiv:2509.02350. https://arxiv.org/abs/2509.02350. arXiv:2509.02350.

[30] Sicheng Feng et al. (2025). Efficient Reasoning Models: A Survey. arXiv preprint arXiv:2504.10903. https://arxiv.org/abs/2504.10903. arXiv:2504.10903.

[31] Maile, Kaitlin and Sacramento, João (2026). Dynamic Parameter Reuse Augments Reasoning via Latent Chain of Thought. In ICLR Blogposts 2026. https://iclr-blogposts.github.io/2026/blog/2026/recur-refine-reason/.

[32] Giannou et al. (2023). Looped Transformers as Programmable Computers. In Proceedings of the 40th International Conference on Machine Learning. pp. 11398–11442. https://proceedings.mlr.press/v202/giannou23a.html.

[33] William Merrill and Ashish Sabharwal (2024). The Expressive Power of Transformers with Chain of Thought. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=NjNGlPh8Wh.

[34] Zhiyuan Li et al. (2024). Chain of Thought Empowers Transformers to Solve Inherently Serial Problems. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=3EWTEy9MTM.

[35] Nowak et al. (2024). On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12510–12548. doi:10.18653/v1/2024.acl-long.676. https://aclanthology.org/2024.acl-long.676/.

[36] William Merrill and Ashish Sabharwal (2025). Exact Expressive Power of Transformers with Padding. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=O1abxStFcy.

[37] Selim Jerad et al. (2026). Context-Free Recognition with Transformers. arXiv preprint arXiv:2601.01754. https://arxiv.org/abs/2601.01754. arXiv:2601.01754.

[38] Kevin Xu and Issei Sato (2025). To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformers. arXiv preprint arXiv:2505.19245. https://arxiv.org/abs/2505.19245. arXiv:2505.19245.

[39] Kevin Xu and Issei Sato (2025). A Formal Comparison Between Chain-of-Thought and Latent Thought. ArXiv. abs/2509.25239. https://api.semanticscholar.org/CorpusID:281681404.

[40] Anej Svete and Ashish Sabharwal (2026). On the Reasoning Abilities of Masked Diffusion Language Models. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=BVnIsh4Nz1.

[41] Ahmadreza Jeddi et al. (2026). LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=RzYXb5YWBs.

[42] Hung-Hsuan Chen (2026). Thinking Deeper, Not Longer: Depth-Recurrent Transformers for Compositional Generalization. arXiv preprint arXiv:2603.21676. https://arxiv.org/abs/2603.21676. arXiv:2603.21676.

[43] Jonas Geiping et al. (2025). Efficient Parallel Samplers for Recurrent-Depth Models and Their Connections to Diffusion Language Models. In NeurIPS 2025 Workshop on Efficient Reasoning. https://openreview.net/forum?id=nA5IRfAfbn.

[44] Boyi Zeng et al. (2026). PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space. arXiv preprint arXiv:2509.23184. https://arxiv.org/abs/2509.23184. arXiv:2509.23184.

[45] Kei-Sing Ng and Qingchen Wang (2024). Loop Neural Networks for Parameter Sharing. arXiv preprint arXiv:2409.14199. https://arxiv.org/abs/2409.14199. arXiv:2409.14199.

[46] Shixiang Song et al. (2026). AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth. arXiv preprint arXiv:2603.01914. https://arxiv.org/abs/2603.01914. arXiv:2603.01914.

[47] Ibraheem Muhammad Moosa et al. (2026). Understanding Dynamic Compute Allocation in Recurrent Transformers. arXiv preprint arXiv:2602.08864. https://arxiv.org/abs/2602.08864. arXiv:2602.08864.

[48] David Raposo et al. (2024). Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258. https://arxiv.org/abs/2404.02258. arXiv:2404.02258.

[49] Alex Ning et al. (2025). Learning When to Stop: Adaptive Latent Reasoning via Reinforcement Learning. https://arxiv.org/abs/2511.21581. arXiv:2511.21581.

[50] Tianyu Fu et al. (2025). Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models. arXiv preprint arXiv:2511.08577. https://arxiv.org/abs/2511.08577. arXiv:2511.08577.

[51] Jonas Knupp et al. (2026). Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves. arXiv preprint arXiv:2601.21582. https://arxiv.org/abs/2601.21582. arXiv:2601.21582.

[52] Sangmin Bae et al. (2025). Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=WwpYSOkkCt.

[53] Sangmin Bae et al. (2025). Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=QuqsEIVWIG.

[54] Chen et al. (2025). Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 28241–28259. doi:10.18653/v1/2025.acl-long.1369. https://aclanthology.org/2025.acl-long.1369/.

[55] Róbert Csordás et al. (2024). MoEUT: Mixture-of-Experts Universal Transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=ZxVrkm7Bjl.

[56] Sean McLeish et al. (2025). Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence. arXiv preprint arXiv:2511.07384. https://arxiv.org/abs/2511.07384. arXiv:2511.07384.

[57] Yeskendir Koishekenov et al. (2025). Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts. arXiv preprint arXiv:2510.07358. https://arxiv.org/abs/2510.07358. arXiv:2510.07358.

[58] Harsh Kohli et al. (2026). Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers. arXiv preprint arXiv:2604.07822. https://arxiv.org/abs/2604.07822. arXiv:2604.07822.

[59] Guo Tang et al. (2026). LoopRPT: Reinforcement Pre-Training for Looped Language Models. arXiv preprint arXiv:2603.19714. https://arxiv.org/abs/2603.19714. arXiv:2603.19714.

[60] Róbert Csordás et al. (2025). Do Language Models Use Their Depth Efficiently?. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=Kz6eUL86XP.

[61] Liu et al. (2025). Tidar: Think in diffusion, talk in autoregression. arXiv preprint arXiv:2511.08923.

[62] Haoqiang Kang et al. (2026). LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=z5cPEZ4n6i.

[63] Xinyi Wang et al. (2024). Guiding Language Model Reasoning with Planning Tokens. In First Conference on Language Modeling. https://openreview.net/forum?id=wi9IffRhVM.

[64] Yunlong Chu et al. (2026). SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models. arXiv preprint arXiv:2603.06222. https://arxiv.org/abs/2603.06222. arXiv:2603.06222.

[65] Jeffrey Cheng and Benjamin Van Durme (2024). Compressed Chain of Thought: Efficient Reasoning Through Dense Representations. arXiv preprint arXiv:2412.13171. https://arxiv.org/abs/2412.13171. arXiv:2412.13171.

[66] Wenhui Tan et al. (2025). Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=AQsko3PPUe.

[67] Jiecong Wang et al. (2026). Latent Chain-of-Thought as Planning: Decoupling Reasoning from Verbalization. arXiv preprint arXiv:2601.21358. https://arxiv.org/abs/2601.21358. arXiv:2601.21358.

[68] Ido Amos et al. (2026). Latent Reasoning with Supervised Thinking States. arXiv preprint arXiv:2602.08332. https://arxiv.org/abs/2602.08332. arXiv:2602.08332.

[69] Ayhan Suleymanzade et al. (2026). MUX: Continuous Reasoning via Multiplexed Tokens. In ICLR 2026 Workshop on Logical Reasoning of Large Language Models. https://openreview.net/forum?id=Ee78Uqsq0a.

[70] Halil Alperen Gozeten et al. (2026). Continuous Chain of Thought Enables Parallel Exploration and Reasoning. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=sTPKDKn5ig.

[71] Houjun Liu et al. (2026). Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space. arXiv preprint arXiv:2510.00219. https://arxiv.org/abs/2510.00219. arXiv:2510.00219.

[72] Wengao Ye et al. (2026). Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=r1WEQzkCQv.

[73] Hengli Li et al. (2026). Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space. arXiv preprint arXiv:2505.13308. https://arxiv.org/abs/2505.13308. arXiv:2505.13308.

[74] Yige Xu et al. (2025). SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning. arXiv preprint arXiv:2505.11484. https://arxiv.org/abs/2505.11484. arXiv:2505.11484.

[75] Long Lian et al. (2025). ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models. arXiv preprint arXiv:2512.07843. https://arxiv.org/abs/2512.07843. arXiv:2512.07843.

[76] Runyang You et al. (2026). Parallel Test-Time Scaling for Latent Reasoning Models. arXiv preprint arXiv:2510.07745. https://arxiv.org/abs/2510.07745. arXiv:2510.07745.

[77] Renyu Fu and Guibo Luo (2026). SeLaR: Selective Latent Reasoning in Large Language Models. arXiv preprint arXiv:2604.08299. https://arxiv.org/abs/2604.08299. arXiv:2604.08299.

[78] Jingcheng Deng et al. (2026). LLM Latent Reasoning as Chain of Superposition. arXiv preprint arXiv:2510.15522. https://arxiv.org/abs/2510.15522. arXiv:2510.15522.

[79] Luyang Liu et al. (2025). Deliberation in Latent Space via Differentiable Cache Augmentation. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=IaUJl5RCOu.

[80] Alok Shah et al. (2025). Language Modeling with Learned Meta-Tokens. In ICML 2025 Workshop on Long-Context Foundation Models. https://openreview.net/forum?id=oaHYnLldHM.

[81] Yinhan He et al. (2025). SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=1ZuzFUMtx6.

[82] DiJia Su et al. (2025). Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=hYfOPXrbUr.

[83] Zhikang Chen et al. (2025). Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thought. arXiv preprint arXiv:2511.07124. https://arxiv.org/abs/2511.07124. arXiv:2511.07124.

[84] Baek et al. (2026). Generative Recursive Reasoning. arXiv preprint arXiv:2605.19376.

[85] Yuyi Zhang et al. (2025). Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought. arXiv preprint arXiv:2512.21711. https://arxiv.org/abs/2512.21711. arXiv:2512.21711.

[86] Michael Rizvi-Martel and Marius Mosbach (2026). The Illusion of Superposition in Latent CoT via Soft Thinking. In Workshop on Latent & Implicit Thinking – Going Beyond CoT Reasoning. https://openreview.net/forum?id=FvPx9Nzvnw.

[87] Yingqian Cui et al. (2026). How Do Latent Reasoning Methods Perform Under Weak and Strong Supervision?. arXiv preprint arXiv:2602.22441. https://arxiv.org/abs/2602.22441. arXiv:2602.22441.

[88] Ayhan Suleymanzade et al. (2026). Limits of Continuous Chain-of-Thought in Multi-Step and Multi-Chain Reasoning. In Logical and Symbolic Reasoning in Language Models @ AAAI 2026. https://openreview.net/forum?id=UQFTJPqJAc.

[89] Leviathan et al. (2023). Fast inference from transformers via speculative decoding. In International Conference on Machine Learning. pp. 19274–19286.

[90] Chen et al. (2023). Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318.

[91] Fu et al. (2024). Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. In International Conference on Machine Learning. pp. 14060–14079.

[92] Cai et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. In International Conference on Machine Learning. pp. 5209–5235.

[93] Li et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In International Conference on Machine Learning. pp. 28935–28948.