ChapterPal ChapterPal
Log In Sign Up
Unlimited OCR Works

Unlimited OCR Works: Welcome the Era of One-shot Long-horizon Parsing

Baidu Inc.

Abstract

Recently, end-to-end OCR models, exemplified by DeepSeek OCR, have once again thrust OCR into the spotlight. A widely held view is that employing a large language model (LLM) as the decoder allows the model to leverage the prior distribution of language, leading to improved OCR performance. However, the downside is equally evident: as the output sequence lengthens, the accumulated KV cache drives up memory consumption and progressively slows down generation. This stands in stark contrast to humans, who exhibit no such decline in efficiency during long-horizon copying tasks. In this technical report, we propose Unlimited OCR, a model designed to emulate human parsing working memory. Taking DeepSeek OCR as the baseline, we replace all attention layers in the decoder with our proposed Reference Sliding Window Attention (R-SWA), which reduces attention computation costs while maintaining a constant KV cache throughout the entire decoding process. By combining the high compression rate of DeepSeek OCR's encoder with our constant KV cache design, Unlimited OCR can transcribe dozens of pages of documents in a single forward pass under a standard maximum length of 32K. More importantly, R-SWA is a general-purpose parsing attention mechanism — beyond OCR, it is equally applicable to tasks such as ASR, translation, etc. Codes and model weights are publicly available at http://github.com/baidu/Unlimited-OCR.

**Figure 1:** Illustration of Reference Sliding Window Attention (R-SWA). Each generated token attends to all reference tokens (visual tokens in OCR) and the preceding $n$ output tokens (128 by default). Compared to standard full attention, R-SWA maintains a constant KV cache throughout decoding. Compared to vanilla SWA, it preserves visual token fidelity by excluding them from state transitions, thereby avoiding progressive blurring.

Executive Summary: Unlimited OCR is a new end-to-end document parsing model from Baidu that can transcribe dozens of pages in a single forward pass. Current vision-language models struggle with multi-page documents because standard attention mechanisms cause memory use and generation speed to degrade sharply as output length grows. This creates practical barriers for high-volume document processing.

The report set out to test whether a simple change to the decoder’s attention could remove these limits while preserving or improving accuracy. Starting from the DeepSeek OCR baseline, the team replaced every attention layer with Reference Sliding Window Attention (R-SWA). R-SWA keeps all visual tokens and the prompt visible at all times but restricts each newly generated token to attending only to the most recent 128 output tokens. Visual features remain fixed after the first encoding step, so memory and compute stay constant regardless of how many pages are processed.

On the OmniDocBench v1.5 document-parsing benchmark, Unlimited OCR reached an overall score of 93 percent, a 6-point gain over the DeepSeek OCR baseline. It also showed higher throughput (about 13 percent faster at typical concurrency) and maintained generation speed even at six thousand output tokens, while the baseline slowed noticeably. In internal tests on books of 2 to 40 pages, edit distance stayed below 0.11 and text remained highly non-repetitive. Performance held across nine document categories, including complex layouts.

These results matter because they remove a key scaling obstacle for document-heavy workflows. Organizations that currently split long files into pages or reset model state between pages can instead process entire documents continuously with stable latency and lower memory. The same attention pattern is described as applicable to other reference-based tasks such as speech recognition and translation.

The authors recommend extending context length to 128K tokens in the near term and exploring a prefill-pool mechanism in the longer term to approach truly unbounded document lengths. They also plan to test R-SWA on additional sequence tasks. The main limitations are the fixed 32K training context and dependence on the encoder’s compression for very large page counts. Results are based on continued training from a single strong baseline rather than training from scratch, so broader validation across model families would increase confidence.

1. Introduction

Section Summary: Current AI models struggle with long tasks like transcribing an entire book because they process information page by page in isolation, resetting their internal state each time rather than maintaining a continuous flow of attention like humans do. To address this, the authors introduce Unlimited OCR, which uses a new Reference Sliding Window Attention mechanism that lets the model keep full access to the source image and prompt while only tracking a small recent window of its own output. This approach reduces memory and computation demands, allows dozens of pages to be handled in one pass, and modestly improves accuracy on document parsing benchmarks.

Humans are remarkably adept at seemingly straightforward long-horizon tasks: transcribing hundreds of book pages, translating hours-long audio recordings, and the like. Yet these are precisely the tasks where current models fall short. Take OCR as an example—no existing model [1, 2, 3, 4] can even parse ten of pages in a single forward pass. Instead, they resort to page-by-page processing in a for-loop fashion, resetting memory at every step. This divergence is far from superficial, and it cannot be reduced to a mere lack of sufficient context. When humans perform such tasks, they maintain a continuous cognitive state in which distant outputs fade softly from memory, while nearby context is used to track progress. The for-loop paradigm, by contrast, erases memory entirely at each page, fragmenting a coherent long-horizon process into isolated short tasks managed by an external scheduler. It works to some extent, but it remains an engineering workaround, not a step toward AGI-purpose intelligence.

Consider the act of transcribing a document. As we copy each character, we do not scan the entire text already written; we simply glance at the immediately surrounding context to stay oriented. This everyday behavior points to an attention pattern fundamentally different from those in current models. It is not standard full attention—the full history is never fully consulted. Nor does it resemble linear attention, since visual/reference tokens undergo no recurrent state updates; such updates would progressively blur the visual features and degrade recognition accuracy. To align more closely with this natural attention flow, and to explore how multimodal large language models (MLLMs) [5, 6, 7, 8] can handle simple long-horizon parsing tasks, we propose Unlimited OCR. Our main contributions are as follows:

  • We introduce Reference Sliding Window Attention (R-SWA), illustrated in Figure 1. For each token, R-SWA attends to all reference tokens—visual tokens and the prompt—while limiting output attention to the preceding $n$ tokens ($n$ defaults to 128). In this way, each token perceives the full image and autonomously tracks OCR progress through state transitions within the causal sliding window. This design keeps the KV cache constant during inference, alleviating memory pressure and reducing the computational cost.
  • Building on R-SWA, we propose Unlimited OCR. Using DeepSeek OCR as our baseline, we retain its DeepEncoder with high image compression rate, modifying all the decoder LLM's attention mechanism to R-SWA. This enables Unlimited OCR to parse dozens of paper pages in a single forward pass. R-SWA also yields a modest improvement in general OCR accuracy. Specifically, Unlimited OCR achieves 93% on the OmniDocBench v1.5 benchmark [9], outperforming the DeepSeek OCR baseline by 6%.
  • We conduct a preliminary validation of MLLM architectures with linear-complexity attention on OCR tasks, particularly in long-horizon scenarios. Rather than brute-force scaling up the training context, we identify an elegant approach that achieves long-horizon OCR. Looking ahead, we see promise in extending R-SWA to ASR, translation, and other reference-based tasks that demand long-horizon dependency modeling.

In summary, we present R-SWA, which substantially reduces both the computational cost of attention and the memory footprint in the long-horizon inference. Building on R-SWA, Unlimited OCR not only enables one-shot parsing of an entire book, but also surpasses the DeepSeek OCR baseline by a large margin on popular document parsing benchmarks. Furthermore, we believe R-SWA holds promise well beyond OCR.

2. Related Works

Section Summary: Recent OCR systems largely fall into two categories. Pipeline methods first detect page elements with a separate model and then recognize text inside them, sometimes using large language models for one or both steps, while end-to-end models combine detection and recognition into a single pass through a vision-language model. The latter approach demands more powerful encoders for heavy image compression and more efficient decoders to control memory use, yet it remains limited by steadily growing key-value caches that slow inference and shorten output length.

2.1 Pipeline-based Framework

Traditional OCR models, particularly those designed for document parsing, typically adopt a pipeline architecture [10, 3, 4, 11, 12]: a detection model first identifies different types of document elements, followed by multiple recognition operators that further parse the content within those blocks. These components are often bridged by a variety of heuristic strategies, such as cropping, rectification, and so on. In recent years, with the powerful decoder capabilities of large language models (LLMs), the pipeline-based OCR paradigm has continued to evolve [11]. The most straightforward adaptation retains the detection model while consolidating the multiple recognition models into a single unified model—a pragmatic hybrid that combines mature traditional detection algorithms with the advanced decoder of an LLM. Beyond this, there is another pipeline variant that invokes the LLM twice, replacing even the detection model with the same LLM [12], so that the entire OCR workflow becomes: LLM detection–cropping strategy–LLM recognition. Thanks to the inherent flexibility in how OCR tasks can be decomposed, pipeline architectures still remain widely adopted to this day.

2.2 End-to-end Model

With the advancement of vision-language models (VLMs) [13, 14, 6, 15, 7], end-to-end OCR, especially dense OCR models [16, 2, 1, 17, 18, 19] are on the rise. This approach fully leverages the powerful decoder capabilities of LLMs by merging text detection and recognition into a single unified function, allowing the entire content of a page to be parsed in a single forward pass. Compared with the pipeline approach, the end-to-end algorithm places higher demands on model capacity and poses greater training challenges. This, in turn, makes research on end-to-end OCR models all the more compelling: innovations in architectural design and iterative improvements in training methodologies can more directly inspire, or even advance, the development of general-purpose VLMs.

2.2.1 High-compression Encoder

In end-to-end models, the encoder is an indispensable module that extracts and compresses image information. To a certain extent, the encoder determines the upper bound of the model: taking generation efficiency as an example, if the input vision tokens are too long—meaning the encoder's token compression ratio is insufficient—the model's decoding efficiency will be hindered by excessively long prefix tokens, thereby affecting decoding speed. The same holds true for effective decoding length. DeepEncoder [1] achieves a 16 $\times$ token compression rate under low activation values by cascading window attention ViT [20] and global attention one [21], making it an ideal choice for multi-page long-horizon OCR.

2.2.2 High-efficiency Decoder

What most directly affects inference cost is the decoder, including the activation value of the LLM and the KV cache size. Regarding the former, current end-to-end OCR models are typically under 3B parameters. In a related vein, DeepSeek OCR [1] uses an MoE architecture [22], keeping its activation at only 500M during inference. As for the KV cache, current models all see it grow continuously with decoding contexts, which limits both generation speed and length. This is exactly the key issue that our Unlimited OCR aims to address.

**Figure 2:** Inspired by the process of humans copying books, we propose the Unlimited OCR. This model features a unified end-to-end architecture, consisting of an encoder and a MoE-LLM decoder in which all attention mechanisms are R-SWA. The KV cache is implemented as a queue with a capacity of $m+n$ —each time a new token is generated, the KV corresponding to the $(m+1)$-th token in the queue is evicted, ensuring that both computational cost and memory usage do not progressively increase during the generation process.

3. Methodology

Section Summary: The methodology for Unlimited OCR draws inspiration from human transcription habits, which rely on limited attention to the source material, recent output, and the next character being produced, along with a form of soft forgetting to handle long tasks like transcribing entire books under low memory load. It builds on the DeepSeek OCR model, retaining its DeepEncoder for efficient visual token compression from page images while replacing standard attention with a new Reference Sliding Window Attention mechanism that keeps a fixed prefix cache for visuals and prompts but limits attention over generated text to a small sliding window of recent tokens. This design, paired with the model's Mixture-of-Experts decoder, drastically reduces KV cache size and computation, enabling one-shot parsing of hundreds of pages without the usual length constraints.

3.1 Long-horizon Parsing

Our humans excel at long-horizon parsing tasks—continuously transcribing an entire book, translating even hundreds of pages in one sitting, or transcribing hours of audio without interruption. This continuous parsing capability appears closely linked to the working memory. As illustrated in Figure 2, when a person copies a book by hand, their attention typically centers on three points: the original source book, a small portion of what has just been written (usually only a few characters), and the next character about to be written. Rather than retaining a complete memory of everything already transcribed, they engage in a form of soft forgetting. This maybe the key to sustaining long-horizon parsing under low cognitive load. Inspired by this observation, we present Unlimited OCR.

3.2 Architecture

As shown in Figure 2, Unlimited OCR adopts DeepSeek OCR as its baseline. Specifically, it comprises the DeepEncoder paired with a Mixture-of-Experts (MoE) architecture that enjoys 3B total and 500M activated parameters. The DeepEncoder stands out for its exceptional visual token compression capability, which can dramatically reduce the KV cache footprint during the prefill stage while preserving robust optical text feature extraction. Departing from the original DeepSeek OCR, we replace the vanilla Multi-Head Attention (MHA) with our proposed R-SWA. With the new proposed attention, long-horizon parsing can be achieved by augmenting the original reference KV cache $m$ with a fixed-capacity output KV buffer of width $n$. We will delve into the technical details in the following sections.

3.3 DeepEncoder

DeepEncoder is originally introduced in DeepSeek OCR [1]. It cascades SAM-ViT [20] with CLIP-ViT [21] and applies 16 $\times$ [15] token compression at the bridge, so that the first half relies entirely on window attention to process the original image tokens, while global attention is reserved exclusively for the compressed tokens. This design keeps the activation values low when encoding high-resolution images, thereby conserving GPU memory. DeepEncoder natively supports five resolution modes; we retain two of them: the "Base" model (1024×1024 for multi-page), and the "Gundam" mode (dynamic resolution for single-page). Specifically, DeepEncoder can compress a 1024 $\times$ 1024 PDF-image to just 256 tokens. This high compression ratio is critically important for unlimited OCR works, because visual tokens do not undergo state transitions alongside the output - they are encoded once and remain static throughout the entire long-horizon parsing process.

3.4 Reference Sliding Window Attention

Despite the satisfactory compression of visual tokens that DeepEncoder achieves on the input side, the real bottleneck for one-shot parsing of an entire book lies in the decoding stage. Assume a compression ratio of 1:10 between visual and text tokens — i.e., one visual token can decode around ten text tokens. In that case, 10K visual tokens (equivalent to roughly $20-30$ pages at 1024 $\times$ 1024 resolution) demand an output length of 100k $+$ tokens for full decoding. This has long been a formidable challenge for vanilla LLM-driven OCR models, due to the massive KV cache storage and attention computation that sequences beyond 128k tokens entail. To address this, we propose Reference Sliding Window Attention (R-SWA).

3.4.1 Attention computation

In essence, R-SWA constrains attention within a two-segment window of size $m+n$, as illustrated in Figure 2. Here, $m$ denotes the window for prefix tokens, which includes both visual tokens and the prompt. During a single inference pass, $m$ remains fixed; it depends only on the number of book pages or the resolution size of the document being decoded, and does not vary with decoding length. The window $n$ for the decode region is also fixed in size and slides in a causal manner. Specifically, the formulation is as follows:

$ \begin{aligned} \mathcal{N}(t) &= \mathcal{P}\cup \mathcal{D}_n(t); \ \ \ \ \mathcal{P} = {1, \dots, L_m}, \ \mathcal{D}_n(t) &= \left{, j \mid \max(L_m+1, ;L_m+t-n)\le j \le L_m+t-1 \right}, \end{aligned} $

where $\mathcal{P}$ denotes the prefix segment of length $L_m$, which is globally visible to all subsequent tokens, and $\mathcal{D}_n(t)$ denotes the causal sliding window of width $n$ over the decode region. The attention weight from token $t$ to position $j\in\mathcal{N}(t)$ is then computed as

$ \begin{aligned} \alpha_{tj} &= \frac{ \exp\left(\frac{\mathbf{q}_t^\top \mathbf{k}j}{\sqrt{d_k}}\right) }{ \sum\limits{i\in\mathcal{N}(t)} \exp\left(\frac{\mathbf{q}_t^\top \mathbf{k}_i}{\sqrt{d_k}}\right) }, \quad j\in\mathcal{N}(t), \end{aligned} $

where $\mathbf{q}_t$, $\mathbf{k}_j$, and $\mathbf{v}_j$ are the query, key, and value vectors, respectively, and $d_k$ is the dimension of the key-vector. The output representation is obtained by aggregating values over the same accessible set:

$ \begin{aligned} \mathbf{o}t &= \sum{j\in\mathcal{N}(t)} \alpha_{tj}\mathbf{v}_j . \end{aligned} $

This formulation makes explicit that each decoding token can attend to all prefix tokens as persistent global context, while only attending locally within a bounded causal window over previously generated tokens. As a result, the model preserves access to the full prefix information while reducing the attention cost over the growing decode sequence.

3.4.2 KV cache management

For DeepSeek OCR baseline, it employs standard Multi-Head Attention (MHA)—the most classical form of attention, which offers strong expressiveness but imposes enormous KV cache pressure, the KV cache size is calculated as follows:

$ \begin{aligned} C_{\mathrm{MHA}}(T) &= L_m + T. \end{aligned} $

In contrast, under R-SWA, the model always retains the full prefix cache of size $L_m$, but for the generated continuation it only needs to keep the most recent $n$ tokens. Therefore, after generating a total of $T$ tokens, the required KV cache size is

$ \begin{aligned} C_{\mathrm{R\text{-}SWA}}(T) &= L_m + \min(n, , T) \le L_m + n. \end{aligned} $

This shows that, unlike standard MHA whose cache size increases unboundedly with $T$, the decode-side cache of R-SWA is upper-bounded by a constant window size. To quantify the reduction, we define the cache ratio

$ \begin{aligned} \rho(T) &= \frac{C_{\mathrm{R\text{-}SWA}}(T)}{C_{\mathrm{MHA}}(T)}

\frac{L_m+\min(n, , T)}{L_m+T}. \end{aligned} $

If the generated length is sufficiently long such that $T\gg n$, then

$ \begin{aligned} \rho(T) &= \frac{L_m+n}{L_m+T}. \end{aligned} $

which decreases as $T$ grows. In particular, when the decode length dominates both the prefix length and the window size, we have

$ \rho(T)\approx \frac{L_m+n}{T}\to 0. $

Therefore, for long-sequence decoding, R-SWA reduces the KV cache requirement from linear growth in $T$ to a bounded quantity $L_m+n$, yielding a substantial memory saving compared with standard MHA. Accordingly, R-SWA serves as the cornerstone to enabling near-unlimited parsing works under limited resources.

3.4.3 Kernel study

**Figure 3:** The latency of the Flash Attention v3 kernel as decoding length increases.

As shown in Figure 3, we plot the per-call duration of the Flash Attention v3 kernel for both the DeepSeek OCR baseline and Unlimited OCR Works (denoted as UOW in the figure). The figure clearly shows that the standard MHA kernel in DeepSeek OCR incurs growing latency with each successive decoding step, whereas in Unlimited OCR the duration remains constant—a direct benefit of adopting R-SWA across all layers of the LLM decoder. The spike in the DeepSeek OCR occurs when the KV cache length crosses a certain alignment boundary, causing an abrupt drop in data transfer efficiency; this issue also does not arise with R-SWA. Besides, the same pattern will hold for GPU memory usage during inference: in the original DeepSeek OCR it scales linearly, while in Unlimited OCR it stays fixed. This joint stability in both computational cost and memory footprint is precisely what makes long-horizon parsing possible.

4. Experimental Settings

Section Summary: Researchers built a training dataset of roughly two million document pages by annotating single-page files with an existing OCR tool and then stitching pages together to create longer multi-page examples, all formatted into fixed-length sequences. Starting from an earlier OCR model, they continued training for four thousand steps on a large GPU cluster while keeping the visual encoder frozen and updating only the language-model weights, using standard optimizers and specialized parallelism tools to handle the long input lengths. For running the final system they added custom memory-handling code that lets both research and production inference engines process documents efficiently at constant speed.

4.1 Data Engine

We construct approximately 2 million document OCR data samples to train Unlimited OCR, with a 9:1 ratio of single-page to multi-page data. For the single-page PDF data, we use Paddle OCR [10] for annotation, concatenating the coordinates and content of each block to construct end-to-end detection and parsing ground truth. The coordinates of each element are normalized to the range of 0–1000. All multi-page data are synthesized by concatenating single-page data. We randomly generate around 200k samples, each consisting of 2 to 50 pages, with used as a separator between pages. All data are packed into a sequence length of 32K tokens.

4.2 Implementation Details

Starting from the DeepSeek OCR checkpoint [1], we continue training Unlimited OCR for 4, 000 steps with a global batch size of 256 and a maximum sequence length of 32K on 8 $\times$ 16 A800 GPUs, using random packing for all data. During training, we freeze the DeepEncoder and only train the LLM parameters, as the DeepEncoder is already sufficiently optimized in DeepSeek OCR. We use the AdamW [23] optimizer and a cosine annealing scheduler [24] with an initial learning rate of 1e-4. To support 32K training, we adopt DeepEP [22], with expert parallelism (EP) set to 4. The entire training pipeline is built on the Megatron-LM [25] framework. For inference, we implement KV cache management for R-SWA in the Transformers library, along with corresponding support and optimizations in the SGLang inference engine. Both inference frameworks can operate Unlimited OCR under constant TPS (tokens/S) and GPU memory.

5. Evaluation

Section Summary: The evaluation of Unlimited OCR relied on the OmniDocBench dataset to test performance on text recognition, formula and table extraction, and reading order across varied document types, plus an internal collection of multi-page books and papers for longer documents. The model reached state-of-the-art accuracy on both single-page and extended tasks, improving key metrics over its DeepSeek OCR baseline while also running faster. These gains held steady across nine document categories and remained reliable even as document length grew to forty pages or more.

5.1 Benchmark and Metrics

We select OmniDocBench [9] as the main benchmark for evaluating foundational document OCR capabilities, and test the Unlimited OCR on both v1.5 and v1.6 versions. OmniDocBench v1.6 includes 296 more test images than v1.5 and represents the latest benchmark, while v1.5 provides official metrics from more classic models—including our baseline DeepSeek OCR—which facilitates performance comparisons. For long-horizon OCR evaluation, an in-house test set is constructed, where we select a number of novels, documents, and papers and divide them by page count to assess the multi-page performance of Unlimited OCR. Specifically, we select books of 2, 5, 10, 20, and 40+ pages for testing, with no fewer than ten books for each category.

OmniDocBench is designed to evaluate document parsing capabilities across multiple dimensions, including text recognition, formula recognition, table structure extraction, and reading order prediction. It adopts task-specific metrics for a well-rounded evaluation: (1) Text Edit Distance (Edit $\downarrow$), which measures character-level accuracy for text recognition; (2) Formula CDM (CDM $\uparrow$), which evaluates the quality of mathematical formula recognition; (3) Table TEDS (TEDS $\uparrow$) and Table TEDS-S (TEDS-S $\uparrow$), which assess table structure extraction accuracy with and without content recognition; and (4) Reading Order Edit Distance (Edit $\downarrow$), which quantifies the correctness of predicted reading sequences. The overall score is then computed as a weighted average across text, formula, and table recognition tasks. For the in-house benchmark, we report both the Distinct-n and the Edit Distance. Distinct-n is the ratio of the number of unique n-grams to the total number of n-grams in the generated text.

::: {caption="Table 1: Comparison on OmniDocBench (v1.5/v1.6). All models in the table are end-to-end VLM-based architectures. v1.5 is primarily intended for comparison with classic end-to-end algorithms and the baseline DeepSeek OCR. v1.6 mainly compares against current end-to-end SOTA models. Except for the proposed Unlimited OCR, all other models are selected from the OmniDocBench repository."}

:::

5.2 Main Results

As shown in Table 1, by continue-training on merely 2M PDF-document-specific data based on DeepSeek OCR, Unlimited OCR achieves end-to-end SOTA performance. This demonstrates the effectiveness of R-SWA on parsing tasks. First, compared with the standard attention in DeepSeek OCR, R-SWA may allow the model to focus more on dense OCR tasks, whereas full attention could lead to divergence as the output length increases. On the other hand, the state transition across intra-page content under R-SWA is both workable and solid. Specifically, on OmniDocBench v1.5, compared with DeepSeek OCR, the text edit distance drops by 0.035, and the table TEDS improves by 5.96%, indicating that historical information is causally and continuously fed into the sliding window, enabling the model to clearly locate its OCR progress even though it sees only a few tokens. On the OmniDocBench v1.6 benchmark, Unlimited OCR again achieves end-to-end SOTA (93.92% on overall metric), further proving that for single-page PDF-level document OCR tasks, replacing all standard attention entirely with R-SWA of width 128 is both effective and lossless.

Moreover, Unlimited OCR gains all the benefits of DeepSeek OCR, such as the MoE architecture with only 0.5B activated parameters, resulting in very high inference efficiency. In the OmniDocBench, Unlimited OCR achieves 5580 TPS (tokens/s/512 concurrency) compared to DeepSeek OCR’s 4951 TPS under ''Base" DeepEncoder mode, representing a 12.7% speed increase. Of course, the average document length in OmniDocBench is relatively short—the longer the output length, the more pronounced the advantage of Unlimited OCR becomes.

5.3 Subcategory Study

OmniDocBench (v1.5) provides 9 types of documents, and conducting a subcategory comparison is crucial for a more systematic and comprehensive analysis of R-SWA. As shown in Table 2, compared to DeepSeek OCR, Unlimited OCR shows clear and consistent gains across every metric, demonstrating that our decoder-side optimization, i.e., R-SWA, delivers a genuine "free lunch"—improvements without compromises. Compared to DeepSeek OCR 2, Unlimited OCR also holds a clear advantage, with seven-ninths of both the text edit distance and reading order scores surpassing those of DeepSeek OCR 2. For documents with complex layouts such as PPT, newspapers, magazines, and note, Unlimited OCR shows no disadvantage either, further demonstrating that replacing all standard attention with R-SWA for LLM-decoder is complete and sound for parsing tasks.

::: {caption="Table 2: Detailed subcategory comparison between Unlimited OCR and the DeepSeek-OCR series across nine document types. R-order denotes reading order. All metrics are edit distances, where lower is better. Red cells indicate that the corresponding metric of DeepSeek-OCR or DeepSeek-OCR 2 is better than that of Unlimited OCR."}

:::

\begin{tabular}{l|cccccc}
  \toprule
  \diagbox[height=0.61cm]{Metric}{Pages} & 2 & 5 & 10 & 15 & 20 & 40+ \\
  \midrule
  Distinct-20 $\uparrow$ & 99.76\% & 99.78\% & 97.49\% & 99.92\% & 98.73\% & 96.08\% \\
  Distinct-35 $\uparrow$ & 99.87\% & 99.98\% & 99.83\% & 99.99\% & 99.89\% & 96.90\% \\

  \midrule
  Edit Distance $\downarrow$ & 0.0362 & 0.0452 & 0.0526 & 0.0787 & 0.0572 & 0.1069 \\

  \bottomrule
  \end{tabular}

5.4 Long-horizon Parsing

Long-horizon parsing is one of the novel capabilities of Unlimited OCR. Two main obstacles have hindered previous models from achieving this: first, excessively long output sequences can easily exceed the maximum token limit; second, output latency grows with sequence length, causing the OCR of documents spanning dozens of pages to become progressively slower. Unlimited OCR, equipped with R-SWA, can prefill tens to hundreds of document pages in a single pass and parse continuously from the first page to the last. Throughout this process, the KV cache remains fixed, so output latency stays constant—making long-horizon parsing feasible. As shown in Table 3, our model delivers satisfactory performance in multi-page one-shot OCR scenarios, maintaining strong results even with 20 pages input simultaneously. At 40+ pages, the edit distance remains below 0.11 along with 97% Distinct-35. We examine the cases with repeated errors and find that most occur where small text in the PDF is difficult to discern, primarily due to the use of DeepEncoder's "Base" mode (1024 $\times$ 1024 resolution) under multi-page conditions, rather than R-SWA losing direction in long-horizon parsing process.

\begin{tabular}{l|ccccccc}
  \toprule
  \diagbox[height=0.7cm]{Model}{TPS} & 256 & 512 & 1024 & 2048 & 3072 & 4096 & 6144 \\
  \midrule
  Deepseek OCR & 7229.32 & 7468.27 & 7422.50 & 7166.85 & 6790.72 & 6430.21 & 5822.87\\
  Unlimited OCR & 7229.52 & 7714.78 & 7840.94& 7881.11 & 7881.93 & 7905.18 & 7847.71 \\

  \bottomrule
  \end{tabular}

6. Efficiency Analysis

Section Summary: A comparison of Unlimited OCR and DeepSeek OCR shows that the two systems generate text at nearly identical speeds for short outputs around 256 tokens. As the output length increases, however, DeepSeek OCR slows down noticeably, falling 35 percent behind Unlimited OCR at 6,000 tokens. This gap highlights the value of the R-SWA technique in Unlimited OCR for maintaining steady performance during lengthy document processing.

As presented in Table 4, we compare the output tokens per second (TPS) of Unlimited OCR and DeepSeek OCR under ideal concurrency conditions. The prefill length is fixed at 10, with all other settings held identical. The results show that at 256 tokens, the inference speeds of the two models are virtually the same. As the output length grows, however, the TPS of DeepSeek OCR steadily declines, and at 6, 000 tokens, it lags behind Unlimited OCR—which incorporates R-SWA—by 35%. These findings further validate the effectiveness of R-SWA and underscore that consistent generation speed is a critical requirement for long-horizon OCR tasks.

7. Limitation and Future Work

Section Summary: The model’s ability to handle growing numbers of document pages is still limited by the length of the initial processing step, even though the image encoder already compresses data efficiently. In the near term the team plans to train versions that accept much longer contexts, such as 128K tokens, so more pages can be processed at once. Over a longer horizon they aim to let the model selectively retrieve stored processing results, mimicking how a person flips through pages, and to apply the same sliding-window technique to speech recognition and translation tasks.

Our model cannot achieve truly unlimited parsing under a finite context length (e.g., 32K), as it is also constrained by the prefill length. Although DeepEncoder already achieves a high compression rate for image tokens, the prefill still becomes very long as the number of pages accumulates. In the short term, we will train models with longer context lengths, such as 128K, to support the prefill of more pages. In the long term, we plan to build a prefill pool and enable the model to learn to automatically fetch prefill KV chunks, thereby simulating the effect of a human flipping through pages, so as to achieve truly unlimited OCR works. In addition, we will also transfer R-SWA to reference-based tasks such as ASR and translation.

8. Conclusion

Section Summary: The report introduces an Unlimited OCR model along with a supporting R-SWA algorithm designed to handle long sequences of text. Experiments showed that swapping the usual attention mechanism in the decoder for this new reference-based approach preserved full accuracy on parsing tasks, suggesting the model can carry forward key details from earlier text much like a person transcribing a book. The authors expect the method to ease computing and memory demands in similar long-sequence tasks going forward.

In this technical report, we propose the Unlimited OCR model and present the R-SWA algorithm to support its capability for long-horizon parsing. We verify that when all standard attention in the decoder of an end-to-end model is replaced with causal reference-based SWA, the model's performance on parsing tasks remains lossless. This indicates that the model learns to continuously pass useful information from historical outputs into the window, and this soft form of forgetting is consistent with how we humans behave when transcribing a book. We believe that R-SWA will be applied to more tasks in the future, making attention computation and memory footprint no longer the bottleneck for long-horizon parsing field.

9. Author List

Section Summary: The author list identifies the full team behind the project and distinguishes a small group of core contributors from a larger set of supporting contributors. It marks Huanhuan Liu as project leader and YY as technical director within the core group that also includes Youyang Yin. The remaining fourteen individuals are listed simply as contributors without special roles.

* indicates project leader; $^\dag$ indicates technical director

Core Contributors: Youyang Yin, Huanhuan Liu*, YY$^\dag$

Contributors: Qunyi Xie, Chaorun Liu, Shiqi Yang, Shaohua Wang, Zhanlong Liu, Hao Zou, Jinyue Chen, Shu Wei, Jingjing Wu, Mingxin Huang, Zhen Wu, Guibin Wang, Tengyu Du, Lei Jia

References

Section Summary: This section compiles dozens of recent research papers, technical reports, and open-source tools mostly published between 2023 and 2026. They focus on advancing optical character recognition, document parsing, and multimodal vision-language models for extracting and understanding text from PDFs, images, and other sources. The cited works include major systems such as DeepSeek-OCR, Qwen-VL, Gemini, and PaddleOCR, along with related benchmarks and training methods.

[1] Wei et al. (2025). Deepseek-ocr: Contexts optical compression. arXiv preprint arXiv:2510.18234.

[2] Wei et al. (2024). General ocr theory: Towards ocr-2.0 via a unified end-to-end model. arXiv preprint arXiv:2409.01704.

[3] Cui et al. (2025). Paddleocr-vl: Boosting multilingual document parsing via a 0.9 b ultra-compact vision-language model. arXiv preprint arXiv:2510.14528.

[4] Wang et al. (2024). Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839.

[5] Team et al. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

[6] Bai et al. (2025). Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923.

[7] Huang et al. (2026). STEP3-VL-10B Technical Report. arXiv preprint arXiv:2601.09668.

[8] OpenAI (2023). GPT-4 Technical Report.

[9] Ouyang et al. (2025). Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 24838–24848.

[10] Cui et al. (2025). Paddleocr 3.0 technical report. arXiv preprint arXiv:2507.05595.

[11] Li et al. (2025). MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm. arXiv preprint arXiv:2506.05218.

[12] Feng et al. (2025). Dolphin: Document image parsing via heterogeneous anchor prompting. arXiv preprint arXiv:2505.14059.

[13] Li et al. (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730–19742.

[14] Bai et al. (2023). Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv preprint arXiv:2308.12966.

[15] Wei et al. (2024). Vary: Scaling up the vision vocabulary for large vision-language model. In European Conference on Computer Vision. pp. 408–424.

[16] Blecher et al. (2023). Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.13418.

[17] Wei et al. (2026). DeepSeek-OCR 2: Visual Causal Flow. arXiv preprint arXiv:2601.20552.

[18] Rednote (2025). dots.ocr. https://github.com/rednote-hilab/dots.ocr.

[19] Poznanski et al. (2025). olmocr: Unlocking trillions of tokens in pdfs with vision language models. arXiv preprint arXiv:2502.18443.

[20] Kirillov et al. (2023). Segment anything. arXiv preprint arXiv:2304.02643.

[21] Radford et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning. pp. 8748–8763.

[22] Liu et al. (2025). Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.

[23] Ilya Loshchilov and Frank Hutter (2019). Decoupled Weight Decay Regularization. In ICLR.

[24] Loshchilov, Ilya and Hutter, Frank (2016). Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983.

[25] Shoeybi et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053.

[26] (2025). OCRFlux. https://github.com/chatdoc-com/OCRFlux.

[27] Zhu et al. (2025). Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479.

[28] Yuan Liu et al. (2025). POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 1576–1601.

[29] Weiyun Wang et al. (2025). InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv preprint arXiv:2508.18265.

[30] (2025). Nanonets-OCR-s. https://huggingface.co/nanonets/Nanonets-OCR-s.

[31] Google AI (2025). Gemini 2.5-Pro. https://gemini.google.com/.

[32] (2025). OCRVerse. https://github.com/DocTron-hub/OCRVerse.

[33] S. Bai et al. (2025). Qwen3-VL Technical Report. arXiv preprint arXiv:2511.21631. https://arxiv.org/abs/2511.21631.

[34] Team et al. (2025). Hunyuanocr technical report. arXiv preprint arXiv:2511.19575.

[35] Wu et al. (2026). Firered-ocr technical report. arXiv preprint arXiv:2603.01840.

[36] alibaba (2026). https://github.com/alibaba/Logics-Parsing.

[37] Dong et al. (2026). Qianfan-ocr: A unified end-to-end model for document intelligence. arXiv preprint arXiv:2603.13398.