ReToken: One Token to Improve Vision–Language Models for Visual Retrieval

Yao Xiao$^{1}$
Reuben Tan$^{2}$
Zhen Zhu$^{1,3,*}$
Yuqun Wu$^{1}$
Jianfeng Gao$^{2}$
Derek Hoiem$^{1}$
$^{1}$University of Illinois at Urbana-Champaign, $^{2}$Microsoft Research, $^{3}$Google DeepMind
$^{1}${yaox11, dhoiem}@illinois.edu

$^{*}$ This work was done while the author was at UIUC (currently at Google DeepMind).

Abstract

Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present RETOKEN, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, RETOKEN yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.0-point gain with Qwen3VL-8B. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken.

Executive Summary: ReToken introduces a lightweight method to help vision-language models handle long visual inputs, such as collections of images or hour-long videos. Current models lose accuracy when most of the input is irrelevant to the question, and processing everything at once often exceeds available GPU memory. Attention signals inside the model itself prove unreliable for identifying relevant frames, while external retrieval systems add complexity and cost.

The work set out to create a simple, internal retrieval mechanism that selects only the most useful visual information before the model generates an answer. Researchers trained a single learnable token, called ReToken, on a modest image question-answering dataset while keeping the underlying vision-language model frozen. The token scores frames by comparing its own embedding against the average value vectors produced by the model’s final layer, then retrieves the top frames for the answer stage. Training and inference both run on one H100 GPU.

The approach yields clear gains. On the Visual Haystacks benchmark, ReToken improves accuracy by 13.4 points for Qwen3VL-8B and 12.4 points for InternVL3.5 when many distractor images are present, representing relative gains above 20 percent. It also transfers zero-shot to long-video tasks, raising accuracy by 8.0 points on LVBench, where videos average over an hour. The method outperforms attention-based retrieval, external embedding models, and prior retrieval-augmented pipelines, especially when only a small number of frames can be used.

These results matter because they show that a minimal addition can make existing models far more effective at long-context multimodal reasoning without retraining the full system or requiring multiple specialized components. The gains are largest precisely when the input contains many irrelevant frames—the realistic case for video archives or image collections. Efficiency remains practical: video encoding happens once and is reused across questions.

The authors recommend deploying the frozen version of ReToken for immediate use on image and video tasks, while exploring partial tuning of early layers only when higher precision on static images is required. Further gains are likely from training on video data and extending retrieval to groups of frames or individual tokens rather than whole frames. The main limitations are the two-pass inference, which adds modest latency, and the current training data being limited to images; results on tasks that require temporal reasoning across many frames are therefore less certain. Overall confidence in the reported improvements is high because the gains appear consistently across multiple models and benchmarks.

1. Introduction

Section Summary: Vision-language models can now process very long image collections or hour-long videos, but they struggle to answer questions when only a few relevant frames matter and full-context processing often exceeds memory limits. Standard attention signals prove unreliable for selecting those frames, yet the authors find that matching text against the model's value projections yields a much stronger retrieval signal than conventional key-based methods. To exploit this, they introduce ReToken, a single trainable embedding appended to the question that learns to identify relevant visuals; the approach improves accuracy on both multi-image and long-video benchmarks while adding almost no parameters.

Long visual contexts, e.g. from image collections or hour-long videos, are now within the input range of vision-language models (VLMs) [1, 2, 3, 4]. However, VLMs have difficulty answering questions from long visual contexts when only a small subset of images or frames is relevant to the prompt [5], and sometimes processing the full context at once is computationally infeasible under GPU memory constraints. So, handling long visual input reduces to a retrieval problem: selecting a small subset of frames or tokens from which the model can produce a correct answer. When processing text, the attention signals produced by large language models (LLMs) can be selective with respect to the input context, thanks to extensive long-context training [6, 7]. For VLMs, we find that the analogous signal is unreliable: attention between text and visual features is weakly correlated with relevance, as shown in Figure 1a, and the attention-based retriever achieves only 5.1% average recall@1 across layers on QAEgo4D $_{\text{Test-MC}}$ for Qwen3VL-8B (Figure 1b).

Looking at what does work, we find a striking asymmetry: matching a precise target phrase against the average visual value projections, rather than against the keys, increases recall@1 from 65.7 to 78.0 on Qwen3VL and from 78.8 to 83.8 on InternVL3.5 in a controlled two-image setting (Table 1). Values carry the content that is actually propagated through attention, and they appear to provide a better space for text-based visual retrieval. However, value–value pooling does not universally outperform query–key scoring for arbitrary retrieval text. Simply averaging over all question tokens introduces noise, reversing the advantage of the value space.

Building on this finding, we propose $\textsc{ReToken}$, a single learnable embedding that is appended to the question and trained explicitly as a retrieval target. $\textsc{ReToken}$ scores each frame by the cosine similarity between its projected embedding and the frame's mean value vector at the final layer, and is supervised with a class-balanced binary cross-entropy loss against ground-truth relevance labels. The token and a single projection matrix are the only added parameters; the VLM is frozen by default.

Despite its minimal footprint, $\textsc{ReToken}$ yields consistent gains across image and video benchmarks. On Visual Haystacks [5], it improves Qwen3VL-8B [3] by 13.4 points and InternVL3.5 [8] by 12.4 points, corresponding to over 20% relative gain. More notably, although trained only on multi-image QA, $\textsc{ReToken}$ generalizes to improve performance on long-video understanding: it transfers zero-shot to long video, yielding an 8.0-point improvement on LVBench [9] with Qwen3VL-8B, where the average video length exceeds an hour.

Our contributions are:

• We identify that retrieval scores computed in the value space, rather than the conventional query-key space, provide a substantially stronger signal for retrieving visual information.
• We introduce $\textsc{ReToken}$, a lightweight learnable retrieval token that enables pretrained VLMs to identify relevant visual information more effectively.
• We show that $\textsc{ReToken}$ trained only on multi-image QA transfers zero-shot to long-video benchmarks, suggesting a practical path toward scalable long-context multimodal reasoning.

2. Related Work

Section Summary: The section surveys prior approaches to visual retrieval and long-context video understanding in vision-language models. Most methods either rely on separate external retrievers such as CLIP-style embedders or agentic pipelines, or they compress and manage visual tokens through memory banks or pruning techniques that operate independently of the specific query. It also reviews learnable-token methods like Q-Formers that aggregate visual information, positioning ReToken as a more efficient, dynamically generated alternative trained with explicit retrieval supervision in the value space rather than relying on attention scores.

Visual Retrieval. The dominant approach to visual retrieval is to train image-text embedding models that align visual and language feature spaces, either with dual encoders such as CLIP [10], Perception Encoder [11], and SigLIP2 [12], or by adapting a VLM into a universal embedder, as in E5-V [13] and LamRA [14]. While effective, all of these operate as external retrievers separate from the VLM: relevant images must first be selected by the retriever and then re-encoded by the VLM for answer generation. A related family in long-video QA chains a separate localizer with an answerer model: SeViLA [15] repurposes BLIP-2 [16] as both a keyframe localizer and an answerer, while VideoAgent [17] and VideoTree [18] build agentic pipelines that iteratively retrieve and caption keyframes for an LLM. A second line of work, originating in long-context language modeling, instead treats the model's own attention scores as a retrieval signal, unifying retrieval and generation within a single forward pass. InfLLM [19] and EM-LLM [20] demonstrate this effectively in NLP by selecting key-value blocks based on query-to-key attention, and ReKV [21] transfers the mechanism to the visual domain without modification. We diagnose the limitations of attention-based retrieval in VLMs and propose $\textsc{ReToken}$, which performs retrieval in the value space via a learnable token.

Long Context Understanding. When handling long visual contexts, prior work follows three main directions. Memory-based methods such as MovieChat [22] and MA-LMM [23] maintain a fixed-size memory bank over streaming input, merging or evicting older content to bound state. Token compression methods such as Chat-UniVi [24], LLaMA-VID [25], LongVU [26], and Video-XL [27] prune, merge, or summarize visual tokens before the LLM to shorten the prefix and reduce attention and KV-cache cost. Both families bound the visual context independently of the query. Retrieval-based methods instead defer context selection to inference time, picking a query-relevant subset to preserve fine-grained evidence: Goldfish [28] chunks the video into clips and retrieves the top- $K$ by caption-query similarity; Video-RAG [29] augments retrieval with visually-aligned auxiliary text from ASR, OCR, and object detectors; and DrVideo [30] converts the video into a long document and retrieves question-relevant passages for the LLM. End-task accuracy in this family, however, is bottlenecked by the retriever rather than the LLM, and our work targets this bottleneck directly.

Learnable Tokens for Visual Aggregation. $\textsc{ReToken}$ is most directly related to methods that introduce learnable tokens into a VLM to aggregate or retrieve visual content. The Q-Former in BLIP-2 [16] and the Perceiver Resampler in Flamingo [31] use a small set of learnable queries to compress variable-length visual features into a fixed-size representation for a frozen LLM, and InstructBLIP [32] extends Q-Former to be text-conditioned. These modules are trained jointly with vision-language alignment as part of the bridge between encoder and LLM, and produce many tokens (typically 32 or 64) intended to carry the visual content forward into generation. SPRING [33] prepends pluggable soft-prompt tokens to externally retrieved passages to help a frozen text-only LLM consume them. The visual summarization token in Video-XL [27] similarly compresses a chunk of visual KVs into a single token. $\textsc{ReToken}$ differs along three axes: (i) operational mechanism. Q-Formers and prompt tuning use learnable tokens as static, query-agnostic input conditions. In contrast, the ReToken is generated dynamically. A first-pass placeholder produces an output token that retrieves visual KV contexts relevant to the query. (ii) supervision. Q-Formers are trained as visual compressors under alignment or generation losses, while $\textsc{ReToken}$ is supervised by an explicit retrieval loss scored directly against final-layer value vectors. (iii) capacity. Our method needs only a single token rather than a set of 32–64. The second half of the contribution is diagnostic: attention (query–key) scores are unreliable for visual retrieval in pretrained VLMs, while the value space carries a much stronger signal.

3. Approach

Section Summary: The section first examines why attention-based methods struggle to identify relevant visual tokens in vision-language models, noting that attention scores are optimized for next-token prediction rather than retrieval and that averaging question tokens provides only a crude signal. It highlights that value projections within attention layers offer a more reliable indicator of content relevance than standard query-key scores. Building on these observations, the authors introduce ReToken, which learns a dedicated token embedding to compute improved frame-level retrieval scores and enable efficient selection of the most pertinent visual information.

In this section, we begin by analyzing the limitations of attention-based retrieval methods for identifying relevant visual tokens in Section 3.1. Motivated by these insights, we then discuss our proposed $\textsc{ReToken}$ approach in Section 3.2, which uses a learnable token embedding to help compute more informative retrieval scores over relevant visual tokens.

3.1 Attention-Based Retrieval

In our empirical setting, we consider a VLM based on a decoder-only LM comprising $N$ transformer layers ${L_1, \dots, L_N}$. Given input question tokens $\mathbf{T}_t$ and visual tokens $\mathbf{I}_v$, the VLM autoregressively generates the response $\mathbf{y}$ based on the conditional probability $p(\mathbf{y} \mid \mathbf{I}_v, \mathbf{T}_t)$. For multi-image and video inputs, the visual tokens are partitioned into $F$ frames, $\mathbf{I}_v = {\mathbf{I}v^{(f)}}{f=1}^{F}$, with each frame contributing $M/F$ tokens (where $M$ the total number of visual tokens). For long videos, $M$ can be prohibitively large, rendering full-context inference infeasible. Thus, our goal is to retrieve a subset of the $K$ most relevant frames, where $K \ll F$, whose tokens suffice to answer the question.

Attention Scores as a Retrieval Signal. At each layer $l \in {1, 2, \dots, N}$, the VLM computes attention scores $\mathbf{A}^{(l)} = \mathbf{Q}^{(l)} \mathbf{K}^{(l)\top} / \sqrt{d}$, which indicate how much each token contributes to the final response. Tokens with higher attention contributions can be interpreted as being more relevant to the question. For our analysis, we use the state-of-the-art ReKV [21] approach that directly uses attention score as the retrieval score and aggregates it at the frame level. At the $l-$ th layer, we compute the mean key vector of each frame's visual tokens and the mean query vector over the question tokens as:

$ \bar{\mathbf{k}}^{(l)}_f = \frac{1}{|\mathbf{I}v^{(f)}|} \sum{i \in \mathbf{I}_v^{(f)}} \mathbf{k}^{(l)}i, \qquad \bar{\mathbf{q}}^{(l)} = \frac{1}{E} \sum{t=1}^{E} \mathbf{q}^{(l)}_t\tag{1} $

where $E$ denotes the number of total question tokens. Then, the frame-level retrieval score $\mathbf{s}_f^{(l)}$ and top- $K$ frame selection are computed as:

$ \mathbf{s}_f^{(l)} = \bar{\mathbf{q}}^{(l)\top} \bar{\mathbf{k}}^{(l)}_f, \qquad \mathcal{S}_K^{(l)} = \mathrm{TopK} (\ \mathbf{s}_f^{(l)})\tag{2} $

where $\mathcal{S}_K^{(l)}$ denotes the $K$ frames most relevant to the question at layer $l$. Given the per-layer retrieval sets ${\mathcal{S}K^{(l)}}{l=1}^{N}$, the model re-runs generation through the frozen VLM with layer-wise sparse attention: at each layer $l$, the question and answer tokens attend only to the visual tokens belonging to $\mathcal{S}_K^{(l)}$. The reduced visual context at layer $l$ is therefore $[\mathbf{I}_v[\mathcal{S}_K^{(l)}]]$, and the answer is decoded autoregressively over this layer-dependent context $[\mathbf{I}_v[\mathcal{S}_K^{(l)}], \mathbf{T}_t]$.

Limitations of Attention-Based Retrieval. While attention-based retrieval is intuitive, two structural issues motivate our approach. First, attention is trained for next-token prediction rather than retrieval, so high-attention tokens are not guaranteed to correspond to query-relevant content. This mismatch is exacerbated by the composition of typical VLM training data: the input images or videos are almost always fully relevant to the question, so the model is never required to select among visual inputs. The consequences are visible in both image and video settings: on Visual Haystacks [5] the layer-wise $\bar{\mathbf{q}}^\top \bar{\mathbf{k}}f$ retriever lands at $63.3%$ recall@1 even in the simplest two-image case (Table 1), and on long-video QAEgo4D ${\text{Test-MC}}$ it averages only $5.1%$ recall@1 across layers (Figure 1b). Second, averaging the question tokens to form a single query is itself a heuristic, and the resulting rankings shift with the choice of retrieval text. As shown in Table 1, replacing the full question sentence with a target phrase that directly names the entity of interest improves Recall@1. Figure 2 illustrates a question sentence and its target phrase.

::: {caption="Table 1: Value × Value is more informative, but it is sensitive to the input. Recall@1 with 2 input images, retrieving 1 image based on the retrieval score."}

:::

Why Values Carry Retrieval Signal. A more reliable signal sits in the value projections. Within a transformer attention layer, the value of a token carries the content propagated to any token that attends to it, while the query-key inner product only determines how that content is aggregated. Pooling the value projections within each frame therefore yields a representation of the content the frame contributes to attending tokens, making value features more sensitive to the retrieval text than key features. Table 1 confirms this consistently across both backbones: with a precise target phrase that names the entity of interest, value-space pooling identifies the ground-truth image at $78.0%$ recall@1 vs. $65.7%$ for the corresponding query-key score on Qwen3VL, and at $83.8%$ vs. $78.8%$ on InternVL3.5. This mirrors observations in visual segmentation, where value features are reported to be more informative while query-key features can be replaced by alternative aggregation signals [34, 35, 36]; TextRegion [37] in particular shows that, in image-text models, value features in the final attention block are rich in visual-language semantics, whereas attention weights primarily serve as an aggregation mechanism. This sensitivity cuts both ways, however: with the full question sentence, averaging mixes in many uninformative tokens, and the resulting noisy query erases or even reverses the value-space advantage. The value space thus rewards a precise retrieval target and penalizes a noisy one.

3.2 ReToken: One Token for Visual Retrieval

To improve retrieval, we introduce $\textsc{ReToken}$, a single learnable embedding $\mathbf{X}_r \in \mathbb{R}^{d}$ that is appended to the question and trained explicitly as a retrieval target. $\textsc{ReToken}$ addresses the rephrasing instability by replacing the question-averaged query with a token learned from data, and strengthens the retrieval score by computing in the value space rather than the query–key space.

Retrieval Score. Given an input sequence with visual tokens $\mathbf{I}_v$, question tokens $\mathbf{T}_t$, and the appended retrieval token $\mathbf{X}_r$, we compute retrieval scores at the final layer $L_N$ via a lightweight projection $\mathbf{W}_r \in \mathbb{R}^{d \times d}$, which together with $\mathbf{X}_r$ constitutes the only added parameters. The retrieval score for the $f$-th frame is the cosine similarity between the projected embedding and the frame's mean value vector $\bar{\mathbf{v}}_f^{(N)}$,

$ \bar{\mathbf{v}}_f^{(N)} = \frac{1}{|\mathbf{I}v^{(f)}|} \sum{i \in \mathbf{I}_v^{(f)}} \mathbf{v}_i^{(N)}, \quad \mathbf{Z}_r=\mathbf{W}_r, \mathbf{X}_r^{(N)}, \quad s_f^{(N)} = \cos!\left(\mathbf{Z}_r, ; \bar{\mathbf{v}}_f^{(N)} \right)\tag{3} $

For inference, we compute $\mathcal{S}_K^{(N)} = \mathrm{TopK}(s_f^{(N)})$ once at the final layer $L_N$ and broadcast it to all layers, rather than maintaining per-layer subsets ${\mathcal{S}_K^{(l)}}$. We do not apply this at training time, since our training data is short-context and retrieval is unnecessary.

Training. We train $\textsc{ReToken}$ with the VLM kept frozen by default; only the retrieval token $\mathbf{X}_r$ and a single final-layer projection $\mathbf{W}r$ are updated. We supervise these parameters with a retrieval loss $\mathcal{L}{\text{ret}}$ that compares the final-layer retrieval scores against ground-truth relevance labels $y_f \in {0, 1}$.

Let $\mathcal{F}^{+} = {f : y_f = 1}$ and $\mathcal{F}^{-} = {f : y_f = 0}$ denote the sets of relevant and irrelevant images, respectively. To prevent the loss from being dominated by the irrelevant images, we apply a class-balanced binary cross-entropy in which the positive and negative terms are averaged separately,

$ \mathcal{L}{\text{ret}} = - \frac{1}{|\mathcal{F}^{+}|} \sum{f \in \mathcal{F}^{+}} \log \sigma(\tau s_f^{(N)}) ;-; \frac{1}{|\mathcal{F}^{-}|} \sum_{f \in \mathcal{F}^{-}} \log \sigma(- \tau s_f^{(N)})\tag{4} $

where $s_f^{(N)}$ is the retrieval score for image $f$ at the final layer $L_N$, $\sigma(\cdot)$ is the sigmoid, and $\tau$ is a learnable logit scale parameter.

We additionally explore a partial-tuning variant in which the early layers of the VLM are tuned to give the visual representation more flexibility. In this setting, training is driven by two complementary losses, illustrated in Figure 3. In addition to the retrieval loss above, we use a generation loss $\mathcal{L}{\text{gen}}$, the standard next-token prediction loss on the answer, which preserves the model's question-answering ability and prevents the unfrozen layers from drifting under $\mathcal{L}{\text{ret}}$ alone. The total loss is

$ \mathcal{L} = \mathcal{L}{\text{ret}} + \lambda , \mathcal{L}{\text{gen}}\tag{5} $

Inference. At test time, we use a two-pass retrieve-then-answer pipeline (Figure 4). The video is first encoded once, and each question then retrieves a subset of visual KV cache for answer generation.

Cache video once: At video ingestion time, the VLM processes the visual tokens and stores the resulting per-layer KV cache in a persistent cache. For short videos, this is a single full-context forward pass. For long videos, we follow ReKV [21] and encode the video chunk by chunk with a sliding-window attention mask: each chunk attends only to the most recent $l_m$ visual tokens, while its newly produced KV states are appended to the cache. Older KV states can be offloaded to CPU memory when GPU memory is limited.

Retrieve then answer: Given a question, we run two forward passes through the frozen VLM over the cached visual context. The first pass computes the retrieval set $\mathcal{S}_K^{(N)}$ from the final layer; the second pass generates the answer conditioned on the selected frames. We use two passes because $\textsc{ReToken}$ is supervised only at $L_N$ Equation (4). The first pass enables the retrieval token to fully integrate textual and visual content, while the second pass enables attention to a consistent set of visual tokens, matching how it was trained.

First pass – retrieval. We append the retrieval token $\mathbf{X}_r$ to the question and run the first pass over the cached visual KV. For short videos $\mathbf{X}_r$ attends to all cached visual tokens at layer $l < N$, and the index $\mathcal{S}_K^{(N)}$ is computed at $L_N$ via Equation 3. For long videos the early-layer attention budget cannot accommodate the full cache, so at each layer $l < N$ we restrict $\mathbf{X}_r^{(l)}$ 's attention as follows: (1) project the contextualized $\mathbf{X}_r^{(l)}$ through that layer's frozen value projection; (2) score every frame by the cosine similarity between this projection and the frame's mean value vector $\bar{\mathbf{v}}_f^{(l)}$ at layer $l$; (3) restrict $\mathbf{X}_r^{(l)}$ 's attention to the top- $K'$ frames under this score before computing layer $l$ 's output. We use $K' = 256$ by default, which keeps each early-layer attention well within GPU memory while leaving headroom for the final-layer ranking to refine the selection. At $L_N$ we then compute Equation 3 over the per-frame value means to obtain $\mathcal{S}_K^{(N)}$, reordered by original timestamp.

We note an asymmetry between this retrieval-pass budget $K'$ and the answer-stage budget $K$: $\textsc{ReToken}$ 's retrieval pass attends to up to $K' = 256$ frames per early layer, which is different from the $K$ used at answer time. $\textsc{ReToken}$ therefore has access to more visual context when ranking candidates, while it still uses only $K$-frame visual context when generating the answer.

Second pass – answer. For each layer $l$, we load only the visual KV cache belonging to frames in $\mathcal{S}_K^{(N)}$, and run a standard generation pass over this reduced visual context together with the question. The answer stage therefore attends to about $K \cdot M / F$ visual tokens instead of all $M$, while the expensive video encoding is performed only once per video. This is particularly efficient when multiple questions are asked about the same video. Full prompt templates and a worked example of how $\textsc{ReToken}$ interacts with the contextual visual tokens and the question are provided in Supp. Appendix A.

4. Experiments

Section Summary: In the experiments, the authors trained ReToken on top of models such as Qwen3VL-8B using a multi-image question-answering dataset based on MIRAGE, applying different setups for frozen versus partially fine-tuned vision-language models along with specific learning rates, batch sizes, and schedules. They tested the method on four benchmarks chosen to measure retrieval skill and long-context understanding: Visual Haystacks for finding relevant images among distractors, plus QAEgo4D, LVBench, and Video-MME for question answering over videos ranging from minutes to hours long. On Visual Haystacks, ReToken delivered substantially higher recall and accuracy than alternative retrievers like SigLIP2 or ReKV, especially when the number of irrelevant images grew large, while also revealing a precision-recall tradeoff compared with attention-based approaches.

4.1 Settings

Implementation Details. We use the greedy decoding configuration for $\textsc{ReToken}$ and every baseline. We train $\textsc{ReToken}$ using Qwen3VL-8B and InternVL3.5-8B on an image question-answering (QA) dataset. For the default frozen-VLM setting, we use an effective batch size of 64 and a learning rate of 3 x 10^-4 to train the learnable token for 3 epochs on a single H100 GPU with Qwen3VL-8B. Since InternVL3.5-8B requires more tokens to represent an image, resulting in substantially higher computational and memory costs, we train InternVL3.5-8B for only one epoch. We adopt a linear warmup schedule followed by cosine learning rate decay. For the partial fine-tuning setting, we use a learning rate of 2 x 10^-5 and an effective batch size of 64. As this setting converges rapidly and begins to overfit after one epoch, we apply early stopping at the end of the first epoch. Training Qwen3VL-8B under this setting takes approximately 4 hours on a single H100 GPU.

For long video inference, we use an encoding chunk size of 128 frames and a sliding-window length of $l_m{=}30{,}000$ tokens, which corresponds to roughly 153 frames at 196 tokens per frame.

Training Datasets. Our default multi-image training dataset comes from the MIRAGE [5] fine-tuning dataset, whose examples are each annotated with a relevant/irrelevant label per image. We use a 95%/5% train/validation split. It is a multi-image QA (MIQA) dataset that combines existing MIQA datasets (RetVQA [38], SlideVQA [39], and WebQA [40]) with synthetic MIQA data adapted from the LLaVA Visual Instruct 150K dataset [41] via clustering and distractor sampling. Please refer to Supp. Appendix B.1 for more details.

Evaluation Datasets. We evaluate $\textsc{ReToken}$ on four benchmarks chosen to probe complementary aspects of retrieval and long-context understanding.

Visual Haystacks (VHs) [5] is a benchmark introduced alongside MIRAGE to evaluate the pure visual recognition ability of vision-language models. Constructed from the COCO dataset [42], it consists of 1,000 question-answer pairs and provides query-relevant image annotations. The answer is always either "Yes" or "No". For each QA example, a varying number of distractor images is added to test the model's ability to locate informative content within a large pool of inputs.

QAEgo4D $_\text{Test-MC}$ [43] is the multiple-choice subset of the QAEgo4D-test benchmark [44], focusing on question answering over long egocentric videos. Each example is annotated with the video segments relevant to the question. Video lengths range from 4 to 20 minutes.

LVBench [9] is a benchmark for extreme long-video understanding, comprising 103 publicly sourced YouTube videos totaling roughly 117 hours, with an average length of 68 minutes and individual videos extending up to 2 hours. This is a multiple-choice dataset, and the metric is accuracy.

Video-MME [45] consists of 900 videos and 2,700 question-answer pairs. Video durations range from 11 seconds to 1 hour, partitioned into short (<2 min), medium (4–15 min), and long (30–60 min) splits. Each question is multiple-choice, and we report accuracy as the evaluation metric.

```latextable {caption="Table 2: **Retrieval Strategy Comparison.** We compare ReToken against different retrievers on Visual Haystacks with retrieval budget $K{=}1$. “GT Cache” means using the KV cache of the ground truth image as input and serves as the oracle upper bound for any retriever. All results are based on Qwen3VL-8B with the VLM frozen."} \begin{tabular}{@lcc|ccc|c@} \toprule Metric {} & Standard {} & GT Cache {} & SigLIP2 {} & ReKV {} & CoT {} & \textbf{\textsc{ReToken}} \ \midrule \multicolumn{3}{@l}{\textbf{\textcolor{gray}{$C=2, K=1$}}} \ Recall & N/A & 100 & 76.4 & 63.3 & 65.3 & \textbf{88.5} \ Accuracy & 82.0 & 86.5 & 82.8 & 73.0 & 77.8 & \textbf{85.9} \ \midrule \multicolumn{3}{@l}{\textbf{\textcolor{gray}{$C=50, K=1$}}} \ Recall & N/A & 100 & 20.8 & 1.8 & 3.1 & \textbf{64.7} \ Accuracy & 58.6 & 80.7 & 60.7 & 51.2 & 51.5 & \textbf{72.0} \ \bottomrule \end{tabular}

### Visual Haystacks Retrieval and Accuracy

We evaluate $\textsc{ReToken}$ against the baselines on the single-needle task of the Visual Haystacks benchmark [5], in which exactly one image in the haystack is relevant to the query. The context size $C$ controls the number of distractor images, with larger $C$ posing a greater challenge to the model. $K$ denotes the number of retrieved images.

**Retrieval Behavior of ReToken and Attention-Based Retrieval**. Figure 5 compares $\textsc{ReToken}$ and attention-based retriever, ReKV [21], across retrieval budgets from $K{=}1$ to $K{=}128$ at $C{=}50$ and $C{=}100$. When $K >= C$, no retrieval is needed since all images are used as input. The two methods exhibit opposite trends in the meaningful retrieval regime ($K < C$): $\textsc{ReToken}$ performs best at small $K$ and degrades as more images (mostly distractors) are retrieved, whereas ReKV starts low and improves with larger $K$. The crossover reflects a precision–recall tradeoff: $\textsc{ReToken}$ is precise per slot, while ReKV needs a wider budget to recall the relevant image.

**Retrieval Strategy Comparison**. We also compute recall to measure the retrieval ability of different strategies on Visual Haystacks. Here, recall measures whether the retriever successfully selects the ground-truth image when $K{=}1$. Table 2 compares $\textsc{ReToken}$ against several retrieval strategies on Visual Haystacks. We compare against four baselines: *Standard* pre-fills all images (the default VLM inference pipeline); *GT Cache* uses the ground-truth image's KV cache as input, serving as an oracle upper bound; *SigLIP2* [12] first retrieves a single image with `siglip2-giant-opt-patch16-384` and feeds its KV cache to the VLM; and *CoT* prompts the VLM to first generate a target search phrase, then uses that phrase as the query for ReKV attention-based retrieval (full prompt in Supp. Appendix B.2).

$\textsc{ReToken}$ substantially outperforms all baselines, retaining most of the GT Cache upper-bound accuracy at large context size ($C=50$) while the others fail.

::: {caption="Table 3: Partial tuning on the retrieval dataset yields a cleaner KV cache. Delta_GT Image - GT Cache measures the degree of distraction. A smaller gap indicates a cleaner stored KV cache."}

![](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/ad3ghtbg/complex_tab_e52a4df0b086.png)

:::

**How the Stored KV Cache Differs From Re-encoding the Visual Input**.

Observing the accuracy for GT Cache in Table 2, we notice an interesting phenomenon: the accuracy degrades with higher C, even though only the cache for the same single ground truth relevant image is used in each case (86.5 for $C=2$ vs. 80.7 for $C=50$). This is because the image tokens have attended to previous images, which are irrelevant in this case, picking up distracting information.

To quantify this distraction, we compare the accuracy when using "GT Image" (re-encoding the ground-truth image alone, yielding a clean KV cache) with "GT Cache" (the fused KV cache for the ground-truth image). Their gap, $\Delta_{\text{GT Image} - \text{GT Cache}}$, measures the degree of distraction (Table 3); a smaller gap indicates a cleaner stored KV cache. Partial tuning on the retrieval dataset encourages the model to produce cleaner KV cache representations (KV cache is not affected when we only train the retrieval token). A cleaner KV cache benefits Visual Haystacks, but may pose a challenge for video understanding, which requires connecting information across adjacent frames.

::: {caption="Table 4: 'ReToken's advantage grows with context size, yielding over 20% relative gain at C=50, when freezing the VLM. Performance on Visual Haystacks. C denotes the number of context images (i.e., images provided as input). E indicates a context overflow or CUDA out-of-memory error."}

![](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/ad3ghtbg/complex_tab_3e63099b168c.png)

:::

**Accuracy Comparison**. Table 4 reports accuracy across context sizes from $C{=}1$ (only the query-relevant image, no distractors) to $C{=}100$. The retrieval budget $K$ is set to 1. At $C{=}1$, $\textsc{ReToken}$ matches the vanilla `Qwen3VL-8B` backbone, as we skip retrieval when $C<=K$.

As the context size grows, the gap between $\textsc{ReToken}$ and the vanilla backbone widens substantially. The one exception is InternVL3.5-8B at $C{=}100$, which we attribute to its 1-epoch training budget (Section 4.1). $\textsc{ReToken}$ also compares favorably against external baselines, surpassing all baselines across all context sizes, including the dedicated multi-image RAG framework MIRAGE.

### Image-to-Video Transfer Evaluation

Having validated $\textsc{ReToken}$ on image data, we now evaluate its transfer to the video setting. Notably, results in this section are obtained with $\textsc{ReToken}$ trained *only* on the multi-image MIRAGE training data, demonstrating strong zero-shot transfer. By default, we evaluate the frozen `Qwen3VL-8B` on video benchmarks with videos sampled at 0.5 FPS. "Uniformly" means loading the KV cache of $K$ uniformly sampled frames as input.

::: {caption="Table 5: Zero-shot performance on QAEgo4D _Test-MC."}
![](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/ad3ghtbg/complex_tab_7ea00094a218.png)

:::

**Accuracy on QAEgo4D $_\text{Test-MC}$**. Table 5 shows that $\textsc{ReToken}$ 's advantage is most pronounced under tight retrieval budgets. With $K{=}1$, $\textsc{ReToken}$ achieves a 6.8-point improvement over uniform sampling, while ReKV actually performs slightly worse than uniform sampling. Unlike multi-image QA where images are independent, neighboring video frames provide complementary context, so accuracy continues to climb as $K$ grows even where $\textsc{ReToken}$ 's lead over uniform sampling narrows.

**Long Video Understanding**. The benefits of retrieval grow with video length (Table 6). On the Short split of Video-MME, $\textsc{ReToken}$ provides no gain, since these videos last only 2 minutes (60 frames at 0.5 FPS), which falls below our retrieval input budget of 100 frames and therefore does not trigger retrieval. As video length grows, however, the gain becomes substantial. $\textsc{ReToken}$ achieves an 8.0-point improvement on LVBench (Table 6).

::: {caption="Table 6: ReToken helps more with long videos, even when trained only with images. Freezing the VLM and only tuning ReToken."}

![](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/ad3ghtbg/complex_tab_99c2f803c193.png)

:::

### Ablations

We ablate the design choices of $\textsc{ReToken}$ with `Qwen3VL-8B` on Visual Haystacks by default, focusing on: (1) retrieval based on visual key or value; (2) training the token only versus partial tuning; (3) the influence of the inference setting; (4) efficiency analysis; and (5) error analysis.

: Table 7: Key vs. Value.

| | × Key | × Value |
| :--- | :---: | :---: |
| Recall | 59.4 | **64.7** |
| Accuracy | 70.6 | **72.0** |

We further ablate design choices on whether a single token is sufficient for retrieval in Supp. Appendix C.1, and whether $\textsc{ReToken}$ can skip attending to visual tokens in Supp. Appendix C.2.

**Retrieval Score Based on Average Vision Key or Value**. We ablate $\textsc{ReToken}$ 's scoring mechanism in Table 7 with $C=50, K=1$: " $\times$ Key" trains $\textsc{ReToken}$ to retrieve via the average image keys, while " $\times$ Value" uses the average image values. " $\times$ Value" pulls clearly ahead in both recall and accuracy. We attribute this to values carrying the content actually propagated through attention, providing a more discriminative signal for distinguishing the relevant image among many distractors. Supp. Appendix B.3 provides further evidence for this.

**Train Only Token or Partial Tuning VLM Layers**. Table 8a ablates the partial-tuning depth. Tuning the first few layers (1–3) gives the best accuracy while preserving high recall, indicating that early layers are where visual features can be best shaped without compromising downstream performance.

Table 8b compares the transfer ability of the frozen and partial-tuning settings. $\textsc{ReToken}$ shows a clear advantage in both. However, overall performance under partial tuning is lower than under the frozen setting. A possible reason is that tuning the LLM layers on image data can cause domain shift, especially given the distribution gap between image training data and video benchmarks.

::: {caption="Table 8: Tuning early layers can improve performance on the image task, but it degrades performance on the video task. (a) Tuning the first few VLM layers (Layer 1–3) gives the best recall–accuracy trade-off at C=50, K=1. (b) Training the VLM on images decreases accuracy on video; evaluated on QAEgo4D _Test-MC with K=16."}

![](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/ad3ghtbg/complex_tab_865dfb5f1374.png)

:::

: Table 9: Two-pass Inference.

| | ReKV | **ReToken** |
| :--- | :---: | :---: |
| Single | **51.2** | 50.4 |
| Two | 50.4 | **72.0** |

**Single or Two-pass Inference**. Since $\textsc{ReToken}$ is trained to retrieve at the last layer, we use two-pass inference to obtain retrieval results at the final layer and then broadcast them to all layers. We ablate its effect and report the accuracy in Table 9 under $C=50, K=1$. "Single" means each layer retrieves its own images and directly generates the response based on that. "Two" means early layers attend to all visual tokens, and the last-layer retrieval result is broadcast to generate the response. The conclusions are different for ReKV and $\textsc{ReToken}$. One possible explanation is that attention-based retrieval is suited for answering directly, so it does not rely on the final layer's results; in contrast, $\textsc{ReToken}$ is only trained with the retrieval loss at the final layer. Therefore, the default setting for ReKV is single-pass, while for $\textsc{ReToken}$ it is two-pass.

::: {caption="Table 10: Runtime vs. quality on long-video QA (QAEgo4D _Test-MC, ∼ 240 frames per video at 0.5 FPS, single H100). Latency is reported per question after one-time video encoding. ReToken achieves higher accuracy with only modest overhead in the retrieval pass, and identical cost for the encode and answer-stage."}

![](https://ittowtnkqtyixxjxrhou.supabase.co/storage/v1/object/public/public-images/ad3ghtbg/complex_tab_66d4e706e663.png)

:::

**Runtime and Memory Usage**. Table 10 breaks down per-question cost into retrieval and answer phases, and reports video encoding separately because encoding produces a persistent KV cache shared across all questions about the same video. Encoding dominates the budget at $\approx$ 14.7 s per video. When few questions are asked of a video, the encoding dominates the compute time, but can be performed as a pre-process. $\textsc{ReToken}$ adds roughly 0.4 seconds to the per-question retrieval and answering time, but significantly improves recall and question accuracy.

```latextable {caption="Table 11: ReToken helps most when the evidence is localized and nameable, and hurts when it is dispersed across the video. Per-type accuracy on LVBench. K =100. \#QA denotes the number of questions per type; a question can carry multiple type labels."}
\begin{tabular}{lcccr}
  \toprule
  Question type & \textit{#QA} & Uniformly & \textbf{\textsc{ReToken}} & $\Delta$ \\
  \midrule
  Key information retrieval & 291 & 42.3 & 57.7 & +15.4 \\
  Entity recognition & 677 & 40.0 & 50.5 & +10.5 \\
  Reasoning & 201 & 40.8 & 46.3 & +5.5 \\
  Temporal grounding & 220 & 35.0 & 38.6 & +3.6 \\
  Event understanding & 647 & 41.4 & 43.3 & +1.9 \\
  Summarization & 58 & 36.2 & 31.0 & -5.2 \\
  \bottomrule
  \end{tabular}

Error Analysis. LVBench annotates each question with a task-type label, allowing us to characterize when $\textsc{ReToken}$ works best and when it fails (Table 11). $\textsc{ReToken}$ is strongest when the evidence is localized and nameable: key information retrieval ($+15.4$) and entity recognition ($+10.5$) are exactly the regimes where a precise retrieval target locks onto the relevant frames. Gains shrink where relevance depends on cross-frame or temporal structure rather than per-frame content, as in temporal grounding and event understanding. $\textsc{ReToken}$ hurts on summarization, where the evidence is dispersed across the entire video; uniform coverage is the better prior for this question type.

5. Conclusion

Section Summary: The paper concludes that diagnosing weaknesses in how vision-language models retrieve visual information led to ReToken, a trainable token that strengthens retrieval by focusing on value-space signals already aligned with text. This approach, trained only on multi-image data, boosts performance on both image and long-video tasks in a zero-shot manner while remaining efficient enough to run on a single GPU. The authors also note remaining limits around memory use and data scope, along with promising directions for handling temporal sequences and finer-grained retrieval in future work.

We diagnose the limitations of attention-based retrieval in VLMs and introduce $\textsc{ReToken}$, a learnable token that improves visual retrieval. Our diagnosis points to a broader principle: in pretrained VLMs, the value space carries a stronger text-aligned signal. Despite being trained only on multi-image data, $\textsc{ReToken}$ yields significant improvements on both image and long-video benchmarks, transferring zero-shot from images to videos. Both training and long-video inference fit on a single H100, making $\textsc{ReToken}$ a practical step toward scalable long-context multimodal reasoning.

Limitations. $\textsc{ReToken}$ requires a two-pass forward and attends to more visual context in early layers, adding slightly to the memory requirements and response time. Our training data is limited to multi-image QA. Training on video data could help $\textsc{ReToken}$ better capture temporal structure and improve video understanding.

Future Work. $\textsc{ReToken}$ scores each frame independently by matching the query content against per-frame value means, and relaxing this design opens several directions. First, retrieval could target sets of consecutive frames whose information emerges from their temporal combination rather than from any single frame. Second, temporally offset queries such as "what happened before I entered the room" would require a scoring mechanism aware of temporal displacement, since content matching alone tends to locate the described event rather than the frames preceding it. Third, computing the retrieval score at the token level instead of the frame level could recover evidence that occupies only a few tokens and is diluted by frame-level mean pooling.

Acknowledgments

Section Summary: The authors thank Xiaodong Liu, Sethuraman T V, and Bolin Lai for their helpful comments and discussions. The work received funding from Microsoft's Agentic AI Research and Innovation grant program and partial support from the Office of Naval Research. It also made use of NVIDIA graphics processors at an NSF-supported computing facility.

We thank Xiaodong Liu, Sethuraman T V, and Bolin Lai for their insightful comments and helpful discussions.

This research project has benefited from the Microsoft Agentic AI Research and Innovation (AARI) grant program, and was partially supported by the Office of Naval Research under grant N00014-23-1-2383. This work also used NVIDIA GPUs at NCSA Delta through allocation CIS240059 and CIS250059 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296.

Appendix

Section Summary: The appendix describes the prompt formats used to train and run the ReToken system, including the original vision-language model template, two variants that add a special retrieval token during training, and a two-stage inference process that first gathers relevant image tokens before generating an answer. It also explains how the training data was filtered to remove memorized or overly simple examples, along with comparisons showing that training on value tokens outperforms key tokens and that using multiple retrieval tokens yields little extra benefit. Additional notes cover a chain-of-thought baseline and the final size of the cleaned dataset.

A. Prompt Templates

In this section, we detail the prompt templates used during training and inference with $\textsc{ReToken}$. To highlight the modifications introduced by $\textsc{ReToken}$, we color-code the content-bearing tokens as follows: All Vision Tokens, Retrieved Vision Tokens, Question Tokens and the token, and the model output Answer.

A.1 Original Prompt Template

The standard prompt format used by vision-language models feeds all visual tokens into the model alongside the textual query:

A.2 Training Prompt Template

During training, we employ two complementary prompt formats depending on the training mode. The first, which appends the token after the question (teaching the model when to invoke retrieval), is used in both frozen and partially fine-tuned settings:

The second format, designed to preserve the model's original QA abilities, is incorporated alongside the first exclusively when the LLM is partially fine-tuned:

A.3 Inference Pipeline

At inference time, we adopt a two-stage retrieve-then-answer pipeline. In Stage 1, at most $K'$ frames are attended during retrieval and the model uses the token to aggregate query-relevant information, performing retrieval at the final layer. In Stage 2, only the retrieved visual tokens Retrieved Vision Tokens are supplied to the model to generate the final answer.

B. Additional Details

B.1 Training Data Filtering

The original MIRAGE fine-tuning set aggregates multiple open-source VQA datasets, many of which were already seen during Qwen3VL pretraining. We observe that on a substantial fraction of these examples, Qwen3VL achieves lower generation loss when conditioned on the full set of distractor images than on the target ground-truth image alone, suggesting that the model has memorized the underlying QA pairs and bypasses the intended retrieval task. To address this, we apply three filters to the MIRAGE-augmented dataset (in which images from different QAs are combined into a single multi-image input):

• Memorization filter. We discard examples where the generation loss conditioned on the target image alone is higher than that conditioned on the full image set, as such examples no longer require retrieval to be answered.
• Difficulty filter. We rank remaining examples by the average attention score between the question query tokens and the target image key tokens, and retain those with higher scores. In a controlled experiment, we find that easier retrieval examples (higher attention scores on the target image) lead to a stronger learned retriever than harder examples.
• Multi-image filter. We remove single-image QA examples (which lack distractor images for retrieval) and OCR-style queries (which test text recognition rather than visual recognition).

::: {caption="Table 12: The influence of training dataset. 'w/o filtered' means directly sampling 70, 686 examples from the original MIRAGE fine-tuning set without filtering, and using them to train ReToken. 'w/ filtered' is our default setting."}

:::

After filtering, there are 70,686 examples in our final training dataset. We note that our filters are applied only to the MIRAGE training split; the Visual Haystacks evaluation split and video datasets are left untouched. The filtering signals (generation loss and attention scores) are computed from the frozen Qwen3VL-8B on training examples only, with no access to evaluation labels or examples. Table 12 shows the influence of filtering.

B.2 Chain-of-Thought (CoT) Retrieval Baseline

Given the following question, what phrase should I search for to find the visual evidence needed to answer it? Respond with the search phrase only, no more than 5 words.

Question: question

B.3 Key or Value? Training Comparison

Figure 6 shows the retrieval loss and retrieval score gap across training steps with the VLM frozen and only $\textsc{ReToken}$ trained: $\times$ Value yields a clear advantage.

C. Design Choice

C.1 Multiple Tokens

Table 13 reports results for training 3 ReTokens on the retrieval task. We append the 3 ReTokens sequentially, and compute the retrieval score by computing the cosine similarity between each projected ReToken and the averaged image value feature. We then average the resulting logits to obtain the final score.

::: {caption="Table 13: Multiple tokens perform on par with a single token. Qwen3VL-8B."}

:::

Performance is on par overall, with small fluctuations in both directions. One reason may lie in our training recipe: the multi-token design treats the embeddings equally, since all three are supervised through one averaged score, nothing encourages them to specialize, so the tokens likely converge on a similar solution. A promising direction could be to empower specialization of different tokens so that a token is more pronounced for its specialized domain. But this is beyond the scope of this work and we leave it for future exploration.

C.2 Query Composition or visual Summarization

Since we append the $\textsc{ReToken}$ to the input, it can attend to both the images and the query, allowing it to serve as both a visual summarization token and a query summarization token. To better understand the role of the $\textsc{ReToken}$, we design an ablation in which the $\textsc{ReToken}$ is only allowed to attend to the query tokens. In this setting, we perform two forward streams. In stream A, we encode the vision tokens and cache their KV. In stream B, we process only the query and the $\textsc{ReToken}$, so that neither the query token nor the $\textsc{ReToken}$ can attend to the visual part. This way, the $\textsc{ReToken}$ sees only the query tokens and learns to summarize what we want to retrieve based on the input query alone. We then compute the retrieval score by computing the cosine similarity between the projected $\textsc{ReToken}$ from the last layer of stream B and the averaged visual value feature from the last layer of stream A.

::: {caption="Table 14: Allowing the ReToken to attend to images yields better results. Question tokens also skip the visual tokens in the 'skip images' setting. The VLM is frozen."}

:::

Table 14 reports the results with and without attention to the images. The results show that skipping the visual tokens entirely still yields decent performance, but underperforms the variant that attends to the images. This suggests that when the $\textsc{ReToken}$ can understand what happens in the input video/images, it achieves better retrieval results.

References

Section Summary: This section consists of a numbered bibliography listing dozens of academic papers and technical reports from recent years. The cited works focus on advancements in vision-language AI models, techniques for handling long videos and extended contexts in machine learning systems, and related benchmarks for tasks like video understanding and retrieval. They represent foundational and cutting-edge sources likely drawn upon in the main document for methods, evaluations, and comparisons.

[1] Liu et al. (2023). Visual instruction tuning. Advances in neural information processing systems. 36. pp. 34892–34916.

[2] Team et al. (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.

[3] Bai et al. (2025). Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.

[4] Chen et al. (2024). Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24185–24198.

[5] Wu et al. (2024). Visual haystacks: A vision-centric needle-in-a-haystack benchmark. arXiv preprint arXiv:2407.13766.

[6] Xiao et al. (2023). Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453.

[7] Han et al. (2024). Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 3991–4008.

[8] Wang et al. (2025). Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.

[9] Wang et al. (2025). Lvbench: An extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22958–22967.

[10] Radford et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning. pp. 8748–8763.

[11] Bolya et al. (2025). Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181.

[12] Tschannen et al. (2025). Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786.

[13] Jiang et al. (2024). E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580.

[14] Liu et al. (2025). Lamra: Large multimodal model as your advanced retrieval assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 4015–4025.

[15] Yu et al. (2023). Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems. 36. pp. 76749–76771.

[16] Li et al. (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730–19742.

[17] Wang et al. (2024). Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision. pp. 58–76.

[18] Wang et al. (2025). Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 3272–3283.

[19] Xiao et al. (2024). Infllm: Training-free long-context extrapolation for llms with an efficient context memory. Advances in neural information processing systems. 37. pp. 119638–119661.

[20] Fountas et al. (2025). Human-inspired episodic memory for infinite context LLMs. In 13th International Conference on Learning Representations Iclr 2025.

[21] Di et al. (2025). Streaming video question-answering with in-context video kv-cache retrieval. arXiv preprint arXiv:2503.00540.

[22] Song et al. (2024). Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18221–18232.

[23] He et al. (2024). Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13504–13514.

[24] Jin et al. (2024). Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13700–13710.

[25] Li et al. (2024). Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision. pp. 323–340.

[26] Shen et al. (2024). Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434.

[27] Shu et al. (2025). Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 26160–26169.

[28] Ataallah et al. (2024). Goldfish: Vision-language understanding of arbitrarily long videos. In European Conference on Computer Vision. pp. 251–267.

[29] Luo et al. (2024). Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093.

[30] Ma et al. (2025). Drvideo: Document retrieval based long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18936–18946.

[31] Alayrac et al. (2022). Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems. 35. pp. 23716–23736.

[32] Dai et al. (2023). Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems. 36. pp. 49250–49267.

[33] Zhu et al. (2025). One token can help! learning scalable and pluggable virtual tokens for retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence. pp. 26166–26174.

[34] Zhou et al. (2022). Extract free dense labels from clip. In European conference on computer vision. pp. 696–712.

[35] Wang et al. (2024). Sclip: Rethinking self-attention for dense vision-language inference. In European conference on computer vision. pp. 315–332.

[36] Lan et al. (2024). Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. In European Conference on Computer Vision. pp. 70–88.

[37] Xiao et al. (2025). TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models. arXiv preprint arXiv:2505.23769.

[38] Penamakuri et al. (2023). Answer mining from a pool of images: towards retrieval-based visual question answering. arXiv preprint arXiv:2306.16713.

[39] Tanaka et al. (2023). Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence. pp. 13636–13645.

[40] Chang et al. (2022). Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16495–16504.

[41] Liu et al. (2024). Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 26296–26306.

[42] Lin et al. (2014). Microsoft coco: Common objects in context. In European conference on computer vision. pp. 740–755.

[43] Di, Shangzhe and Xie, Weidi (2024). Grounded question-answering in long egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12934–12943.

[44] Bärmann, Leonard and Waibel, Alex (2022). Where did I leave my keys? — Episodic-Memory-Based Question Answering on Egocentric Videos. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1559-1567. doi:10.1109/CVPRW56347.2022.00162.

[45] Fu et al. (2025). Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 24108–24118.

[46] Team et al. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

[47] Chen et al. (2024). Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.

[48] Khosla et al. (2025). REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders. arXiv preprint arXiv:2505.18153.

[49] Comanici et al. (2025). Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.

[50] Singh et al. (2025). Openai gpt-5 system card. arXiv preprint arXiv:2601.03267.

[51] Anthropic (2025). System Card Addendum: Claude Opus 4.1. https://www-cdn.anthropic.com/9fa30625273bafdf5af82c93719d7ca606485a16.pdf.