Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

keyframe selection

Keyframe selection is the process of identifying and extracting a representative subset of still frames from a video sequence that effectively captures its primary visual content, structure, and events. By filtering out redundant or uninformative frames, this technique reduces the overall volume of video data to decrease computational complexity and memory consumption in downstream applications such as video summarization, indexing, retrieval, and multimodal video analysis. Selection algorithms typically evaluate factors including visual diversity, temporal coverage across the duration of the video, motion dynamics, and semantic relevance to specific queries or tasks, ensuring that critical visual information is preserved within a constrained frame budget.

1 item

Adaptive Keyframe Sampling for Long Video Understanding

Adaptive Keyframe Sampling for Long Video Understanding

Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, Qixiang Ye

OrganizationsUniversity at BuffaloUniversity of Chinese Academy of Sciences

Why you should read this

Proposes a plug-and-play keyframe selection algorithm that balances prompt relevance with temporal coverage to improve long video question-answering accuracy in multimodal large language models without exceeding token limits.

Multimodal large language models (MLLMs) have enabled open-world visual understanding by injecting visual input as extra tokens into large language models (LLMs) as contexts. However, when the visual input changes from a single image to a long video, the above paradigm encounters difficulty because the vast amount of video tokens has significantly exceeded the maximal capacity of MLLMs. Therefore, existing video-based MLLMs are mostly established upon sampling a small portion of tokens from input data, which can cause key information to be lost and thus produce incorrect answers. This paper presents a simple yet effective algorithm named Adaptive Keyframe Sampling (AKS). It inserts a plug-and-play module known as keyframe selection, which aims to maximize the useful information with a fixed number of video tokens. We formulate keyframe selection as an optimization involving (1) the relevance between the keyframes and the prompt, and (2) the coverage of the keyframes over the video, and present an adaptive algorithm to approximate the best solution. Experiments on two long video understanding benchmarks validate that AKS improves video QA accuracy (beyond strong baselines) upon selecting informative keyframes. Our study reveals the importance of information pre-filtering in video-based MLLMs. Our codes are available at https://github.com/ncTimTang/AKS

Added

2026-09-26