Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

temporal saliency

Temporal saliency refers to the measure of importance, prominence, or informativeness assigned to specific points in time or segments within a dynamic visual sequence such as a video. Unlike spatial saliency, which locates distinctive regions within an individual image or frame, temporal saliency focuses on the temporal dimension to identify when significant events, distinctive motions, or concept-relevant transitions take place. In computer vision and video analysis tasks such as action recognition, video summarization, and multimodal modeling, estimating temporal saliency enables algorithms to prioritize critical frames while filtering out redundant or uninformative intervals, resulting in more effective and focused video representations.

1 item

Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models

Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models

Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, Wanli Ouyang

OrganizationsBaiduShanghai Artificial Intelligence LaboratoryUniversity of Chinese Academy of SciencesUniversity of SydneyZhejiang University

Why you should read this

Proposes a bidirectional framework called BIKE that transfers pre-trained vision-language knowledge into video recognition by retrieving complementary textual attributes and using category concepts to capture frame-level temporal saliency.

Vision-language models (VLMs) pre-trained on large-scale image-text pairs have demonstrated impressive transferability on various visual tasks. Transferring knowledge from such powerful VLMs is a promising direction for building effective video recognition models. However, current exploration in this field is still limited. We believe that the greatest value of pre-trained VLMs lies in building a bridge between visual and textual domains. In this paper, we propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge: i) We introduce the Video Attribute Association mechanism, which leverages the Video-to-Text knowledge to generate textual auxiliary attributes for complementing video recognition. ii) We also present a Temporal Concept Spotting mechanism that uses the Text-to-Video expertise to capture temporal saliency in a parameter-free manner, leading to enhanced video representation. Extensive studies on six popular video datasets, including Kinetics-400 & 600, UCF-101, HMDB-51, ActivityNet and Charades, show that our method achieves state-of-the-art performance in various recognition scenarios, such as general, zero-shot, and few-shot video recognition. Our best model achieves a state-of-the-art accuracy of 88.6% on the challenging Kinetics-400 using the released CLIP model. The code is available at https://github.com/whwu95/BIKE.

Added

2026-09-26