Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

pre-trained visual encoder

A pre-trained visual encoder is a deep neural network component that has been optimized on large collections of image or video data to extract and transform raw visual inputs into rich, meaningful numerical representations called feature embeddings. Commonly implemented using architectures such as convolutional neural networks or vision transformers, it captures a spectrum of visual patterns ranging from low-level textures and edges to high-level semantic concepts. In broader machine learning systems and vision-language models, this component serves as a foundational visual backbone that can be kept frozen or fine-tuned to transfer learned visual knowledge to downstream tasks, including image recognition, video classification, and cross-modal alignment, without needing to train the visual feature extractor from scratch.

1 item

Revisiting Classifier: Transferring Vision-Language Models for Video Recognition

Revisiting Classifier: Transferring Vision-Language Models for Video Recognition

Wenhao Wu, Zhun Sun, Wanli Ouyang

OrganizationsBaiduShanghai Artificial Intelligence LaboratoryUniversity of Sydney

Why you should read this

Proposes replacing the traditional randomly initialized visual classifier with frozen text embeddings from pre-trained vision-language models, drastically boosting video recognition accuracy and convergence speed across zero-shot, few-shot, and fully supervised benchmarks.

Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. Along with the growth of computational capacity, we now have open-source vision-language pre-trained models in large scales of the model architecture and amount of data. In this study, we focus on transferring knowledge for video classification tasks. Conventional methods randomly initialize the linear classifier head for vision classification, but they leave the usage of the text encoder for downstream visual recognition tasks undiscovered. In this paper, we revise the role of the linear classifier and replace the classifier with different knowledge from the pre-trained model. We utilize the well-pre-trained language model to generate a good semantic target for efficient transferring learning. The empirical study shows that our method improves both the performance and the training speed of video classification, with a negligible change in the model. Our simple yet effective tuning paradigm achieves state-of-the-art performance and efficient training on various video recognition scenarios, i.e., zero-shot, few-shot, and general recognition. In particular, our paradigm achieves the state-of-the-art accuracy of 87.8% on Kinetics-400, and also surpasses previous methods by 20~50% absolute top-1 accuracy under zero-shot, few-shot settings on five video datasets. Code and models are available at https://github.com/whwu95/Text4Vis.

Added

2026-09-26