Built independently by an author, for readers. Read the story and support ChapterPal

keyword

Self-Supervised Vision Representation Learning

Self-supervised vision representation learning is a machine learning approach in computer vision that trains neural networks to extract meaningful visual features from unlabeled images or videos without relying on human annotations. Instead of using manual labels, this paradigm generates supervisory signals directly from the inherent structure of the visual data through pretext tasks such as masked image modeling, contrastive learning, and instance discrimination across transformed views. By learning to predict missing visual regions, reconstruct content, or align representations across different augmentations of the same input, models capture both high-level semantic relationships and fine-grained spatial structures. The resulting representations provide a general-purpose feature foundation that can be transferred or fine-tuned for a wide variety of downstream vision tasks, including image classification, object detection, and semantic segmentation.

1 item

Siamese Image Modeling for Self-Supervised Vision Representation Learning

Siamese Image Modeling for Self-Supervised Vision Representation Learning

Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, Jifeng Dai

OrganizationsSenseTimeShanghai Artificial Intelligence LaboratoryThe Chinese University of Hong KongTsinghua UniversityUniversity of Science and Technology of China

Why you should read this

Proposes Siamese Image Modeling, a self-supervised framework that bridges the gap between instance discrimination and masked image modeling by predicting dense representations across differently augmented views to achieve both semantic alignment and spatial sensitivity.

Self-supervised learning (SSL) has delivered superior performance on a variety of downstream vision tasks. Two main-stream SSL frameworks have been proposed, i.e., Instance Discrimination (ID) and Masked Image Modeling (MIM). ID pulls together representations from different views of the same image, while avoiding feature collapse. It lacks spatial sensitivity, which requires modeling the local structure within each image. On the other hand, MIM reconstructs the original content given a masked image. It instead does not have good semantic alignment, which requires projecting semantically similar views into nearby representations. To address this dilemma, we observe that (1) semantic alignment can be achieved by matching different image views with strong augmentations; (2) spatial sensitivity can benefit from predicting dense representations with masked images. Driven by these analysis, we propose Siamese Image Modeling (SiameseIM), which predicts the dense representations of an augmented view, based on another masked view from the same image but with different augmentations. SiameseIM uses a Siamese network with two branches. The online branch encodes the first view, and predicts the second view's representation according to the relative positions between these two views. The target branch produces the target by encoding the second view. SiameseIM can surpass both ID and MIM on a wide range of downstream tasks, including ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation. The improvement is more significant in few-shot, long-tail and robustness-concerned scenarios. Code shall be released.

Added

2026-09-26