Built independently by an author, for readers. Read the story and support ChapterPal

keyword

patch-wise visual features

Patch-wise visual features are numerical representations extracted from localized spatial sub-regions, or patches, of an image. In modern computer vision architectures such as Vision Transformers, an input image is partitioned into a grid of distinct patches, which are linearly projected and processed through neural network layers. Each resulting feature vector corresponds to a specific patch, capturing local visual patterns such as color, texture, and shape while integrating global contextual information through self-attention mechanisms. By maintaining localized spatial information across the image grid, these features enable neural networks to perform fine-grained visual recognition, cross-modal alignment, and dense prediction tasks such as semantic segmentation and object detection.

1 item

ReSTR: Convolution-free Referring Image Segmentation Using Transformers

ReSTR: Convolution-free Referring Image Segmentation Using Transformers

Namyup Kim, Dongwon Kim, Suha Kwak, Cuiling Lan, Wenjun Zeng

OrganizationsEIT Institute for Advanced StudyMicrosoftPohang University of Science and Technology

Why you should read this

Proposes ReSTR, the first purely transformer-based, convolution-free model for referring image segmentation that unifies vision and language processing through self-attention to capture long-range cross-modal dependencies and achieve state-of-the-art performance across major benchmarks.

Referring image segmentation is an advanced semantic segmentation task where target is not a predefined class but is described in natural language. Most of existing methods for this task rely heavily on convolutional neural networks, which however have trouble capturing long-range dependencies between entities in the language expression and are not flexible enough for modeling interactions between the two different modalities. To address these issues, we present the first convolution-free model for referring image segmentation using transformers, dubbed ReSTR. Since it extracts features of both modalities through transformer encoders, it can capture long-range dependencies between entities within each modality. Also, ReSTR fuses features of the two modalities by a self-attention encoder, which enables flexible and adaptive interactions between the two modalities in the fusion process. The fused features are fed to a segmentation module, which works adaptively according to the image and language expression in hand. ReSTR is evaluated and compared with previous work on all public benchmarks, where it outperforms all existing models.

Added

2026-09-26