Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

visual-linguistic encoder

A visual-linguistic encoder is a neural network component that processes and integrates visual data and natural language text into a shared multimodal representation. Often built using transformer architectures with self-attention or cross-attention mechanisms, it extracts features from both images and textual descriptions, capturing dependencies within each separate modality as well as complex interactions between them. By aligning words with corresponding visual regions or objects, the encoder enables models to understand context across different data types, serving as a core foundation for downstream vision-language tasks such as referring image segmentation, visual question answering, and cross-modal retrieval.

1 item

ReSTR: Convolution-free Referring Image Segmentation Using Transformers

ReSTR: Convolution-free Referring Image Segmentation Using Transformers

Namyup Kim, Dongwon Kim, Suha Kwak, Cuiling Lan, Wenjun Zeng

OrganizationsEIT Institute for Advanced StudyMicrosoftPohang University of Science and Technology

Why you should read this

Proposes ReSTR, the first purely transformer-based, convolution-free model for referring image segmentation that unifies vision and language processing through self-attention to capture long-range cross-modal dependencies and achieve state-of-the-art performance across major benchmarks.

Referring image segmentation is an advanced semantic segmentation task where target is not a predefined class but is described in natural language. Most of existing methods for this task rely heavily on convolutional neural networks, which however have trouble capturing long-range dependencies between entities in the language expression and are not flexible enough for modeling interactions between the two different modalities. To address these issues, we present the first convolution-free model for referring image segmentation using transformers, dubbed ReSTR. Since it extracts features of both modalities through transformer encoders, it can capture long-range dependencies between entities within each modality. Also, ReSTR fuses features of the two modalities by a self-attention encoder, which enables flexible and adaptive interactions between the two modalities in the fusion process. The fused features are fed to a segmentation module, which works adaptively according to the image and language expression in hand. ReSTR is evaluated and compared with previous work on all public benchmarks, where it outperforms all existing models.

Added

2026-09-26