Built independently by an author, for readers. Read the story and support ChapterPal

keyword

region-word similarity

Region-word similarity is a metric in multimodal artificial intelligence that measures the degree of semantic alignment and correspondence between a localized region of an image and an individual word in a text sequence. It is typically calculated by mapping visual feature representations of image regions, such as objects or background elements, alongside word embeddings into a shared semantic space, commonly using vector operations like cosine similarity. This fine-grained measurement allows computational models to capture local vision-language interactions, serving as a foundational element for cross-modal attention, visual grounding, and comprehensive image-text retrieval tasks.

1 item

Stacked Cross Attention for Image-Text Matching

Stacked Cross Attention for Image-Text Matching

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, Xiaodong He

OrganizationsJD AI ResearchMicrosoft

Why you should read this

Proposes a stacked cross-attention network that aligns visual regions with corresponding sentence words, achieving significant gains in bidirectional image-text retrieval accuracy on MS-COCO and Flickr30K.

In this paper, we study the problem of image-text matching. Inferring the latent semantic alignment between objects or other salient stuff (e.g. snow, sky, lawn) and the corresponding words in sentences allows to capture fine-grained interplay between vision and language, and makes image-text matching more interpretable. Prior work either simply aggregates the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or uses a multi-step attentional process to capture limited number of semantic alignments which is less interpretable. In this paper, we present Stacked Cross Attention to discover the full latent alignments using both image regions and words in a sentence as context and infer image-text similarity. Our approach achieves the state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K, our approach outperforms the current best methods by 22.1% relatively in text retrieval from image query, and 18.2% relatively in image retrieval with text query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1 using the 5K test set). Code has been made available at: this https URL.

Added

2026-09-25