Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

hierarchical co-attention model

A hierarchical co-attention model is a multimodal deep learning architecture that jointly coordinates attention across two distinct data modalities, such as text and images, while processing at least one modality across multiple granularities. Commonly applied in visual question answering, this architecture simultaneously determines which visual regions to focus on and which textual elements to prioritize, establishing bidirectional relevance between language and visual features. The hierarchical mechanism structures the textual input across progressive levels of representation, typically spanning word, phrase, and sentence levels, which enables the network to align both fine-grained lexical details and broader semantic context with corresponding visual regions for more accurate reasoning.

1 item

Hierarchical Question-Image Co-Attention for Visual Question Answering

Hierarchical Question-Image Co-Attention for Visual Question Answering

Jiasen Lu, Jianwei Yang, Dhruv Batra, Devi Parikh

OrganizationsGeorgia Institute of TechnologyVirginia Tech

Why you should read this

Introduces a hierarchical co-attention model for visual question answering that jointly computes attention over relevant image regions and multi-level language structures across word, phrase, and question representations.

A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.

Added

2026-09-24