Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

visual hierarchy

Visual hierarchy in computer vision and representation learning refers to the structured organization of visual information across multiple levels of abstraction, spanning from fine-grained, localized details to high-level semantic concepts. Within artificial neural networks and visual understanding systems, this structure is captured by progressively aggregating low-level visual inputs, such as edges, textures, and constituent parts, into unified entities, categories, and holistic scene representations. By organizing visual data as a multi-layered compositional structure rather than a flat collection of features, computational models can better interpret the relationships between individual elements and overarching contexts, facilitating more effective visual reasoning, object recognition, and multimodal alignment with language.

1 item

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, Yongfeng Zhang

OrganizationsByteDanceRutgers University

Why you should read this

Introduces HiCLIP to integrate hierarchy-aware attention into contrastive vision-language pretraining, enabling the unsupervised discovery of multi-level semantics across images and text to improve cross-modal alignment on downstream tasks.

The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision-language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks.

Added

2026-09-26