Back

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Shijie GengJianbo YuanYu TianYuxiao ChenYongfeng Zhang

article2023ICLR59 citations

Introduces HiCLIP to integrate hierarchy-aware attention into contrastive vision-language pretraining, enabling the unsupervised discovery of multi-level semantics across images and text to improve cross-modal alignment on downstream tasks.

ByteDanceRutgers Universityhierarchical structuresinferencelanguage model
arXiv: 2303.02995jeykigung/HiCLIP
Cover for HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Abstract

The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision-language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks.

Topics

hierarchical structuresinferencelanguage modelreasoningunsupervised learning methodprobabilistic inferencesupervised machine learning

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Hierarchy-aware Attention in CLIP
  • 3.1 A Framework of Hierarchical Information Aggregation
  • 3.1.1 Hierarchy Induction for Language Branch
  • 3.1.2 Hierarchy Induction for Visual Branch
  • 3.2 Hierarchy-aware CLIP
  • 4 Experiments
  • 4.1 Experimental Settings
  • 4.2 Visual Recognition
  • 4.3 Performance Comparison on Vision-Language Tasks
  • 4.4 Ablation Study
  • 4.5 Unsupervised Hierarchy Induction with Pretrained HiCLIP model
  • 5 Conclusion and Future Work
  • References
  • A Prompts Engineering for Zero-shot Visual Recognition
  • B Linear Probe Performance
  • C Additional Pretraining Implementation Details
  • D More Visualization Results & Visualization Process
  • E Visualization of Learned Feature Space
  • F Detailed Illustration of the Computation of CC

Knowls

  1. Knowl 1 — Hierarchy-Aware Attention Mechanism

    model/method

    Hierarchy-aware attention modifies the standard multi-head self-attention mechanism in Transformers by incorporating a hierarchy aggregation mask CC as an inductive bias. The standard Transformer attention for query QQ, key KK, and value VV with head dimension dhd_h is defined as:

    Attention(Q,K,V)=softmax(QKTdh)V\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_h}}\right)V

    Hierarchy-aware attention introduces an attention mask matrix C∈[0,1]N×NC \in [0, 1]^{N \times N} (where NN is the sequence length) that element-wise modulates the attention weights:

    Hierarchy_Attention(Q,K,V)=(C⊙softmax(QKTdh))V\text{Hierarchy\_Attention}(Q, K, V) = \left( C \odot \text{softmax}\left(\frac{QK^T}{\sqrt{d_h}}\right) \right) V

    The attention mask CC is shared across all attention heads within a layer and is updated bottom-up layer-by-layer to satisfy two core priors:

    1. Tendency to merge: Spatially and semantically adjacent tokens or image patches that share high similarity are assigned higher affinity weights to encourage grouping into higher-level semantic constituents (such as phrases or visual objects).
    2. Non-splittability: Once constituents are merged at lower layers, they cannot be separated at subsequent higher layers. This ensures that semantic clustering progresses monotonically without degrading as the network depth increases.

  2. Knowl 2 — Group Transformer for 2D Visual Hierarchy Induction

    model/method

    Group Transformer generalizes hierarchy-aware attention to 2D image grids by progressively aggregating image patches into semantic object groups.

    Let ti,jt_{i,j} denote the feature representation of an image patch at 2D grid position (i,j)(i, j). For each patch, its local neighborhood A\mathcal{A} is defined by its 4-connected spatial neighbors:

    A(i,j)={(i+δ,j+η)∣δ,η∈{−1,+1},∣δ∣+∣η∣=1}\mathcal{A}(i, j) = \{(i + \delta, j + \eta) \mid \delta, \eta \in \{-1, +1\}, |\delta| + |\eta| = 1\}

    1. Neighboring Attention Score: The merging tendency between patch ti,jt_{i,j} and an adjacent neighbor ti′,j′∈A(i,j)t_{i',j'} \in \mathcal{A}(i, j) is computed via learnable projection matrices WQ′′W_Q'' and WK′′W_K'':

    s(i,j),(i′,j′)=(ti,jWQ′′)⋅(ti′,j′WK′′)σvs_{(i,j),(i',j')} = \frac{(t_{i,j} W_Q'') \cdot (t_{i',j'} W_K'')}{\sigma_v}

    where σv\sigma_v is a scaling hyperparameter (set to 256).

    1. Normalized Merging Tendency and Affinity: Merging tendencies are normalized across the four neighbors using softmax:

    {p(i,j),(i′,j′)}=softmax({s(i,j),(i′,j′)∣(i′,j′)∈A(i,j)})\{p_{(i,j),(i',j')}\} = \text{softmax}(\{s_{(i,j),(i',j')} \mid (i', j') \in \mathcal{A}(i, j)\})

    The symmetric neighboring affinity score is defined as the geometric mean:

    a^(i,j),(i′,j′)=p(i,j),(i′,j′)⋅p(i′,j′),(i,j)\hat{a}_{(i,j),(i',j')} = \sqrt{p_{(i,j),(i',j')} \cdot p_{(i',j'),(i,j)}}

    1. Non-splittable Layer-wise Update: At layer ll, the cumulative affinity score a(i,j),(i′,j′)la^l_{(i,j),(i',j')} is recursively updated from layer l−1l-1 to enforce monotonic non-decreasing affinities (al≥al−1a^l \ge a^{l-1}):

    a(i,j),(i′,j′)l=a(i,j),(i′,j′)l−1+(1−a(i,j),(i′,j′)l−1)a^(i,j),(i′,j′)la^l_{(i,j),(i',j')} = a^{l-1}_{(i,j),(i',j')} + \left(1 - a^{l-1}_{(i,j),(i',j')}\right) \hat{a}^l_{(i,j),(i',j')}

    1. 2D Grid Propagation for Mask CC: To compute the attention mask entry C(i1,j1),(i2,j2)C_{(i_1,j_1),(i_2,j_2)} between any two arbitrary patches (i1,j1)(i_1, j_1) and (i2,j2)(i_2, j_2), affinities are propagated along two single-turn orthogonal paths (horizontal-first C1C_1 and vertical-first C2C_2):

    C1=(∏n=i1i2−1a(n,j1),(n+1,j1))(∏m=j1j2−1a(i2,m),(i2,m+1))C_1 = \left( \prod_{n=i_1}^{i_2-1} a_{(n,j_1),(n+1,j_1)} \right) \left( \prod_{m=j_1}^{j_2-1} a_{(i_2,m),(i_2,m+1)} \right)

    C2=(∏m=j1j2−1a(i1,m),(i1,m+1))(∏n=i1i2−1a(n,j2),(n+1,j2))C_2 = \left( \prod_{m=j_1}^{j_2-1} a_{(i_1,m),(i_1,m+1)} \right) \left( \prod_{n=i_1}^{i_2-1} a_{(n,j_2),(n+1,j_2)} \right)

    The composite pairwise mask weight is then chosen as the maximum over the two paths:

    C(i1,j1),(i2,j2)=max⁡(C1,C2)C_{(i_1,j_1),(i_2,j_2)} = \max(C_1, C_2)

  3. Knowl 3 — Language Hierarchy Induction via Tree Transformer

    model/method

    In the language branch of HiCLIP, 1D textual hierarchy is modeled using a Tree Transformer formulation to induce constituent tree structures without parse annotations.

    1. Neighboring Attention Score: For adjacent token representations (ti,ti+1)(t_i, t_{i+1}), the affinity score is computed using query and key projection matrices WQ0,WK0W_Q^0, W_K^0:

    si,i+1=(tiWQ0)⋅(ti+1WK0)σts_{i,i+1} = \frac{(t_i W_Q^0) \cdot (t_{i+1} W_K^0)}{\sigma_t}

    where σt\sigma_t is a temperature scaling factor (set to 256).

    1. Softmax Normalization & Affinity Score: The directional merging probability of token tit_i toward its left and right neighbors is normalized via softmax:

    pi,i+1,pi,i−1=softmax(si,i+1,si,i−1)p_{i,i+1}, p_{i,i-1} = \text{softmax}(s_{i,i+1}, s_{i,i-1})

    The bidirectional neighboring affinity a^i,i+1\hat{a}_{i,i+1} is computed as the geometric mean of the mutual merging probabilities:

    a^i,i+1=pi,i+1⋅pi+1,i\hat{a}_{i,i+1} = \sqrt{p_{i,i+1} \cdot p_{i+1,i}}

    1. Non-splittable Layer Update: To prevent merged constituents from splitting in deeper layers, layer ll updates the accumulated affinity monotonically:

    ai,i+1l=ai,i+1l−1+(1−ai,i+1l−1)a^i,i+1la_{i,i+1}^l = a_{i,i+1}^{l-1} + \left(1 - a_{i,i+1}^{l-1}\right) \hat{a}_{i,i+1}^l

    where ai,i+1l≥ai,i+1l−1a_{i,i+1}^l \ge a_{i,i+1}^{l-1}.

    1. Attention Mask Computation: The pairwise hierarchy mask entry Ci,jC_{i,j} between tokens ii and jj (i≤ji \le j) is computed by the chained product of intermediate neighboring affinities:

    Ci,j=∏k=ij−1ak,k+1C_{i,j} = \prod_{k=i}^{j-1} a_{k,k+1}

    with Ci,j=Cj,iC_{i,j} = C_{j,i} ensuring matrix symmetry.

  4. Knowl 4 — HiCLIP Architecture and Contrastive Pretraining Formulation

    model/method

    Hierarchy-aware CLIP (HiCLIP) pairs a Group Transformer as the visual encoder and a Tree Transformer as the language encoder. Both encoders generate dd-dimensional normalized feature vectors viv_i (image) and uiu_i (text) for an image-text pair ii.

    The model is pretrained on a batch of NN image-text pairs using a symmetric vision-language InfoNCE contrastive objective:

    L=−1N∑i=1Nlog⁡exp⁡(viTui/τ)∑j=1Nexp⁡(viTuj/τ)−1N∑i=1Nlog⁡exp⁡(uiTvi/τ)∑j=1Nexp⁡(uiTvj/τ)\mathcal{L} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(v_i^T u_i / \tau)}{\sum_{j=1}^N \exp(v_i^T u_j / \tau)} - \frac{1}{N} \sum_{i=1}^N \log \frac{\exp(u_i^T v_i / \tau)}{\sum_{j=1}^N \exp(u_i^T v_j / \tau)}

    where τ\tau is a learnable temperature parameter initialized to 0.07.

    By optimizing cross-modal alignment on top of layer-wise hierarchy-aware attention masks, HiCLIP induces visual and textual representations to construct and align structured semantic constituents at multiple abstraction levels without explicit parse tree or bounding box annotations.

  5. Knowl 5 — Unsupervised Visual Hierarchy Induction Algorithm

    algorithm

    During inference, HiCLIP induces visual patch hierarchies without supervision by analyzing the accumulated layer-wise neighboring affinity scores a(i,j),(i′,j′)la_{(i,j),(i',j')}^l in a top-down greedy fashion.

    Input: Neighboring affinity scores a(i,j),(i′,j′)la_{(i,j),(i',j')}^l for layers l∈{1,…,N}l \in \{1, \dots, N\}, layer threshold sequence {θ1,…,θN}\{\theta_1, \dots, \theta_N\}
    Output: Hierarchy tree / connected constituent regions across layers
    l←Nl \leftarrow N
    Initialize nested list B={B1,…,BN}B = \{B_1, \dots, B_N\} to store broken edges per layer
    while l>0l > 0 do
        for each adjacent edge (i,j),(i′,j′)(i, j), (i', j') in the 2D patch grid do
            if a(i,j),(i′,j′)l<θla_{(i,j),(i',j')}^l < \theta_l then
                if l==Nl == N then
                    Append edge (i,j),(i′,j′)(i, j), (i', j') to BlB_l
                else
                    if edge (i,j),(i′,j′)(i, j), (i', j') not in Bl+1B_{l+1} then
                        Append edge (i,j),(i′,j′)(i, j), (i', j') to BlB_l
                    end if
                end if
            end if
        end for
        l←l−1l \leftarrow l - 1
    end while
    Construct visual hierarchy from BB and remove redundant edges via connected component analysis

    For a 12-layer Vision Transformer, the layer-wise breaking thresholds {θ1,…,θ12}\{\theta_1, \dots, \theta_{12}\} are configured to {0.35,0.5,0.5,0.6,0.8,0.85,0.9,0.9,0.9,0.9,0.9,0.9}\{0.35, 0.5, 0.5, 0.6, 0.8, 0.85, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9\}.

  6. Knowl 6 — Zero-Shot Visual Classification Benchmarks of HiCLIP

    data/table

    Zero-shot top-1 classification performance (%) across 11 visual recognition benchmarks using a ViT-B/32 backbone pretrained on YFCC15M (15M) and a 30M combination dataset (YFCC15M + CC3M + CC12M). The benchmarks include CIFAR-10 (C10), CIFAR-100 (C100), Food101 (F101), Oxford-IIIT Pets (Pets), Flowers102 (Flow), SUN397 (SUN), Stanford Cars (Cars), Describable Textures Dataset (DTD), Caltech101 (Cal), FGVC-Aircraft (Air), and ImageNet (IN).

    Model Data C10 C100 F101 Pets Flow SUN Cars DTD Cal Air IN Avg
    CLIP 15M 63.7 33.2 34.6 20.1 50.1 35.7 2.6 15.5 59.9 1.2 32.8 31.8
    SLIP 15M 50.7 25.5 33.3 23.5 49.0 34.7 2.8 14.4 59.9 1.7 34.3 30.0
    FILIP 15M 65.5 33.5 43.1 24.1 52.7 50.7 3.3 24.3 68.8 3.2 39.5 37.2
    HiCLIP 15M 74.1 46.0 51.2 37.8 60.9 50.6 4.5 23.1 67.4 3.6 40.5 41.8
    DeCLIP 15M 66.7 38.7 52.5 33.8 60.8 50.3 3.8 27.7 74.7 2.1 43.2 41.3
    DeFILIP 15M 70.1 46.8 54.5 40.3 63.7 52.4 4.6 30.2 75.0 3.3 45.0 44.2
    HiDeCLIP 15M 65.1 39.4 56.3 43.6 64.1 55.4 5.4 34.0 77.0 4.6 45.9 44.6
    CLIP 30M 77.3 48.1 59.1 58.5 58.2 52.6 17.7 28.0 80.8 3.2 48.8 48.4
    HiCLIP 30M 77.6 56.2 63.9 65.6 62.5 60.7 22.2 38.0 82.4 5.5 52.9 53.4
    DeCLIP 30M 84.0 57.1 67.3 71.7 65.0 62.5 23.0 39.5 86.1 5.3 55.3 56.1
    HiDeCLIP 30M 80.4 54.2 68.9 73.5 66.1 65.2 26.8 44.2 87.8 7.2 56.9 57.4

    HiCLIP achieves a 10.0% gain in 11-dataset average accuracy over standard CLIP under 15M pretraining (41.8% vs 31.8%) and a 5.0% average gain under 30M pretraining (53.4% vs 48.4%). Adding self-supervised multi-task objectives (HiDeCLIP) improves performance further, achieving 44.6% average top-1 accuracy on 15M data and 57.4% on 30M data.

  7. Knowl 7 — Performance on Vision-Language Downstream Tasks

    data/table

    Evaluation on downstream vision-language tasks including zero-shot image-text retrieval on MSCOCO (5K test set) and fine-tuned visual reasoning on VQAv2 (test-dev) and SNLI-VE (val+test). All models use ViT-B/32 image backbones.

    Text Retrieval Image Retrieval VQA (test-dev) SNLI
    Method Data R@1 R@5 R@10 R@1 R@5 R@10 RSum Y/N Num. Other All Acc
    CLIP 15M 21.4 44.7 56.4 13.7 32.4 42.9 211.5 67.3 30.5 32.7 46.7 62.5
    HiCLIP 15M 34.2 60.3 70.9 20.6 43.8 55.3 285.1 69.4 33.7 37.2 50.1 67.7
    DeCLIP 15M 29.1 55.2 66.6 19.0 41.2 53.1 264.2 70.3 34.9 36.9 50.4 66.1
    HiDeCLIP 15M 38.7 64.4 74.8 23.9 48.2 60.1 310.1 72.4 36.1 40.9 53.3 70.5
    CLIP 30M 34.8 63.3 73.9 23.3 46.9 58.6 300.8 69.7 34.8 37.8 50.6 66.9
    HiCLIP 30M 43.9 69.1 78.8 27.0 51.8 62.9 333.5 72.2 36.1 40.9 53.2 70.1
    DeCLIP 30M 41.3 68.8 79.3 25.6 50.7 62.3 328.0 71.3 35.4 39.7 52.2 69.0
    HiDeCLIP 30M 48.6 74.1 82.7 29.6 54.9 66.3 356.2 73.3 37.0 42.5 54.6 72.5

    HiCLIP outperforms baseline CLIP by 73.6 points in MSCOCO RSum on 15M data (285.1 vs 211.5) and by 32.7 points on 30M data (333.5 vs 300.8). Furthermore, HiCLIP alone outperforms DeCLIP (which incorporates auxiliary self-supervised learning losses) on zero-shot retrieval and visual entailment (SNLI-VE: 67.7% vs 66.1% at 15M, and 70.1% vs 69.0% at 30M), demonstrating that hierarchy modeling directly enhances cross-modal semantic alignment and collaborative multimodal reasoning.

  8. Knowl 8 — Synergy of Group Transformer and Tree Transformer Components

    empirical result

    An ablation isolating the vision branch (Group Transformer, G-Trans) and the language branch (Tree Transformer, T-Trans) on models pretrained on YFCC15M demonstrates that both hierarchical attention mechanisms are complementary:

    ImageNet 11 Datasets Text Retrieval Image Retrieval COCO
    Method Encoder G-Trans T-Trans Acc Avg R@1 R@5 R@10 R@1 R@5 R@10 RSum
    CLIP ViT-B/32 - - 32.8 31.8 21.4 44.7 56.4 13.7 32.4 42.9 211.5
    HiCLIP ViT-B/32 - ✓ 37.1 38.4 28.7 53.8 65.7 17.2 38.5 50.3 254.2
    HiCLIP ViT-B/32 ✓ - 36.2 35.3 22.9 47.5 59.4 14.8 34.2 45.1 223.9
    HiCLIP ViT-B/32 ✓ ✓ 40.5 41.8 34.2 60.3 70.9 20.6 43.8 55.3 285.1
    CLIP ViT-B/16 - - 39.3 35.5 26.1 52.0 64.6 16.5 37.3 48.5 245.0
    HiCLIP ViT-B/16 - ✓ 40.4 39.6 32.8 58.5 69.7 19.6 42.3 54.0 276.9
    HiCLIP ViT-B/16 ✓ - 42.2 37.7 28.5 53.2 65.2 18.1 39.8 51.3 256.1
    HiCLIP ViT-B/16 ✓ ✓ 45.2 44.9 39.0 65.7 76.4 24.0 48.7 60.1 313.9

    Applying either G-Trans or T-Trans alone yields substantial improvements over baseline CLIP, but combining both yields a synergistic gain. For instance, on ViT-B/32, applying T-Trans alone achieves 254.2 COCO RSum and G-Trans alone achieves 223.9 RSum, while enabling both achieves 285.1 RSum (a 73.6-point gain over CLIP's 211.5).

  9. Knowl 9 — Effect of Visual Patch Granularity and Training Data Scale

    empirical result

    Evaluating HiCLIP and HiDeCLIP across different visual encoder patch granularities (ViT-B/32 vs. ViT-B/16) and pretraining dataset sizes (15M vs. 30M) reveals distinct scaling behaviors:

    Method Encoder Data ImageNet Acc 11 Datasets Avg COCO Rsum VQA Acc SNLI Acc
    CLIP ViT-B/32 15M 32.8 31.8 211.5 46.7 62.5
    HiCLIP ViT-B/32 15M 40.5 41.8 285.1 50.1 67.7
    DeCLIP ViT-B/32 15M 43.2 41.3 264.2 50.4 66.1
    HiDeCLIP ViT-B/32 15M 45.9 44.6 310.1 53.3 70.5
    CLIP ViT-B/16 15M 39.3 35.5 245.0 48.8 63.8
    HiCLIP ViT-B/16 15M 45.2 44.9 313.9 51.2 69.0
    DeCLIP ViT-B/16 15M 48.2 43.7 290.3 51.5 67.3
    HiDeCLIP ViT-B/16 15M 51.1 48.3 339.6 54.4 71.3
    CLIP ViT-B/32 30M 48.8 48.4 300.8 50.6 66.9
    HiCLIP ViT-B/32 30M 52.9 53.4 333.5 53.2 70.1
    DeCLIP ViT-B/32 30M 55.3 56.1 328.0 52.2 69.0
    HiDeCLIP ViT-B/32 30M 56.9 57.4 356.2 54.6 72.5
    1. Patch Granularity: Finer patch granularity (ViT-B/16 vs ViT-B/32) consistently boosts the performance of Group Transformer by providing a richer patch graph for local semantic clustering (e.g., HiCLIP 15M COCO Rsum rises from 285.1 to 313.9, and ImageNet accuracy rises from 40.5% to 45.2%).
    2. Data Scale vs. Task Type: As pretraining data scales from 15M to 30M, vision-language tasks (MSCOCO retrieval, VQA, SNLI-VE) show consistent, strong performance improvements across models. For zero-shot visual recognition, while self-supervised objectives in DeCLIP benefit vision-only categorization, HiCLIP scales better on vision-language understanding and reasoning tasks.

  10. Knowl 10 — Limitation of Static Layer-Wise Thresholds in Hierarchy Parsing

    limitation

    Unsupervised hierarchy induction in HiCLIP relies on a top-down greedy splitting heuristic that checks layer-wise neighboring affinity scores ala^l against a fixed, hand-crafted threshold list {θ1,…,θN}\{\theta_1, \dots, \theta_N\} (with higher layers assigned strictly higher threshold values, e.g., reaching 0.9 at top layers).

    This approach has two main limitations:

    1. Lack of Sample Adaptability: The statically chosen threshold sequence is uniform across all inputs, meaning it may not fit images or sentences with complex, irregular, or fine-grained constituent boundaries.
    2. Sensitivity to Hyperparameters: The induced visual segments and syntactic parse trees vary significantly depending on small changes in the threshold values, indicating that a dynamic, data-dependent affinity thresholding or graph segmentation algorithm is needed to parse the attention mask CC reliably across arbitrary inputs.

Coverage note — Linear probe classification results on frozen image features (Appendix Table 7) and qualitative t-SNE feature visualizations were omitted as they reinforce the primary findings captured in the main benchmark tables.

References

  1. 1.Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6077–6086, 2018.
  2. 2.Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425–2433, 2015.
  3. 3.James K Baker. Trainable grammars for speech recognition. The Journal of the Acoustical Society of America, 65(S1):S132–S132, 1979.
  4. 4.Hedi Ben-Younes, Remi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker ´ fusion for visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2612–2620, 2017.
  5. 5.Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In European conference on computer vision, pp. 446–461. Springer, 2014.
  6. 6.Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558–3568, 2021.
  7. 7.Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and ´ C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  8. 8.Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104–120. Springer, 2020.
  9. 9.Yuxiao Chen, Jianbo Yuan, Yu Tian, Shijie Geng, Xinyu Li, Ding Zhou, Dimitris N. Metaxas, and Hongxia Yang. Revisiting multimodal representation in contrastive learning: from patch and token embeddings to finite discrete tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  10. 10.Noam Chomsky. Three models for the description of language. IRE Transactions on information theory, 2(3):113–124, 1956.
  11. 11.Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606–3613, 2014.
  12. 12.Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive language-image pre-training: A clip benchmark of data, model, and supervision. arXiv preprint arXiv:2203.05796, 2022.
  13. 13.Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1601–1610, 2021.
  14. 14.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  15. 15.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  16. 16.Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18166–18176, 2022.
  17. 17.Andrew Drozdov, Patrick Verga, Yi-Pei Chen, Mohit Iyyer, and Andrew McCallum. Unsupervised labeled parsing with deep inside-outside recursive autoencoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1507–1512, 2019.
  18. 18.Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178–178. IEEE, 2004.
  19. 19.Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp. 267–285. Springer, 1982.
  20. 20.Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6639–6648, 2019.
  21. 21.Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021.
  22. 22.Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12976–12985, 2021.
  23. 23.David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology, 195(1):215–243, 1968.
  24. 24.Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10236–10247, 2020.
  25. 25.Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. Advances in neural information processing systems, 31, 2018.
  26. 26.Yoon Kim, Chris Dyer, and Alexander Rush. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2369–2385, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1228. URL https://aclanthology.org/P19-1228.
  27. 27.Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554–561, 2013.
  28. 28.Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32–73, 2017.
  29. 29.Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
  30. 30.Ilya Kuzovkin, Raul Vicente, Mathilde Petton, Jean-Philippe Lachaux, Monica Baciu, Philippe Kahane, Sylvain Rheims, Juan R Vidal, and Jaan Aru. Activations of deep convolutional neural networks are aligned with gamma band activity of human visual cortex. Communications biology, 1(1):1–12, 2018.
  31. 31.Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021.
  32. 32.Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121–137. Springer, 2020.
  33. 33.Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In ICLR, 2022.
  34. 34.Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. ´ Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125, 2017.
  35. 35.Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022, 2021.
  36. 36.Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525–11538, 2020.
  37. 37.Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  38. 38.Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, 2019.
  39. 39.Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  40. 40.Christopher D. Manning. Human Language Understanding & Reasoning. Daedalus, 151(2):127–138, 05 2022. ISSN 0011-5266. doi: 10.1162/daed a 01905. URL https://doi.org/10.1162/daed_a_01905.
  41. 41.Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint arXiv:2112.12750, 2021.
  42. 42.Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722–729. IEEE, 2008.
  43. 43.Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498–3505. IEEE, 2012.
  44. 44.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 2019.
  45. 45.Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.
  46. 46.Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18082–18091, 2022.
  47. 47.Itiroo Sakai. Syntax in universal translation. In Proceedings of the International Conference on Machine Translation and Applied Language Analysis, 1961.
  48. 48.Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565, 2018.
  49. 49.Yikang Shen, Zhouhan Lin, Chin wei Huang, and Aaron Courville. Neural language modeling by jointly learning syntax and lexicon. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rkgOLb-0W.
  50. 50.Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1l6qiR5F7.
  51. 51.Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. Visually grounded neural syntax acquisition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1842–1861, 2019.
  52. 52.Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
  53. 53.Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5100–5111, 2019.
  54. 54.Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016.
  55. 55.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
  56. 56.Bo Wan, Wenjuan Han, Zilong Zheng, and Tinne Tuytelaars. Unsupervised vision-language grammar induction with shared structure modeling. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=N0n_QyQ5lBF.
  57. 57.Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022.
  58. 58.Yau-Shian Wang, Hung-Yi Lee, and Yun-Nung Chen. Tree transformer: Integrating tree structures into self-attention. arXiv preprint arXiv:1909.06639, 2019.
  59. 59.Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485–3492. IEEE, 2010.
  60. 60.Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8392–8401, 2021.
  61. 61.Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706, 2019.
  62. 62.Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
  63. 63.Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18134–18144, 2022.
  64. 64.Qingxiong Yang. Recursive bilateral filtering. In European Conference on Computer Vision, pp. 399–413. Springer, 2012.
  65. 65.Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=cpDhcsEDC2.
  66. 66.Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hatwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means mask transformer. In ECCV, 2022.
  67. 67.Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, and Qianru Sun. Feature pyramid transformer. In European conference on computer vision, pp. 323–339. Springer, 2020.
  68. 68.Ji Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11535–11543, 2019.
  69. 69.Yanpeng Zhao and Ivan Titov. Visually grounded compound pcfgs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4369–4379, 2020.
  70. 70.Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. In BMVC, 2021.
  71. 71.Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16793–16803, 2022.
  72. 72.Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.

Citation

MLA
Geng, S., et al. “HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention”. arXiv, 2023, http://arxiv.org/abs/2303.02995v1.
APA
Geng, S., Yuan, J., Tian, Y., Chen, Y., & Zhang, Y. (2023). HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention. arXiv. http://arxiv.org/abs/2303.02995v1
Chicago
Geng, S., J. Yuan, Y. Tian, Y. Chen, and Y. Zhang. 2023. “HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention”. arXiv. http://arxiv.org/abs/2303.02995v1.
Harvard
Geng, S. et al. (2023) “HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2303.02995v1.
Vancouver
1. Geng S, Yuan J, Tian Y, Chen Y, Zhang Y (2023) HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention. arXiv

BibTeX

@article{geng2023hiclip,
  title = {HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention},
  author = {Geng, Shijie and Yuan, Jianbo and Tian, Yu and Chen, Yuxiao and Zhang, Yongfeng},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2303.02995v1},
  eprint = {2303.02995}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: Authors