Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

joint embeddings

Joint embeddings are machine learning representations where data from multiple distinct modalities or domains, such as text, images, and audio, are mapped into a single, shared mathematical vector space. In this unified space, semantically related or paired inputs from different sources are positioned close to one another, allowing geometric distance metrics to measure relationships across diverse types of data. By aligning disparate data types within a common coordinate system, joint embeddings enable models to perform cross-modal tasks such as searching for images using audio or text queries, transferring knowledge between modalities, and facilitating zero-shot recognition and multimodal analysis.

1 item

ImageBind One Embedding Space to Bind Them All

ImageBind One Embedding Space to Bind Them All

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra

OrganizationsMeta

Why you should read this

Demonstrates that aligning various modalities (audio, depth, thermal) to images automatically aligns them to each other, creating a universal embedding space.

Part of:Must-read papers in multimodal AI

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications 'out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.

Added

2026-01-28