keyword
modality-invariant representations
Modality-invariant representations are machine learning feature embeddings that encode shared semantic information across different data types or sensory channels, such as text, audio, images, and video, independent of the specific format used to express that information. In multimodal artificial intelligence systems, data originating from heterogeneous sources inherently differs in structure and feature distributions, creating a discrepancy known as the modality gap. To overcome this, models map inputs from distinct modalities into a shared latent space where semantically equivalent concepts or events occupy nearby coordinates regardless of whether they were captured visually, acoustically, or textually. By capturing commonalities and discarding modality-specific superficial differences, these representations enable models to perform cross-modal retrieval, align disparate signals, and fuse heterogeneous information for downstream prediction tasks.
2 items
Cross-Modal Discrete Representation Learning
Alexander H. Liu, SouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, James R. Glass
Why you should read this
Presents a self-supervised framework that uses vector quantization and code matching across modalities to learn fine-grained discrete representations, enabling unsupervised concept localization and boosting retrieval performance.
In contrast to recent advances focusing on high-level representation learning across modalities, in this work we present a self-supervised learning framework that is able to learn a representation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities. Beyond the shared embedding space, we propose a Cross-Modal Code Matching objective that forces the representations from different views (modalities) to have a similar distribution over the discrete embedding space such that cross-modal objects/actions localization can be performed without direct supervision. We show that the proposed discretized multi-modal fine-grained representation (e.g., pixel/word/frame) can complement high-level summary representations (e.g., video/sentence/waveform) for improved performance on cross-modal retrieval tasks. We also observe that the discretized representation uses individual clusters to represent the same semantic concept across modalities.
Added
2026-09-26
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Devamanyu Hazarika, Roger Zimmermann, Soujanya Poria
Why you should read this
Proposes a multimodal representation framework, MISA, that factorizes signals into invariant and modality-specific subspaces, effectively resolving heterogeneous modality gaps to achieve state-of-the-art performance in sentiment analysis and humor detection.
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Added
2026-09-25