Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

deep canonical correlation

Deep canonical correlation is a representation learning technique that applies deep neural networks to map multiple distinct views or modalities of data into shared, lower-dimensional latent spaces such that their linear correlation is maximized. Extending classical linear and kernel-based canonical correlation analysis, this approach passes each data view through separate deep networks whose parameters are jointly optimized end-to-end to capture complex, non-linear relationships across the inputs. By identifying and aligning the common information shared between heterogeneous sources, deep canonical correlation produces coordinated, modality-invariant representations that facilitate multi-view fusion, cross-modal retrieval, and downstream predictive tasks.

1 item

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Devamanyu Hazarika, Roger Zimmermann, Soujanya Poria

OrganizationsNational University of SingaporeSingapore University of Technology and Design

Why you should read this

Proposes a multimodal representation framework, MISA, that factorizes signals into invariant and modality-specific subspaces, effectively resolving heterogeneous modality gaps to achieve state-of-the-art performance in sentiment analysis and humor detection.

Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.

Added

2026-09-25