Back

Multimodal Transformer for Unaligned Multimodal Language Sequences

Yao-Hung Hubert TsaiShaojie BaiPaul Pu LiangJ. Zico KolterLouis-Philippe MorencyRuslan Salakhutdinov

article2019ACL2,383 citations

Introduces the Multimodal Transformer to model asynchronous language, audio, and visual streams end-to-end via directional crossmodal attention without requiring explicit word-level alignment preprocessing.

Listen

Analyzing human communication requires interpreting multiple interacting data streams simultaneously, including spoken words, facial gestures, and vocal tone. A persistent challenge in automated multimodal analysis is that these different streams operate at varying sampling rates, resulting in unaligned time series where signals do not match on a simple step-by-step basis. Standard practices attempt to force visual and acoustic data into rigid word-level alignment before training, a process that requires extensive domain engineering, requires precise timing metadata that is often unavailable, and fails to capture long-range interactions across time.

The article aims to introduce and evaluate the Multimodal Transformer, an end-to-end deep learning model designed to process unaligned language, video, and audio streams directly. It set out to demonstrate that pairwise crossmodal attention can adapt information across modalities and capture long-range contextual dependencies without manual preprocessing or alignment steps.

To evaluate this framework, the authors conducted empirical experiments across three standard benchmark datasets: CMU-MOSI and CMU-MOSEI for sentiment analysis, and IEMOCAP for emotion recognition. The evaluation tested the model on both traditionally pre-aligned datasets and raw, unaligned sequences where video and audio streams spanned up to more than one thousand time steps. The model was benchmarked against established competitive architectures—such as recurrent fusion networks and cyclic translation models—augmented with temporal classification alignment techniques where necessary, while keeping total parameter counts comparable at around two hundred thousand parameters.

The experimental findings show that the Multimodal Transformer consistently achieves state-of-the-art results across tasks. In the challenging unaligned setting, it outperformed prior competitive methods across most evaluation metrics by 10% to 15%, demonstrating higher classification accuracy and lower prediction errors. In addition, the model achieved state-of-the-art results in traditional word-aligned settings, outperforming existing baselines by 5% to 15% on most metrics. The model also demonstrated faster convergence during training compared to other baselines. Ablation and qualitative analyses confirmed that crossmodal attention successfully learns meaningful correlations between spoken words and visual or acoustic cues, with adaptations targeted toward language features providing the strongest predictive gains.

These results indicate that artificial alignment steps can be eliminated without compromising predictive accuracy. For engineering and product teams, this removes the need for costly, labor-intensive preprocessing pipelines and domain-specific feature engineering. The architecture also reduces deployment complexity by using a unified end-to-end framework that is more resilient to asynchronous data streams in real-world environments.

Organizations developing multimodal applications should consider adopting direct crossmodal attention architectures over rigid word-alignment pipelines, particularly when ingesting asynchronous, real-time data streams. Future efforts should explore applying this architecture to broader domains involving mixed static and time-varying signals, such as visual question answering.

Confidence in these findings is supported by consistent empirical gains across multiple diverse benchmarks and comprehensive ablation testing. However, decision-makers should note that model performance across all evaluated methods still experiences some degradation when shifting from aligned to raw, unaligned sequences, reflecting the inherent difficulty of filtering noise and identifying relevant features across lengthy, asynchronous inputs.

Bosch Center for AICarnegie Mellon Universityattention mechanismempirical analysisencoder-decoderMultimodalCrossmodalLanguage
arXiv: 1906.00295yaohungt/Multimodal-Transformer
  • Paper: Attention Is All You Need, Ashish Vaswani et al. (2017). Reading the foundational Transformer architecture paper first is essential for understanding the self-attention and cross-attention mechanisms adapted in the source model.
  • Paper: Multimodal Machine Learning: A Survey and Taxonomy, Tadas Baltrušaitis et al. (2017). This comprehensive taxonomy of multimodal machine learning provides the necessary background on data alignment challenges that the source paper directly addresses.
Cover for Multimodal Transformer for Unaligned Multimodal Language Sequences

Abstract

Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise crossmodal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.

Topics

attention mechanismempirical analysisencoder-decodernatural languagesfacial expressionacoustic signalshand gesturebilingualdata streamintegrated data

Keywords

MultimodalCrossmodalLanguageAttentionTransformerMulTCrossmodal AttentionSequencesMultimodal LanguageUnaligned

Table of Contents

  • 1 Introduction
  • 2 Related Works
  • 3 Proposed Method
  • 3.1 Crossmodal Attention
  • 3.2 Overall Architecture
  • 3.3 Discussion about Attention & Alignment
  • 4 Experiments
  • 4.1 Datasets and Evaluation Metrics
  • 4.2 Baselines
  • 4.3 Quantitative Analysis
  • 4.4 Qualitative Analysis
  • 5 Discussion
  • References
  • A Positional Embedding
  • B Connectionist Temporal Classification
  • C Hyperparameters
  • D Features

Knowls

  1. Knowl 1 — Multimodal Transformer Architecture

    model/method

    The Multimodal Transformer (MulT) is an end-to-end neural network architecture for modeling unaligned multimodal time-series data across language (LL), video (VV), and audio (AA) modalities without requiring explicit word-level alignment or temporal resampling.

    The architecture operates in four stages:

    1. Temporal Convolution and Positional Embedding: Each modality sequence Xm∈RTm×dmX_m \in \mathbb{R}^{T_m \times d_m} (where m∈{L,V,A}m \in \{L, V, A\}, TmT_m is sequence length, and dmd_m is feature dimension) passes through a 1D temporal convolution layer to project features into a shared dimension dd and aggregate local neighborhood context. Fixed sinusoidal positional embeddings are added to produce position-aware low-level representations Zm[0]∈RTm×dZ_m^{[0]} \in \mathbb{R}^{T_m \times d}.

    2. Pairwise Crossmodal Transformers: For every ordered pair of distinct modalities (β,α)(\beta, \alpha) with α,β∈{L,V,A}\alpha, \beta \in \{L, V, A\}, a dedicated crossmodal transformer β→α\beta \to \alpha of depth DD repeatedly adapts and reinforces the representation of target modality α\alpha using the low-level representation of source modality β\beta. With three modalities, there are 6 directional crossmodal transformers in total: V→LV \to L, A→LA \to L, L→VL \to V, A→VA \to V, L→AL \to A, and V→AV \to A.

    3. Modality Representation Aggregation: For each target modality α\alpha, the outputs from the crossmodal transformers targeting α\alpha are concatenated along the feature dimension:

    Zα=[Zβ1→α[D];Zβ2→α[D]]∈RTα×2dZ_\alpha = \left[ Z_{\beta_1 \to \alpha}^{[D]} ; Z_{\beta_2 \to \alpha}^{[D]} \right] \in \mathbb{R}^{T_\alpha \times 2d}

    where β1,β2\beta_1, \beta_2 are the two remaining modalities.

    1. Sequence Modeling and Prediction: Each aggregated sequence ZαZ_\alpha is processed through a standard self-attention transformer to gather temporal context. The final time-step representations from the sequence models are concatenated and passed through fully-connected layers to produce the task prediction y^\hat{y}.

  2. Knowl 2 — Directional Crossmodal Attention

    equation

    Given a target sequence Xα∈RTα×dαX_\alpha \in \mathbb{R}^{T_\alpha \times d_\alpha} from modality α\alpha and a source sequence Xβ∈RTβ×dβX_\beta \in \mathbb{R}^{T_\beta \times d_\beta} from modality β\beta, where TαT_\alpha and TβT_\beta denote sequence lengths and dαd_\alpha and dβd_\beta denote feature dimensions, directional crossmodal attention computes a latent adaptation CMβ→α(Xα,Xβ)∈RTα×dv\text{CM}_{\beta \to \alpha}(X_\alpha, X_\beta) \in \mathbb{R}^{T_\alpha \times d_v} that expresses modality α\alpha using features from modality β\beta.

    Queries are computed from the target modality α\alpha, while Keys and Values are computed from the source modality β\beta:

    Qα=XαWQα,Kβ=XβWKβ,Vβ=XβWVβQ_\alpha = X_\alpha W_{Q_\alpha}, \quad K_\beta = X_\beta W_{K_\beta}, \quad V_\beta = X_\beta W_{V_\beta}

    where WQα∈Rdα×dkW_{Q_\alpha} \in \mathbb{R}^{d_\alpha \times d_k}, WKβ∈Rdβ×dkW_{K_\beta} \in \mathbb{R}^{d_\beta \times d_k}, and WVβ∈Rdβ×dvW_{V_\beta} \in \mathbb{R}^{d_\beta \times d_v} are learnable projection matrices, and dkd_k is the key/query dimension.

    The single-head crossmodal attention is defined as:

    CMβ→α(Xα,Xβ)=softmax(QαKβ⊤dk)Vβ=softmax(XαWQαWKβ⊤Xβ⊤dk)XβWVβ\text{CM}_{\beta \to \alpha}(X_\alpha, X_\beta) = \text{softmax}\left( \frac{Q_\alpha K_\beta^\top}{\sqrt{d_k}} \right) V_\beta = \text{softmax}\left( \frac{X_\alpha W_{Q_\alpha} W_{K_\beta}^\top X_\beta^\top}{\sqrt{d_k}} \right) X_\beta W_{V_\beta}

    The attention score matrix softmax(QαKβ⊤dk)∈RTα×Tβ\text{softmax}\left( \frac{Q_\alpha K_\beta^\top}{\sqrt{d_k}} \right) \in \mathbb{R}^{T_\alpha \times T_\beta} computes the affinity between the ii-th time step of modality α\alpha and the jj-th time step of modality β\beta. The output has sequence length TαT_\alpha (matching target modality α\alpha) while existing in the feature space of modality β\beta. Multi-head crossmodal attention extends this formulation by computing hh parallel attention heads of dimension d/hd/h and projecting their concatenation.

  3. Knowl 3 — Crossmodal Transformer Layer with Low-Level Source Feature Injection

    model/method

    A crossmodal transformer β→α\beta \to \alpha comprises a stack of DD crossmodal attention blocks. In each block i∈{1,…,D}i \in \{1, \dots, D\}, the intermediate representation of the target modality Zβ→α[i−1]Z_{\beta \to \alpha}^{[i-1]} is updated by attending directly to the low-level initial representation of the source modality Zβ[0]Z_\beta^{[0]} (rather than its intermediate layer outputs Zβ[i−1]Z_\beta^{[i-1]}) and without using self-attention.

    Formally, the forward computation for layers i=1,…,Di = 1, \dots, D proceeds as:

    Zβ→α[0]=Zα[0]Z_{\beta \to \alpha}^{[0]} = Z_\alpha^{[0]}

    Z^β→α[i]=CMβ→α[i],mul(LN(Zβ→α[i−1]),LN(Zβ[0]))+LN(Zβ→α[i−1])\hat{Z}_{\beta \to \alpha}^{[i]} = \text{CM}_{\beta \to \alpha}^{[i], \text{mul}}\left(\text{LN}\left(Z_{\beta \to \alpha}^{[i-1]}\right), \text{LN}\left(Z_\beta^{[0]}\right)\right) + \text{LN}\left(Z_{\beta \to \alpha}^{[i-1]}\right)

    Zβ→α[i]=fθβ→α[i](LN(Z^β→α[i]))+LN(Z^β→α[i])Z_{\beta \to \alpha}^{[i]} = f_{\theta_{\beta \to \alpha}}^{[i]}\left(\text{LN}\left(\hat{Z}_{\beta \to \alpha}^{[i]}\right)\right) + \text{LN}\left(\hat{Z}_{\beta \to \alpha}^{[i]}\right)

    where:

    • LN(⋅)\text{LN}(\cdot) denotes Layer Normalization.
    • CMβ→α[i],mul(Q,KV)\text{CM}_{\beta \to \alpha}^{[i], \text{mul}}(Q, KV) denotes the multi-head crossmodal attention module at layer ii, where Queries are computed from LN(Zβ→α[i−1])\text{LN}(Z_{\beta \to \alpha}^{[i-1]}) and Keys/Values are computed from LN(Zβ[0])\text{LN}(Z_\beta^{[0]}).
    • fθβ→α[i]f_{\theta_{\beta \to \alpha}}^{[i]} is a positionwise feed-forward neural network sublayer with learnable parameters θβ→α[i]\theta_{\beta \to \alpha}^{[i]}.

    Attending directly to the low-level representation Zβ[0]Z_\beta^{[0]} at every layer preserves original modality-specific low-level signals while allowing the target representation to be repeatedly refined.

  4. Knowl 4 — 1D Temporal Convolution and Sinusoidal Positional Encoding for Multimodal Sequences

    model/method

    To handle unaligned multimodal streams with disparate sampling frequencies and feature dimensions, input sequences are preprocessed with 1D temporal convolutions followed by sinusoidal positional embeddings before crossmodal attention.

    For each modality m∈{L,V,A}m \in \{L, V, A\} with raw input sequence Xm∈RTm×dmX_m \in \mathbb{R}^{T_m \times d_m} (where TmT_m is sequence length and dmd_m is raw feature dimension):

    1. 1D Temporal Convolution:

    X^m=Conv1D(Xm,km)∈RTm×d\hat{X}_m = \text{Conv1D}(X_m, k_m) \in \mathbb{R}^{T_m \times d}

    where kmk_m is the 1D convolution kernel size for modality mm (e.g., kL∈{1,3}k_L \in \{1, 3\}, kV=3k_V = 3, kA∈{3,5}k_A \in \{3, 5\}), and dd is a unified hidden dimension common across all modalities (e.g., d=40d=40). The convolution integrates local temporal context around each time step and projects features into a matching dimension suitable for inner-product attention.

    1. Sinusoidal Positional Embedding:

    Zm[0]=X^m+PE(Tm,d)∈RTm×dZ_m^{[0]} = \hat{X}_m + \text{PE}(T_m, d) \in \mathbb{R}^{T_m \times d}

    where PE(Tm,d)\text{PE}(T_m, d) is defined elementwise for time step index i∈{1,…,Tm}i \in \{1, \dots, T_m\} and channel index j∈{0,…,⌊d/2⌋−1}j \in \{0, \dots, \lfloor d/2 \rfloor - 1\} by:

    PE[i,2j]=sin⁡(i100002j/d),PE[i,2j+1]=cos⁡(i100002j/d)\text{PE}[i, 2j] = \sin\left(\frac{i}{10000^{2j / d}}\right), \quad \text{PE}[i, 2j + 1] = \cos\left(\frac{i}{10000^{2j / d}}\right)

    This yields order- and position-aware initial features Zm[0]Z_m^{[0]} across asynchronous time scales.

  5. Knowl 5 — Crossmodal Attention as Generalized Non-Monotonic Alignment

    definition

    In multimodal sequence processing, crossmodal attention generalizes classical forced word-level alignment:

    1. Forced Word-Level Alignment: Manually segments continuous asynchronous modalities (such as video frames and acoustic frames) into discrete time windows defined by word boundary timestamps using tools like P2FA, averaging features within each window. In attention matrix form, forced alignment corresponds to a constrained, piecewise-constant block diagonal (monotonic) matrix.

    2. Crossmodal Attention: Computes a full, unconstrained dense attention score matrix A∈RTα×TβA \in \mathbb{R}^{T_\alpha \times T_\beta} between sequence α\alpha of length TαT_\alpha and sequence β\beta of length TβT_\beta. Crossmodal attention removes the requirement for word boundary metadata or manual feature averaging, allowing elements in modality α\alpha to attend dynamically to non-local, off-diagonal events in modality β\beta (e.g., associating a spoken word with delayed or preceding facial gestures and vocal inflections).

  6. Knowl 6 — Multimodal Sentiment Analysis Results on CMU-MOSI and CMU-MOSEI

    data/table

    The Multimodal Transformer (MulT) was evaluated on the CMU-MOSI and CMU-MOSEI multimodal sentiment analysis benchmarks under both word-aligned and unaligned settings. CMU-MOSI contains 2,199 video monologue clips, and CMU-MOSEI contains 23,454 video clips. Evaluation metrics include 7-class accuracy (Acc7\text{Acc}_7, higher is better), binary accuracy (Acc2\text{Acc}_2, higher is better), F1 score (higher is better), Mean Absolute Error (MAE, lower is better), and Pearson correlation (Corr\text{Corr}, higher is better).

    Model Acc7↑\text{Acc}_7 \uparrow Acc2↑\text{Acc}_2 \uparrow F1↑\text{F1} \uparrow MAE↓\text{MAE} \downarrow Corr↑\text{Corr} \uparrow
    CMU-MOSI (Word Aligned)
    EF-LSTM 33.7 75.3 75.2 1.023 0.608
    LF-LSTM 35.3 76.8 76.7 1.015 0.625
    RMFN 38.3 78.4 78.0 0.922 0.681
    MFM 36.2 78.1 78.1 0.951 0.662
    RAVEN 33.2 78.0 76.6 0.915 0.691
    MCTN 35.6 79.3 79.1 0.909 0.676
    MulT (ours) 40.0 83.0 82.8 0.871 0.698
    CMU-MOSI (Unaligned)
    CTC + EF-LSTM 31.0 73.6 74.5 1.078 0.542
    LF-LSTM 33.7 77.6 77.8 0.988 0.624
    CTC + MCTN 32.7 75.9 76.4 0.991 0.613
    CTC + RAVEN 31.7 72.7 73.1 1.076 0.544
    MulT (ours) 39.1 81.1 81.0 0.889 0.686
    CMU-MOSEI (Word Aligned)
    EF-LSTM 47.4 78.2 77.9 0.642 0.616
    LF-LSTM 48.8 80.6 80.6 0.619 0.659
    Graph-MFN 45.0 76.9 77.0 0.710 0.540
    RAVEN 50.0 79.1 79.5 0.614 0.662
    MCTN 49.6 79.8 80.6 0.609 0.670
    MulT (ours) 51.8 82.5 82.3 0.580 0.703
    CMU-MOSEI (Unaligned)
    CTC + EF-LSTM 46.3 76.1 75.9 0.680 0.585
    LF-LSTM 48.8 77.5 78.2 0.624 0.656
    CTC + RAVEN 45.5 75.4 75.7 0.664 0.599
    CTC + MCTN 48.2 79.3 79.7 0.631 0.645
    MulT (ours) 50.7 81.6 81.6 0.591 0.694

    MulT outperforms baseline methods across all metrics in both aligned and unaligned settings with approximately 200K parameters, maintaining high performance in the unaligned regime where other methods suffer larger degradation.

  7. Knowl 7 — Multimodal Emotion Recognition Results on IEMOCAP

    data/table

    Multimodal emotion recognition performance was evaluated on the IEMOCAP dataset (10,000 video clips) across four emotion categories: Happy, Sad, Angry, and Neutral. Evaluation is performed under both word-aligned and unaligned settings, reporting binary classification accuracy (Acc\text{Acc}, in %) and F1 score (F1\text{F1}, in %) for each class.

    Happy Sad Angry Neutral
    Model Acc↑\text{Acc} \uparrow F1↑\text{F1} \uparrow Acc↑\text{Acc} \uparrow F1↑\text{F1} \uparrow Acc↑\text{Acc} \uparrow F1↑\text{F1} \uparrow Acc↑\text{Acc} \uparrow F1↑\text{F1} \uparrow
    IEMOCAP (Word Aligned)
    EF-LSTM 86.0 84.2 80.2 80.5 85.2 84.5 67.8 67.1
    LF-LSTM 85.1 86.3 78.9 81.7 84.7 83.0 67.1 67.6
    RMFN 87.5 85.8 83.8 82.9 85.1 84.6 69.5 69.1
    MFM 90.2 85.8 88.4 86.1 87.5 86.7 72.1 68.1
    RAVEN 87.3 85.8 83.4 83.1 87.3 86.7 69.7 69.3
    MCTN 84.9 83.1 80.5 79.6 79.7 80.4 62.3 57.0
    MulT (ours) 90.7 88.6 86.7 86.0 87.4 87.0 72.4 70.7
    IEMOCAP (Unaligned)
    CTC + EF-LSTM 76.2 75.7 70.2 70.5 72.7 67.1 58.1 57.4
    LF-LSTM 72.5 71.8 72.9 70.4 68.6 67.9 59.6 56.2
    CTC + RAVEN 77.0 76.8 67.6 65.6 65.0 64.1 62.0 59.5
    CTC + MCTN 80.5 77.5 72.0 71.7 64.9 65.6 49.4 49.3
    MulT (ours) 84.8 81.9 77.7 74.1 73.9 70.2 62.5 59.7

    In the word-aligned setting, MulT achieves the highest scores on Happy, Neutral, and Angry F1. In the unaligned setting, MulT outperforms all prior baseline methods across all four emotion categories by margins of up to 4.3% in accuracy and 4.4% in F1.

  8. Knowl 8 — Ablation Analysis of Multimodal Transformer Components

    data/table

    An ablation study on unaligned CMU-MOSEI evaluates the impact of individual modalities, fusion strategies, directional crossmodal configurations, and source feature levels in MulT. Metrics reported are 7-class accuracy (Acc7\text{Acc}_7), binary accuracy (Acc2\text{Acc}_2), F1 score, Mean Absolute Error (MAE), and Pearson correlation (Corr\text{Corr}).

    Model Variant Acc7↑\text{Acc}_7 \uparrow Acc2↑\text{Acc}_2 \uparrow F1↑\text{F1} \uparrow MAE↓\text{MAE} \downarrow Corr↑\text{Corr} \uparrow
    Unimodal Transformers
    Language only 46.5 77.4 78.2 0.653 0.631
    Audio only 41.4 65.6 68.8 0.764 0.310
    Vision only 43.5 66.4 69.3 0.759 0.343
    Fusion without Crossmodal Attention
    LF-Transformer 47.9 78.6 78.5 0.636 0.658
    EF-Transformer 47.8 78.9 78.8 0.648 0.647
    Multimodal Transformers
    Only [V,A→L][V, A \to L] 50.5 80.1 80.4 0.605 0.670
    Only [L,A→V][L, A \to V] 48.2 79.7 80.2 0.611 0.651
    Only [L,V→A][L, V \to A] 47.5 79.2 79.7 0.620 0.648
    MulT mixing intermediate features 50.3 80.5 80.6 0.602 0.674
    Full MulT (low-level features) 50.7 81.6 81.6 0.591 0.691

    Key findings include:

    1. Language is the strongest single modality (77.4% Acc277.4\% \text{ Acc}_2).
    2. Crossmodal attention modules outperform both Early-Fusion (EF-Transformer) and Late-Fusion (LF-Transformer) baselines.
    3. The crossmodal transformer targeting language ([V,A→L][V, A \to L]) achieves the best performance among single-target sub-networks (80.1% Acc280.1\% \text{ Acc}_2).
    4. Injecting low-level source features (Zβ[0]Z_\beta^{[0]}) at each crossmodal layer outperforms injecting intermediate-level representations (Zβ[i−1]Z_\beta^{[i-1]}) (81.6%81.6\% vs. 80.5% Acc280.5\% \text{ Acc}_2).

  9. Knowl 9 — Connectionist Temporal Classification (CTC) Alignment for Unaligned Baselines

    model/method

    To adapt baseline models designed for word-aligned multimodal inputs (e.g., EF-LSTM, RAVEN, and MCTN) to unaligned multimodal time-series, Connectionist Temporal Classification (CTC) is employed as an auxiliary alignment module.

    The CTC alignment process operates as follows:

    1. Alignment Predictor: A recurrent network (such as an LSTM) processes the unaligned source sequence (e.g., audio [a1,…,aTA][a_1, \dots, a_{T_A}] of length TAT_A) and predicts, for each time step, a probability distribution over the target sequence tokens (e.g., text tokens [w1,…,wTL][w_1, \dots, w_{T_L}]) plus a blank token ϵ\epsilon.

    2. CTC Loss Optimization: The alignment predictor is trained end-to-end to minimize the CTC negative log-likelihood loss, which integrates probabilities over all valid alignment paths mapping the source sequence to the target sequence while allowing token repetitions and blanks.

    3. Pseudo-Aligned Feature Generation: Excluding the blank token probabilities, the probability distributions from the alignment predictor are multiplied by the source sequence features. This projects the source modality into a pseudo-aligned representation with sequence length matching target sequence length TLT_L, which is then fed into the baseline models.

Coverage note — None was omitted; all primary contributions, mathematical formulations, experimental comparisons, ablation studies, and baseline adaptation mechanisms have been captured.

References

  1. 1.Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
  2. 2.Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In International Conference on Learning Representations (ICLR).
  3. 3.Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335.
  4. 4.Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In ACL.
  5. 5.Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2018. Transformer-xl: Language modeling with longer-term dependency.
  6. 6.Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarepa collaborative voice analysis repository for speech technologies. In ICASSP. IEEE.
  7. 7.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  8. 8.Sri Harsha Dumpala, Rupayan Chakraborty, and Sunil Kumar Kopparapu. 2019. Audio-visual fusion for sentiment classification using cross-modal autoencoder. NIPS.
  9. 9.Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200.
  10. 10.Paul Ekman, Wallace V Freisen, and Sonia Ancoli. 1980. Facial signs of emotional experience. Journal of personality and social psychology, 39(6):1125.
  11. 11.Kathleen R Gibson, Kathleen Rita Gibson, and Tim Ingold. 1994. Tools, language and cognition in human evolution. Cambridge University Press.
  12. 12.Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML.
  13. 13.Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
  14. 14.iMotions. 2017. Facial expression analysis.
  15. 15.Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. arXiv preprint arXiv:1501.02598.
  16. 16.Paul Pu Liang, Ziyin Liu, Amir Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. EMNLP.
  17. 17.Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
  18. 18.Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Prismatic Inc, Steven J. Bethard, and David Mcclosky. 2014. The stanford corenlp natural language processing toolkit. In In ACL, System Demonstrations.
  19. 19.Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696.
  20. 20.Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933.
  21. 21.Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  22. 22.Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabas Poczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. AAAI.
  23. 23.Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 873–883.
  24. 24.Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations.
  25. 25.Imran Sheikh, Sri Harsha Dumpala, Rupayan Chakraborty, and Sunil Kumar Kopparapu. 2018. Sentiment analysis using imperfect views from spoken language and acoustic modalities. In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), pages 35–39.
  26. 26.Nitish Srivastava and Ruslan R Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pages 2222–2230.
  27. 27.Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038. Association for Computational Linguistics.
  28. 28.Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? a targeted evaluation of neural machine translation architectures. arXiv preprint arXiv:1808.08946.
  29. 29.Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Learning factorized multimodal representations. ICLR.
  30. 30.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
  31. 31.Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803.
  32. 32.Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. AAAI.
  33. 33.Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online segment to segment neural transduction. arXiv preprint arXiv:1609.08194.
  34. 34.Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America, 123(5):3878.
  35. 35.Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multiview sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
  36. 36.Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82–88.
  37. 37.AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018b. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL.

Citation

MLA
Tsai, Y.-H. H., et al. “Multimodal Transformer for Unaligned Multimodal Language Sequences”. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 6558–69, https://doi.org/10.18653/v1/P19-1656.
APA
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019). Multimodal Transformer for Unaligned Multimodal Language Sequences. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 6558–6569. https://doi.org/10.18653/v1/P19-1656
Chicago
Tsai, Y.-H. H., S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov. 2019. “Multimodal Transformer for Unaligned Multimodal Language Sequences”. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 6558–69. https://doi.org/10.18653/v1/P19-1656.
Harvard
Tsai, Y.-H.H. et al. (2019) “Multimodal Transformer for Unaligned Multimodal Language Sequences”, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 6558–6569. Available at: https://doi.org/10.18653/v1/P19-1656.
Vancouver
1. Tsai Y-HH, Bai S, Liang PP, Kolter JZ, Morency L-P, Salakhutdinov R (2019) Multimodal Transformer for Unaligned Multimodal Language Sequences. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 6558–6569

BibTeX

@inproceedings{Tsai_2019, title={Multimodal Transformer for Unaligned Multimodal Language Sequences}, url={http://dx.doi.org/10.18653/v1/P19-1656}, DOI={10.18653/v1/p19-1656}, booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, publisher={Association for Computational Linguistics}, author={Tsai, Yao-Hung Hubert and Bai, Shaojie and Liang, Paul Pu and Kolter, J. Zico and Morency, Louis-Philippe and Salakhutdinov, Ruslan}, year={2019}, pages={6558–6569} }
Metadata:Crossref

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/