Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

audio-visual speech recognition

Audio-visual speech recognition is a multimodal technology that automatically transcribes spoken language into text by combining auditory signals with visual cues, most notably the movement of a speaker's lips and face. By supplementing acoustic data with visual information, which is unaffected by background noise or acoustic distortion, these systems resolve phonetic ambiguities and achieve greater transcription accuracy than traditional audio-only approaches. Modern systems typically use multimodal deep learning architectures to align and fuse features extracted from both audio waveforms and video streams, emulating the human ability to integrate sound and sight for speech comprehension in noisy or complex listening environments.

1 item

Multimodal Machine Learning: A Survey and Taxonomy

Multimodal Machine Learning: A Survey and Taxonomy

Tadas Baltrušaitis, Chaitanya Ahuja, Louis-Philippe Morency

OrganizationsCarnegie Mellon University

Why you should read this

Establishes a comprehensive taxonomy for multimodal machine learning by structuring the field around five fundamental technical challenges: representation, translation, alignment, fusion, and co-learning.

Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.

Added

2026-09-10