Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

speech feature extraction

Speech feature extraction is the computational process of converting raw acoustic speech signals into structured numerical representations that capture essential phonetic, prosodic, linguistic, or emotional properties. By transforming continuous, high-dimensional audio waveforms into compact feature vectors or learned neural embeddings, this process filters out redundant noise and reduces data complexity while preserving salient characteristics of the utterance. These derived features can represent a wide range of attributes, including spectral energy distributions, fundamental frequency, phonetic content, speaker identity, and affective state. As a foundational stage in speech processing pipelines, speech feature extraction supplies the informative representations required for downstream machine learning tasks such as automatic speech recognition, speaker verification, speech synthesis, and emotion recognition.

1 item

SECap: Speech Emotion Captioning with Large Language Model

SECap: Speech Emotion Captioning with Large Language Model

Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi-Xiong Zhang, Guangzhi Li, Yi Luo, Rongzhi Gu

OrganizationsTencentThe Chinese University of Hong KongTsinghua University

Why you should read this

Proposes SECap, a framework that integrates HuBERT audio representations with LLaMA through a Q-Former to generate rich natural language descriptions of complex speech emotions rather than relying on discrete classification labels.

Speech emotions are crucial in human communication and are extensively used in fields like speech synthesis and natural language understanding. Most prior studies, such as speech emotion recognition, have categorized speech emotions into a fixed set of classes. Yet, emotions expressed in human speech are often complex, and categorizing them into predefined groups can be insufficient to adequately represent speech emotions. On the contrary, describing speech emotions directly by means of natural language may be a more effective approach. Regrettably, there are not many studies available that have focused on this direction. Therefore, this paper proposes a speech emotion captioning framework named SECap, aiming at effectively describing speech emotions using natural language. Owing to the impressive capabilities of large language models in language comprehension and text generation, SECap employs LLaMA as the text decoder to allow the production of coherent speech emotion captions. In addition, SECap leverages HuBERT as the audio encoder to extract general speech features and Q-Former as the Bridge-Net to provide LLaMA with emotion-related speech features. To accomplish this, Q-Former utilizes mutual information learning to disentangle emotion-related speech features and speech contents, while implementing contrastive learning to extract more emotion-related speech features. The results of objective and subjective evaluations demonstrate that: 1) the SECap framework outperforms the HTSAT-BART baseline in all objective evaluations; 2) SECap can generate high-quality speech emotion captions that attain performance on par with human annotators in subjective mean opinion score tests.

Added

2026-09-26