BEATs: Audio Pre-Training with Acoustic Tokenizers

Sanyuan ChenYu WuChengyi WangShujie LiuDaniel TompkinsZhuo ChenWanxiang CheXiangzhan YuFuru Wei

article2023ICML549 citations

Proposes an iterative self-supervised pre-training framework that jointly optimizes an acoustic tokenizer and an audio transformer via discrete label prediction, achieving state-of-the-art classification performance on AudioSet-2M and ESC-50 while using fewer parameters and less data than previous reconstruction-based methods.

Listen

Building powerful artificial intelligence models to understand general audio is challenging due to the diverse, unstructured nature of environmental sounds, background noise, and speech. While existing self-supervised audio models typically rely on reconstructing low-level acoustic details—such as raw spectrograms—this approach often retains background noise and fails to extract high-level semantic meaning. The article addresses this limitation by developing an iterative audio pre-training framework called BEATS (Bidirectional Encoder representation from Audio Transformers), which shifts the training objective from reconstructing low-level acoustic details to predicting high-level discrete acoustic tokens.

The framework pairs an acoustic tokenizer with a transformer-based audio encoder in an alternating, iterative training loop. Initially, a random-projection tokenizer creates baseline discrete labels. The audio encoder learns to predict these discrete tokens for masked audio segments. Once trained, the encoder acts as a teacher to guide and refine a new, self-distilled acoustic tokenizer through knowledge distillation. The newly refined tokenizer generates richer semantic labels to train the subsequent audio encoder, and the process repeats. The article pre-trained the framework on roughly two million clips from the AudioSet database and evaluated the resulting models across six standard audio and speech classification benchmarks.

The experimental findings show that BEATS consistently outperforms previous approaches while using fewer resources. First, single BEATS models established new state-of-the-art results on major benchmarks, achieving a 48.6% mean Average Precision (mAP) on the full AudioSet-2M dataset and a 25% relative error reduction on the ESC-50 environmental sound benchmark (reaching 98.1% accuracy). Second, BEATS outperformed previous leading models that required over three times as many parameters (90 million versus 304 million) or extensive pre-training on external visual datasets like ImageNet. Third, ensembling ten fine-tuned BEATS models achieved a record 50.6% mAP on AudioSet-2M without using any external data. Finally, visualization and disturbance analyses confirmed that the acoustic tokenizers successfully ignore background noise and reverberation, isolating meaningful semantic content far better than reconstruction-based methods.

These findings indicate that discrete label prediction enables more semantically focused and computationally efficient audio representation learning. By matching or exceeding larger systems with significantly smaller model architectures, this approach lowers deployment costs and reduces dependence on massive supervised labels or cross-modal pre-training data. Furthermore, using discrete tokens aligns audio pre-training methods with those used in natural language, computer vision, and speech processing, offering a viable pathway toward building unified multimodal foundation models.

Stakeholders and engineering teams should consider adopting the released BEATS pre-trained models and acoustic tokenizers to optimize sound classification workflows, lower labeling overhead, and accelerate downstream application development. For future work, development efforts should focus on scaling the architecture beyond 90 million parameters, expanding pre-training data beyond AudioSet to broader audio domains, and integrating the tokenizer framework into multimodal systems combining audio, text, and vision. While the reported performance gains are statistically robust across multiple standard benchmarks, the main practical trade-off is the linear increase in training compute required during the iterative tokenizer-model optimization cycles.

Harbin Institute of Technology Microsoft Nankai University audio audio classification audio stream acoustic tokenizers audio pre-training BEATs

arXiv: 2212.09058 microsoft/unilm

Paper: BEiT: BERT Pre-Training of Image Transformers, Hangbo Bao et al. (2022). BEiT established the masked prediction pre-training paradigm using discrete visual tokenizers, providing the conceptual foundation for BEATs' acoustic tokenizer and discrete label prediction framework.
Paper: HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, Wei-Ning Hsu et al. (2021). HuBERT introduced iterative self-supervised representation learning via masked prediction of discrete hidden cluster units, which directly motivates the iterative tokenizer and SSL optimization design in BEATs.
Paper: AST: Audio Spectrogram Transformer, Yuan Gong et al. (2021). AST pioneered the use of pure Vision Transformer architectures on spectrogram patches for audio classification, establishing the baseline neural backbone used in BEATs.
Paper: wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, Alexei Baevski et al. (2020). wav2vec 2.0 provides essential background on self-supervised audio pre-training via latent quantization and masked temporal modeling.
Paper: WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing, Sanyuan Chen et al. (2021). WavLM demonstrates large-scale masked self-supervised speech modeling, setting the benchmark for full-stack acoustic pre-training that BEATs generalizes to broader audio classification.
Paper: SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition, Daniel S. Park et al. (2019). SpecAugment introduced time and frequency spectrogram masking techniques that form the core input corruption strategies in BEATs' pre-training and fine-tuning.
Paper: CNN architectures for large-scale audio classification, Shawn Hershey et al. (2016). This paper establishes the large-scale YouTube/AudioSet audio classification setup and evaluation protocol upon which BEATs benchmarks its state-of-the-art representations.

Paper: Contrastive Audio-Visual Masked Autoencoder, Yuan Gong et al. (2023). CAV-MAE extends masked autoencoding and self-supervised acoustic representations into multimodal audio-visual classification and cross-modal retrieval on AudioSet.
Paper: MAViL: Masked Audio-Video Learners, Po-Yao Huang et al. (2023). MAViL builds on self-supervised audio-visual learning by integrating masked representations with iterative student-teacher training across AudioSet benchmarks.
Paper: Pengi: An Audio Language Model for Audio Tasks, Soham Deshmukh et al. (2023). Pengi builds upon general-purpose audio representation models like BEATs to unify audio understanding tasks into a generative audio-language modeling framework.
Paper: AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension, Qian Yang et al. (2024). AIR-Bench provides an extensive downstream benchmark for evaluating advanced audio and audio-language models across diverse perceptual and conversational sound tasks.
Paper: MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark, S. Sakshi et al. (2025). MMAU extends the evaluation of foundational audio understanding representations to complex multi-task reasoning and expert domain comprehension.

Cover for BEATs: Audio Pre-Training with Acoustic Tokenizers

Abstract

We introduce a self-supervised learning (SSL) framework BEATs for general audio representation pre-training, where we optimize an acoustic tokenizer and an audio SSL model by iterations. Unlike the previous audio SSL models that employ reconstruction loss for pre-training, our audio SSL model is trained with the discrete label prediction task, where the labels are generated by a semantic-rich acoustic tokenizer. We propose an iterative pipeline to jointly optimize the tokenizer and the pre-trained model, aiming to abstract high-level semantics and discard the redundant details for audio. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art (SOTA) results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new SOTA mAP 50.6% on AudioSet-2M without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.

Topics

audio audio classification audio stream classification reconstruction semantics contrastive learning acoustic signals

Keywords

acoustic tokenizers audio pre-training BEATs self-distilled tokenizer audio classification AudioSet environmental sound classification audio pattern recognition

1. Introduction
2. Related Work
3. BEATS
3.1. Iterative Audio Pre-training
3.2. Acoustic Tokenizers
3.2.1. COLD START: RANDOM-PROJECTION TOKENIZER
3.2.2. ITERATION: SELF-DISTILLED TOKENIZER
3.3. Audio SSL Model
3.3.1. BACKBONE
3.3.2. PRE-TRAINING
3.3.3. FINE-TUNING
4. Experiment
4.1. Setup
4.2. Comparing with the SOTA Single Models
4.3. Comparing Different BEATS Tokenizers
4.4. Comparing Different Pre-Training Targets via Visualization
4.5. Comparing with the SOTA Ensemble Models
5. Conclusion, Limitations, and Future Work
References
A. Convergence Analysis
A.1. Mathematical Formulation
A.2. Convergence Proof
B. Datasets
C. Hyperparamter Settings

Knowls

Knowl 1 — Iterative Audio Pre-Training Framework (BEATS)
model/method
The BEATS (Bidirectional Encoder representation from Audio Transformers) framework optimizes an acoustic tokenizer and an audio self-supervised learning (SSL) model alternately across multiple iterations:
1. Cold-Start Iteration ( $t=1$ ): A frozen random-projection tokenizer clusters continuous audio patch features into discrete acoustic labels. An audio SSL model is pre-trained to predict these discrete labels on masked audio patches.
2. Subsequent Iterations ( $t \ge 2$ ): A self-distilled acoustic tokenizer is trained via knowledge distillation using the audio SSL model from iteration $t-1$ (which can be either a pre-trained SSL model or a model fine-tuned on downstream labeled data) as the teacher. The newly trained tokenizer then quantizes the unlabeled audio dataset to produce refined discrete semantic labels. A new audio SSL model is trained from scratch using the updated discrete labels as masked prediction targets.
This alternating optimization process allows the acoustic tokenizer to learn high-level semantic abstractions from the SSL teacher while enabling the SSL model to learn rich discrete representations from the improved tokenizer.
Knowl 2 — Self-Distilled Acoustic Tokenizer Architecture and Training Loss
model/method
From the second pre-training iteration of BEATS, a self-distilled acoustic tokenizer converts continuous audio features into semantic discrete tokens using knowledge distillation from an audio SSL teacher model obtained in the previous iteration.

Given an input audio clip represented as a sequence of regular 2D spectrogram patches $X = \{x_t\}_{t=1}^T$ :
1. A 12-layer Transformer encoder maps $X$ to continuous latent representations $E = \{e_t\}_{t=1}^T$ .
2. Each vector $e_t$ is quantized by finding the nearest neighbor in a learnable codebook $V = \{v_i\}_{i=1}^K$ containing $K=1024$ embeddings of dimension 256 using $\ell_2$ -normalized Euclidean distance: $\hat{z}_t = \arg\min_{i \in \{1,\dots,K\}} \|\ell_2(v_i) - \ell_2(e_t)\|_2^2$ The quantized sequence is $E^q = \{v_{\hat{z}_t}\}_{t=1}^T$ . During backpropagation, straight-through gradient estimation copies gradients directly from $E^q$ to $E$ , and the codebook embeddings are updated via exponential moving average (EMA).
3. A 3-layer Transformer estimator takes $E^q$ as input and predicts representation vectors $\{o_t\}_{t=1}^T$ matching the teacher model's final-layer representations $\{\hat{o}_t\}_{t=1}^T$ .
The training objective maximizes cosine similarity to the teacher outputs while minimizing vector quantization error with stop-gradient ( $\text{sg}[\cdot]$ ): $\mathcal{L}_{\text{tokenizer}} = \sum_{X \in \mathcal{D}} \sum_{t=1}^T \cos(o_t, \hat{o}_t) - \|\text{sg}[\ell_2(e_t)] - \ell_2(v_{\hat{z}_t})\|_2^2 - \|\ell_2(e_t) - \text{sg}[\ell_2(v_{\hat{z}_t})]\|_2^2$ where $\mathcal{D}$ is the pre-training dataset and $\cos(\cdot, \cdot)$ is cosine similarity. During inference, the estimator is discarded and the encoder-codebook pair is used to generate patch-level discrete labels $\hat{Z} = \{\hat{z}_t\}_{t=1}^T$ .
Knowl 3 — Masked Audio Modeling Pre-Training for BEATS
model/method
The BEATS audio self-supervised learning (SSL) model uses a 12-layer Vision Transformer (ViT) backbone (768 hidden dimension, 8 attention heads, 90M parameters) equipped with a convolution-based relative position embedding layer, gated relative position bias, and DeepNorm initialization.

The input waveform is sampled at 16 kHz and converted to 128-dimensional Mel-filter bank features (25 ms Povey window, 10 ms hop size, normalized to zero mean and standard deviation 0.5), which are partitioned into non-overlapping $16 \times 16$ time-frequency patches $X = \{x_t\}_{t=1}^T$ .

During Masked Audio Modeling (MAM) pre-training:
1. A random subset $\mathcal{M} \subset \{1, \dots, T\}$ comprising 75% of the patch sequence is masked.
2. Only the unmasked patches $X^U = \{x_t : t \notin \mathcal{M}\}$ are linearly projected and fed into the ViT encoder to produce unmasked representations $R^U = \{r_t : t \notin \mathcal{M}\}$ , reducing computational cost.
3. The unmasked representations are recombined with zero embeddings at the masked positions: $\{r_t : t \notin \mathcal{M}\} \cup \{0 : t \in \mathcal{M}\}$ .
4. A linear label predictor predicts discrete target labels $\hat{Z} = \{\hat{z}_t\}_{t=1}^T$ generated by the acoustic tokenizer.
5. The model is optimized using cross-entropy loss over the masked positions: $\mathcal{L}_{\text{MAM}} = -\sum_{t \in \mathcal{M}} \log p(\hat{z}_t | X^U)$
Knowl 4 — Random-Projection Acoustic Tokenizer for Cold Start
model/method
In the initial pre-training iteration of BEATS ( $t=1$ ), when no trained teacher model is available, a random-projection acoustic tokenizer is used to discretize continuous audio spectrogram patches.

Given an input patch sequence $X = \{x_t\}_{t=1}^T$ , the tokenizer uses a randomly initialized, frozen linear projection matrix $W$ and a frozen codebook of $K=1024$ randomly initialized vectors $V = \{v_i\}_{i=1}^K \subset \mathbb{R}^{256}$ . The patch $x_t$ is linearly projected to $W x_t$ , and the discrete token $\hat{z}_t$ is assigned as the index of the nearest codebook vector: $\hat{z}_t = \arg\min_{i \in \{1,\dots,K\}} \|v_i - W x_t\|_2^2$ This non-learned clustering step provides the cold-start target labels needed to train the first-iteration audio SSL model.
Knowl 5 — Expectation-Maximization Formulation and Convergence of Iterative Pre-training
theoretical result
The alternating pre-training of the acoustic tokenizer parameters $\delta$ and the audio SSL model parameters $\theta$ can be formulated as an Expectation-Maximization (EM) procedure maximizing the observable data log-likelihood $\log p(X | \theta, \delta)$ with latent variables given by the discrete labels $Z$ and SSL hidden representations $R$ .

Under this formulation:
1. Tokenizer optimization (E-step over $R$ , M-step over $\delta$ ): Training the tokenizer at iteration $t+1$ to maximize the joint distribution $p(X, R | \theta^{(t)}, \delta)$ with $R \sim p(R | X, \theta^{(t)})$ guarantees that the likelihood is non-decreasing: $\log p(X | \theta^{(t)}, \delta^{(t+1)}) \ge \log p(X | \theta^{(t)}, \delta^{(t)})$
2. SSL model optimization (E-step over $Z$ , M-step over $\theta$ ): Training the SSL model at iteration $t+1$ to maximize $p(X, Z | \theta, \delta^{(t+1)})$ with $Z \sim p(Z | X, \delta^{(t+1)})$ guarantees: $\log p(X | \theta^{(t+1)}, \delta^{(t+1)}) \ge \log p(X | \theta^{(t)}, \delta^{(t+1)})$
Combining both steps yields the non-decreasing log-likelihood property across iterations: $\log p(X | \theta^{(t+1)}, \delta^{(t+1)}) \ge \log p(X | \theta^{(t)}, \delta^{(t)})$ This ensures convergence of the iterative pre-training framework to a local maximum or saddle point of the data log-likelihood.
Knowl 6 — Downstream Fine-Tuning and Evaluation Methodology
experimental setup
For downstream task adaptation, the pre-trained BEATS label predictor is discarded, and a linear classification head with mean pooling is placed on top of the ViT encoder: $p(C) = \text{Softmax}(\text{MeanPool}(W_c R))$ where $R = \{r_t\}_{t=1}^T$ are the encoded representations of the full patch sequence $X$ (augmented with SpecAugment), and $W_c$ is a task-specific linear projection matrix. Cross-entropy loss is used for single-label tasks, and binary cross-entropy loss is used for multi-label tasks or mixup augmentation.

Evaluation tasks include:
- AudioSet-2M (AS-2M): 2M 10-second YouTube clips, 527 classes (multi-label), evaluated using mean Average Precision (mAP).
- AudioSet-20K (AS-20K): 21K balanced subset of AudioSet, evaluated using mAP.
- ESC-50: 2,000 5-second environmental recordings across 50 classes, evaluated using 5-fold cross-validation accuracy.
- Speech Commands V1 (KS1): 12 classes (10 keyword classes, 1 silence class, 1 unknown class), evaluated by classification accuracy under the SUPERB benchmark split.
- Speech Commands V2 (KS2): 105,829 1-second clips across 35 word classes, evaluated by classification accuracy.
- IEMOCAP (ER): Emotion recognition across 4 emotion classes (~12 hours), evaluated by 5-fold cross-validation accuracy.

Knowl 7 — Audio and Speech Classification Benchmark Performance

data/table

Single-model classification results comparing BEATS models across pre-training iterations with prior supervised and self-supervised models on six audio and speech benchmarks. Evaluation metrics are mean Average Precision (mAP) for AS-2M and AS-20K, and accuracy (%) for ESC-50, KS1, KS2, and ER.

Model	# Param	Pre-train Data	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
Out-of-Domain / In-Domain Supervised Pre-Training
AST	86M	ImageNet	45.9	34.7	88.7	95.5	98.1	56.0
PaSST	86M	ImageNet+AudioSet	47.1	-	96.8	-	-	-
HTS-AT	31M	ImageNet+AudioSet	47.1	-	97.0	-	98.0	-
Audio-MAE (Sup.)	86M	AudioSet	-	-	97.4	-	-	-
Self-Supervised Pre-Training
SS-AST	89M	AudioSet+LibriSpeech	-	31.0	88.8	96.0	98.0	59.6
MSM-MAE	86M	AudioSet	-	-	85.6	-	87.3	-
MaskSpec	86M	AudioSet	47.1	32.3	89.6	-	97.7	-
MAE-AST	86M	AudioSet+LibriSpeech	-	30.6	90.0	95.8	97.9	59.8
data2vec	94M	AudioSet	-	34.5	-	-	-	-
Audio-MAE	86M	AudioSet	47.3	37.1	94.1	96.9	98.3	-
Audio-MAE Large	304M	AudioSet	47.4	37.6	-	-	-	-
CAV-MAE	86M	AudioSet+ImageNet	44.9	34.2	-	-	-	-
BEATS (Ours)
$\text{BEATS}_{\text{iter1}}$	90M	AudioSet	47.9	36.0	94.0	98.0	98.3	65.9
$\text{BEATS}_{\text{iter2}}$	90M	AudioSet	48.1	38.3	95.1	97.7	98.3	66.1
$\text{BEATS}_{\text{iter3}}$	90M	AudioSet	48.0	38.3	95.6	97.7	98.3	64.5
$\text{BEATS}_{\text{iter3+}}$	90M	AudioSet	48.6	38.9	98.1	98.1	98.1	65.0

$\text{BEATS}_{\text{iter1}}$ (trained with the random-projection tokenizer) outperforms prior SSL methods on five tasks. Self-distilled iterations improve performance further, and $\text{BEATS}_{\text{iter3+}}$ (which uses a supervised fine-tuned teacher) achieves a state-of-the-art single-model mAP of 48.6% on AS-2M (outperforming 304M-parameter Audio-MAE Large at 47.4%) and 98.1% accuracy on ESC-50 (with AS-2M pre-training), reducing relative classification error by 25%.

Knowl 8 — AudioSet-2M State-of-the-Art Ensemble Performance
empirical result
Model ensembling of fine-tuned BEATS models on AudioSet-2M (AS-2M) demonstrates state-of-the-art performance without utilizing external supervised pre-training data such as ImageNet:
- BEATS (5 models): An ensemble of five fine-tuned BEATS models achieves 50.4% mAP on AS-2M, outperforming prior best ensemble models (PaSST with ImageNet pre-training at 49.6% mAP, HTS-AT at 48.7% mAP, AST at 48.5% mAP, and PSLA at 47.4% mAP).
- BEATS (10 models): Re-running AS-2M fine-tuning of the five BEATS SSL models with a learning rate of $5 \times 10^{-5}$ for 100k steps and ensembling all ten models yields a new state-of-the-art 50.6% mAP on AS-2M.

Knowl 9 — Ablation of Tokenizer Architectures and Teacher Supervision

data/table

Detailed comparison of different BEATS tokenizer variants and teacher supervision sources across audio classification benchmarks. ESC-50 results are reported without additional supervised pre-training on AudioSet-2M.

Model	Tokenizer Type	Tokenizer Teacher	SSL Data	SL Data	AS-2M (mAP)	AS-20K (mAP)	ESC-50 (Acc)
$\text{BEATS}_{\text{iter1}}$	Random-Projection	N/A	AS	-	47.9	36.0	94.0
$\text{BEATS}_{\text{iter2}}$	Self-Distilled	$\text{BEATS}_{\text{iter1}}$	AS	-	48.1	38.3	95.1
$\text{BEATS}_{\text{iter3}}$	Self-Distilled	$\text{BEATS}_{\text{iter2}}$	AS	-	48.0	38.3	95.6
$\text{BEATS}_{\text{iter3+}}$	Self-Distilled	$\text{BEATS}_{\text{iter2}}$ (fine-tuned on AS-20K)	AS	AS-20K	48.0	38.9	96.2
$\text{BEATS}_{\text{iter3+}}$	Self-Distilled	$\text{BEATS}_{\text{iter2}}$ (fine-tuned on AS-2M)	AS	AS	48.6	41.8*	97.1

*Note: The 41.8% AS-20K score uses AS-2M supervised data during tokenizer pre-training and AS-20K for downstream fine-tuning.

Key takeaways:

Self-distilled tokenizers provide substantial improvements over the random-projection tokenizer, especially on data-scarce setups (AS-20K mAP increases from 36.0% to 38.3%, ESC-50 accuracy increases from 94.0% to 95.1%).
The self-distilled tokenizer performance is robust across iterations of self-supervised teachers ( $\text{BEATS}_{\text{iter1}}$ vs. $\text{BEATS}_{\text{iter2}}$ yield matching 38.3% mAP on AS-20K).
Using a supervised fine-tuned teacher model distills task-specific semantic information into the tokenizer, achieving the highest performance across all tasks (48.6% mAP on AS-2M, 97.1% accuracy on ESC-50 without AS-2M supervised pre-training).

Knowl 10 — Semantic Clustering and Perturbation Robustness of BEATS Token Targets
empirical result
T-SNE 2D visualizations comparing pre-training targets on ESC-50 samples subjected to Room Impulse Response (RIR) reverberation and Deep Noise Suppression (DNS) background noise show distinct differences between reconstruction-based targets and BEATS discrete targets:
- Reconstruction-based SSL targets (continuous acoustic Mel-filter bank features): Highly sensitive to random acoustic variations. Disturbed variants of the same audio sample scatter widely across feature space, while distinct semantic classes overlap closely. This demonstrates that continuous reconstruction objectives focus heavily on low-level time-frequency details and lack high-level semantic abstraction.
- $\text{BEATS}_{\text{iter3}}$ discrete targets (self-supervised teacher distillation): Cluster perturbed versions of audio samples belonging to the same semantic class together, filtering out background noise and reverberation.
- $\text{BEATS}_{\text{iter3+}}$ discrete targets (supervised fine-tuned teacher distillation): Form tight, well-separated clusters corresponding strictly to semantic class identities, ignoring acoustic perturbations entirely and providing robust semantic targets for masked audio modeling.
Knowl 11 — Computational, Data, and Scale Limitations of BEATS
limitation
The BEATS framework has three primary stated limitations:
1. Computational Overhead: The iterative pre-training framework introduces a computational overhead that scales linearly with the number of iteration rounds (pre-training each 90M SSL model requires ~75 hours on 16 Tesla V100 GPUs, and training each self-distilled tokenizer requires ~45 hours on 8 Tesla V100 GPUs).
2. Data Coverage: Pre-training relies solely on AudioSet-2M, which limits data diversity and coverage across broader acoustic, music, and multilingual speech domains.
3. Model Scalability: Model size in this study is constrained to 90M parameters (96M total including heads), which is considerably smaller than modern foundation models in speech, vision, and natural language processing.

Coverage note — None was omitted; all key architectural components, training algorithms, theoretical convergence results, empirical benchmark performances, tokenizer ablation comparisons, target visualizations, and stated limitations are covered.

References

1.Al-Tahan, H. and Mohsenzadeh, Y. Clar: Contrastive learning of auditory representations. In International Conference on Artificial Intelligence and Statistics, pp. 2530–2538. PMLR, 2021.
2.Baade, A., Peng, P., and Harwath, D. Mae-ast: Masked autoencoding audio spectrogram transformer. arXiv preprint arXiv:2203.16691, 2022.
3.Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449–12460, 2020.
4.Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
5.Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pretraining of image transformers. In International Conference on Learning Representations, 2021.
6.Bao, H., Wang, W., Dong, L., and Wei, F. Vl-beit: Generative vision-language pretraining. arXiv preprint arXiv:2206.01127, 2022.
7.Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. S. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359, 2008.
8.Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 646–650. IEEE, 2022a.
9.Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022b.
10.Chi, Z., Huang, S., Dong, L., Ma, S., Zheng, B., Singhal, S., Bajaj, P., Song, X., Mao, X.-L., Huang, H.-Y., et al. Xlm-e: Cross-lingual language model pre-training via electra. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6170–6182, 2022.
11.Chiu, C.-C., Qin, J., Zhang, Y., Yu, J., and Wu, Y. Self-supervised learning with random-projection quantizer for speech recognition. arXiv preprint arXiv:2202.01855, 2022.
12.Chong, D., Wang, H., Zhou, P., and Zeng, Q. Masked spectrogram prediction for self-supervised audio pre-training. arXiv preprint arXiv:2204.12768, 2022.
13.Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
14.Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, 2019.
15.Dieleman, S., van den Oord, A., and Simonyan, K. The challenge of realistic music generation: modelling raw audio at scale. Advances in Neural Information Processing Systems, 31, 2018.
16.Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
17.Elizalde, B., Deshmukh, S., Ismail, M. A., and Wang, H. Clap: Learning audio concepts from natural language supervision. arXiv preprint arXiv:2206.04769, 2022.
18.Fonseca, E., Ortego, D., McGuinness, K., O’Connor, N. E., and Serra, X. Unsupervised contrastive learning of sound event representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 371–375. IEEE, 2021.
19.Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776–780. IEEE, 2017.
20.Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778, 2021a.
21.Gong, Y., Chung, Y.-A., and Glass, J. Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3292–3306, 2021b.
22.Gong, Y., Lai, C.-I., Chung, Y.-A., and Glass, J. Ssast: Self-supervised audio spectrogram transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10699–10709, 2022a.
23.Gong, Y., Rouditchenko, A., Liu, A. H., Harwath, D., Karlinsky, L., Kuehne, H., and Glass, J. Contrastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022b.
24.Guzhov, A., Raue, F., Hees, J., and Dengel, A. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976–980. IEEE, 2022.
25.Harb, H. and Chen, L. A general audio classifier based on human perception motivated model. Multimedia Tools and Applications, 34:375–395, 2007.
26.He, K., Chen, X., Xie, S., Li, Y., Dollar, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009, 2022.
27.Hinton, G., Vinyals, O., Dean, J., et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.
28.Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
29.Kong, Q., Cao, Y., Iqbal, T., Wang, Y., Wang, W., and Plumbley, M. D. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020.
30.Koutini, K., Schluter, J., Eghbal-zadeh, H., and Widmer, G. Efficient training of audio transformers with patchout. arXiv preprint arXiv:2110.05069, 2021.
31.Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.
32.Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
33.Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
34.Ma, K. W., Wong, H. M., and Mak, C. M. A systematic review of human perceptual dimensions of sound: Meta-analysis of semantic differential method applications to indoor and outdoor sounds. Building and Environment, 133:123–150, 2018.
35.Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., and Sun, C. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34:14200–14213, 2021.
36.Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., and Kashino, K. Byol for audio: Self-supervised learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE, 2021.
37.Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., and Kashino, K. Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation. arXiv preprint arXiv:2204.12260, 2022.
38.Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206–5210. IEEE, 2015.
39.Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779, 2019.
40.Patterson, K., Nestor, P. J., and Rogers, T. T. Where do you know what you know? the representation of semantic knowledge in the human brain. Nature reviews neuroscience, 8(12):976–987, 2007.
41.Peng, Z., Dong, L., Bao, H., Ye, Q., and Wei, F. BEiT v2: Masked image modeling with vector-quantized visual tokenizers. 2022.
42.Piczak, K. J. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015–1018, 2015.
43.Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.
44.Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821–8831. PMLR, 2021.
45.Ravanelli, M. and Bengio, Y. Learning speaker representations with mutual information. arXiv preprint arXiv:1812.00271, 2018.
46.Reddy, C. K., Dubey, H., Koishida, K., Nair, A., Gopal, V., Cutler, R., Braun, S., Gamper, H., Aichner, R., and Srinivasan, S. Interspeech 2021 deep noise suppression challenge. arXiv preprint arXiv:2101.01902, 2021.
47.Saeed, A., Grangier, D., and Zeghidour, N. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3875–3879. IEEE, 2021.
48.Schneider, S., Baevski, A., Collobert, R., and Auli, M. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019.
49.Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
50.Tagliasacchi, M., Gfeller, B., de Chaumont Quitry, F., and Roblek, D. Pre-training audio representations with self-supervision. IEEE Signal Processing Letters, 27:600–604, 2020.
51.Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
52.Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
53.Verbitskiy, S., Berikov, V., and Vyshegorodtsev, V. Eranns: Efficient residual audio neural networks for audio pattern recognition. Pattern Recognition Letters, 161:38–44, 2022.
54.Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., and Wei, F. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555, 2022a.
55.Wang, L. and Oord, A. v. d. Multi-format contrastive learning of audio representations. arXiv preprint arXiv:2103.06508, 2021.
56.Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022b.
57.Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.
58.wen Yang, S., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., Huang, T.-H., Tseng, W.-C., tik Lee, K., Liu, D.-R., Huang, Z., Dong, S., Li, S.-W., Watanabe, S., Mohamed, A., and yi Lee, H. SUPERB: Speech Processing Universal PERformance Benchmark. In Proc. Interspeech 2021, pp. 1194–1198, 2021. doi: 10.21437/Interspeech.2021-1775.
59.Wu, H.-H., Seetharaman, P., Kumar, K., and Bello, J. P. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4563–4567. IEEE, 2022.
60.Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., Feichtenhofer, C., et al. Masked autoencoders that listen. arXiv preprint arXiv:2207.06405, 2022.
61.Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021.
62.Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
63.Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., et al. Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition. IEEE Journal of Selected Topics in Signal Processing, 16(6):1519–1532, 2022.

Citation

MLA

Chen, S., et al. “BEATs: Audio Pre-Training with Acoustic Tokenizers”. International Conference on Machine Learning, vol. 202, 2023, pp. 5178–93, https://proceedings.mlr.press/v202/chen23ag.html.

APA

Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., Che, W., Yu, X., & Wei, F. (2023). BEATs: Audio Pre-Training with Acoustic Tokenizers. International Conference on Machine Learning, 202, 5178–5193. https://proceedings.mlr.press/v202/chen23ag.html

Chicago

Chen, S., Y. Wu, C. Wang, et al. 2023. “BEATs: Audio Pre-Training with Acoustic Tokenizers”. International Conference on Machine Learning 202: 5178–93. https://proceedings.mlr.press/v202/chen23ag.html.

Harvard

Chen, S. et al. (2023) “BEATs: Audio Pre-Training with Acoustic Tokenizers”, International Conference on Machine Learning. PMLR, pp. 5178–5193. Available at: https://proceedings.mlr.press/v202/chen23ag.html.

Vancouver

1. Chen S, Wu Y, Wang C, Liu S, Tompkins D, Chen Z, Che W, Yu X, Wei F (2023) BEATs: Audio Pre-Training with Acoustic Tokenizers. In: International Conference on Machine Learning. PMLR, pp 5178–5193

BibTeX

@InProceedings{pmlr-v202-chen23ag,
  title = 	 {{BEAT}s: Audio Pre-Training with Acoustic Tokenizers},
  author =       {Chen, Sanyuan and Wu, Yu and Wang, Chengyi and Liu, Shujie and Tompkins, Daniel and Chen, Zhuo and Che, Wanxiang and Yu, Xiangzhan and Wei, Furu},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {5178--5193},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/chen23ag/chen23ag.pdf},
  url = 	 {https://proceedings.mlr.press/v202/chen23ag.html},
  abstract = 	 {We introduce a self-supervised learning (SSL) framework BEATs for general audio representation pre-training, where we optimize an acoustic tokenizer and an audio SSL model by iterations. Unlike the previous audio SSL models that employ reconstruction loss for pre-training, our audio SSL model is trained with the discrete label prediction task, where the labels are generated by a semantic-rich acoustic tokenizer. We propose an iterative pipeline to jointly optimize the tokenizer and the pre-trained model, aiming to abstract high-level semantics and discard the redundant details for audio. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art (SOTA) results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new SOTA mAP 50.6% on AudioSet-2M without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

BEATs: Audio Pre-Training with Acoustic Tokenizers

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Iterative Audio Pre-Training Framework (BEATS)

Knowl 2 — Self-Distilled Acoustic Tokenizer Architecture and Training Loss

Knowl 3 — Masked Audio Modeling Pre-Training for BEATS

Knowl 4 — Random-Projection Acoustic Tokenizer for Cold Start

Knowl 5 — Expectation-Maximization Formulation and Convergence of Iterative Pre-training

Knowl 6 — Downstream Fine-Tuning and Evaluation Methodology