Back

Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

Pete Warden

article2018arXiv2,088 citations

Introduces the Speech Commands benchmark dataset alongside standardized evaluation metrics and baseline models to advance limited-vocabulary speech recognition and on-device keyword spotting.

audioautomatic speech recognitionspeech signalsSpeechSpeech CommandsDataset
arXiv: 1804.03209
Cover for Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

Abstract

Describes an audio dataset of spoken words designed to help train and evaluate keyword spotting systems. Discusses why this task is an interesting challenge, and why it requires a specialized dataset that is different from conventional datasets used for automatic speech recognition of full sentences. Suggests a methodology for reproducible and comparable accuracy metrics for this task. Describes how the data was collected and verified, what it contains, previous versions and properties. Concludes by reporting baseline results of models trained on this dataset.

Topics

audioautomatic speech recognitionspeech signalsvocabularyspeech synthesisspeech recognition systems

Keywords

SpeechSpeech CommandsDatasetRecognitionDataSpeech RecognitionAudioModelsTestAvailable

Table of Contents

  • 1 Abstract
  • 2 Introduction
  • 3 Related Work
  • 4 Motivations
  • 5 Collection
  • 5.1 Requirements
  • 5.2 Word Choice
  • 5.3 Implementation
  • 5.4 Quality Control
  • 5.5 Extract Loudest Section
  • 5.6 Manual Review
  • 5.7 Release Process
  • 5.8 Background Noise
  • 6 Properties
  • 7 Evaluation
  • 7.1 Top-One Error
  • 7.2 Streaming Error Metrics
  • 7.3 Historical Evaluations
  • 7.4 Applications
  • 8 Conclusion
  • 9 Acknowledgements
  • References

Knowls

  1. Knowl 1 — Speech Commands Dataset Overview and Specification

    definition

    The Speech Commands dataset is an open-access audio dataset designed for training and evaluating keyword spotting and limited-vocabulary speech recognition systems on short spoken words under real-world acoustic conditions.

    Key properties of Version 2 of the dataset include:

    • Total Utterances: 105,829 audio files across 35 word categories.
    • Number of Speakers: 2,618 distinct speakers, each identified by an anonymized 8-digit hexadecimal identifier.
    • Audio Format: 16-bit linear PCM single-channel WAV format sampled at 16 kHz16\text{ kHz}.
    • Clip Duration: 1.0 second or less per file.
    • Storage: Approximately 3.8 GB uncompressed (2.7 GB as a gzip-compressed tar archive).
    • Licensing: Released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
    • Background Noise: Includes dedicated multi-minute ambient noise recordings and mathematically generated white and pink noise files stored in a _background_noise_ directory.

  2. Knowl 2 — Speech Commands Vocabulary and Per-Word Utterance Distribution

    data/table

    The Speech Commands dataset (version 2) consists of 35 words organized into core command words, digits, and auxiliary/distractor words designed to test discrimination against phonetically similar non-target speech. The utterance counts per word category are:

    Word Number of Utterances
    Backward 1,664
    Bed 2,014
    Bird 2,064
    Cat 2,031
    Dog 2,128
    Down 3,917
    Eight 3,787
    Five 4,052
    Follow 1,579
    Forward 1,557
    Four 3,728
    Go 3,880
    Happy 2,054
    House 2,113
    Learn 1,575
    Left 3,801
    Marvin 2,100
    Nine 3,934
    No 3,941
    Off 3,745
    On 3,845
    One 3,890
    Right 3,778
    Seven 3,998
    Sheila 2,022
    Six 3,860
    Stop 3,872
    Three 3,727
    Tree 1,759
    Two 3,880
    Up 3,723
    Visual 1,592
    Wow 2,123
    Yes 4,044
    Zero 4,052
    Total 105,829

    The core vocabulary contains 20 words (digits 0 to 9 plus "Yes", "No", "Up", "Down", "Left", "Right", "On", "Off", "Stop", "Go"), supplemented by 4 robotics/IoT command words added in version 2 ("Backward", "Forward", "Follow", "Learn"). The remaining auxiliary words provide phonetically diverse confounders (e.g., "Tree" sounds similar to "Three").

  3. Knowl 3 — Audio Collection and Curation Pipeline

    model/method

    The data collection and verification pipeline for the Speech Commands dataset is designed to capture realistic, speaker-independent speech across diverse acoustic environments using consumer hardware:

    1. Audio Capture: A web application using the WebAudio API records audio via phone or laptop microphones. Speakers are prompted with randomly ordered target words, each displayed for 1.5 seconds followed by a 1.0-second pause. Core vocabulary words are prompted 5 times and auxiliary words 1 time, yielding 135 total recordings per contributor session (~6 minutes).
    2. Technical File Filtering: Raw recordings returned as compressed OGG files with a file size smaller than 5 KB are automatically discarded as near-silent or empty.
    3. Format Normalization: Surviving OGG files are decoded and resampled to single-channel 16-bit linear PCM WAV files at 16 kHz16\text{ kHz}.
    4. Energy Filtering and Trimming: An automated volume extraction tool computes the mean absolute sample amplitude. Clips with mean amplitude below 0.0040.004 (on a [−1.0,1.0][-1.0, 1.0] floating-point scale) are eliminated. The highest-energy 1.0-second continuous window is extracted to center the spoken word.
    5. Manual Verification via Crowdsourcing: Each 1.0-second audio clip is submitted to a crowdsourced worker who transcribes the heard word. If the transcription does not match the prompt label, the clip is discarded.
    6. Anonymization and Noise Augmentation: Contributor session IDs are hashed into 8-digit hexadecimal strings without storing demographic or personally identifiable information. Background noise files (ambient environment recordings, synthetic white noise, and synthetic pink noise) are added to support noise-robustness training and silence modeling.

  4. Knowl 4 — Extract Loudest Section Alignment and Silence Removal Algorithm

    algorithm

    To remove near-silent utterances and automatically align spoken words within unsegmented 1.5 s1.5\text{ s} audio recordings without requiring phonetic forced alignment, the Extract Loudest Section procedure computes sample amplitudes and extracts the 1.0-second window containing the maximum integrated volume.

    Input: Discrete audio signal x=[x1,x2,…,xN]x = [x_1, x_2, \dots, x_N] sampled at fs=16000 Hzf_s = 16000\text{ Hz}, threshold θ=0.004\theta = 0.004, window length L=16000L = 16000 (corresponding to 1.0 s1.0\text{ s})
    Output: Trimmed 1.0-second audio clip yy of length LL, or REJECT
    Convert 16-bit integer samples xi∈[−32768,32767]x_i \in [-32768, 32767] to normalized floats si∈[−1.0,1.0]s_i \in [-1.0, 1.0]:
    for i←1i \leftarrow 1 to NN do
        si←xi/32768.0s_i \leftarrow x_i / 32768.0
    end for
    Compute the overall mean absolute volume:
    vˉ←1N∑i=1N∣si∣\bar{v} \leftarrow \frac{1}{N} \sum_{i=1}^N |s_i|
    if vˉ<θ\bar{v} < \theta then
        return REJECT (clip is too quiet or silent)
    end if
    Find the starting sample index k∗k^* that maximizes the volume over a window of length LL:
    k∗←arg⁡max⁡1≤k≤N−L+1∑j=0L−1∣sk+j∣k^* \leftarrow \arg\max_{1 \le k \le N - L + 1} \sum_{j=0}^{L-1} |s_{k + j}|
    Extract the 1.0-second window:
    y←[xk∗,xk∗+1,…,xk∗+L−1]y \leftarrow [x_{k^*}, x_{k^* + 1}, \dots, x_{k^* + L - 1}]
    return yy

  5. Knowl 5 — Deterministic Hash-Based Dataset Partitioning

    model/method

    To ensure reproducible and leak-free benchmarking across successive dataset releases, partitioning of audio files into training, validation, and test subsets is computed deterministically from file paths/names using a stable hash function rather than random sampling.

    The download distribution provides two predefined lists:

    • validation_list.txt: audio files allocated for validation during training and hyperparameter tuning.
    • testing_list.txt: audio files strictly reserved for final model evaluation.
    • All remaining audio files not present in either list comprise the training set.

    Because the subset assignment is a deterministic hash of the file path, existing utterances remain consistently assigned to their respective partitions when new utterances or speakers are added in subsequent versions, preventing cross-contamination between training and testing splits across dataset versions.

  6. Knowl 6 — Standard 12-Class Keyword Spotting Evaluation Benchmark

    experimental setup

    The standard evaluation benchmark for the Speech Commands dataset evaluates models on a 12-class closed/open-world classification task:

    • Target Categories (10 classes): "Yes", "No", "Up", "Down", "Left", "Right", "On", "Off", "Stop", and "Go".
    • Unknown Word (1 class): Utterances randomly sampled from the remaining 25 non-target word classes in the dataset.
    • Silence (1 class): 1.0-second clips randomly sampled from the background noise audio files in _background_noise_.

    The canonical evaluation set is class-balanced, providing an equal number of test examples for each of the 12 categories (each category comprising approximately 8.33%8.33\% of the evaluation data). Model performance is measured by Top-One accuracy (the proportion of test audio clips where the model's highest-probability predicted category matches the ground truth label).

  7. Knowl 7 — Continuous Audio Streaming Evaluation Methodology and Metrics

    experimental setup

    To evaluate keyword spotting under operational streaming conditions where word boundaries are unknown and background noise is continuous, models are evaluated against continuous audio streams (e.g., a 10-minute or 1-hour audio file containing randomly placed keyword utterances interspersed with silence and background noise).

    The streaming evaluation protocol employs the following metrics:

    • Matched Percentage: The proportion of ground-truth spoken keyword occurrences detected within a specified time tolerance window (default is 750 ms750\text{ ms}).
    • Correct Percentage: The proportion of matched detections where the predicted keyword class matches the ground-truth label.
    • Wrong Percentage: The proportion of speech events correctly distinguished from background noise but assigned an incorrect keyword class label.
    • False-Positive Percentage: The number of keyword detections triggered during audio segments where no target speech was present.

    To prevent multiple false triggers from a single utterance, a command recognition logic suppresses repeated recognitions within a short sliding time window.

  8. Knowl 8 — Baseline Top-One Classification Accuracy Across Dataset Versions

    empirical result

    A baseline convolutional neural network (CNN) for small-footprint keyword spotting was trained and evaluated on Version 1 (64,727 utterances, 1,881 speakers) and Version 2 (105,829 utterances, 2,618 speakers) of the Speech Commands dataset. The Top-One classification accuracies on the standard 12-class benchmark are:

    Test Set V1 Training V2 Training
    V1 Test 85.4% 89.7%
    V2 Test 82.7% 88.2%

    Training on the larger Version 2 dataset improves Top-One accuracy by 4.3%4.3\% on the V1 test set (from 85.4%85.4\% to 89.7%89.7\%) and by 5.5%5.5\% on the V2 test set (from 82.7%82.7\% to 88.2%88.2\%), demonstrating the impact of increased speaker and utterance diversity.

  9. Knowl 9 — Baseline Streaming Keyword Spotting Performance

    empirical result

    When evaluating the standard baseline convolutional neural network trained on Version 2 of the Speech Commands dataset against a continuous 10-minute streaming test audio file containing background noise and randomly timed utterances (with a match time tolerance of 750 ms750\text{ ms}), the model achieves:

    • Matched: 49.0%49.0\%
    • Correctly identified: 46.0%46.0\%
    • Wrong class label: 3.0%3.0\%
    • False positives: 0.0%0.0\%

    This indicates that while the model achieves 0.0%0.0\% false alarms in background noise segments, continuous unaligned recognition matches only 49.0%49.0\% of spoken target keywords, reflecting the gap between isolated 1-second Top-One classification (88.2%88.2\%) and streaming deployment.

Coverage note — Omitted literature review of related speech datasets (Common Voice, LibriSpeech, TIDIGITS, CHiME-5) and summary of third-party downstream research papers that utilized Version 1 of the dataset, as these do not constitute the paper's novel contributions.

References

  1. 1.(2018) Speech commands dataset version 2. [Online]. Available: http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
  2. 2.(2017) Speech commands dataset version 1. [Online]. Available: http://download.tensorflow.org/data/speech_commands_v0.01.tar.gz
  3. 3.(2018) Linguistic data consortium. [Online]. Available: https://www.ldc.upenn.edu/
  4. 4.J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR09, 2009.
  5. 5.(2018) Creative commons international attribution international 4.0 license. [Online]. Available: https://creativecommons.org/licenses/by/4.0/
  6. 6.(2017) Mozilla common voice. [Online]. Available: https://voice.mozilla.org/en
  7. 7.V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015.
  8. 8.R. G. Leonard and G. R. Doddington. (1992) A speaker-independent connected-digit database. [Online]. Available: https://catalog.ldc.upenn.edu/docs/LDC93S10/tidigits.readme.html
  9. 9.(2018) The 5th chime speech separation and recognition challenge. [Online]. Available: http://spandh.dcs.shef.ac.uk/chime_challenge/data.html
  10. 10.(2017) Hey siri: An on-device dnn-powered voice trigger for apple’s personal assistant. [Online]. Available: https://machinelearning.apple.com/2017/10/01/hey-siri.html
  11. 11.(2015) Web audio api. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API
  12. 12.(2018) Implementation of set assignment algorithm. [Online]. Available: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/input_data.py#L61
  13. 13.(2018) Speech commands dataset test set version 2. [Online]. Available: http://download.tensorflow.org/data/speech_commands_test_set_v0.02.tar.gz
  14. 14.(2017) Speech commands dataset test set version 1. [Online]. Available: http://download.tensorflow.org/data/speech_commands_test_set_v0.01.tar.gz
  15. 15.(2017) Tensorflow audio recognition tutorial. [Online]. Available: https://www.tensorflow.org/tutorials/audio_recognition
  16. 16.(2018) test_streaming_accuracy.cc source file. [Online]. Available: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/test_streaming_accuracy.cc
  17. 17.(2018) recognize_commands.cc source file. [Online]. Available: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/recognize_commands.cc
  18. 18.(2018) Speech commands dataset streaming test version 2. [Online]. Available: http://download.tensorflow.org/data/speech_commands_streaming_test_v0.02.tar.gz
  19. 19.T. N. Sainath and C. Parada, “Convolutional Neural Networks for Small-Footprint Keyword Spotting,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2015/papers/i15_1478.pdf
  20. 20.(2018) Speech commands tutorial checkpoints. [Online]. Available: https://storage.googleapis.com/download.tensorflow.org/models/speech_commands_checkpoints.tar.gz
  21. 21.L. Lai, N. Suda, and V. Chandra, “ CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs,” ArXiv e-prints, Jan. 2018.
  22. 22.B. McMahan and D. Rao, “Listening to the World Improves Speech Command Recognition,” ArXiv e-prints, Oct. 2017.
  23. 23.J. Salamon, C. Jacoby, and J. P. Bello, “ A dataset and taxonomy for urban sound research,” in Proceedings of the 22Nd ACM International Conference on Multimedia, ser. MM ’14. New York, NY, USA: ACM, 2014, pp. 1041–1044. [Online]. Available: http://doi.acm.org/10.1145/2647868.2655045
  24. 24.M. Alzantot, B. Balaji, and M. Srivastava, “ Did you hear that? Adversarial Examples Against Automatic Speech Recognition,” ArXiv e-prints, Jan. 2018.
  25. 25.R. Tang and J. Lin, “ Deep Residual Learning for Small-Footprint Keyword Spotting,” ArXiv e-prints, Oct. 2017.
  26. 26.J. Lee, T. Kim, J. Park, and J. Nam, “ Raw Waveform-based Audio Classification Using Sample-level CNN Architectures,” ArXiv e-prints, Dec. 2017.
  27. 27.S. Krishna Gouda, S. Kanetkar, D. Harrison, and M. K. Warmuth, “ Speech Recognition: Keyword Spotting Through Image Recognition,” ArXiv e-prints, Mar. 2018.

Citation

MLA
Warden, P. “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition”. arXiv, 2018, http://arxiv.org/abs/1804.03209v1.
APA
Warden, P. (2018). Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. arXiv. http://arxiv.org/abs/1804.03209v1
Chicago
Warden, P. 2018. “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition”. arXiv. http://arxiv.org/abs/1804.03209v1.
Harvard
Warden, P. (2018) “Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1804.03209v1.
Vancouver
1. Warden P (2018) Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. arXiv

BibTeX

@article{warden2018speech,
  title = {Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition},
  author = {Warden, Pete},
  year = {2018},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1804.03209v1},
  eprint = {1804.03209}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/