Listen, Attend and Spell

William ChanNavdeep JaitlyQuoc V. LeOriol Vinyals

article2015ICASSP2,475 citations

Introduces an end-to-end speech recognition framework that directly transcribes audio spectra into character sequences using a pyramidal recurrent encoder and an attention-based decoder, removing the need for pronunciation models and conditional independence assumptions.

Listen

The article addresses the challenge of building speech recognition systems that avoid the complexity and separate training of traditional acoustic, pronunciation, and language models, which often rely on hidden Markov models and conditional independence assumptions. These limitations reduce accuracy on varied speech and make handling out-of-vocabulary words or alternative spellings difficult. Recent end-to-end approaches like connectionist temporal classification have improved integration but still fall short on sequence dependencies.

The work evaluates whether a single neural network can directly map audio filter bank features to character sequences without those assumptions or external dictionaries. The Listen, Attend and Spell model uses a pyramidal bidirectional LSTM encoder to compress long audio inputs into higher-level features and an attention-based LSTM decoder to generate characters one at a time, trained jointly on roughly three million Google voice search utterances.

On a held-out test set the model reached 14.1 percent word error rate without a language model and 10.3 percent after rescoring the top beams; the comparable state-of-the-art CLDNN-HMM system achieved 8.0 percent. The network automatically produced multiple valid spellings for the same audio and correctly repeated words, behaviors that are harder for prior end-to-end methods.

These results show that end-to-end attention models can approach conventional performance while eliminating hand-crafted components and pronunciation lexicons. The approach therefore simplifies deployment and may improve robustness on rare or noisy utterances, though it remains a few points behind the hybrid baseline.

Further gains appear possible by adding convolutional layers, location-based attention to handle longer utterances, and stronger language model integration during decoding. The main limitations are higher error rates on very short or very long utterances, greater variance on infrequent words, and evaluation on a single English voice-search domain; readers should treat the reported margins as indicative rather than definitive until broader testing confirms them.

Carnegie Mellon University Google filter banks hmm models language model speech recognition attention model Connectionist Temporal Classification

arXiv: 1508.01211

Paper: Neural Machine Translation by Jointly Learning to Align and Translate, Dzmitry Bahdanau et al. (2015). Introduces the attention-based sequence-to-sequence framework that Listen, Attend and Spell adapts to map acoustic filter bank sequences directly to character transcriptions.
Paper: Attention-Based Models for Speech Recognition, Jan Chorowski et al. (2015). Provides the foundational demonstration of applying attention mechanisms to speech features, which Listen, Attend and Spell scales to end-to-end character decoding.
Paper: Sequence to Sequence Learning with Neural Networks, Ilya Sutskever et al. (2014). Establishes the core encoder-decoder LSTM paradigm for variable-length sequence transduction that serves as the baseline architecture for speech recognition in LAS.
Paper: Towards End-To-End Speech Recognition with Recurrent Neural Networks, Alex Graves et al. (2014). Demonstrates early end-to-end recurrent neural network speech recognition without intermediate phonetic representations, highlighting the sequence-modeling limitations that attention in LAS resolves.
Paper: Speech Recognition with Deep Recurrent Neural Networks, Alex Graves et al. (2013). Shows the effectiveness of deep bidirectional LSTM networks for acoustic modeling and sequence transcription, foundational to the pyramidal encoder used in LAS.
Paper: Deep Speech: Scaling up end-to-end speech recognition, Awni Y. Hannun et al. (2014). Presents a scaled-up end-to-end deep learning framework for speech recognition, providing the large-scale pipeline benchmark that LAS improves upon by removing alignment assumptions.
Paper: Effective Approaches to Attention-based Neural Machine Translation, Minh-Thang Luong et al. (2015). Explores global and local attention architectures in sequence transduction that directly inform the attention scoring and alignment mechanisms used in LAS.
Paper: Long Short-Term Memory, Sepp Hochreiter et al. (1997). Introduces the Long Short-Term Memory cell architecture that forms the fundamental recurrent building blocks of both the encoder and decoder in LAS.

Paper: Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, Dario Amodei et al. (2016). Scales end-to-end deep neural speech recognition to multi-language domains with high-performance computing, advancing beyond earlier single-domain neural recognizers.
Paper: Tacotron: Towards End-to-End Speech Synthesis, Yuxuan Wang et al. (2017). Applies the sequence-to-sequence attention architecture pioneered in speech recognition in reverse to generate speech spectrograms directly from text.
Paper: Conformer: Convolution-augmented Transformer for Speech Recognition, Anmol Gulati et al. (2020). Combines self-attention with convolutional layers to enhance acoustic modeling efficiency and error rates over earlier recurrent attention models like LAS.
Paper: Attention Is All You Need, Ashish Vaswani et al. (2017). Replaces recurrent encoder-decoder structures with pure multi-head self-attention, significantly improving training parallelism for sequence-to-sequence tasks.
Paper: Robust Speech Recognition via Large-Scale Weak Supervision, Alec Radford et al. (2023). Demonstrates modern large-scale weakly supervised sequence-to-sequence speech recognition that generalizes robustly across diverse acoustic domains and languages.
Paper: wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, Alexei Baevski et al. (2020). Advances speech recognition beyond fully supervised end-to-end models by learning rich acoustic representations through self-supervised pre-training on unlabeled audio.
Paper: Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions, Jonathan Shen et al. (2017). Builds upon attention-driven spectrogram generation and neural vocoding to produce human-level synthetic speech directly from character inputs.
Paper: HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, Wei-Ning Hsu et al. (2021). Extends self-supervised acoustic representation learning using masked prediction of hidden discrete units for downstream speech recognition.

Abstract

We present Listen, Attend and Spell (LAS), a neural network that learns to transcribe speech utterances to characters. Unlike traditional DNN-HMM models, this model learns all the components of a speech recognizer jointly. Our system has two components: a listener and a speller. The listener is a pyramidal recurrent network encoder that accepts filter bank spectra as inputs. The speller is an attention-based recurrent network decoder that emits characters as outputs. The network produces character sequences without making any independence assumptions between the characters. This is the key improvement of LAS over previous end-to-end CTC models. On a subset of the Google voice search task, LAS achieves a word error rate (WER) of 14.1% without a dictionary or a language model, and 10.3% with language model rescoring over the top 32 beams. By comparison, the state-of-the-art CLDNN-HMM model achieves a WER of 8.0%.

Topics

filter banks hmm models language model speech recognition systems vocabulary word error rate recurrent networks search tasks

Keywords

speech recognition attention model Connectionist Temporal Classification decoder RNN Neural Machine Translation Clockwork RNN voice search task language model sequence learning

1 Introduction
2 Related Work
3 Model
3.1 Listen
3.2 Attend and Spell
3.3 Learning
3.4 Decoding and Rescoring
4 Experiments
4.1 Attention Visualization
4.2 Effects of Beam Width
4.3 Effects of Utterance Length
4.4 Word Frequency
4.5 Interesting Decoding Examples
5 Conclusions
References
A Alignment Examples

Knowls

Knowl 1 — Listen, Attend and Spell (LAS) Architecture
model/method
Listen, Attend and Spell (LAS) is an end-to-end neural network architecture that maps an acoustic feature sequence directly to a sequence of characters without intermediate phonetic representations, pronunciation dictionaries, or conditional independence assumptions between output tokens.

Given an input sequence of acoustic filter bank spectra $\mathbf{x} = (x_1, \ldots, x_T)$ , the model defines the conditional probability of an output character sequence $\mathbf{y} = (\langle\text{sos}\rangle, y_1, \ldots, y_S, \langle\text{eos}\rangle)$ via the chain rule:

$P(\mathbf{y} \mid \mathbf{x}) = \prod_{i=1}^{S+1} P(y_i \mid \mathbf{x}, y_{<i})$

where each character $y_i \in \mathcal{V} = \{a, b, \ldots, z, 0, \ldots, 9, \langle\text{space}\rangle, \langle\text{comma}\rangle, \langle\text{period}\rangle, \langle\text{apostrophe}\rangle, \langle\text{unk}\rangle\}$ , and $\langle\text{sos}\rangle$ and $\langle\text{eos}\rangle$ denote start-of-sentence and end-of-sentence tokens, respectively. LAS consists of two jointly trained sub-modules:
1. The Listener: an acoustic encoder function $\mathbf{h} = \text{Listen}(\mathbf{x})$ that compresses the acoustic input sequence into a sequence of higher-level feature vectors $\mathbf{h} = (h_1, \ldots, h_U)$ where $U \le T$ .
2. The Speller: an attention-based character decoder function $P(\mathbf{y} \mid \mathbf{x}) = \text{AttendAndSpell}(\mathbf{h}, \mathbf{y})$ that computes categorical probability distributions over characters conditioned on the encoder representations and historical output characters.
Knowl 2 — Pyramidal Bidirectional LSTM Listener
model/method
In acoustic speech processing, input utterances often contain hundreds to thousands of acoustic frames. Standard Bidirectional Long Short-Term Memory (BLSTM) encoders produce output sequences that are too long for attention mechanisms to efficiently query and converge upon. To address this and reduce the attention mechanism complexity from $O(TS)$ to $O(US)$ , the Listener component uses a stacked pyramidal BLSTM (pBLSTM).

In standard deep BLSTM layers, the hidden state at layer $j$ and time $i$ is:

$h_i^j = \text{BLSTM}(h_{i-1}^j, h_i^{j-1})$

In contrast, each pBLSTM layer downsamples time by concatenating the hidden representations from two successive time steps of the underlying layer before processing:

$h_i^j = \text{pBLSTM}\left(h_{i-1}^j, [h_{2i}^{j-1}, h_{2i+1}^{j-1}]\right)$

In the LAS model, 3 pBLSTM layers (each containing 512 nodes, with 256 units per direction) are stacked on top of a bottom standard BLSTM layer. This reduces the time resolution by $2^3 = 8$ times ( $U = T/8$ ) while enabling the network to learn nonlinear acoustic feature representations.
Knowl 3 — Speller Attention Mechanism and Character Probability Formulation
equation
The Speller decoder in Listen, Attend and Spell is an attention-based recurrent neural network (a 2-layer LSTM with 512 nodes per layer) that computes the conditional character distribution at each decoding step $i$ using a content-based attention mechanism over listener representations $\mathbf{h} = (h_1, \ldots, h_U)$ .

The decoder computation at time step $i$ is defined by the following system of equations:

$s_i = \text{RNN}(s_{i-1}, y_{i-1}, c_{i-1})$

$e_{i, u} = \langle \phi(s_i), \psi(h_u) \rangle$

$\alpha_{i, u} = \frac{\exp(e_{i, u})}{\sum_{u'=1}^U \exp(e_{i, u'})}$

$c_i = \sum_{u=1}^U \alpha_{i, u} h_u$

$P(y_i \mid \mathbf{x}, y_{<i}) = \text{Softmax}(\text{MLP}(s_i, c_i))$

where:
- $s_i$ is the decoder RNN hidden state at output step $i$ .
- $y_{i-1}$ is the character token emitted at step $i-1$ .
- $c_i$ is the context vector representing the blend of acoustic features relevant to emitting $y_i$ .
- $e_{i, u}$ is the scalar attention energy between decoder state $s_i$ and encoder state $h_u$ , computed via the dot product of multi-layer perceptron projections $\phi(s_i)$ and $\psi(h_u)$ .
- $\alpha_{i, u}$ is the softmax alignment weight across encoder time frames $u \in \{1, \ldots, U\}$ .
- $\text{Softmax}(\text{MLP}(s_i, c_i))$ denotes a multi-layer perceptron with softmax output over the character vocabulary $\mathcal{V}$ .
Knowl 4 — Training with Character Distribution Sampling
model/method
Standard teacher-forced sequence-to-sequence training optimizes the log-likelihood of target characters conditioned strictly on ground-truth prior characters:

$\max_{\theta} \sum_i \log P(y_i \mid \mathbf{x}, y_{<i}^*; \theta)$

where $y_{<i}^*$ are the true historical tokens. During inference, ground-truth history is unavailable, and early mistakes can cause cascading errors (exposure bias).

To bridge this train-test discrepancy, LAS applies a sampling technique during training where the previous token input $\tilde{y}_{i-1}$ fed into the decoder RNN is sampled from the model's predicted character distribution with a fixed probability:

$\tilde{y}_i \sim \text{CharacterDistribution}(s_i, c_i)$

$\max_{\theta} \sum_i \log P(y_i \mid \mathbf{x}, \tilde{y}_{<i}; \theta)$

where $\tilde{y}_{i-1}$ is either the true target or a sample from the model's distribution. Rather than using an annealing schedule, a constant sampling rate of $10\%$ is maintained throughout training.
Knowl 5 — Length-Normalized Beam Search Decoding with Language Model Rescoring
algorithm
Inference in Listen, Attend and Spell finds the most likely character sequence $\hat{\mathbf{y}}$ given the acoustic input $\mathbf{x}$ using left-to-right beam search of width $\beta$ . Because character sequence models exhibit a slight length bias toward shorter transcripts, hypotheses are scored by normalizing the sequence log-probability by character count $|\mathbf{y}|_c$ and linearly combining it with an external language model score $P_{\text{LM}}(\mathbf{y})$ :

$s(\mathbf{y} \mid \mathbf{x}) = \frac{\log P(\mathbf{y} \mid \mathbf{x})}{|\mathbf{y}|_c} + \lambda \log P_{\text{LM}}(\mathbf{y})$

where $\lambda$ is the language model weight tuned on a validation set.
Input: Acoustic feature sequence $\mathbf{x}$ , beam width $\beta$ , language model weight $\lambda$ , external language model $P_{\text{LM}}$
Output: Most likely rescored character sequence $\hat{\mathbf{y}}$
$\mathbf{h} \leftarrow \text{Listen}(\mathbf{x})$
Initialize beam $B \leftarrow [(\langle\text{sos}\rangle, \text{score}=0.0)]$
Initialize completed hypotheses $C \leftarrow []$
while $B$ is not empty and max length not reached:
     $B_{\text{next}} \leftarrow []$
    for each candidate $(\mathbf{y}_{\text{prefix}}, \text{score})$ in $B$ :
        if last token of $\mathbf{y}_{\text{prefix}}$ is $\langle\text{eos}\rangle$ :
            Add $(\mathbf{y}_{\text{prefix}}, \text{score})$ to $C$
            continue
        Compute $P(y \mid \mathbf{x}, \mathbf{y}_{\text{prefix}})$ via $\text{AttendAndSpell}(\mathbf{h}, \mathbf{y}_{\text{prefix}})$
        for each character token $y \in \mathcal{V} \cup \{\langle\text{eos}\rangle\}$ :
             $\mathbf{y}_{\text{new}} \leftarrow \mathbf{y}_{\text{prefix}} \circ y$
             $\text{score}_{\text{new}} \leftarrow \text{score} + \log P(y \mid \mathbf{x}, \mathbf{y}_{\text{prefix}})$
            Add $(\mathbf{y}_{\text{new}}, \text{score}_{\text{new}})$ to $B_{\text{next}}$
     $B \leftarrow \text{top } \beta \text{ candidates in } B_{\text{next}} \text{ sorted by } \text{score}_{\text{new}}$
for each completed hypothesis $\mathbf{y}$ in $C$ :
    Compute final score $s(\mathbf{y} \mid \mathbf{x}) = \frac{\log P(\mathbf{y} \mid \mathbf{x})}{|\mathbf{y}|_c} + \lambda \log P_{\text{LM}}(\mathbf{y})$
return $\hat{\mathbf{y}} = \arg\max_{\mathbf{y} \in C} s(\mathbf{y} \mid \mathbf{x})$

Knowl 6 — Word Error Rate Comparison on Google Voice Search

data/table

The Listen, Attend and Spell (LAS) model was evaluated on approximately 3 million Google voice search utterances (2000 hours), augmented 20-fold with room impulse and environmental noise simulations. Testing was performed on a 22K-utterance (~16 hours) clean test set and a matching noisy test set. All LAS models were decoded using beam search with beam width $\beta = 32$ . Language model (LM) rescoring applied an external $n$ -gram LM with weight $\lambda = 0.008$ .

Standard LAS trained without sampling or LM rescoring achieves 16.2% WER on clean speech. Adding 10% sampling during training improves unrescored WER to 14.1%. When combined with language model rescoring over the top 32 beams, LAS achieves 10.3% WER on clean speech (2.3% absolute gap from the state-of-the-art CLDNN-HMM) and 12.0% on noisy speech (3.1% absolute gap from CLDNN-HMM). Constraining the search space with a dictionary during decoding yielded no improvement.

Knowl 7 — Beam Width Scaling and Oracle Word Error Rates in LAS
empirical result
Evaluation of the decode beam width $\beta$ during inference on the clean Google voice search dataset demonstrates the following properties:
- Without a language model or dictionary, increasing beam width $\beta$ from 1 to 16 reduces the Word Error Rate (WER) monotonically from ~16.5% to 14.1%. Increasing $\beta$ further to 32 yields no additional WER reduction.
- With language model rescoring, WER improves with beam width up to $\beta=32$ , reaching 10.3%.
- Oracle WER (evaluating the minimum WER among hypotheses in the top $\beta$ beams) drops sharply with beam size: at $\beta=16$ , oracle WER is ~5.0%, and at $\beta=32$ , oracle WER reaches 4.3% on clean speech and 5.5% on noisy speech.
The large margin between top-1 WER (10.3%) and oracle WER (4.3%) at $\beta=32$ indicates that correct hypotheses are generated within the beam candidates, but acoustic and language model scoring functions are insufficient to rank them first.
Knowl 8 — Utterance Length Dependency and Error Modes in LAS
empirical result
Analysis of recognition performance as a function of the number of words per utterance (from 1 to 25+ words) identifies distinct error characteristics across length regimes:
- Short utterances (1–2 words): Despite high frequency in the training data, short utterances show elevated error rates (~18%–22% WER). Substitutions and insertions predominate, caused by the character decoder over-segmenting words.
- Moderate-length utterances (3–15 words): The model attains its lowest WER (~5%–10%), corresponding to the bulk of the training distribution.
- Long utterances ( $\ge 16$ words, and especially $\ge 20$ words): Error rates increase sharply, exceeding 30% WER at 18 words and reaching ~80% WER for utterances with 25 or more words. Deletion errors dominate this regime, indicating that the attention mechanism fails to sustain tracking over long sequences and terminates decoding prematurely.
Knowl 9 — End-to-End Learning of Multi-Spelling Variants Without Pronunciation Dictionaries
empirical result
Because Listen, Attend and Spell models output sequences directly at the character level using the exact chain rule $P(\mathbf{y}\mid\mathbf{x}) = \prod_i P(y_i \mid \mathbf{x}, y_{<i})$ , the decoder learns an implicit character language model capable of producing multiple valid spelling variants for the same acoustic input without a pronunciation dictionary or grapheme-to-phoneme component.

For the spoken acoustic input corresponding to "call triple a roadside assistance" (where "aaa" appeared only once and "triple a" appeared 4 times in the 2000-hour training corpus), unconstrained beam search produced both representations in its top candidates:
- Beam 1: "call aaa roadside assistance" ( $\log P = -0.5740$ , 0.00% WER)
- Beam 2: "call triple a roadside assistance" ( $\log P = -1.5399$ , 50.00% WER)
- Beam 3: "call trip way roadside assistance" ( $\log P = -3.5012$ , 50.00% WER)
- Beam 4: "call xxx roadside assistance" ( $\log P = -4.4375$ , 25.00% WER)
Similarly, the model generated both "st" and "saint" for the spoken input "st mary's animal clinic". CTC models struggle to generate such diverse transcripts due to conditional independence assumptions across frames, and hybrid DNN-HMM systems require explicit lexicon entries for each variation.
Knowl 10 — Relationship Between Training Frequency, Acoustic Uniqueness, and Word Recall
empirical result
An evaluation of word recall (whether a word appears anywhere in the decoded transcript) relative to word frequency in the training set demonstrates that performance is governed by both linguistic exposure and acoustic uniqueness:
- High-frequency words with phonetically ambiguous acoustics can have lower recall: the word "and" occurred ~85,000 times in training but achieved only an 80% recall rate on the test set, frequently mis-transcribed as "in" (which achieved 95% recall).
- Low-frequency words with distinctive acoustic profiles can achieve perfect recall: the rare word "walkerville" occurred only once in the training distribution but achieved 100% recall on the test set.
- Across the test set, rare words show significantly higher recall variance than frequent words.

Coverage note — Acoustic encoder pre-training experiments using frame-level phoneme targets and multi-task loss functions were deliberately omitted as they yielded no performance improvements over training from scratch.

References

1.Nathaniel Morgan and Herve Bourlard. Continuous Speech Recognition using Multilayer Perceptrons with Hidden Markov Models. In IEEE International Conference on Acoustics, Speech and Signal Processing, 1990.
2.Abdel-rahman Mohamed, George E. Dahl, and Geoffrey E. Hinton. Deep belief networks for phone recognition. In Neural Information Processing Systems: Workshop on Deep Learning for Speech Recognition and Related Applications, 2009.
3.George E. Dahl, Dong Yu, Li Deng, and Alex Acero. Large vocabulary continuous speech recognition with context-dependent dbn-hmms. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2011.
4.Abdel-rahman Mohamed, George E. Dahl, and Geoffrey Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):14–22, 2012.
5.Navdeep Jaitly, Patrick Nguyen, Andrew W. Senior, and Vincent Vanhoucke. Application of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition. In INTERSPEECH, 2012.
6.Tara Sainath, Abdel-rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep Convolutional Neural Networks for LVCSR. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2013.
7.Kanishka Rao, Fuchun Peng, Hasim Sak, and Francoise Beaufays. Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2015.
8.Kaisheng Yao and Geoffrey Zweig. Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion. 2015.
9.Tomas Mikolov, Karafiat Martin, Burget Luka, Eernocky Jan, and Khudanpur Sanjeev. Recurrent neural network based language model. In INTERSPEECH, 2010.
10.Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmiduber. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In International Conference on Machine Learning, 2006.
11.Alex Graves. Sequence Transduction with Recurrent Neural Networks. In International Conference on Machine Learning: Representation Learning Workshop, 2012.
12.Alex Graves and Navdeep Jaitly. Towards End-to-End Speech Recognition with Recurrent Neural Networks. In International Conference on Machine Learning, 2014.
13.Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Ng. Deep Speech: Scaling up end-to-end speech recognition. In http://arxiv.org/abs/1412.5567, 2014.
14.Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results. In Neural Information Processing Systems: Workshop Deep Learning and Representation Learning Workshop, 2014.
15.Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-Based Models for Speech Recognition. In http://arxiv.org/abs/1506.07503, 2015.
16.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations, 2015.
17.Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to Sequence Learning with Neural Networks. In Neural Information Processing Systems, 2014.
18.Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwen, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing, 2014.
19.Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In http://arxiv.org/abs/1506.03099, 2015.
20.Tara N. Sainath, Oriol Vinyals, Andrew Senior, and Hasim Sak. Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2015.
21.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Neural Information Processing Systems, 2012.
22.Leonard E. Baum and Ted Petrie. Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics, 37:1554–1563, 1966.
23.John Lafferty, Andrew McCallum, and Fernando Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In International Conference on Machine Learning, 2001.
24.Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the Rare Word Problem in Neural Machine Translation. In Association for Computational Linguistics, 2015.
25.Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On Using Very Large Target Vocabulary for Neural Machine Translation. In Association for Computational Linguistics, 2015.
26.Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and Tell: A Neural Image Caption Generator. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
27.Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In International Conference on Machine Learning, 2015.
28.Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey E. Hinton. Grammar as a foreign language. In http://arxiv.org/abs/1412.7449, 2014.
29.Oriol Vinyals and Quoc V. Le. A Neural Conversational Model. In International Conference on Machine Learning: Deep Learning Workshop, 2015.
30.Sepp Hochreiter and Jurgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, November 1997.
31.Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. Hybrid Speech Recognition with Bidirectional LSTM. In Automatic Speech Recognition and Understanding Workshop, 2013.
32.Salah Hihi and Yoshua Bengio. Hierarchical Recurrent Neural Networks for Long-Term Dependencies. In Neural Information Processing Systems, 1996.
33.Jan Koutnik, Klaus Greff, Faustino Gomez, and Jurgen Schmidhuber. A Clockwork RNN. In International Conference on Machine Learning, 2014.
34.Navdeep Jaitly, Vincent Vanhoucke, and Geoffrey Hinton. Autoregressive product of multi-frame predictions can improve the accuracy of hybrid models. In INTERSPEECH, 2014.
35.Hasim Sak, Andrew Senior, Kanishka Rao, and Francoise Beaufays. Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition. In INTERSPEECH, 2015.
36.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Neural Information Processing Systems, 2013.
37.Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannenmann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. The Kaldi Speech Recognition Toolkit. In Automatic Speech Recognition and Understanding Workshop, 2011.
38.Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large Scale Distributed Deep Networks. In Neural Information Processing Systems, 2012.

Citation

MLA

Chan, W., et al. “Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition”. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 4960–64, https://doi.org/10.1109/icassp.2016.7472621.

APA

Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016). Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4960–4964. https://doi.org/10.1109/icassp.2016.7472621

Chicago

Chan, W., N. Jaitly, Q. Le, and O. Vinyals. 2016. “Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition”. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4960–64. https://doi.org/10.1109/icassp.2016.7472621.

Harvard

Chan, W. et al. (2016) “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition”, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 4960–4964. Available at: https://doi.org/10.1109/icassp.2016.7472621.

Vancouver

1. Chan W, Jaitly N, Le Q, Vinyals O (2016) Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 4960–4964

BibTeX

@inproceedings{Chan_2016, title={Listen, attend and spell: A neural network for large vocabulary conversational speech recognition}, url={http://dx.doi.org/10.1109/icassp.2016.7472621}, DOI={10.1109/icassp.2016.7472621}, booktitle={2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, publisher={IEEE}, author={Chan, William and Jaitly, Navdeep and Le, Quoc and Vinyals, Oriol}, year={2016}, month=Mar, pages={4960–4964} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Model	Clean WER (%)	Noisy WER (%)
CLDNN-HMM	8.0	8.9
LAS	16.2	19.0
LAS + LM Rescoring	12.6	14.7
LAS + Sampling	14.1	16.5
LAS + Sampling + LM Rescoring	10.3	12.0

Listen, Attend and Spell

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Listen, Attend and Spell (LAS) Architecture

Knowl 2 — Pyramidal Bidirectional LSTM Listener

Knowl 3 — Speller Attention Mechanism and Character Probability Formulation

Knowl 4 — Training with Character Distribution Sampling

Knowl 5 — Length-Normalized Beam Search Decoding with Language Model Rescoring

Knowl 6 — Word Error Rate Comparison on Google Voice Search

Knowl 7 — Beam Width Scaling and Oracle Word Error Rates in LAS

Knowl 8 — Utterance Length Dependency and Error Modes in LAS

Knowl 9 — End-to-End Learning of Multi-Spelling Variants Without Pronunciation Dictionaries

Knowl 10 — Relationship Between Training Frequency, Acoustic Uniqueness, and Word Recall

References

Citation

BibTeX

Access the Paper