Attention-Based Models for Speech Recognition

Jan ChorowskiDzmitry BahdanauDmitriy SerdyukKyunghyun ChoYoshua Bengio

article2015NeurIPS2,777 citations

Introduces location-aware attention mechanisms for end-to-end speech recognition, resolving alignment failures across long acoustic inputs and achieving competitive phoneme error rates on TIMIT.

Listen

The article addresses the challenge of building fully end-to-end neural systems for speech recognition, where traditional hybrid approaches rely on hand-crafted pronunciation dictionaries, triphone HMMs, and separate language models. Attention-based recurrent sequence generators had succeeded on shorter-sequence tasks such as machine translation, yet they struggled with the much longer and noisier inputs typical of speech, raising questions about whether they could scale reliably.

The article set out to evaluate whether attention mechanisms could be adapted for phoneme recognition on the TIMIT corpus and to develop extensions that would allow the models to handle utterances many times longer than those seen during training.

The authors trained and compared three attention-based recurrent sequence generators on the standard TIMIT train-dev-test split, using 40 mel-filterbank features plus energy and deltas. They started with a content-only attention baseline previously used for translation, then introduced a hybrid mechanism that augments attention scores with convolutional features extracted from the previous alignment, and finally tested a smoothing modification that replaces softmax normalization with a sigmoid to encourage the model to aggregate information from multiple frames. Performance was measured both on the original short test utterances and on artificially concatenated sequences up to ten times longer.

The baseline content-based model reached a competitive 18.7 percent phoneme error rate on the standard test set but rapidly degraded on longer inputs, often aligning only the first 40–50 phones before jumping to the end and cycling. The hybrid model with convolutional location features improved to 18.0 percent error on the original test set and maintained error rates below 20 percent even on utterances formed by repeating or mixing up to eleven original recordings. Adding the smoothing modification further lowered the error to 17.6 percent while preserving robustness to length.

These results demonstrate that a purely neural attention architecture can match or approach the accuracy of conventional pipelines on phoneme recognition without requiring separate alignment or lexicon stages, and that explicit location awareness makes the approach practical for longer audio. The improvement matters because it removes a key obstacle to scaling end-to-end speech systems and suggests the same attention extensions could benefit other sequence tasks that involve long or repetitive inputs.

The article recommends incorporating a language model directly into the generator, extending the approach to direct character or word transcription, and applying the convolutional location features to neural Turing machines or image captioning. Further validation on larger, more varied corpora and real-world long-form audio is needed before deployment decisions can be made; the current experiments rely on a relatively small phoneme dataset and artificially constructed long utterances.

CIFAR Jacobs University Bremen Université de Montréal University of Wrocław attention mechanism frames machine translations attention-based models speech recognition hybrid attention

arXiv: 1506.07503

Paper: Neural Machine Translation by Jointly Learning to Align and Translate, Dzmitry Bahdanau et al. (2015). Introduces the foundational content-based soft attention mechanism in sequence-to-sequence neural networks that the source paper evaluates and extends for acoustic modeling.
Paper: Sequence to Sequence Learning with Neural Networks, Ilya Sutskever et al. (2014). Establishes the core encoder-decoder recurrent framework for mapping variable-length input sequences to variable-length target sequences.
Paper: Towards End-To-End Speech Recognition with Recurrent Neural Networks, Alex Graves et al. (2014). Demonstrates early end-to-end recurrent architectures for speech recognition, providing the motivation to eliminate traditional multi-stage phonetic pipelines.
Paper: Speech Recognition with Deep Recurrent Neural Networks, Alex Graves et al. (2013). Pioneers deep bidirectional LSTM networks and alignment-free objectives on the TIMIT phoneme recognition benchmark used as the baseline in the source.
Paper: Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation, Kyunghyun Cho et al. (2014). Presents the foundational RNN Encoder-Decoder model and gated recurrent units for sequence generation that underpin attentional decoders.
Paper: On the Properties of Neural Machine Translation: Encoder–Decoder Approaches, Kyunghyun Cho et al. (2014). Examines the severe performance degradation of standard encoder-decoder models on long sequence inputs, highlighting the exact length-generalization problem solved in the source.
Paper: Generating Sequences With Recurrent Neural Networks, Alex Graves (2013). Introduces recurrent sequence generation with location-sensitive Gaussian mixture attention mechanisms that inspired location-based alignments in speech.
Paper: Long Short-Term Memory, Sepp Hochreiter et al. (1997). Develops the gated Long Short-Term Memory cell architecture necessary for training deep recurrent sequence models over long temporal spans.

Paper: Listen, Attend and Spell, William Chan et al. (2015). Extends attention-based end-to-end speech recognition to large-scale, full character transcription using a pyramidal encoder and integrated sequence decoding.
Paper: Deep Speech 2: End-to-End Speech Recognition in English and Mandarin, Dario Amodei et al. (2016). Scales end-to-end neural speech recognition to multi-language industrial settings using deep convolutional recurrent networks.
Paper: Tacotron: Towards End-to-End Speech Synthesis, Yuxuan Wang et al. (2017). Applies location-aware attention and sequence-to-sequence recurrent modeling to text-to-speech synthesis to generate spectrogram frames from character inputs.
Paper: Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions, Jonathan Shen et al. (2017). Further refines attention-based acoustic sequence generation for speech synthesis by coupling recurrent mel-spectrogram predictors with neural vocoding.
Paper: Conformer: Convolution-augmented Transformer for Speech Recognition, Anmol Gulati et al. (2020). Combines convolutional modules with self-attention architectures to modernize end-to-end speech recognition models.
Paper: Robust Speech Recognition via Large-Scale Weak Supervision, Alec Radford et al. (2023). Scales attention-based sequence-to-sequence speech recognition across massive multilingual and weakly supervised web audio datasets.
Paper: wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, Alexei Baevski et al. (2020). Develops self-supervised representations for raw audio sequences that improve end-to-end speech recognition efficiency in low-resource conditions.

Cover for Attention-Based Models for Speech Recognition

Abstract

Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks in- cluding machine translation, handwriting synthesis and image caption gen- eration. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the at- tention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.

Topics

attention mechanism frames machine translations phone recognition

Keywords

attention-based models speech recognition hybrid attention location-based attention mechanism connectionist temporal classification phoneme recognition task recurrent networks acoustic model language model

1 Introduction
2 Attention-Based Model for Speech Recognition
2.1 General Framework
2.2 Proposed Model: ARSG with Convolutional Features
2.3 Score Normalization: Sharpening and Smoothing
3 Related Work
4 Experimental Setup
4.1 Training Procedure
4.2 Details of Evaluated Models
5 Results
5.1 Forced Alignment of Long Utterances
5.2 Decoding Long Utterances
6 Conclusions
References
A Additional Figures
B Detailed results of experiments

Knowls

Knowl 1 — Attention-Based Recurrent Sequence Generator Framework for Speech Recognition
model/method
The Attention-Based Recurrent Sequence Generator (ARSG) framework models speech recognition as an end-to-end sequence transduction problem mapping an acoustic feature sequence $x = (x_1, \dots, x_{L'})$ to a phoneme target sequence $y = (y_1, \dots, y_T)$ .

An encoder, implemented as a deep bidirectional recurrent neural network (BiRNN), processes the input frames into a sequential context representation $h = (h_1, \dots, h_L)$ where $L = L'$ .

At each output step $i \in \{1, \dots, T\}$ , decoding proceeds as follows:
1. An attention module computes an alignment vector $\alpha_i \in \mathbb{R}^L$ over encoder states: $\alpha_i = \mathrm{Attend}(s_{i-1}, \alpha_{i-1}, h)$ where $s_{i-1}$ is the recurrent generator state at step $i-1$ , and $\alpha_{i-1}$ is the previous step's alignment.
2. A glimpse (context vector) $g_i \in \mathbb{R}^{\dim(h)}$ is computed as the weighted sum of encoder representations: $g_i = \sum_{j=1}^L \alpha_{i,j} h_j$
3. The current output token $y_i$ is sampled or selected from the output distribution: $y_i \sim \mathrm{Generate}(s_{i-1}, g_i)$
4. The generator state is updated by a recurrent transition function (such as a GRU or LSTM): $s_i = \mathrm{Recurrency}(s_{i-1}, g_i, y_i)$
Knowl 2 — Location-Aware Hybrid Attention via 1D Convolutional Features
model/method
Purely content-based attention mechanisms select encoder representations $h_j$ solely by scoring their match with the decoder state $s_{i-1}$ , making them susceptible to confusion when acoustically similar sounds appear at multiple positions in an utterance. To enforce location awareness, the attention mechanism is extended to condition scoring on the previous alignment vector $\alpha_{i-1} \in \mathbb{R}^L$ .

First, $k$ -dimensional convolutional feature vectors $f_{i,j} \in \mathbb{R}^k$ are extracted for every input position $j \in \{1, \dots, L\}$ by convolving the previous alignment $\alpha_{i-1}$ with a 1D learnable convolution matrix $F \in \mathbb{R}^{k \times r}$ containing $k$ filters of width $r$ : $f_i = F * \alpha_{i-1}$

Second, the unnormalized alignment energy score $e_{i,j}$ for frame $j$ at decoder step $i$ is computed by combining the generator state $s_{i-1}$ , encoder frame $h_j$ , and the local alignment feature $f_{i,j}$ : $e_{i,j} = w^\top \tanh(W s_{i-1} + V h_j + U f_{i,j} + b)$ where $w \in \mathbb{R}^{d_{\mathrm{att}}}$ and $b \in \mathbb{R}^{d_{\mathrm{att}}}$ are parameter vectors, and $W, V, U$ are learnable weight matrices projecting $s_{i-1}$ , $h_j$ , and $f_{i,j}$ into a shared attention hidden space.
Knowl 3 — Smooth Focus Score Normalization Using Logistic Sigmoid
model/method
Standard attention mechanisms compute alignment weights $\alpha_{i,j}$ using a softmax normalization: $\alpha_{i,j} = \frac{\exp(e_{i,j})}{\sum_{j'=1}^L \exp(e_{i,j'})}$ Because the exponential function is unbounded, softmax tends to concentrate the alignment weight sharply onto a single acoustic frame $h_j$ . For speech inputs, however, aggregating contextual acoustic information across multiple neighboring top-scored frames provides more useful representations for output phoneme prediction.

To prevent the attention mechanism from collapsing onto a single frame and to produce a broader, smoother focus, the unbounded exponential function is replaced with the bounded logistic sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$ : $\alpha_{i,j} = \frac{\sigma(e_{i,j})}{\sum_{j'=1}^L \sigma(e_{i,j'})}$ This normalization smooths the alignment distribution across neighboring frames, yielding better generalization on phoneme recognition tasks.
Knowl 4 — Attention Windowing for Long Sequence Decoding
model/method
Computing attention over all $L$ acoustic frames at every decoding step $i \in \{1, \dots, T\}$ incurs an $O(LT)$ computational complexity and allows noisy acoustic frames from distant parts of long utterances to degrade the context vector $g_i$ .

Attention windowing restricts the candidate frames evaluated by the attention mechanism at decoder step $i$ to a localized window centered around the median position $p_i$ of the previous alignment $\alpha_{i-1}$ : $\tilde{h} = (h_{p_i - w}, \dots, h_{p_i + w - 1})$ where $w \ll L$ is a predefined half-window width and $p_i = \min \{ m : \sum_{j=1}^m \alpha_{i-1,j} \ge 0.5 \}$ .

Scores $e_{i,j}$ for frames outside this window ( $j \notin [p_i - w, p_i + w - 1]$ ) are set to zero without evaluation. This reduces computational complexity from $O(LT)$ to $O(L + T)$ and prevents irrelevant frames from adding noise to the glimpse vector during the decoding of long utterances.
Knowl 5 — TIMIT Phoneme Recognition Architecture and Training Protocol
experimental setup
Evaluation is performed on the TIMIT continuous speech corpus using the standard Kaldi s5 train-dev-test split: 462 speakers for training (with all SA sentence records removed), 50 speakers for development early-stopping, and the 24-speaker core test set.
- Acoustic Features: 40 Mel-scale filterbank channel energies, log frame energy, plus their first and second temporal differences ( $\{ \Delta, \Delta\Delta \}$ ), totaling 123 normalized features per 20 ms frame. An all-zero frame is appended to indicate utterance termination.
- Targets: 61 phonemes plus a special \langle eos \rangle token during training and decoding; final scoring maps outputs to the standard 39 phoneme set.
- Model Dimensions:
  - Encoder: 3-layer bidirectional RNN with 256 Gated Recurrent Units (GRU) per direction (512 units total in the top-layer representation $h$ ).
  - Decoder / Generator: 1 recurrent layer with 256 GRU units.
  - Emission Network: Hidden layer of 64 maxout units.
  - Scoring Network: 512 hidden units.
  - Convolutional Attention: $k = 10$ filters with filter length $r = 201$ .
- Optimization & Regularization:
  - Parameters are initialized from $\mathcal{N}(0, 0.01^2)$ with orthogonalized recurrent weights.
  - Optimized using AdaDelta with $\rho = 0.95$ , initial $\epsilon = 10^{-8}$ , and batch size 1.
  - A column-norm constraint (maximum norm 1) is applied during initial training until development negative log-likelihood reaches a minimum.
  - Next, column-norm constraints are disabled, adaptive weight noise regularization is activated, model complexity cost $L_C$ is scaled down by 10, and fine-tuning continues with $\epsilon = 10^{-10}$ until development phoneme error rate does not improve for 100k updates.

Knowl 6 — Phoneme Error Rate Comparison on TIMIT

data/table

The table below presents Phoneme Error Rates (PER) on the TIMIT development and test sets across different attention-based sequence generator configurations and benchmark end-to-end architectures. Scoring is performed on the 39-phoneme standard set after decoding with the 61-phoneme set:

Model	Dev PER (%)	Test PER (%)
Baseline Model (Content-based only)	15.9	18.7
Baseline + Conv. Features	16.1	18.0
Baseline + Conv. Features + Smooth Focus	15.8	17.6
RNN Transducer (Graves et al., 2013)	N/A	17.7
HMM over Time and Frequency Convolutional Net (Tóth, 2014)	13.9	16.7

Adding location-aware convolutional features improves test PER by 3.7% relative (from 18.7% to 18.0%). Incorporating smooth focus (logistic sigmoid normalization) provides an additional 5.9% relative reduction to 17.6% PER, outperforming the RNN Transducer baseline without requiring dynamic programming over unaligned paths.

Knowl 7 — Failure Modes of Content-Based versus Location-Aware Attention on Extended Utterances
empirical result
When attention models trained on single short TIMIT utterances (under 5 seconds) are evaluated on long sequences constructed by concatenating up to 15 utterances or repeating utterances, they exhibit distinct failure modes during forced alignment:
- Content-Based Attention: Learns an implicit tracking of absolute position in the input sequence. On long sequences, it correctly aligns the first $\approx 40$ phonemes; once its tracking saturates beyond lengths observed during training, it jumps directly to the end of the recording and cycles over the final 10 phonemes. Applying local windowing during decoding fails to fix this because the model confounds repeated acoustic content inside the window.
- Location-Aware Attention (Convolutional Features): Maintains alignment for up to $\approx 200$ phonemes. When it fails without windowing, it fails by diffusing attention across all frames due to the accumulation of small positive scores across hundreds of irrelevant frames (noisy glimpse problem), rather than jumping to the end. When paired with alignment windowing (e.g., window width $\pm 75$ or $\pm 150$ frames), this failure is prevented, enabling correct forced alignment on sequences over 600 phonemes long.
Knowl 8 — Long Utterance Decoding Robustness with Location-Aware Attention and Windowing
empirical result
On long speech inputs created by concatenating up to 11 TIMIT test utterances (either repetitions of the same sentence or distinct random utterances joined with 50 ms silence intervals):
1. The baseline content-based attention model fails to decode long utterances, with phoneme error rates exceeding 24% and deteriorating rapidly as sequence length increases, even when constrained by local alignment windows.
2. Location-aware ARSG models incorporating convolutional features and smooth focus normalization, combined with an attention window of $\pm 150$ frames, decode sequences up to 11 times longer than training utterances while maintaining a stable phoneme error rate of approximately 20% (compared to 17.6% on single utterances).
Knowl 9 — Beam Search Characteristics and Decoding Efficiency in ARSG
empirical result
In ARSG models, the attention mechanism deterministically aligns input speech frames and output phonemes at each time step, unlike CTC and RNN Transducer models which treat alignments as latent variables requiring marginalization over multiple paths.

Consequently, ARSG models require very small beam widths for optimal decoding:
- Phoneme error rate stabilizes at a beam width of 10, with larger beam sizes (20, 50, and 100) offering negligible error rate reductions.
- Models utilizing smooth focus normalization perform well even with greedy decoding (beam width of 1).

Knowl 10 — Impact of Attention Sharpening and Windowing Strategies on Standard Decoding

data/table

The table below presents Phoneme Error Rates (PER) on TIMIT development and test sets across different attention models under various decoding modifications, including plain decoding, top- $k$ frame filtering (Keep 1, Keep 10, Keep 50), inverse temperature sharpening ( $\beta = 2$ ), and windowing (Win $\pm 75$ , Win $\pm 150$ frames):

Model	Set	Plain	Keep 1	Keep 10	Keep 50	$\mathbf{\beta = 2}$	Win $\pm 75$	Win $\pm 150$
Baseline	Dev	15.9%	17.6%	15.9%	15.9%	16.1%	15.9%	15.9%
	Test	18.7%	20.2%	18.7%	18.7%	18.9%	18.7%	18.6%
Conv Feats	Dev	16.1%	19.4%	16.2%	16.1%	16.7%	16.0%	16.1%
	Test	18.0%	22.3%	17.9%	18.0%	18.7%	18.0%	18.0%
Smooth Focus	Dev	15.8%	21.6%	16.5%	16.1%	16.2%	16.2%	16.0%
	Test	17.6%	24.7%	18.7%	17.8%	18.4%	17.7%	17.6%

Hard sharpening via Keep 1 (forcing the model to look at only the single top-scoring frame) degrades performance significantly across all architectures (e.g., from 17.6% to 24.7% for Smooth Focus). Windowing with $\pm 150$ frames preserves test performance (17.6%) while lowering decoding complexity from $O(LT)$ to $O(L+T)$ .

Coverage note — No substantial contributed material was omitted. All primary architectures, mathematical formulas, experimental configurations, failure mode analyses, and empirical results are covered.

References

1.Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850, August 2013.
2.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, September 2014.
3.Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv:1502.03044, February 2015.
4.Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pages 2204–2212, 2014.
5.Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end continuous speech recognition using attention-based recurrent NN: First results. arXiv:1412.1602 [cs, stat], December 2014.
6.Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv:1410.5401, 2014.
7.Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv:1410.3916, 2014.
8.Mark Gales and Steve Young. The application of hidden markov models in speech recognition. Found. Trends Signal Process., 1(3):195–304, January 2007.
9.G. Hinton, Li Deng, Dong Yu, G.E. Dahl, A Mohamed, N. Jaitly, A Senior, V. Vanhoucke, P. Nguyen, T.N. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82–97, November 2012.
10.Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deepspeech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567, 2014.
11.S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural. Comput., 9(8):1735–1780, 1997.
12.Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP 2014, October 2014. to appear.
13.Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML-06, 2006.
14.Alex Graves. Sequence transduction with recurrent neural networks. In ICML-12, 2012.
15.Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient based learning applied to document recognition. Proc. IEEE, 1998.
16.Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In ICASSP 2013, pages 6645–6649. IEEE, 2013.
17.Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML-14, pages 1764–1772, 2014.
18.Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. Weakly supervised memory networks. arXiv preprint arXiv:1503.08895, 2015.
19.J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. DARPA TIMIT acoustic phonetic continuous speech corpus, 1993.
20.Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, and others. The kaldi speech recognition toolkit. In Proc. ASRU, pages 1–4, 2011.
21.Matthew D Zeiler. ADADELTA: An adaptive learning rate method. arXiv:1212.5701, 2012.
22.Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348–2356. Curran Associates, Inc., 2011.
23.Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
24.Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215, 2014.
25.László Tóth. Combining time-and frequency-domain convolution in convolutional neural network-based phone recognition. In ICASSP 2014, pages 190–194, 2014.
26.Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535, 2015.
27.James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Presentation.
28.Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
29.Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu, James Bergstra, Frédéric Bastien, and Yoshua Bengio. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013.
30.Bart van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde-Farley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning. arXiv:1506.00619 [cs, stat], June 2015.

Citation

MLA

Chorowski, J., et al. “Attention-Based Models for Speech Recognition”. arXiv, 2015, https://doi.org/10.48550/arxiv.1506.07503.

APA

Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015). Attention-Based Models for Speech Recognition. arXiv. https://doi.org/10.48550/arxiv.1506.07503

Chicago

Chorowski, J., D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. 2015. “Attention-Based Models for Speech Recognition”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1506.07503.

Harvard

Chorowski, J. et al. (2015) “Attention-Based Models for Speech Recognition”. arXiv. Available at: https://doi.org/10.48550/arxiv.1506.07503.

Vancouver

1. Chorowski J, Bahdanau D, Serdyuk D, Cho K, Bengio Y (2015) Attention-Based Models for Speech Recognition. https://doi.org/10.48550/arxiv.1506.07503

BibTeX

@misc{https://doi.org/10.48550/arxiv.1506.07503,
  doi = {10.48550/ARXIV.1506.07503},
  url = {https://arxiv.org/abs/1506.07503},
  author = {Chorowski, Jan and Bahdanau, Dzmitry and Serdyuk, Dmitriy and Cho, Kyunghyun and Bengio, Yoshua},
  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), Neural and Evolutionary Computing (cs.NE), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Attention-Based Models for Speech Recognition},
  publisher = {arXiv},
  year = {2015},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission

Attention-Based Models for Speech Recognition

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Attention-Based Recurrent Sequence Generator Framework for Speech Recognition

Knowl 2 — Location-Aware Hybrid Attention via 1D Convolutional Features

Knowl 3 — Smooth Focus Score Normalization Using Logistic Sigmoid

Knowl 4 — Attention Windowing for Long Sequence Decoding

Knowl 5 — TIMIT Phoneme Recognition Architecture and Training Protocol

Knowl 6 — Phoneme Error Rate Comparison on TIMIT

Knowl 7 — Failure Modes of Content-Based versus Location-Aware Attention on Extended Utterances

Knowl 8 — Long Utterance Decoding Robustness with Location-Aware Attention and Windowing

Knowl 9 — Beam Search Characteristics and Decoding Efficiency in ARSG

Knowl 10 — Impact of Attention Sharpening and Windowing Strategies on Standard Decoding

References

Citation

BibTeX

Access the Paper