Back

A Critical Review of Recurrent Neural Networks for Sequence Learning

Zachary C. LiptonJohn BerkowitzCharles Elkan

article2015arXiv2,674 citations

Synthesizes three decades of recurrent neural network research by reconciling conflicting notation and explaining the architectural and optimization advancements that made models like LSTMs effective for sequence learning.

Listen

The article addresses the challenge of modeling sequential data in machine learning tasks such as language translation, image captioning, speech synthesis, and video analysis, where standard neural networks fail because they assume independence among data points and fixed-length inputs. Sequential dependencies matter in real-world applications like dialogue systems and time-series prediction, and without explicit modeling of time or order, even powerful classifiers cannot handle extended interactions or long-range patterns.

The article sets out to review and synthesize three decades of research on recurrent neural networks, reconcile inconsistent notation across papers, and explain how recent architectural and training advances made large-scale sequence learning practical.

It proceeds through a literature survey that covers foundational designs from the 1980s, formal definitions of sequences and networks, training difficulties, and modern variants, drawing on primary sources from cognitive modeling to empirical machine-learning results.

The review finds that long short-term memory units with input, forget, and output gates, together with bidirectional architectures, overcome vanishing and exploding gradients and enable networks to capture dependencies across dozens or hundreds of time steps; these models now match or exceed prior state-of-the-art systems on translation (BLEU scores above 34), handwriting recognition (word accuracy above 80 percent), and image captioning. It also shows that external-memory extensions such as neural Turing machines further improve performance on algorithmic tasks. The core practical message is that gradient-based training of recurrent networks has become reliable when combined with modern optimization heuristics and hardware.

These results matter because they remove a long-standing barrier to applying neural networks to any domain where order or timing carries information, lowering the cost of building interactive systems and improving accuracy on high-value tasks such as automated translation and assistive captioning.

Next steps supported by the article include automating the search over network architectures, developing more reliable evaluation metrics than BLEU or METEOR, and extending the same sequence-to-sequence approach to longer documents and full dialogue systems while retaining full conversation history.

The main limitations are that reported gains rely on imperfect automatic metrics whose correlation with human judgment is only moderate at the sentence level, and that most experiments use fixed-length or segmented sequences rather than truly open-ended streams; readers should therefore treat headline numbers as indicative rather than definitive until corroborated by human evaluation and longer-context tests.

University of California, San Diegodialoguefeedforward neural networkshandwriting recognitionRecurrent Neural NetworkLong Short-Term MemoryBidirectional RNN
arXiv: 1506.00019
Cover for A Critical Review of Recurrent Neural Networks for Sequence Learning

Abstract

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades first yielded and then made practical these powerful learning models. When appropriate, we reconcile conflicting notation and nomenclature. Our goal is to provide a self-contained explication of the state of the art together with a historical perspective and references to primary research.

Topics

dialoguefeedforward neural networkshandwriting recognitionlong short term memory neural networksnatural languagesnetwork architecturenomenclatureoptimizationoptimization problemsparallel computation

Keywords

Recurrent Neural NetworkLong Short-Term MemoryBidirectional RNNBackpropagation Through TimeVanishing GradientExploding GradientNeural Turing MachineSequence LearningBLEU ScoreGradient Descent

Table of Contents

  • 1 Introduction
  • 1.1 Why model sequentiality explicitly?
  • 1.2 Why not use Markov models?
  • 1.3 Are RNNs too expressive?
  • 1.4 Comparison to prior literature
  • 2 Background
  • 2.1 Sequences
  • 2.2 Neural networks
  • 2.3 Feedforward networks and backpropagation
  • 3 Recurrent neural networks
  • 3.1 Early recurrent network designs
  • 3.2 Training recurrent networks
  • 4 Modern RNN architectures
  • 4.1 Long short-term memory (LSTM)
  • 4.2 Bidirectional recurrent neural networks (BRNNs)
  • 4.3 Neural Turing machines
  • 5 Applications of LSTMs and BRNNs
  • 5.1 Representations of natural language inputs and outputs
  • 5.2 Evaluation methodology
  • 5.3 Natural language translation
  • 5.4 Image captioning
  • 5.5 Further applications
  • 6 Discussion
  • 7 Acknowledgements
  • References

Knowls

  1. Knowl 1 — Standard Recurrent Neural Network Forward Pass

    model/method

    A simple recurrent neural network (RNN) processes an input sequence (x(1),x(2),…,x(T))(x^{(1)}, x^{(2)}, \dots, x^{(T)}) where each x(t)∈Rdx^{(t)} \in \mathbb{R}^d by maintaining a hidden state vector h(t)∈Rmh^{(t)} \in \mathbb{R}^m across discrete time steps t∈{1,…,T}t \in \{1, \dots, T\}.

    At each time step tt, the hidden state h(t)h^{(t)} and predicted output distribution y^(t)\hat{y}^{(t)} are computed via: h(t)=σ(Whxx(t)+Whhh(t−1)+bh)h^{(t)} = \sigma\left(W_{hx} x^{(t)} + W_{hh} h^{(t-1)} + b_h\right) y^(t)=softmax(Wyhh(t)+by)\hat{y}^{(t)} = \text{softmax}\left(W_{yh} h^{(t)} + b_y\right) where Whx∈Rm×dW_{hx} \in \mathbb{R}^{m \times d} is the input-to-hidden weight matrix, Whh∈Rm×mW_{hh} \in \mathbb{R}^{m \times m} is the recurrent hidden-to-hidden weight matrix connecting adjacent time steps, and Wyh∈RK×mW_{yh} \in \mathbb{R}^{K \times m} is the hidden-to-output matrix for KK classes. The vectors bh∈Rmb_h \in \mathbb{R}^m and by∈RKb_y \in \mathbb{R}^K represent bias parameters.

    The activation function σ(⋅)\sigma(\cdot) is applied element-wise (commonly a sigmoid σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}} or hyperbolic tangent ϕ(z)=tanh⁡(z)\phi(z) = \tanh(z)). Unfolding this computation across time turns the recurrent network into a feedforward network with one layer per time step and shared parameter weights across all steps, enabling training via backpropagation through time (BPTT).

  2. Knowl 2 — Modern Long Short-Term Memory Network with Forget Gates

    model/method

    The Long Short-Term Memory (LSTM) architecture replaces standard hidden units with memory cells equipped with multiplicative gating mechanisms to prevent vanishing and exploding gradients. For an input vector x(t)x^{(t)} and previous hidden output h(t−1)h^{(t-1)} at time step tt, the forward pass equations for an LSTM cell with forget gates are: g(t)=ϕ(Wgxx(t)+Wghh(t−1)+bg)g^{(t)} = \phi\left(W_{gx} x^{(t)} + W_{gh} h^{(t-1)} + b_g\right) i(t)=σ(Wixx(t)+Wihh(t−1)+bi)i^{(t)} = \sigma\left(W_{ix} x^{(t)} + W_{ih} h^{(t-1)} + b_i\right) f(t)=σ(Wfxx(t)+Wfhh(t−1)+bf)f^{(t)} = \sigma\left(W_{fx} x^{(t)} + W_{fh} h^{(t-1)} + b_f\right) o(t)=σ(Woxx(t)+Wohh(t−1)+bo)o^{(t)} = \sigma\left(W_{ox} x^{(t)} + W_{oh} h^{(t-1)} + b_o\right) s(t)=g(t)⊙i(t)+s(t−1)⊙f(t)s^{(t)} = g^{(t)} \odot i^{(t)} + s^{(t-1)} \odot f^{(t)} h(t)=ϕ(s(t))⊙o(t)h^{(t)} = \phi\left(s^{(t)}\right) \odot o^{(t)} where ⊙\odot denotes the element-wise (Hadamard) product, σ(z)=11+e−z\sigma(z) = \frac{1}{1 + e^{-z}} is the logistic sigmoid function mapping activations to [0,1][0, 1], and ϕ(z)=tanh⁡(z)=ez−e−zez+e−z\phi(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} maps activations to [−1,1][-1, 1].

    The components of each memory cell operate as follows:

    • Input node g(t)g^{(t)}: computes candidate updates from current input x(t)x^{(t)} and previous hidden output h(t−1)h^{(t-1)}.
    • Input gate i(t)i^{(t)}: controls how much of the candidate activation enters the cell state.
    • Forget gate f(t)f^{(t)}: modulates how much of the previous cell state is retained or flushed (setting f(t)=1f^{(t)} = 1 recovers the original LSTM without forget gates).
    • Internal state s(t)s^{(t)}: maintains a linear recurrence with a constant error carousel of fixed unit weight (11), enabling error signals to bridge long time intervals without exponential decay.
    • Output gate o(t)o^{(t)}: regulates what proportion of the transformed internal state is emitted into the hidden output vector h(t)h^{(t)}.

  3. Knowl 3 — Bidirectional Recurrent Neural Network Architecture

    model/method

    A Bidirectional Recurrent Neural Network (BRNN) computes sequence representations by combining context from both past and future time steps for every position in a fixed-length sequence of length TT. The model splits the hidden state into a forward layer h(t)h^{(t)} and a backward layer z(t)z^{(t)}: h(t)=σ(Whxx(t)+Whhh(t−1)+bh)h^{(t)} = \sigma\left(W_{hx} x^{(t)} + W_{hh} h^{(t-1)} + b_h\right) z(t)=σ(Wzxx(t)+Wzzz(t+1)+bz)z^{(t)} = \sigma\left(W_{zx} x^{(t)} + W_{zz} z^{(t+1)} + b_z\right) y^(t)=softmax(Wyhh(t)+Wyzz(t)+by)\hat{y}^{(t)} = \text{softmax}\left(W_{yh} h^{(t)} + W_{yz} z^{(t)} + b_y\right) where x(t)x^{(t)} is the input vector at time step tt, WhxW_{hx} and WzxW_{zx} map input vectors to forward and backward states respectively, WhhW_{hh} and WzzW_{zz} are directional recurrent weight matrices, and Wyh,WyzW_{yh}, W_{yz} map the concatenated or combined directional hidden states to the output node activations.

    A limitation of BRNNs is that they require fixed sequence boundaries in both past and future directions; hence, they cannot run continually in online, real-time streaming environments where future tokens are unobserved.

  4. Knowl 4 — Vanishing and Exploding Gradient Mechanisms in Recurrent Networks

    theoretical result

    When calculating the gradient of an error at time step tt with respect to an input or hidden activation at time step τ\tau (where t≫τt \gg \tau), backpropagation through time requires multiplying error derivatives across t−τt - \tau intermediate steps by the recurrent weight matrix WhhW_{hh}.

    Because the recurrent weights are tied across all time steps:

    1. If the dominant eigenvalue or spectral radius of WhhW_{hh} satisfies ∣wjj∣<1|w_{jj}| < 1, or when using saturating activation functions (such as σ(z)\sigma(z) whose derivative is bounded by 0.250.25), the backpropagated error signal decays exponentially fast toward zero as t−τt - \tau grows large. This prevents the network from learning long-term dependencies (the vanishing gradient problem).
    2. If the effective weights satisfy ∣wjj∣>1|w_{jj}| > 1 (which easily occurs when using unbounded activations like rectified linear units max⁡(0,x)\max(0, x)), the error gradients can grow exponentially fast, causing numerical instability and wild parameter updates (the exploding gradient problem).

    Truncated Backpropagation Through Time (TBPTT) mitigates exploding gradients by restricting error propagation to a fixed maximum time-step cutoff, but sacrifices the ability to capture dependencies longer than the cutoff window. LSTM architectures resolve the vanishing gradient issue by using an internal linear state with fixed unit-weight recurrent edges (constant error carousel).

  5. Knowl 5 — Jordan and Elman Recurrent Neural Network Architectures

    model/method

    Early recurrent networks incorporated historical sequential context into feedforward structures using dedicated context units:

    • Jordan Network (1986): Extends a single-hidden-layer feedforward network with context units that receive activation fed back directly from the output units at the preceding time step. These context units have self-recurrent loops and connect forward into the hidden layer at the next time step, allowing past output actions to condition future hidden states.
    • Elman Network (1990): Connects each hidden unit jj to a corresponding context unit j′j' via a fixed unit weight wj′j=1w_{j'j} = 1. The context units then project back into the same hidden units at time t+1t+1 through standard trainable weights. This setup is structurally equivalent to a simple RNN with self-connected hidden units and provided the conceptual precursor to the unit-weight recurrent connection in LSTM memory cells.

  6. Knowl 6 — Neural Turing Machine Architecture and Algorithmic Generalization

    model/method

    A Neural Turing Machine (NTM) extends recurrent networks by pairing a neural network controller with an addressable external memory matrix M∈RN×MM \in \mathbb{R}^{N \times M}, where NN is the number of memory slots and MM is the vector dimensionality of each slot.

    The system consists of:

    1. A controller network (implemented as a feedforward or LSTM network) that interfaces with external input/output and coordinates memory operations.
    2. Differentiable read and write heads that attend to memory locations using continuous weighting mechanisms based on content and location addressing.

    Because every read, write, and addressing operation is fully differentiable, the entire system can be trained end-to-end via gradient descent using backpropagation through time. On algorithmic tasks such as copying binary sequences and priority sorting, NTMs learn algorithmic logic that generalizes to sequences of lengths substantially greater than those seen during training, whereas standard LSTMs without external memory degrade on longer test sequences.

  7. Knowl 7 — Sequence-to-Sequence Translation with Encoder-Decoder LSTMs

    model/method

    The sequence-to-sequence model transforms an input sequence (x(1),…,x(T))(x^{(1)}, \dots, x^{(T)}) into an output sequence (y(1),…,y(T′))(y^{(1)}, \dots, y^{(T')}) of potentially different length using two separate multilayered LSTMs:

    1. Encoder LSTM: Consumes the source sequence one token at a time without emitting predictions. Reversing the order of the source sequence tokens (e.g., inputting (x(T),…,x(1))(x^{(T)}, \dots, x^{(1)})) substantially improves translation quality by reducing the lag between source and target words at the start of the sentence. The final internal state vector of the encoder acts as the summary representation of the source input.
    2. Decoder LSTM: Initialized with the final hidden state of the encoder, the decoder receives an initial start token and predicts target tokens sequentially via softmax probability distributions over the vocabulary. During generation, each chosen word is fed as input to the next decoder time step until an end-of-sentence token (⟨EOS⟩\langle\text{EOS}\rangle) is emitted.
    3. Inference: A left-to-right beam search explores high-probability token combinations at each time step rather than greedy argmax selection.

  8. Knowl 8 — BLEU and METEOR Formulations for Natural Language Generation Evaluation

    equation

    Evaluating generated text candidates against reference texts in sequence translation and captioning tasks relies on precision/recall metrics:

    BLEU Score: Computes the geometric mean of modified nn-gram precisions pnp_n up to order NN (typically N=4N=4), scaled by a brevity penalty BB: BLEU=B⋅exp⁡(1N∑n=1Nlog⁡pn)\text{BLEU} = B \cdot \exp\left(\frac{1}{N} \sum_{n=1}^N \log p_n\right) where pnp_n is the ratio of candidate nn-grams found in any reference to total candidate nn-grams. Given average candidate length cc and average reference length rr, the brevity penalty is defined as: B={1if c>re1−r/cif c≤rB = \begin{cases} 1 & \text{if } c > r \\ e^{1 - r/c} & \text{if } c \le r \end{cases}

    METEOR Score: Evaluates explicit unigram mappings using exact matches, stemming, and synonym dictionaries. It calculates an F-score FαF_\alpha from unigram precision PP and recall RR: Fα=P⋅RαP+(1−α)RF_\alpha = \frac{P \cdot R}{\alpha P + (1 - \alpha) R} It then applies a fragmentation penalty M∝cmM \propto \frac{c}{m} based on the minimum number of adjacent chunk fragments cc and total matched unigrams mm: METEOR=(1−M)⋅Fα\text{METEOR} = (1 - M) \cdot F_\alpha

    METEOR correlates better with human evaluations at the single-sentence level than BLEU, but requires strict replication of stemming and synonym lookup resources to reproduce reported scores.

  9. Knowl 9 — Preponderance of Saddle Points over Local Minima in Deep Network Optimization

    theoretical result

    On the non-convex error surfaces of large neural networks, including deep feedforward and recurrent architectures, the ratio of saddle points to true local minima grows exponentially with the parameter dimensionality of the network.

    This landscape structure poses a challenge for standard second-order optimization: classic Newton's method is attracted to critical points where the gradient vanishes, including saddle points where the Hessian contains negative eigenvalues. In contrast, saddle-free Newton algorithms identify negative curvature directions by taking the absolute value of the Hessian's eigenvalues, enabling the optimizer to escape saddle points and accelerate training on recurrent networks when applied in hybrid combination with stochastic gradient descent (SGD).

  10. Knowl 10 — Multimodal Convolutional-Recurrent Architecture for Image and Video Description

    model/method

    Multimodal recurrent architectures bridge computer vision and natural language processing by combining convolutional neural networks (CNNs) for visual encoding with recurrent decoders (such as LSTMs) for text generation:

    1. Image Captioning: A convolutional network pretrained on image classification processes an input image xx into a dense feature vector (e.g., from its topmost layer). This representation is fed into an LSTM decoder either as an initial hidden state or as the first input token. The LSTM then generates descriptive words sequentially via a softmax layer until an end-of-sentence (⟨EOS⟩\langle\text{EOS}\rangle) token is generated.
    2. Video Captioning: A sequence of video frames is processed frame-by-frame through a CNN to produce a sequence of visual embeddings. An encoding LSTM reads these feature vectors to summarize temporal dynamics, and a decoding LSTM converts the final representation into a natural language sentence.

Coverage note — Omitted were brief historical overviews of biological plausibility in cognitive science, basic definitions of standard feedforward SGD optimizers (AdaGrad, RMSprop, AdaDelta), general descriptions of word2vec/GloVe embeddings, and peripheral application summaries (such as pen-coordinate online handwriting recognition datasets) that did not introduce novel architectures or formalisms.

References

  1. 1.Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. Joint language and translation modeling with recurrent neural networks. In EMNLP, pages 1044–1054, 2013.
  2. 2.Alan Baddeley, Sergio Della Sala, and T.W. Robbins. Working memory and executive control [and discussion]. Philosophical Transactions of the Royal Society B: Biological Sciences, 351(1346):1397–1404, 1996.
  3. 3.Pierre Baldi and Gianluca Pollastri. The principled design of large-scale recursive neural network architectures–DAG-RNNs and the protein structure prediction problem. The Journal of Machine Learning Research, 4:575–602, 2003.
  4. 4.Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, 2005.
  5. 5.Justin Bayer, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. Evolving memory cell structures for sequence learning. In Artificial Neural Networks–ICANN 2009, pages 755–764. Springer, 2009.
  6. 6.Richard K. Belew, John McInerney, and Nicol N. Schraudolph. Evolving networks: Using the genetic algorithm with connectionist learning. In In. Citeseer, 1990.
  7. 7.Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166, 1994.
  8. 8.Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155, 2003.
  9. 9.Yoshua Bengio, Nicolas Boulanger-Lewandowski, and Razvan Pascanu. Advances in optimizing recurrent networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 8624–8628. IEEE, 2013.
  10. 10.James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), volume 4, page 3. Austin, TX, 2010.
  11. 11.Avrim L. Blum and Ronald L. Rivest. Training a 3-node neural network is NP-complete. In Machine Learning: From Theory to Applications, pages 9–28. Springer, 1993.
  12. 12.Bob Carpenter. Lazy sparse stochastic gradient descent for regularized multinomial logistic regression. Alias-i, Inc., Tech. Rep, pages 1–20, 2008.
  13. 13.Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011.
  14. 14.Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in Neural Information Processing Systems, pages 2933–2941, 2014.
  15. 15.Wim De Mulder, Steven Bethard, and Marie-Francine Moens. A survey on the application of recurrent neural networks to statistical language modeling. Computer Speech & Language, 30(1):61–98, 2015.
  16. 16.John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011.
  17. 17.Charles Elkan. Learning meanings for sentences. http://cseweb.ucsd.edu/~elkan/250B/learningmeaning.pdf, 2015. Accessed: 2015-05-18.
  18. 18.Jeffrey L. Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990.
  19. 19.Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1915–1929, 2013.
  20. 20.Felix A. Gers. Long short-term memory in recurrent neural networks. Unpublished PhD dissertation, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, 2001.
  21. 21.Felix A. Gers and Jürgen Schmidhuber. Recurrent nets that time and count. In Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on, volume 3, pages 189–194. IEEE, 2000.
  22. 22.Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with LSTM. Neural computation, 12(10):2451–2471, 2000.
  23. 23.Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume, volume 15, pages 315–323, 2011.
  24. 24.Yoav Goldberg and Omer Levy. word2vec explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.
  25. 25.Alex Graves. Supervised sequence labelling with recurrent neural networks, volume 385. Springer, 2012.
  26. 26.Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5):602–610, 2005.
  27. 27.Alex Graves, Marcus Liwicki, Santiago Fernández, Roman Bertolami, Horst Bunke, and Jürgen Schmidhuber. A novel connectionist system for unconstrained handwriting recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 31(5):855–868, 2009.
  28. 28.Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. arXiv preprint arXiv:1410.5401, 2014.
  29. 29.Frdric Gruau, L’universite Claude Bernard lyon I, Of A Diplome De Doctorat, M. Jacques Demongeot, Examinators M. Michel Cosnard, M. Jacques Mazoyer, M. Pierre Peretto, and M. Darell Whitley. Neural network synthesis using cellular encoding and the genetic algorithm., 1994.
  30. 30.Steven A. Harp and Tariq Samad. Optimizing neural networks with genetic algorithms. In Proceedings of the 54th American Power Conference, Chicago, volume 2, 2013.
  31. 31.Geoffrey E. Hinton. Learning distributed representations of concepts, 1986.
  32. 32.Sepp Hochreiter and Jurgen Schmidhuber. Bridging long time lags by weight guessing and “long short-term memory”. Spatiotemporal Models in Biological and Artificial Systems, 37:65–72, 1996.
  33. 33.Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  34. 34.Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. A field guide to dynamical recurrent neural networks, 2001.
  35. 35.John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982.
  36. 36.Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
  37. 37.Michael I. Jordan. Serial order: A parallel distributed processing approach. Technical Report 8604, Institute for Cognitive Science, University of California, San Diego, 1986.
  38. 38.Andrej Karpathy. The unreasonable effectiveness of recurrent neural networks. http://karpathy.github.io/2015/05/21/rnn-effectiveness/, 2015. Accessed: 2015-08-13.
  39. 39.Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306, 2014.
  40. 40.Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012.
  41. 41.John Langford, Lihong Li, and Tong Zhang. Sparse online learning via truncated gradient. In Advances in Neural Information Processing Systems, pages 905–912, 2009.
  42. 42.Yann Le Cun, B. Boser, John S. Denker, D. Henderson, Richard E. Howard, W. Hubbard, and Lawrence D. Jackel. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems. Citeseer, 1990.
  43. 43.Zachary C. Lipton and Charles Elkan. Efficient elastic net regularization for sparse linear models. CoRR, abs/1505.06449, 2015. URL http://arxiv.org/abs/1505.06449.
  44. 44.Zachary C. Lipton, Charles Elkan, and Balakrishnan Naryanaswamy. Optimal thresholding of classifiers to maximize F1 measure. In Machine Learning and Knowledge Discovery in Databases, pages 225–239. Springer, 2014.
  45. 45.Marcus Liwicki, Alex Graves, Horst Bunke, and Jürgen Schmidhuber. A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks. In Proc. 9th Int. Conf. on Document Analysis and Recognition, volume 1, pages 367–371, 2007.
  46. 46.Andrew L. Maas, Quoc V. Le, Tyler M. O’Neil, Oriol Vinyals, Patrick Nguyen, and Andrew Y. Ng. Recurrent neural networks for noise reduction in robust ASR. In INTERSPEECH. Citeseer, 2012.
  47. 47.Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-RNN). arXiv preprint arXiv:1412.6632, 2014.
  48. 48.James Martens and Ilya Sutskever. Learning recurrent neural networks with Hessian-free optimization. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1033–1040, 2011.
  49. 49.Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
  50. 50.Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 807–814, 2010.
  51. 51.Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.
  52. 52.Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2012.
  53. 53.Barak A. Pearlmutter. Gradient calculations for dynamic recurrent neural networks: A survey. Neural Networks, IEEE Transactions on, 6(5):1212–1228, 1995.
  54. 54.Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP 2014), 12, 2014.
  55. 55.David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. Technical report, DTIC Document, 1985.
  56. 56.Mike Schuster and Kuldip K. Paliwal. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681, 1997.
  57. 57.Hava T. Siegelmann and Eduardo D. Sontag. Turing computability with neural nets. Applied Mathematics Letters, 4(6):77–80, 1991.
  58. 58.Yoram Singer and John C. Duchi. Efficient learning using forward-backward splitting. In Advances in Neural Information Processing Systems, pages 495–503, 2009.
  59. 59.Richard Socher, Christopher D. Manning, and Andrew Y. Ng. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop, pages 1–9, 2010.
  60. 60.Richard Socher, Eric H. Huang, Jeffrey Pennin, Christopher D. Manning, and Andrew Y. Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems, pages 801–809, 2011a.
  61. 61.Richard Socher, Cliff C. Lin, Chris Manning, and Andrew Y. Ng. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 129–136, 2011b.
  62. 62.Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. Association for Computational Linguistics, 2011c.
  63. 63.Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207–218, 2014.
  64. 64.Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using LSTMs. arXiv preprint arXiv:1502.04681, 2015.
  65. 65.Ruslan L. Stratonovich. Conditional markov processes. Theory of Probability & Its Applications, 5(2):156–178, 1960.
  66. 66.Ilya Sutskever, James Martens, and Geoffrey E. Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024, 2011.
  67. 67.Ilya Sutskever, James Martens, George Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 1139–1147, 2013.
  68. 68.Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112, 2014.
  69. 69.Tijmen Tieleman and Geoffrey E. Hinton. Lecture 6.5- RMSprop: Divide the gradient by a running average of its recent magnitude. https://www.youtube.com/watch?v=LGA-gRkLEsI, 2012.
  70. 70.Alan M. Turing. Computing machinery and intelligence. Mind, pages 433–460, 1950.
  71. 71.Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence–video to text. arXiv preprint arXiv:1505.00487, 2015.
  72. 72.Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164, 2015.
  73. 73.Andrew J. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. Information Theory, IEEE Transactions on, 13(2):260–269, 1967.
  74. 74.Paul J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990.
  75. 75.Wikipedia. Backpropagation — Wikipedia, the free encyclopedia, 2015. URL http://en.wikipedia.org/wiki/Backpropagation. [Online; accessed 18-May-2015].
  76. 76.Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270–280, 1989.
  77. 77.Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014.
  78. 78.Matthew D. Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
  79. 79.Matthew D. Zeiler, M. Ranzato, Rajat Monga, M. Mao, K. Yang, Quoc V. Le, Patrick Nguyen, A. Senior, Vincent Vanhoucke, Jeffrey Dean, et al. On rectified linear units for speech processing. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 3517–3521. IEEE, 2013.

Citation

MLA
Lipton, Z. C., et al. “A Critical Review of Recurrent Neural Networks for Sequence Learning”. arXiv, 2015, https://doi.org/10.48550/arxiv.1506.00019.
APA
Lipton, Z. C., Berkowitz, J., & Elkan, C. (2015). A Critical Review of Recurrent Neural Networks for Sequence Learning. arXiv. https://doi.org/10.48550/arxiv.1506.00019
Chicago
Lipton, Z. C., J. Berkowitz, and C. Elkan. 2015. “A Critical Review of Recurrent Neural Networks for Sequence Learning”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1506.00019.
Harvard
Lipton, Z.C., Berkowitz, J. and Elkan, C. (2015) “A Critical Review of Recurrent Neural Networks for Sequence Learning”. arXiv. Available at: https://doi.org/10.48550/arxiv.1506.00019.
Vancouver
1. Lipton ZC, Berkowitz J, Elkan C (2015) A Critical Review of Recurrent Neural Networks for Sequence Learning. https://doi.org/10.48550/arxiv.1506.00019

BibTeX

@misc{https://doi.org/10.48550/arxiv.1506.00019,
  doi = {10.48550/ARXIV.1506.00019},
  url = {https://arxiv.org/abs/1506.00019},
  author = {Lipton, Zachary C. and Berkowitz, John and Elkan, Charles},
  keywords = {Machine Learning (cs.LG), Neural and Evolutionary Computing (cs.NE), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {A Critical Review of Recurrent Neural Networks for Sequence Learning},
  publisher = {arXiv},
  year = {2015},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors