Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Alex Sherstinsky

article2018Physica D: Nonlinear Phenomena5,266 citations

Establishes a rigorous mathematical foundation for recurrent neural networks and LSTMs by deriving their formulations from differential equations, formally proving the unrolling method, and providing complete training equations alongside a generalized model variant.

Listen

The article addresses the challenge that recurrent neural networks and long short-term memory networks, despite their widespread use in sequential data tasks such as language modeling and speech recognition, are frequently presented without complete derivations or training formulas, leaving practitioners without a unified reference. It sets out to derive the canonical forms of these systems rigorously from first principles, justify key techniques such as unrolling, explain training difficulties, and construct an enhanced LSTM variant.

The work begins by modeling state evolution with nonlinear delay differential equations drawn from physics and biology, then discretizes them via the backward Euler method to obtain the canonical RNN equations. It proves that an infinite impulse response sequence can be approximated by a finite impulse response model when the target output can be partitioned into mutually independent segments, thereby justifying the standard unrolling procedure. Training via back-propagation through time is examined next, revealing that long-range dependencies produce vanishing or exploding gradients because the product of repeated Jacobian matrices decays or grows exponentially.

To overcome these limitations, the analysis systematically augments the RNN cell with multiplicative gates that separately regulate retention of the prior state, injection of new candidate updates, and exposure of the readout signal. The resulting Vanilla LSTM cell recirculates error gradients through a constant-error-carousel mode when the state-control gate saturates at unity, allowing parameter updates to reflect distant dependencies. The article further augments this architecture with non-causal convolutional context windows on the input, a recurrent projection layer for dimensionality reduction, and an explicit input-control gate, producing the most general LSTM formulation presented to date.

These constructions matter because they supply every forward- and backward-pass equation in consistent vector notation, enabling direct implementation and modular reuse in larger sequence models. The findings imply that practitioners can now incorporate long-range context and adaptive input filtering without resorting to ad-hoc modifications, potentially improving accuracy and training stability on tasks that require modeling extended dependencies.

Recommended next steps include coding the augmented model in a high-performance framework, benchmarking it against the vanilla LSTM, bidirectional LSTM, and transformer baselines on language-representation and customer-support datasets, and measuring the contribution of each extension. The principal limitations are that the unrolling proof assumes segment independence—an assumption that must be validated empirically for each dataset—and that the work remains theoretical; concrete performance gains require systematic experimentation before deployment decisions can be made with high confidence.

Massachusetts Institute of Technology blogs differential equations inference Recurrent Neural Network Long Short-Term Memory Vanilla LSTM

arXiv: 1808.03314

Paper: Long Short-Term Memory, Sepp Hochreiter et al. (1997). It introduces the fundamental Long Short-Term Memory architecture and constant error carousel mechanism that the source formally derives and extends.
Paper: On the difficulty of training recurrent neural networks, Razvan Pascanu et al. (2012). It provides the foundational mathematical and dynamical analysis of vanishing and exploding gradients in recurrent networks during backpropagation through time.
Paper: A Critical Review of Recurrent Neural Networks for Sequence Learning, Zachary C. Lipton et al. (2015). It offers a comprehensive review of recurrent sequence learning paradigms, network formulations, and gated mechanisms referenced throughout the source's derivations.
Paper: LSTM: A Search Space Odyssey, Klaus Greff et al. (2015). It systematically explores and compares vanilla LSTM components and their mathematical variants, motivating the search for a unified, canonical formulation.
Paper: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Junyoung Chung et al. (2014). It evaluates the empirical behavior and structural trade-offs between standard recurrent units, LSTMs, and gated variants on sequence modeling tasks.
Paper: Understanding LSTM Networks, Christopher Olah (2015). It breaks down the step-by-step gating operations and internal state updates of standard LSTM cells that the source formalizes into rigorous mathematical equations.
Paper: Learning representations by back-propagating errors, David E. Rumelhart et al. (1986). It establishes the foundational backpropagation algorithm necessary for understanding backpropagation through time in unrolled recurrent networks.

Paper: Neural Ordinary Differential Equations, Ricky T. Q. Chen et al. (2018). It generalizes the continuous-time dynamical system and differential equation perspective used by the source to derive RNNs into fully continuous-depth neural ordinary differential equations.
Paper: Efficiently Modeling Long Sequences with Structured State Spaces, Albert Gu et al. (2022). It extends continuous-time state-space representations to modern structured state space models for ultra-long sequence processing beyond traditional recurrent architectures.
Paper: An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling, Shaojie Bai et al. (2018). It provides a direct empirical comparison between canonical recurrent networks like LSTMs and generic temporal convolutional alternatives for sequence modeling.
Paper: Relational recurrent neural networks, Adam Santoro et al. (2018). It enhances recurrent memory cells with multi-head self-attention mechanisms to enable relational reasoning across sequential memory states.
Paper: Transformer-XL: Attentive Language Models beyond a Fixed-Length Context, Zihang Dai et al. (2019). It incorporates segment-level recurrence into self-attention models to overcome fixed-length context barriers and capture longer-range dependencies.
Paper: Memory Caching: RNNs with Growing Memory, Ali Behrouz et al.. It augments recurrent networks with segmented memory caching to overcome fixed-capacity state limits during long-context processing.
Paper: Titans: Learning to Memorize at Test Time, Ali Behrouz et al. (2024). It develops advanced test-time memorization modules to overcome the fixed-state compression bottlenecks inherent in classic recurrent architectures.

Cover for Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Abstract

Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training formulas are omitted altogether. In addition, the technique of "unrolling" an RNN is routinely presented without justification throughout the literature. The goal of this paper is to explain the essential RNN and LSTM fundamentals in a single document. Drawing from concepts in signal processing, we formally derive the canonical RNN formulation from differential equations. We then propose and prove a precise statement, which yields the RNN unrolling technique. We also review the difficulties with training the standard RNN and address them by transforming the RNN into the "Vanilla LSTM" network through a series of logical arguments. We provide all equations pertaining to the LSTM system together with detailed descriptions of its constituent entities. Albeit unconventional, our choice of notation and the method for presenting the LSTM system emphasizes ease of understanding. As part of the analysis, we identify new opportunities to enrich the LSTM system and incorporate these extensions into the Vanilla LSTM network, producing the most general LSTM variant to date. The target reader has already been exposed to RNNs and LSTM networks through numerous available resources and is open to an alternative pedagogical approach. A Machine Learning practitioner seeking guidance for implementing our new augmented LSTM model in software for experimentation and research will find the insights and derivations in this tutorial valuable as well.

Topics

blogs differential equations inference long short term memory neural networks recurrent neural networks signal processing software network architecture

Keywords

Recurrent Neural Network Long Short-Term Memory Vanilla LSTM Back Propagation Through Time Unroll Vanishing gradient Exploding gradient Derive differential equation Gate Augmented LSTM

I Introduction
II The Roots of RNN
III RNN Unfolding/Unrolling
IV RNN Training Difficulties
V From RNN to Vanilla LSTM Network
VI The Vanilla LSTM Network Mechanism in Detail
VI-A Overview
VI-B Notation
VI-C Control/Throttling (“Gate”) Nodes
VI-D Data Set Standardization
VI-E Warping (Activation) Functions
VI-F Vanilla LSTM Cell Model Parameters
VI-F1 Parameters of the accumulation node, a→cu[n]\vec{a}_{cu}[n], of the gate that controls the fractional amount of the update candidate signal, u→[n]\vec{u}[n], used to comprise the state signal of the cell at the present step with the index, nn
VI-F2 Parameters of the accumulation node, a→cs[n]\vec{a}_{cs}[n], of the gate that controls the fractional amount of the state signal of the cell, s→[n−1]\vec{s}[n-1], at the adjacent lower-indexed step, n−1n-1, used to comprise the state signal of the cell at the present step with the index, nn
VI-F3 Parameters of the accumulation node, a→cr[n]\vec{a}_{cr}[n], of the gate that controls the fractional amount of the readout candidate signal, r→[n]\vec{r}[n], used to release as the externally-accessible (observable) value signal of the cell at the present step with the index, nn
VI-F4 Parameters of the accumulation node, a→du[n]\vec{a}_{du}[n], for the data warping function that produces the update candidate signal, u→[n]\vec{u}[n], of the cell at the present step with the index, nn
VI-F5 All model parameters, which must be learned, combined (for notational convenience)
VI-G Summary of the main entities (generalized)
VI-H Vanilla LSTM System Equations (“Forward Pass”)
VI-I Vanilla LSTM System Derivatives (“Backward Pass”)
VI-J Error Gradient Sequences in Vanilla LSTM System
VII Extensions to the Vanilla LSTM Network
VII-A External Input Context Windows
VII-B Recurrent Projection Layer
VII-C Controlling External Input with a New Gate
VII-D Augmented LSTM System Equations (“Forward Pass”)
VII-E Augmented LSTM System Derivatives: Backward Pass
VIII Conclusions and Future Work
References

Knowls

Knowl 1 — Canonical Recurrent Neural Network Derivation from Delay Differential Equations
model/method
The canonical Recurrent Neural Network (RNN) formulation can be formally derived as the backward Euler discretization of a first-order nonlinear non-homogeneous ordinary delay differential equation (DDE) with discrete delay:

$\frac{d\vec{s}(t)}{dt} = A\vec{s}(t) + B\vec{r}(t - \tau_0) + C\vec{x}(t) + \vec{\phi}$

$\vec{r}(t) = G(\vec{s}(t))$

where $\vec{s}(t) \in \mathbb{R}^d$ is the continuous-time state vector, $\vec{x}(t) \in \mathbb{R}^d$ is the external input signal vector, $\vec{r}(t) \in \mathbb{R}^d$ is the readout signal vector produced by an element-wise saturating warping function $G(z) = \tanh(z)$ , $\vec{\phi} \in \mathbb{R}^d$ is a constant bias vector, $A, B, C \in \mathbb{R}^{d \times d}$ are parameter matrices, and $\tau_0 > 0$ is a discrete delay constant.

Applying the backward Euler discretization rule with sampling period $\Delta T$ set equal to the delay $\tau_0$ ( $t = n\Delta T$ , $\tau_0 = \Delta T$ ) produces the discrete difference equations:

$\vec{s}[n] = W_s\vec{s}[n-1] + W_r\vec{r}[n-1] + W_x\vec{x}[n] + \vec{\theta}_s$

$\vec{r}[n] = G(\vec{s}[n])$

where the discrete weight matrices and bias vector are defined by:

$W_s = (I - (\Delta T)A)^{-1}$

$W_r = (\Delta T)W_s B$

$W_x = (\Delta T)W_s C$

$\vec{\theta}_s = (\Delta T)W_s \vec{\phi}$

When $A$ is a diagonal matrix with large negative diagonal entries ( $a_{ii} \ll 0$ ), $W_s \approx -A^{-1} \approx 0$ , which suppresses the direct feedback from $\vec{s}[n-1]$ and reduces the canonical model to the standard RNN formulation:

$\vec{s}[n] = W_r\vec{r}[n-1] + W_x\vec{x}[n] + \vec{\theta}_s$

$\vec{r}[n] = G(\vec{s}[n])$
Knowl 2 — Approximation of Long Sequences by Unrolled Recurrent Neural Networks
theoretical result
Let a standard RNN be parameterized by $\Theta = \{W_r, W_x, \vec{\theta}_s\}$ with recurrence $\vec{s}[n] = W_r\vec{r}[n-1] + W_x\vec{x}[n] + \vec{\theta}_s$ and $\vec{r}[n] = G(\vec{s}[n])$ . Let $\mathcal{L}(\langle \vec{y}[n] \rangle, \langle \vec{v}[n] \rangle)$ be an objective function measuring the discrepancy between observable model outputs $\vec{y}[n] = Q(\vec{r}[n])$ (where $Q$ is an invertible mapping) and ground-truth sequence $\vec{v}[n]$ over an $N$ -length sequence ( $0 \le n \le N-1$ ).

Assume that:
1. There exists an optimal parameter set $\Theta$ optimizing $\mathcal{L}$ within an acceptable tolerance bound;
2. The ground truth output sequence $\vec{v}[n]$ can be partitioned into $M$ non-overlapping, mutually independent segment-level subsequences $\vec{v}_m[n]$ of finite lengths $K_m < N$ for $0 \le m \le M-1$ ;
3. The initial state of each segment is statistically independent of all other segments, enforced by setting $\vec{s}_m[j(m)-1] = \vec{0}$ , where $j(m) = \sum_{i=0}^{m-1} K_i$ for $m \ge 1$ and $j(0) = 0$ .
Then, a single reusable RNN cell, unrolled for an adjustable number of steps $K_m$ :

$\vec{s}_m[n = -1] = \vec{0}$

$\vec{s}_m[n] = W_r \vec{r}_m[n-1] + W_x \vec{x}[n + j(m)] + \vec{\theta}_s, \quad 0 \le n \le K_m - 1$

$\vec{r}_m[n] = G(\vec{s}_m[n]), \quad 0 \le n \le K_m - 1$

is computationally sufficient for finding the parameters $\Theta$ that optimize $\mathcal{L}$ over the training set and for inferring outputs from unseen inputs. Furthermore, the mutual independence of segments permits parallel evaluation across all $M$ segments.
Knowl 3 — Error Gradient Dynamics and Vanishing Gradients in Standard RNNs
theoretical result
For a standard RNN trained by Backpropagation Through Time (BPTT) with total objective function $E = \sum_{n=0}^{K_m-1} E(\vec{r}[n])$ , the error gradient with respect to the state signal $\vec{\psi}[n] \equiv \nabla_{\vec{s}[n]} E$ follows the backward recurrence:

$\vec{\psi}[n] = \left( \frac{\partial E(\vec{r}[n])}{\partial \vec{r}[n]} + W_r^T \vec{\psi}[n+1] \right) \odot \left. \frac{dG(\vec{z})}{d\vec{z}} \right|_{\vec{z}=\vec{s}[n]}$

with terminal condition $\vec{\psi}[K_m] = \vec{0}$ , where $\odot$ denotes element-wise multiplication and $G(\vec{z})$ is the element-wise hyperbolic tangent activation.

The sensitivity of the error gradient at step $n$ with respect to a perturbation at a distant step $l$ ( $l \gg n$ , $l \le K_m - 1$ ) is given by the chain product of Jacobians:

$\frac{\partial \vec{\psi}[n]}{\partial \vec{\psi}[l]} = \prod_{k=n+1}^l \left( W_r^T \operatorname{diag}\left( \left. \frac{dG(\vec{z})}{d\vec{z}} \right|_{\vec{z}=\vec{s}[k]} \right) \right)$

Because $\left| \frac{dG(z)}{dz} \right| = 1 - \tanh^2(z) \le 1$ and stability in the small-signal regime requires all eigenvalues $\mu_i$ of $W_r$ to satisfy $0 < \mu_i < 1$ (implying $\|W_r\| < 1$ ):

$\left\| \frac{\partial \vec{\psi}[n]}{\partial \vec{\psi}[l]} \right\| \sim \|W_r\|^{l-n} \cdot \left| \frac{dG}{dz} \right|^{l-n} \to 0 \quad \text{as } (l - n) \to \infty$

Consequently, the error gradient dissipates exponentially over long temporal distances (the vanishing gradient problem), preventing standard RNNs from learning long-range dependencies.
Knowl 4 — Forward Pass Equations of the Vanilla LSTM Network Cell
model/method
The Vanilla Long Short-Term Memory (LSTM) cell operates at discrete step $n \in \{0, \dots, K-1\}$ on input vector $\vec{x}[n] \in \mathbb{R}^{d_x}$ , internal state vector $\vec{s}[n] \in \mathbb{R}^{d_s}$ , and observable value vector $\vec{v}[n] \in \mathbb{R}^{d_s}$ . It incorporates peephole connections from the cell state to the gating accumulation nodes.

The forward pass is defined by the following sequence of operations:
1. Accumulation nodes for update control, state control, readout control, and candidate data update:
$\vec{a}_{cu}[n] = W_{xcu}\vec{x}[n] + W_{scu}\vec{s}[n-1] + W_{vcu}\vec{v}[n-1] + \vec{b}_{cu}$

$\vec{a}_{cs}[n] = W_{xcs}\vec{x}[n] + W_{scs}\vec{s}[n-1] + W_{vcs}\vec{v}[n-1] + \vec{b}_{cs}$

$\vec{a}_{cr}[n] = W_{xcr}\vec{x}[n] + W_{scr}\vec{s}[n] + W_{vcr}\vec{v}[n-1] + \vec{b}_{cr}$

$\vec{a}_{du}[n] = W_{xdu}\vec{x}[n] + W_{vdu}\vec{v}[n-1] + \vec{b}_{du}$
1. Activation functions:
$\vec{u}[n] = G_d(\vec{a}_{du}[n]) = \tanh(\vec{a}_{du}[n])$

$\vec{g}_{cu}[n] = G_c(\vec{a}_{cu}[n]) = \sigma(\vec{a}_{cu}[n])$

$\vec{g}_{cs}[n] = G_c(\vec{a}_{cs}[n]) = \sigma(\vec{a}_{cs}[n])$

$\vec{g}_{cr}[n] = G_c(\vec{a}_{cr}[n]) = \sigma(\vec{a}_{cr}[n])$

where $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the logistic sigmoid control warping function and $\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$ is the data warping function.
1. Cell state and observable value computation:
$\vec{s}[n] = \vec{g}_{cs}[n] \odot \vec{s}[n-1] + \vec{g}_{cu}[n] \odot \vec{u}[n]$

$\vec{r}[n] = G_d(\vec{s}[n]) = \tanh(\vec{s}[n])$

$\vec{v}[n] = \vec{g}_{cr}[n] \odot \vec{r}[n]$

where $\odot$ represents element-wise vector multiplication. The weight matrices have dimensions $W_{x\star} \in \mathbb{R}^{d_s \times d_x}$ , $W_{s\star} \in \mathbb{R}^{d_s \times d_s}$ , $W_{v\star} \in \mathbb{R}^{d_s \times d_s}$ , and bias vectors $\vec{b}_\star \in \mathbb{R}^{d_s}$ .
Knowl 5 — Backward Pass and Parameter Gradient Computation for Vanilla LSTM
model/method
The backward pass for the Vanilla LSTM unrolled for $K$ steps ( $n = K-1, \dots, 0$ ) computes total partial derivatives of objective $E$ with respect to cell signals and parameters via Backpropagation Through Time:
1. Border error gradients and readout gate accumulation derivative:
$\vec{\chi}[n] = \left( \frac{\partial \vec{y}[n]}{\partial \vec{v}[n]} \right)^T \frac{\partial E}{\partial \vec{y}[n]} + \vec{f}_\chi[n+1]$

$\vec{\alpha}_{cr}[n] = \vec{\chi}[n] \odot \vec{r}[n] \odot \left. \frac{dG_c(z)}{dz} \right|_{z=\vec{a}_{cr}[n]}$

$\vec{\psi}[n] = \vec{\chi}[n] \odot \vec{g}_{cr}[n] \odot \left. \frac{dG_d(z)}{dz} \right|_{z=\vec{s}[n]} + W_{scr}^T \vec{\alpha}_{cr}[n] + \vec{f}_\psi[n+1]$
1. Accumulation node error derivatives:
$\vec{\alpha}_{cs}[n] = \vec{\psi}[n] \odot \vec{s}[n-1] \odot \left. \frac{dG_c(z)}{dz} \right|_{z=\vec{a}_{cs}[n]}$

$\vec{\alpha}_{cu}[n] = \vec{\psi}[n] \odot \vec{u}[n] \odot \left. \frac{dG_c(z)}{dz} \right|_{z=\vec{a}_{cu}[n]}$

$\vec{\alpha}_{du}[n] = \vec{\psi}[n] \odot \vec{g}_{cu}[n] \odot \left. \frac{dG_d(z)}{dz} \right|_{z=\vec{a}_{du}[n]}$

where $\frac{dG_c(z)}{dz} = G_c(z)(1 - G_c(z))$ and $\frac{dG_d(z)}{dz} = 1 - (G_d(z))^2$ .
1. Future step backward contributions with boundary initialization $\vec{f}_\chi[K] = \vec{0}$ , $\vec{f}_\psi[K] = \vec{0}$ , $\vec{\psi}[K] = \vec{0}$ :
$\vec{f}_\chi[n+1] = W_{vcu}^T \vec{\alpha}_{cu}[n+1] + W_{vcs}^T \vec{\alpha}_{cs}[n+1] + W_{vcr}^T \vec{\alpha}_{cr}[n+1] + W_{vdu}^T \vec{\alpha}_{du}[n+1]$

$\vec{f}_\psi[n+1] = W_{scu}^T \vec{\alpha}_{cu}[n+1] + W_{scs}^T \vec{\alpha}_{cs}[n+1] + \vec{g}_{cs}[n+1] \odot \vec{\psi}[n+1]$
1. Step-level parameter gradients ( $0 \le n \le K-1$ ):
$\frac{\partial E}{\partial W_{x\star}}[n] = \vec{\alpha}_\star[n] \vec{x}^T[n], \quad \frac{\partial E}{\partial W_{s\star}}[n] = \vec{\alpha}_\star[n] \vec{s}^T[n-1] \ (\text{or } \vec{s}^T[n] \text{ for } W_{scr}), \quad \frac{\partial E}{\partial W_{v\star}}[n] = \vec{\alpha}_\star[n] \vec{v}^T[n-1], \quad \frac{\partial E}{\partial \vec{b}_\star}[n] = \vec{\alpha}_\star[n]$
1. Total parameter gradients accumulated across the sequence:
$\frac{dE}{d\Theta} = \sum_{n=0}^{K-1} \frac{\partial E}{\partial \Theta}[n]$
Knowl 6 — Constant Error Carousel and Error Gradient Dynamics in Vanilla LSTM
theoretical result
In the Vanilla LSTM network, the single-step error gradient Jacobian matrix with respect to state error gradients decomposes into two additive terms:

$\frac{\partial \vec{\psi}[k-1]}{\partial \vec{\psi}[k]} = Q(k-1, k; \tilde{\Theta}) + \operatorname{diag}(\vec{g}_{cs}[k])$

where $\tilde{\Theta} = \{W_{scu}, W_{vcu}, W_{scs}, W_{vcs}, W_{scr}, W_{vdu}\}$ and $Q(k-1, k; \tilde{\Theta})$ collects all indirect gradient propagation paths through the update gate, state gate, candidate update, and readout gate accumulation nodes.

Expanding across $l - n$ steps ( $l \gg n$ ):

$\frac{\partial \vec{\psi}[n]}{\partial \vec{\psi}[l]} = \prod_{k=n+1}^l \left( Q(k-1, k; \tilde{\Theta}) + \operatorname{diag}(\vec{g}_{cs}[k]) \right)$

When parameters or activations satisfy conditions under which $\|Q(k-1, k; \tilde{\Theta})\| < 1$ (such as when gate activations or data warping functions saturate, or when weight matrix norms are bounded by $1/2$ ), the long-range propagation is governed by the state gate:

$\left\| \frac{\partial \vec{\psi}[n]}{\partial \vec{\psi}[l]} \right\| \sim \prod_{k=n+1}^l \|\operatorname{diag}(\vec{g}_{cs}[k])\|$

In the asymptotic operating mode where $\vec{g}_{cs}[k] = \vec{1}$ and $\vec{g}_{cu}[k] = \vec{g}_{cr}[k] = \vec{0}$ , the Jacobian satisfies $\frac{\partial \vec{\psi}[k-1]}{\partial \vec{\psi}[k]} = I$ and $\vec{\psi}[n] = \vec{\psi}[l]$ for all steps. This operating regime is termed the Constant Error Carousel (CEC), in which the error gradient flows backward unattenuated across arbitrarily long sequences.
Knowl 7 — Augmented LSTM Architecture with Context Windows, Input Gating, and Recurrent Projection
model/method
The Augmented LSTM extends the Vanilla LSTM cell with three structural enhancements:
1. Non-Causal External Input Context Windows: Matrix multiplications $W_x \vec{x}[n]$ are replaced by discrete convolutions with non-causal linear FIR filters $W_x[n]$ having $L$ matrix coefficients ( $0 \le l \le L-1$ ):
$\vec{\xi}_x[n] = W_x[n] * \vec{x}[n] = \sum_{l=0}^{L-1} W_x[l] \vec{x}[n + l]$

This look-ahead mechanism incorporates $L$ future input samples $\vec{x}[n], \dots, \vec{x}[n+L-1]$ into the accumulation signals.
1. Dedicated Control Input Gate $\vec{g}_{cx}[n]$ : A fifth gate modulates the external input contribution entering the candidate data update node:
$\vec{a}_{cx}[n] = \vec{\xi}_{xcx}[n] + W_{scx}\vec{s}[n-1] + W_{vcx}\vec{v}[n-1] + \vec{b}_{cx}$

$\vec{g}_{cx}[n] = G_c(\vec{a}_{cx}[n])$

$\vec{a}_{du}[n] = \vec{g}_{cx}[n] \odot \vec{\xi}_{xdu}[n] + W_{vdu}\vec{v}[n-1] + \vec{b}_{du}$

allowing the cell to selectively suppress noisy or irrelevant external input features.
1. Recurrent Projection Layer $W_{qdr} \in \mathbb{R}^{d_v \times d_s}$ : Projects the internal qualifier signal $\vec{q}[n] \in \mathbb{R}^{d_s}$ onto an observable value signal $\vec{v}[n] \in \mathbb{R}^{d_v}$ with $d_v \le d_s$ :
$\vec{q}[n] = \vec{g}_{cr}[n] \odot \vec{r}[n]$

$\vec{v}[n] = W_{qdr}\vec{q}[n]$

reducing parameter dimensionality for all recurrent connections involving $\vec{v}[n-1]$ from $d_s$ to $d_v$ .
Knowl 8 — Forward Pass System Equations of the Augmented LSTM Network Cell
equation
The forward pass of the Augmented LSTM cell at discrete step $n \in \{0, \dots, K-1\}$ with context window length $L$ , hidden dimension $d_s$ , input dimension $d_x$ , and projection dimension $d_v \le d_s$ is specified by:

$\vec{\xi}_{xcu}[n] = \sum_{l=0}^{L-1} W_{xcu}[l]\vec{x}[n+l], \quad \vec{a}_{cu}[n] = \vec{\xi}_{xcu}[n] + W_{scu}\vec{s}[n-1] + W_{vcu}\vec{v}[n-1] + \vec{b}_{cu}, \quad \vec{g}_{cu}[n] = G_c(\vec{a}_{cu}[n])$

$\vec{\xi}_{xcs}[n] = \sum_{l=0}^{L-1} W_{xcs}[l]\vec{x}[n+l], \quad \vec{a}_{cs}[n] = \vec{\xi}_{xcs}[n] + W_{scs}\vec{s}[n-1] + W_{vcs}\vec{v}[n-1] + \vec{b}_{cs}, \quad \vec{g}_{cs}[n] = G_c(\vec{a}_{cs}[n])$

$\vec{\xi}_{xcr}[n] = \sum_{l=0}^{L-1} W_{xcr}[l]\vec{x}[n+l], \quad \vec{a}_{cr}[n] = \vec{\xi}_{xcr}[n] + W_{scr}\vec{s}[n] + W_{vcr}\vec{v}[n-1] + \vec{b}_{cr}, \quad \vec{g}_{cr}[n] = G_c(\vec{a}_{cr}[n])$

$\vec{\xi}_{xcx}[n] = \sum_{l=0}^{L-1} W_{xcx}[l]\vec{x}[n+l], \quad \vec{a}_{cx}[n] = \vec{\xi}_{xcx}[n] + W_{scx}\vec{s}[n-1] + W_{vcx}\vec{v}[n-1] + \vec{b}_{cx}, \quad \vec{g}_{cx}[n] = G_c(\vec{a}_{cx}[n])$

$\vec{\xi}_{xdu}[n] = \sum_{l=0}^{L-1} W_{xdu}[l]\vec{x}[n+l], \quad \vec{a}_{du}[n] = \vec{g}_{cx}[n] \odot \vec{\xi}_{xdu}[n] + W_{vdu}\vec{v}[n-1] + \vec{b}_{du}, \quad \vec{u}[n] = G_d(\vec{a}_{du}[n])$

$\vec{s}[n] = \vec{g}_{cs}[n] \odot \vec{s}[n-1] + \vec{g}_{cu}[n] \odot \vec{u}[n]$

$\vec{r}[n] = G_d(\vec{s}[n])$

$\vec{q}[n] = \vec{g}_{cr}[n] \odot \vec{r}[n]$

$\vec{v}[n] = W_{qdr}\vec{q}[n]$

where $G_c(z) = \frac{1}{1 + e^{-z}}$ , $G_d(z) = \tanh(z)$ , $W_{x\star}[l] \in \mathbb{R}^{d_s \times d_x}$ , $W_{s\star} \in \mathbb{R}^{d_s \times d_s}$ , $W_{v\star} \in \mathbb{R}^{d_s \times d_v}$ , $\vec{b}_\star \in \mathbb{R}^{d_s}$ , and $W_{qdr} \in \mathbb{R}^{d_v \times d_s}$ .
Knowl 9 — Backward Pass and Parameter Derivatives for the Augmented LSTM Cell
model/method
The backward pass equations for the Augmented LSTM cell unrolled for $K$ steps with context window length $L$ are defined for $n = K-1, \dots, 0$ as follows:
1. Output and projection gradients:
$\vec{\chi}[n] = \left( \frac{\partial \vec{y}[n]}{\partial \vec{v}[n]} \right)^T \frac{\partial E}{\partial \vec{y}[n]} + \vec{f}_\chi[n+1], \quad \vec{\beta}[n] = W_{qdr}^T \vec{\chi}[n]$
1. Gate and state error derivatives:
$\vec{\alpha}_{cr}[n] = \vec{\beta}[n] \odot \vec{r}[n] \odot G_c'(\vec{a}_{cr}[n])$

$\vec{\psi}[n] = \vec{\beta}[n] \odot \vec{g}_{cr}[n] \odot G_d'(\vec{s}[n]) + W_{scr}^T \vec{\alpha}_{cr}[n] + \vec{f}_\psi[n+1]$

$\vec{\alpha}_{cs}[n] = \vec{\psi}[n] \odot \vec{s}[n-1] \odot G_c'(\vec{a}_{cs}[n])$

$\vec{\alpha}_{cu}[n] = \vec{\psi}[n] \odot \vec{u}[n] \odot G_c'(\vec{a}_{cu}[n])$

$\vec{\alpha}_{du}[n] = \vec{\psi}[n] \odot \vec{g}_{cu}[n] \odot G_d'(\vec{a}_{du}[n])$

$\vec{\alpha}_{cx}[n] = \vec{\alpha}_{du}[n] \odot \vec{\xi}_{xdu}[n] \odot G_c'(\vec{a}_{cx}[n])$
1. Future recurrent contributions with boundary initialization $\vec{f}_\chi[K] = \vec{0}$ , $\vec{f}_\psi[K] = \vec{0}$ , $\vec{\psi}[K] = \vec{0}$ :
$\vec{f}_\chi[n+1] = W_{vcu}^T \vec{\alpha}_{cu}[n+1] + W_{vcs}^T \vec{\alpha}_{cs}[n+1] + W_{vcr}^T \vec{\alpha}_{cr}[n+1] + W_{vcx}^T \vec{\alpha}_{cx}[n+1] + W_{vdu}^T \vec{\alpha}_{du}[n+1]$

$\vec{f}_\psi[n+1] = W_{scu}^T \vec{\alpha}_{cu}[n+1] + W_{scs}^T \vec{\alpha}_{cs}[n+1] + W_{scx}^T \vec{\alpha}_{cx}[n+1] + \vec{g}_{cs}[n+1] \odot \vec{\psi}[n+1]$
1. Parameter gradients at step $n$ for $0 \le l \le L-1$ :
$\frac{\partial E}{\partial W_{x\star}[l]}[n] = \vec{\alpha}_\star[n]\vec{x}^T[n+l], \quad \frac{\partial E}{\partial W_{s\star}}[n] = \vec{\alpha}_\star[n]\vec{s}^T[n-1] \ (\text{or } \vec{s}^T[n] \text{ for } W_{scr}), \quad \frac{\partial E}{\partial W_{v\star}}[n] = \vec{\alpha}_\star[n]\vec{v}^T[n-1], \quad \frac{\partial E}{\partial \vec{b}_\star}[n] = \vec{\alpha}_\star[n]$

$\frac{\partial E}{\partial W_{qdr}}[n] = \vec{\chi}[n]\vec{q}^T[n]$
1. Total parameter gradients accumulated across all steps:
$\frac{dE}{d\Theta} = \sum_{n=0}^{K-1} \frac{\partial E}{\partial \Theta}[n]$
Knowl 10 — Standardization of Training Input Sequences for Recurrent Models
model/method
Before supplying sequence data to an RNN or LSTM network, raw input sample vectors $\vec{x}_0[n] \in \mathbb{R}^{d_x}$ from a training dataset containing $N$ total samples are standardized to zero mean and unit variance per feature dimension:

$\vec{\mu} = \frac{1}{N} \sum_{n=0}^{N-1} \vec{x}_0[n]$

$V = \frac{1}{N - 1} \sum_{n=0}^{N-1} (\vec{x}_0[n] - \vec{\mu}) (\vec{x}_0[n] - \vec{\mu})^T$

$\vec{x}[n] = \left[ \operatorname{diag}\left( \sqrt{V_{ii}} \right) \right]^{-1} (\vec{x}_0[n] - \vec{\mu})$

where $\vec{\mu} \in \mathbb{R}^{d_x}$ is the sample mean vector, $V \in \mathbb{R}^{d_x \times d_x}$ is the sample auto-covariance matrix, and $\operatorname{diag}\left(\sqrt{V_{ii}}\right)$ is a diagonal matrix containing the sample standard deviation $\sqrt{V_{ii}}$ for each input component $i \in \{1, \dots, d_x\}$ . Validation and test sets are standardized using the mean $\vec{\mu}$ and standard deviations $\sqrt{V_{ii}}$ computed from the training set.

Coverage note — No substantial contributed material was omitted; the knowls comprehensively cover the continuous-time DDE derivation of canonical and standard RNNs, the unrolling approximation theorem, the vanishing/exploding gradient analysis, the forward and backward passes for Vanilla and Augmented LSTM models, the Constant Error Carousel derivation, and training input standardization.

References

1.A.V. Oppenheim, R.W. Schafer. Discrete-time signal processing. Prentice-Hall, 1989.
2.Amar Gopal Bose. A Theory of Nonlinear Systems. PhD thesis, Massachusetts Institute of Technology, 1956.
3.Bo Chang, Minmin Chen, Eldad Haber, and Ed H. Chi. AntisymmetricRNN: A dynamical system view on recurrent neural networks. In International Conference on Learning Representations, Feb 2019.
4.Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, AAAI, pages 2811–2818. AAAI Press, 2018.
5.Edwin Chen. Exploring LSTMs. http://blog.echen.me/2017/05/30/exploring-lstms, 2017.
6.Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicoló Cesa-Bianchi, and Roman Garnett, editors, NeurIPS, pages 6572–6583, Dec 2018.
7.Leon O. Chua and Lin Yang. Cellular neural networks: Applications. IEEE Transactions on Circuits and Systems, 35:1273–1290, 1988.
8.Leon O. Chua and Lin Yang. Cellular neural networks: Theory. IEEE Transactions on Circuits and Systems, 35:1257–1272, 1988.
9.Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino J. Gomez. Nais-net: Stable deep networks from non-autonomous differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicoló Cesa-Bianchi, and Roman Garnett, editors, NeurIPS, pages 3029–3039, Dec 2018.
10.B. de Vries and J. C. Principe. A theory for neural networks with time delays. In R. P. Lippmann, J. E. Moody, and D. S. Touretzky, editors, Advances in Neural Information Processing Systems 3, pages 162–168. Morgan Kaufmann, 1991.
11.J.L. Elman. Finding structure in time. Cognitive Science, 14:179–211, 1990.
12.Felix Gers. Long Short-Term Memory in Recurrent Neural Networks. PhD thesis, École Polytechnique Fédérale de Lausanne, 2001.
13.A. Graves and J. Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6):602–610, 2005.
14.A. Graves and J. Schmidhuber. Framewise phoneme classification with bidirectional LSTM networks. In Proc. Int. Joint Conf. on Neural Networks IJCNN 2005, 2005.
15.Alex Graves. Supervised sequence labelling with recurrent neural networks. PhD thesis, Technical University Munich, 2008.
16.Alex Graves. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850, 2013.
17.Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. LSTM: A Search Space Odyssey. CoRR, abs/1503.04069, 2015.
18.S. Grossberg. Recurrent neural networks. Scholarpedia, 8(2):1888, 2013. revision #138057.
19.Stephen Grossberg. Nonlinear neural networks: Principles, mechanisms, and architectures. Neural Networks, 1:17–61, 1988.
20.Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.
21.Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term memory. Neural computation, 9(8):1735–1780, 1997.
22.Sepp Hochreiter, Bengio Yoshua, Frasconi Paolo, and Jürgen Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In Kremer and Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
23.J. J. Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81:3088–3092, 1984.
24.Varuna Jayasiri. Vanilla LSTM with numpy. http://blog.varunajayasiri.com/numpy_lstm.html, 2017.
25.M. I. Jordan. Serial order: A parallel distributed processing approach. Technical Report ICS Report 8604, Institute for Cognitive Science, University of California, San Diego, 1986.
26.Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, László Lukács, Marina Ganea, Peter Young, and Vivek Ramavajjala. Smart reply: Automated response suggestion for email. CoRR, abs/1606.04870, 2016.
27.Andrej Karpathy. The unreasonable effectiveness of recurrent neural networks. http://karpathy.github.io/2015/05/21/rnn-effectiveness, 2015.
28.Yuliya Kyrychko and Stephen Hogan. On the use of delay equations in engineering applications. 16:943–960, Jul 2010.
29.Omer Levy, Kenton Lee, Nicholas FitzGerald, and Luke Zettlemoyer. Long short-term memory as a dynamically computed element-wise weighted sum. CoRR, abs/1805.03716, May 2018.
30.Qianli Liao and Tomaso A. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. CoRR, abs/1604.03640, Apr 2016.
31.Henry W. Lin and Max Tegmark. Criticality in formal languages and statistical physics. Entropy, 19(7):299, Aug 2017.
32.Zachary Chase Lipton. A critical review of recurrent neural networks for sequence learning. CoRR, abs/1506.00019, 2015.
33.Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In Jennifer G. Dy and Andreas Krause, editors, ICML, volume 80 of Proceedings of Machine Learning Research, pages 3282–3291. PMLR, Nov 2018.
34.Arun Mallya. Introduction to RNNs. http://slazebni.cs.illinois.edu/spring17/lec02_rnn.pdf, 2017.
35.Arun Mallya. LSTM Forward and Backward Pass. http://arunmallya.github.io/writeups/nn/lstm/index.html, 2017.
36.Arun Mallya. Some RNN Variants. http://slazebni.cs.illinois.edu/spring17/lec03_rnn.pdf, 2017.
37.J. H. McClellan, T. W. Parks, and L. R. Rabiner. A computer program for designing optimum fir linear phase digital filters. IEEE Transactions on Audio Electroacoustics, 21(6):506–526, 1973.
38.N. Metropolis, A. Rosenbluth, M. Rosenbluth, A. Teller, and E. Teller. Equation of state calculations by fast computing machines. J. Chem. Phys., 21:1087, 1953.
39.M.L. Minsky and S.A. Papert. Perceptrons. MIT Press, Cambridge MA, second edition, 1990.
40.Christopher Olah. Understanding LSTM Networks. http://colah.github.io/posts/2015-08-Understanding-LSTMs, 2015.
41.Oleksii Ostroverkhyi. Neural network processing of multidimensional signals. PhD thesis, Kharkiv National University of Radioelectronics, 2010.
42.Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab K. Ward. Deep sentence embedding using the long short term memory network: Analysis and application to information retrieval. CoRR, abs/1502.06922, 2015.
43.Razvan Pascanu. On Recurrent and Deep Neural Networks. PhD thesis, Université de Montréal, 2014.
44.Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310–1318, 2013.
45.Barak A. Pearlmutter. Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2):263–269, 1989.
46.Barak A. Pearlmutter. Dynamic recurrent neural networks. Technical Report CMU-CS-90-196, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA, 1990.
47.F. J. Pineda. Generalization of backpropagation to recurrent neural networks. Physical Review Letters, 59(19):2229–2232, 1987.
48.Fernando L. Pineda. Generalization of backpropagation to recurrent and higher order neural networks. In Dana Z. Anderson, editor, Neural Information Processing Systems, pages 602–611. New York: American Institute of Physics, 1987.
49.L. R. Rabiner. Techniques for designing finite-duration impulse-response digital filters. IEEE Transactions on Communications Technologies, 19(2):188–195, 1971.
50.Paul Renvoisé. Machine learning spotlight i: Investigating recurrent neural networks. https://recast.ai/blog/ml-spotlight-rnn/, 2017.
51.Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent odes for irregularly-sampled time series. Jul 2019. cite arxiv:1907.03907.
52.D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. Technical Report DTIC Document, University of California San Diego, 1985.
53.D. E. Rumelhart and J. L. Mcclelland, editors. Parallel Distributed Processing, volume 1. MIT Press, 1986.
54.Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. CoRR, abs/1804.04272, 2018.
55.Hasim Sak, Andrew W. Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Haizhou Li, Helen M. Meng, Bin Ma, Engsiong Chng, and Lei Xie, editors, Interspeech, pages 338–342. ISCA, 2014.
56.Hojjat Salehinejad, Julianne Baarbe, Sharan Sankar, Joseph Barfett, Errol Colak, and Shahrokh Valaee. Recent advances in recurrent neural networks. CoRR, abs/1801.01078, 2018.
57.Terence D. Sanger. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2:459–473, 1989.
58.Mike Schuster and Kuldip K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45:2673–2681, November 1997.
59.Alex Sherstinsky. M-Lattice: A System For Signal Synthesis And Processing Based On Reaction-Diffusion. PhD thesis, Massachusetts Institute of Technology, 1994.
60.Alex Sherstinsky. Deriving the Recurrent Neural Network Definition and RNN Unrolling Using Signal Processing. In Critiquing and Correcting Trends in Machine Learning Workshop at Neural Information Processing Systems 31 (NeurIPS 2018), Dec 2018. Organizers: Benjamin Bloem-Reddy, Brooks Paige, Matt J. Kusner, Rich Caruana, Tom Rainforth, and Yee Whye Teh.
61.Alex Sherstinsky. Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network, 2018. cite arxiv:1808.03314.
62.Alex Sherstinsky and Rosalind W. Picard. M-lattice: from morphogenesis to image processing. IEEE Transactions on Image Processing, 5(7):1137–1149, 1996.
63.Alex Sherstinsky and Rosalind W. Picard. On the efficiency of the orthogonal least squares training method for radial basis function networks. IEEE Transactions on Neural Networks, 7(1):195–200, 1996.
64.Alex Sherstinsky and Rosalind W. Picard. On stability and equilibria of the M-Lattice. IEEE Transactions on Circuits and Systems – I: Fundamental Theory and Applications, 45(4):408–415, 1998.
65.S. H. Strogatz. Nonlinear Dynamics and Chaos. Westview Press, Cambridge, MA, 1994.
66.Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM Neural Networks for Language Modeling. In Interspeech, pages 194–197, 2012.
67.Ilya Sutskever. Training Recurrent Neural Networks. PhD thesis, University of Toronto, 2012.
68.Ilya Sutskever, James Martens, and Geoffrey E. Hinton. Generating text with recurrent neural networks. In Lise Getoor and Tobias Scheffer, editors, ICML, pages 1017–1024. Omnipress, 2011.
69.Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112, 2014.
70.Lloyd N. Trefethen. Finite Difference and Spectral Methods for Ordinary and Partial Differential Equations. unpublished text, Cambridge, MA, 1996.
71.Oriol Vinyals. Beyond Deep Learning: Scalable Methods and Models for Learning. PhD thesis, EECS Department, University of California, Berkeley, Dec 2013.
72.Yu Wang. A new concept using LSTM neural networks for dynamic system identification. In ACC, pages 5324–5329. IEEE, 2017.
73.P. Werbos. Backpropagation through time: what does it do and how to do it. In Proceedings of IEEE, volume 78, pages 1550–1560, 1990.
74.P. J. Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 1988.
75.Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1:270–280, 1989.
76.Y. Yamamoto, B. D. O. Anderson, M. Nagahara, and Y. Koyanagi. Optimizing fir approximation for discrete-time iir filters. IEEE Signal Processing Letters, 10(9):273–276, Sep 2003.
77.Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization, 2014. cite arxiv:1409.2329.
78.Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. CoRR, abs/1606.04199, 2016.

Citation

MLA

Sherstinsky, A. “Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network”. Physica D: Nonlinear Phenomena, vol. 404, 2020, p. 132306, https://doi.org/10.1016/j.physd.2019.132306.

APA

Sherstinsky, A. (2020). Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306. https://doi.org/10.1016/j.physd.2019.132306

Chicago

Sherstinsky, A. 2020. “Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network”. Physica D: Nonlinear Phenomena 404: 132306. https://doi.org/10.1016/j.physd.2019.132306.

Harvard

Sherstinsky, A. (2020) “Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network”, Physica D: Nonlinear Phenomena, 404, p. 132306. Available at: https://doi.org/10.1016/j.physd.2019.132306.

Vancouver

1. Sherstinsky A (2020) Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Physica D: Nonlinear Phenomena 404:132306

BibTeX

@article{Sherstinsky_2020, title={Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network}, volume={404}, ISSN={0167-2789}, url={http://dx.doi.org/10.1016/j.physd.2019.132306}, DOI={10.1016/j.physd.2019.132306}, journal={Physica D: Nonlinear Phenomena}, publisher={Elsevier BV}, author={Sherstinsky, Alex}, year={2020}, month=Mar, pages={132306} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Canonical Recurrent Neural Network Derivation from Delay Differential Equations

Knowl 2 — Approximation of Long Sequences by Unrolled Recurrent Neural Networks

Knowl 3 — Error Gradient Dynamics and Vanishing Gradients in Standard RNNs

Knowl 4 — Forward Pass Equations of the Vanilla LSTM Network Cell

Knowl 5 — Backward Pass and Parameter Gradient Computation for Vanilla LSTM

Knowl 6 — Constant Error Carousel and Error Gradient Dynamics in Vanilla LSTM

Knowl 7 — Augmented LSTM Architecture with Context Windows, Input Gating, and Recurrent Projection

Knowl 8 — Forward Pass System Equations of the Augmented LSTM Network Cell

Knowl 9 — Backward Pass and Parameter Derivatives for the Augmented LSTM Cell

Knowl 10 — Standardization of Training Input Sequences for Recurrent Models

References

Citation

BibTeX

Access the Paper