Back

Gate-variants of Gated Recurrent Unit (GRU) neural networks

Rahul DeyFathi M. Salem

article2017Midwest Symposium on Circuits and Systems2,179 citations

Proposes three parameter-efficient Gated Recurrent Unit variants that modify the update and reset gates to reduce computational cost while matching standard GRU performance.

Listen

Modern artificial intelligence applications such as natural language processing and speech recognition rely heavily on recurrent neural networks designed to process sequential data. While popular architectures like the Gated Recurrent Unit (GRU) effectively solve long-term memory challenges, their internal gating mechanisms introduce substantial computational overhead by calculating and storing large parameter sets. As AI models scale, reducing these computational demands while preserving high accuracy is critical for improving hardware efficiency and lowering deployment costs.

The article evaluates whether simplifying the internal gating signals of the GRU architecture by removing redundant parameter calculations can significantly reduce computational expense without degrading task performance.

To demonstrate this, the authors designed three simplified variants—termed GRU1, GRU2, and GRU3—which progressively strip away external inputs, recurrent connections, and internal bias terms from the standard gating equations. They evaluated these variants alongside the baseline GRU on two benchmark datasets across three distinct sequential tasks: a long pixel-by-pixel image recognition task (784-step sequence) using MNIST, a short row-by-row sequence task (28-step sequence) using MNIST, and a natural language sentiment analysis task using 50,000 movie reviews from the IMDB dataset.

The findings show that GRU1 and GRU2 achieve accuracy indistinguishable from the baseline model across all benchmarks while eliminating a significant fraction of gate parameters. On the IMDB sentiment task, all three variants achieved comparable test accuracy to the baseline (approximately 84.5% to 84.8%), even as GRU3 reduced total model parameters by roughly two-thirds (33,152 parameters versus 98,688 in the baseline). On the row-wise MNIST task, all variants maintained comparable test accuracy near 98.8%. However, on the extremely long pixel-wise MNIST task, the most stripped-down variant (GRU3) struggled to converge within 100 epochs, reaching only 59.6% accuracy at lower learning rates compared to over 98% for the other models.

These results indicate that gating mechanisms in standard GRU models carry substantial redundancy, because the network's internal recurrent state and optimization process inherently carry sufficient information to guide gate activation. Organizations deploying recurrent networks can lower memory footprints and operational costs by adopting simplified architectures like GRU1 and GRU2 without sacrificing predictive quality. Highly simplified designs like GRU3 offer dramatic parameter savings but require careful tuning and potentially longer training schedules on long sequence data.

Decision-makers should consider piloting GRU1 and GRU2 architectures in production pipelines where sequence modeling latency or memory constraints are primary concerns. Before widely adopting the most compact variant, GRU3, technical teams should conduct additional testing across extended training cycles and more diverse real-world datasets to identify when parameter reduction begins to compromise convergence speed.

Circuits, Systems, and Neural Networks (CSANN) LABMichigan State UniversityUniversity of Illinois at Urbana-Champaigngated recurrent unitrecurrent neural networkslong short term memory neural networksGated Recurrent UnitGRU RNN variantsrecurrent neural networks
arXiv: 1701.05923
  • Paper: GLU Variants Improve Transformer, Noam Shazeer. This book chapter extends the exploration of gate-variant architectures by demonstrating how gated linear units improve modern transformer models.
  • Paper: DeepLoop: Depth Scaling for Looped Transformers, Shuzhen Li et al. (2026). This follow-up research generalises recurrent depth scaling techniques to looped architectures, building directly upon foundational insights in recurrent gating.
  • Paper: Generative Recursive Reasoning, Junyeob Baek et al. (2026). This ebook chapter continues the investigation of recurrent reasoning frameworks by introducing probabilistic transitions into iterative latent updates.
Cover for Gate-variants of Gated Recurrent Unit (GRU) neural networks

Abstract

The paper evaluates three variants of the Gated Recurrent Unit (GRU) in recurrent neural networks (RNN) by reducing parameters in the update and reset gates. We evaluate the three variant GRU models on MNIST and IMDB datasets and show that these GRU-RNN variant models perform as well as the original GRU RNN model while reducing the computational expense.

Topics

gated recurrent unitrecurrent neural networkslong short term memory neural networks

Keywords

Gated Recurrent UnitGRU RNN variantsrecurrent neural networksLong Short-Term MemoryMinimal Gated UnitMNIST datasetactivation functiongradient descent

Table of Contents

  • I. INTRODUCTION
  • II. BACKGROUND: RNN, LSTM AND GRU
  • A. Long Short-Term Memory (LSTM) RNN
  • B. Gated Recurrent Unit (GRU) RNN
  • III. THE VARIANT GRU ARCHITECTURES
  • IV. RESULTS AND DISCUSSION
  • A. Application to MNIST Dataset - pixel-wise sequences
  • B. Application to MNIST Dataset - row-wise sequences
  • C. Application to the IMDB Dataset- natural sequence
  • V. CONCLUSION
  • REFERENCES

Knowls

  1. Knowl 1 — Gate-Variant Gated Recurrent Unit Architectures (GRU1, GRU2, GRU3)

    model/method

    Standard Gated Recurrent Unit (GRU0) models use an update gate zt∈Rnz_t \in \mathbb{R}^n and a reset gate rt∈Rnr_t \in \mathbb{R}^n to compute the hidden state ht∈Rnh_t \in \mathbb{R}^n from an input vector xt∈Rmx_t \in \mathbb{R}^m:

    ht=(1−zt)⊙ht−1+zt⊙h~th_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t

    h~t=g(Whxt+Uh(rt⊙ht−1)+bh)\tilde{h}_t = g(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)

    where ⊙\odot denotes the element-wise Hadamard product, gg is an activation function (such as ReLU), Wh∈Rn×mW_h \in \mathbb{R}^{n \times m}, Uh∈Rn×nU_h \in \mathbb{R}^{n \times n}, and bh∈Rnb_h \in \mathbb{R}^n. Standard GRU0 gates are defined as:

    zt=σ(Wzxt+Uzht−1+bz)z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)

    rt=σ(Wrxt+Urht−1+br)r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)

    where σ\sigma is the logistic sigmoid function, requiring 3(n2+nm+n)3(n^2 + nm + n) total parameters.

    Three parameter-reduced gate variants simplify the gating equations while leaving the hidden candidate state equation unchanged:

    1. GRU1 (Recurrent State and Bias Only): Gating signals exclude the direct input projection terms:

    zt=σ(Uzht−1+bz)z_t = \sigma(U_z h_{t-1} + b_z)

    rt=σ(Urht−1+br)r_t = \sigma(U_r h_{t-1} + b_r)

    Parameter count is reduced by 2nm2nm, resulting in 3n2+nm+3n3n^2 + nm + 3n total parameters.

    1. GRU2 (Recurrent State Only): Gating signals exclude both direct input projections and gate bias vectors:

    zt=σ(Uzht−1)z_t = \sigma(U_z h_{t-1})

    rt=σ(Urht−1)r_t = \sigma(U_r h_{t-1})

    Parameter count is reduced by 2(nm+n)2(nm + n), resulting in 3n2+nm+n3n^2 + nm + n total parameters.

    1. GRU3 (Bias Only): Gating signals exclude all input and recurrent weight matrices, retaining only learnable gate bias vectors:

    zt=σ(bz)z_t = \sigma(b_z)

    rt=σ(br)r_t = \sigma(b_r)

    Parameter count is reduced by 2(nm+n2)2(nm + n^2), resulting in n2+nm+3nn^2 + nm + 3n total parameters.

  2. Knowl 2 — Cost-Dependent Exponential Learning Rate Decay

    equation

    To accelerate recurrent network optimization when using adaptive optimizers such as RMSProp, the learning rate η\eta is decayed exponentially at each epoch based on the network's loss from the prior epoch:

    η=η0e−cost\eta = \eta_0 e^{-\text{cost}}

    where η0\eta_0 is the constant base learning rate and cost\text{cost} is the scalar training objective loss (such as categorical cross-entropy or binary cross-entropy) evaluated at the preceding epoch.

  3. Knowl 3 — Experimental Configuration for GRU Gate-Variant Evaluation

    experimental setup

    The GRU variants (GRU1, GRU2, GRU3) and the baseline GRU (GRU0) are evaluated on three sequence tasks using a single recurrent layer:

    • MNIST Pixel-Wise Classification: 60,000 training and 10,000 testing 28×2828 \times 28 grayscale handwritten digit images formatted as 1D pixel sequences of length 784 (m=1m = 1). Hidden units n=100n = 100, ReLU hidden activation, categorical cross-entropy loss, trained for 100 epochs.
    • MNIST Row-Wise Classification: The same MNIST dataset presented as 28 time steps of 28-dimensional row vectors (m=28m = 28). Hidden units n=100n = 100, ReLU hidden activation, categorical cross-entropy loss, trained for 50 epochs.
    • IMDB Sentiment Classification: 25,000 training and 25,000 test movie reviews categorized for binary sentiment, with reviews limited to a maximum sequence length of 80 words from a 20,000-word vocabulary. Input embedding dimension m=128m = 128, hidden units n=128n = 128, ReLU hidden activation, binary cross-entropy loss, trained for 100 epochs.

    All models utilize logistic sigmoid gate activations, RMSprop optimizer with cost-dependent exponential decay, 20% dropout, batch size 32, and an appropriate terminal output layer (Softmax for MNIST, logistic sigmoid for IMDB).

  4. Knowl 4 — Classification Performance on MNIST Pixel-Wise Sequences

    data/table

    On long sequential inputs (length 784, input dimension m=1m=1, hidden dimension n=100n=100), GRU1 and GRU2 achieve accuracy indistinguishable from baseline GRU0 while using fewer parameters. GRU3 requires smaller base learning rates η0≤10−4\eta_0 \le 10^{-4} to make steady progress, reaching 59.60% test accuracy at η0=5×10−5\eta_0 = 5\times 10^{-5} after 100 epochs without saturating.

    Architecture η0=10−3\mathbf{\eta_0 = 10^{-3}} η0=5×10−4\mathbf{\eta_0 = 5\times 10^{-4}} 1e-4 5e-5 # Params
    Train (%) Test (%) Train (%) Test (%) Train (%) Test (%)
    GRU0 99.19 98.59 98.59 98.04 – – 30,600
    GRU1 98.88 98.37 98.52 – – – 30,400
    GRU2 98.90 98.10 98.61 – – – 30,200
    GRU3 – – 10.44 – 60.97 59.60 10,400

  5. Knowl 5 — Classification Performance on MNIST Row-Wise Sequences

    data/table

    On short vector sequences (length 28, input dimension m=28m=28, hidden dimension n=100n=100), all three GRU variants demonstrate performance comparable to standard GRU0 across various base learning rates over 50 epochs. GRU3 achieves 98.85% test accuracy at η0=10−2\eta_0 = 10^{-2} while requiring only 13,100 parameters, saving approximately 66% of the parameters of GRU0 (38,700 parameters).

    Architecture η0=10−2\mathbf{\eta_0 = 10^{-2}} η0=10−3\mathbf{\eta_0 = 10^{-3}} η0=10−4\mathbf{\eta_0 = 10^{-4}} # Params
    Train (%) Test (%) Train (%) Test (%) Train (%) Test (%)
    GRU0 96.99 98.49 98.14 98.85 93.02 96.66 38,700
    GRU1 97.24 98.55 97.46 98.93 91.54 96.58 33,100
    GRU2 96.95 98.71 97.33 98.93 91.20 96.23 32,900
    GRU3 97.19 98.85 97.04 97.39 80.33 87.96 13,100

  6. Knowl 6 — Sentiment Classification Performance on the IMDB Dataset

    data/table

    For natural sequence sentiment classification on the IMDB benchmark (m=128m=128, n=128n=128, maximum sequence length 80), all three GRU variants yield test accuracies comparable to standard GRU0 over 100 epochs, providing substantial parameter and computational reductions.

    Architecture η0=10−3\mathbf{\eta_0 = 10^{-3}} η0=10−4\mathbf{\eta_0 = 10^{-4}} # Params
    Train (%) Test (%) Train (%) Test (%)
    GRU0 95.3 83.7 87.4 84.8 98,688
    GRU1 94.5 84.1 87.0 84.8 65,920
    GRU2 94.5 84.2 86.9 84.6 65,664
    GRU3 92.3 83.2 86.8 84.5 33,152

    At η0=10−4\eta_0 = 10^{-4}, GRU3 achieves 84.5% test accuracy with 33,152 parameters, reducing the parameter count by 66.4% relative to baseline GRU0 (98,688 parameters).

  7. Knowl 7 — Sequence Length Sensitivity and Learning Dynamics of Bias-Only Gating (GRU3)

    empirical result

    The bias-only gate variant GRU3 (zt=σ(bz),rt=σ(br)z_t = \sigma(b_z), r_t = \sigma(b_r)) exhibits strong sensitivity to sequence length and learning rate dynamics:

    • Sequence Length Impact: On moderate-to-short sequence lengths (length 28 on MNIST row-wise and length ≤80\le 80 on IMDB), GRU3 achieves classification accuracy comparable to standard GRU0 (98.85% vs. 98.49% on MNIST row-wise; 84.5% vs. 84.8% on IMDB). On long sequences (length 784 on MNIST pixel-wise), GRU3 lags behind within standard epoch budgets.
    • Learning Rate Sensitivity: On long sequences, GRU3 fails to train at larger base learning rates (η0=10−3\eta_0 = 10^{-3} and 5×10−45\times 10^{-4}), but exhibits consistent learning when η0\eta_0 is reduced to ≤10−4\le 10^{-4}.
    • Non-Saturation: Across evaluations where GRU3 accuracy lags behind GRU0 at the end of the epoch limit, its accuracy curve maintains a positive slope without plateauing, indicating that additional training epochs and smaller learning rates allow it to continue closing the performance gap.

Coverage note — No substantial contributed material was omitted.

References

  1. 1.Bengio, Y., Simard, P., and Frasconi, P. Learning Longterm Dependencies with Gradient Descent is Difficult. IEEE Trans.Neural Networks, 5(2):157–166, 1994. H. Simpson, Dumb Robots, 3rd ed., Springfield: UOS Press, 2004, pp.6-9.
  2. 2.Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv preprint arXiv:1412.3555, 2014. Gers, F. A., Schraudolph, N. N., and Schmidhuber, J. Learning Precise Timing with LSTM Recurrent Networks. Journal of Machine Learning Research, 3:115–143, 2002. J.-G. Lu, “Title of paper with only the first word capitalized,” J. Name Stand. Abbrev., in press.
  3. 3.Hochreiter, S. and Schmidhuber, J. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, 1997
  4. 4.Jozefowicz, R., Zaremba, W., and Sutskever, I. An Empirical Exploration of Recurrent Network Architectures. In Proc., Int’l Conf. on Machine Learning, pp. 2342–2350, 2015.
  5. 5.Le, Q. V., Jaitly, N., and Hinton, G. E. A Simple Way to Initialize Recurrent Networks of Rectified Linear Units. arXiv preprint arXiv:1504.00941, 2015.
  6. 6.Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning Word Vectors for Sentiment Analysis. In Proc. 49th Annual Meeting of the ACL, pp. 142–150, 2011.
  7. 7.Mikolov, T., Joulin, A., Chopra, S., Mathieu, M., and Ranzato, M. Learning Longer Memory in Recurrent Neural Networks. In Int’l Conf Learning Represenations, 2015.
  8. 8.Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent Neural Network Regularization. arXiv preprint arXiv:1409.2329, 2014.
  9. 9.Boulanger-Lewandowski, Nicolas, Bengio, Yoshua, and Vincent, Pascal. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. arXiv preprint arXiv:1206.6392, 2012.
  10. 10.Gers, Felix A, Schmidhuber, J¨urgen, and Cummins, Fred. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471, 2000.
  11. 11.Mikolov, Tomas, Joulin, Armand, Chopra, Sumit, Mathieu, Michael, and Ranzato, Marc’Aurelio. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753, 2014.
  12. 12.Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2012.
  13. 13.Zhou G. B., Wu J., Zhang C. L., and Zhou Z. H. Minimal Gated Unit for Recurrent Neural Networks. arXiv preprint arXiv:1603.09420v1 [cs.NE] 31 Mar 2016
  14. 14.https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py.
  15. 15.F. M. Salem, `“Reduced Parameterization in Gated Recurrent Neural Networks,” Memorandum 7.11.2016, MSU, Nov 2016.
  16. 16.F. M. Salem, “A Basic Recurrent Neural Network Model,” arXiv Preprint arXiv: 1612.09022, Dec. 2016.

Citation

MLA
Dey, R., and F. M. Salem. “Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks”. arXiv, 2017, http://arxiv.org/abs/1701.05923v1.
APA
Dey, R., & Salem, F. M. (2017). Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks. arXiv. http://arxiv.org/abs/1701.05923v1
Chicago
Dey, R., and F. M. Salem. 2017. “Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks”. arXiv. http://arxiv.org/abs/1701.05923v1.
Harvard
Dey, R. and Salem, F.M. (2017) “Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1701.05923v1.
Vancouver
1. Dey R, Salem FM (2017) Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks. arXiv

BibTeX

@article{dey2017gate,
  title = {Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks},
  author = {Dey, Rahul and Salem, Fathi M.},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1701.05923v1},
  eprint = {1701.05923}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF