Back

On the Variance of the Adaptive Learning Rate and Beyond

Liyuan LiuHaoming JiangPengcheng HeWeizhu ChenXiaodong LiuJianfeng GaoJiawei Han

article2019ICLR2,306 citations

Proposes RAdam, a variant of the Adam optimizer that dynamically corrects the high variance of early adaptive learning rates to stabilize training without requiring manual warmup heuristics.

Listen

Training deep neural networks using adaptive optimization algorithms, such as Adam and RMSprop, often suffers from early-stage instability and poor convergence unless practitioners manually apply a learning rate warmup heuristic. Because warmup relies on empirical trial-and-error without a clear theoretical justification, selecting optimal training parameters requires substantial engineering effort and compute resources.

The article investigates the root cause of early training instability in adaptive optimizers and evaluates a principled variance-reduction method to eliminate the reliance on manual warmup schedules.

The researchers conducted theoretical statistical analyses modeling gradient updates and verified their findings through empirical experiments across diverse tasks, including language modeling on the One Billion Word dataset, image classification on CIFAR-10 and ImageNet, and neural machine translation on IWSLT'14 and WMT'16 datasets. They evaluated baseline optimizers (standard Adam and Stochastic Gradient Descent), heuristic warmup schedules, and a novel variance-rectified optimizer named Rectified Adam (RAdam).

The analysis produced several key findings: First, the primary cause of early-stage training failure is the undesirably large, sometimes unbounded, variance of the adaptive learning rate when sample sizes are small in initial iterations. Second, the article demonstrates mathematically and empirically that warmup functions primarily as a variance-reduction mechanism. Third, the proposed RAdam optimizer dynamically rectifies this variance by deactivating the adaptive rate when variance is intractable and scaling it when tractable, matching the translation performance of tuned warmup baselines (e.g., 34.76 BLEU on IWSLT'14 DE-EN compared to 34.66 for warmup). Finally, RAdam demonstrates superior robustness, maintaining high test accuracy across learning rates from 0.003 to 0.1 on CIFAR-10 where standard Adam and SGD performance degrades significantly.

These findings indicate that manual tuning of warmup schedules is unnecessary when variance is explicitly controlled. By removing an empirical hyperparameter, RAdam reduces experimentation cycles, lowers compute costs, and avoids severe early gradient distortion across different model architectures.

Engineering and research teams should consider adopting RAdam or similar variance-rectification techniques in place of standard Adam with manual warmup to streamline training pipelines. Future development should explore extending variance stabilization by sharing second-moment estimations across related network parameters.

The conclusions are well-supported across multiple benchmarks and theoretical derivations, though some approximations rely on standard statistical assumptions about gradient distributions. While RAdam consistently outperforms standard Adam and matches heuristic warmup, practitioners training convolutional vision models should note that standard Stochastic Gradient Descent may still achieve slightly higher final test generalization despite RAdam's superior training loss.

Georgia Institute of TechnologyMicrosoftUniversity of Illinois Urbana-Champaignadaptive learningadaptive learning ratesclassificationLearning RateAdaptive Learning RateVariance
arXiv: 1908.03265LiyuanLucasLiu/RAdam
  • Paper: Adam: A Method for Stochastic Optimization, Diederik P. Kingma et al. (2015). Reading the original Adam paper is essential because the source directly analyzes its early-stage learning rate variance and proposes an improved variant.
  • Paper: On the Convergence of Adam and Beyond, Sashank J. Reddi et al. (2018). This paper examines convergence issues in adaptive optimizers like Adam, providing the foundational critique that motivates the variance analysis in the source.
  • Paper: Decoupled Weight Decay Regularization, Ilya Loshchilov et al. (2019). This paper builds on Adam-family optimization by demonstrating the importance of decoupled weight decay to fix generalization gaps.
Cover for On the Variance of the Adaptive Learning Rate and Beyond

Abstract

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: this https URL.

Topics

adaptive learningadaptive learning ratesclassificationimage classificationlanguage modelmachine translationsoptimizationstochasticstochastic optimization algorithmonline learning

Keywords

Learning RateAdaptive Learning RateVarianceAdamWarmupRAdamTrainingVarFigureBeyond

Table of Contents

  • 1 Introduction
  • 2 Preliminaries and Motivations
  • 3 Variance of the Adaptive Learning Rate
  • 3.1 Warmup as Variance Reduction
  • 3.2 Analysis of Adaptive Learning Rate Variance
  • 4 Rectified Adaptive Learning Rate
  • 4.1 Estimation of ρ\rho
  • 4.2 Variance Estimation and Rectification
  • 4.3 In Comparison with Warmup and Other Stabilization Techniques
  • 5 Experiments
  • 5.1 Comparing to Vanilla Adam
  • 5.2 Comparing to Heuristic Warmup
  • 5.3 Simulated Verification
  • 6 Conclusion
  • References
  • A Proof of Theorem
  • B Implementation Details
  • B.1 Language Modeling
  • B.2 Imageine Classification
  • B.3 Neural Machine Translation
  • C Downgrading to SGDM

Knowls

  1. Knowl 1 — Rectified Adam Optimization Algorithm

    algorithm

    Rectified Adam (RAdam) is an adaptive stochastic optimization algorithm that explicitly stabilizes early training by rectifying the variance of Adam's adaptive learning rate. When the estimated degrees of freedom ρt\rho_t of the second moment estimator is small (ρt≤4\rho_t \le 4), the variance of the adaptive learning rate is theoretically divergent, and RAdam turns off the adaptive learning rate term, updating parameters via stochastic gradient descent with momentum alone. When ρt>4\rho_t > 4, RAdam computes a variance rectification factor rtr_t to scale the adaptive learning rate so that its variance remains consistent throughout training.

    Input: Step size sequence {αt\alpha_t}t=1T_{t=1}^T, decay rates β1,β2∈[0,1)\beta_1, \beta_2 \in [0, 1), initial parameter vector θ0∈Rd\theta_0 \in \mathbb{R}^d, stochastic objective function ft(θ)f_t(\theta).
    Output: Parameter vector θT\theta_T.
    m0←0m_0 \leftarrow 0
    v0←0v_0 \leftarrow 0
    ρ∞←21−β2−1\rho_\infty \leftarrow \frac{2}{1 - \beta_2} - 1
    for t=1t = 1 to TT do
        gt←∇θft(θt−1)g_t \leftarrow \nabla_\theta f_t(\theta_{t-1})
        vt←β2vt−1+(1−β2)gt2v_t \leftarrow \beta_2 v_{t-1} + (1 - \beta_2) g_t^2
        mt←β1mt−1+(1−β1)gtm_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t
        m^t←mt/(1−β1t)\hat{m}_t \leftarrow m_t / (1 - \beta_1^t)
        ρt←ρ∞−2tβ2t1−β2t\rho_t \leftarrow \rho_\infty - \frac{2 t \beta_2^t}{1 - \beta_2^t}
        if ρt>4\rho_t > 4 then
            lt←(1−β2t)/vtl_t \leftarrow \sqrt{(1 - \beta_2^t) / v_t}
            rt←(ρt−4)(ρt−2)ρ∞(ρ∞−4)(ρ∞−2)ρtr_t \leftarrow \sqrt{\frac{(\rho_t - 4)(\rho_t - 2)\rho_\infty}{(\rho_\infty - 4)(\rho_\infty - 2)\rho_t}}
            θt←θt−1−αtrtm^t⊙lt\theta_t \leftarrow \theta_{t-1} - \alpha_t r_t \hat{m}_t \odot l_t
        else
            θt←θt−1−αtm^t\theta_t \leftarrow \theta_{t-1} - \alpha_t \hat{m}_t
    return θT\theta_T

    All vector operations (powers, element-wise products ⊙\odot, and divisions) are executed element-wise. Common default decay parameters are β1=0.9\beta_1 = 0.9 and β2=0.999\beta_2 = 0.999. When β2≤0.6\beta_2 \le 0.6, ρ∞≤4\rho_\infty \le 4, which degenerates RAdam entirely to stochastic gradient descent with momentum.

  2. Knowl 2 — Variance Rectification Factor for Adaptive Learning Rates

    equation

    In Rectified Adam (RAdam), the variance of the adaptive learning rate term ψ(g1,…,gt)=1−β2tvt\psi(g_1, \dots, g_t) = \sqrt{\frac{1 - \beta_2^t}{v_t}} is equalized across time steps to match its minimal asymptotic variance Cvar=Var[ψ]∣ρt=ρ∞C_{\text{var}} = \left. \text{Var}[\psi] \right|_{\rho_t = \rho_\infty} by multiplying the parameter update by the variance rectification factor rtr_t:

    rt=(ρt−4)(ρt−2)ρ∞(ρ∞−4)(ρ∞−2)ρtr_t = \sqrt{\frac{(\rho_t - 4)(\rho_t - 2)\rho_\infty}{(\rho_\infty - 4)(\rho_\infty - 2)\rho_t}}

    where β2∈(0,1)\beta_2 \in (0, 1) is the exponential moving average decay parameter for the squared gradient second moment, t∈N+t \in \mathbb{N}^+ is the optimization time step, ρ∞=21−β2−1\rho_\infty = \frac{2}{1 - \beta_2} - 1 is the maximum asymptotic degrees of freedom of the approximated moving average, and ρt=ρ∞−2tβ2t1−β2t\rho_t = \rho_\infty - \frac{2t\beta_2^t}{1 - \beta_2^t} is the effective degrees of freedom at step tt. This rectification term is computed and applied when ρt>4\rho_t > 4.

  3. Knowl 3 — Monotonicity and Analytical Form of Adaptive Learning Rate Variance

    theoretical result

    Assuming stochastic gradients gi∼N(0,σ2)g_i \sim \mathcal{N}(0, \sigma^2) are independent and identically distributed, the squared adaptive rate ψ2(g1,…,gt)=1−β2t(1−β2)∑i=1tβ2t−igi2\psi^2(g_1, \dots, g_t) = \frac{1 - \beta_2^t}{(1 - \beta_2)\sum_{i=1}^t \beta_2^{t-i} g_i^2} follows a scaled inverse chi-square distribution Scale-inv-χ2(ρ,1/σ2)\text{Scale-inv-}\chi^2(\rho, 1/\sigma^2) with ρ>0\rho > 0 degrees of freedom.

    For any ρ>4\rho > 4, the exact variance of ψ(g1,…,gt)\psi(g_1, \dots, g_t) is given by:

    Var[ψ(.)]=τ2(ρρ−2−ρ22ρ−5πB(ρ−12,ρ−12)2)\text{Var}[\psi(.)] = \tau^2 \left( \frac{\rho}{\rho - 2} - \frac{\rho}{2^{2\rho - 5} \pi} B\left(\frac{\rho - 1}{2}, \frac{\rho - 1}{2}\right)^2 \right)

    where τ2=1/σ2\tau^2 = 1/\sigma^2 and B(⋅,⋅)B(\cdot, \cdot) is the beta function. Var[ψ(.)]\text{Var}[\psi(.)] monotonically decreases as ρ\rho increases.

    Using a first-order Taylor expansion approximation for ψ2(.)\sqrt{\psi^2(.)}, the variance is approximated by:

    Var[ψ(.)]≈ρ2(ρ−2)(ρ−4)σ2\text{Var}[\psi(.)] \approx \frac{\rho}{2(\rho - 2)(\rho - 4)\sigma^2}

    which decays at rate O(1/ρ)\mathcal{O}(1/\rho). For ρ≤4\rho \le 4, the theoretical variance of the adaptive learning rate is intractable or divergent.

  4. Knowl 4 — Estimation of Effective Degrees of Freedom via Moving Average Center of Mass

    model/method

    To estimate the effective degrees of freedom ρt\rho_t for the exponential moving average (EMA) of squared gradients at step tt, the EMA is approximated by a simple moving average (SMA) of window length f(t,β2)f(t, \beta_2). The length f(t,β2)f(t, \beta_2) is determined by matching the center of mass of the EMA weights with that of an SMA of length f(t,β2)f(t, \beta_2):

    (1−β2)∑i=1tβ2t−i⋅i1−β2t=∑i=1f(t,β2)(t+1−i)f(t,β2)\frac{(1 - \beta_2) \sum_{i=1}^t \beta_2^{t-i} \cdot i}{1 - \beta_2^t} = \frac{\sum_{i=1}^{f(t, \beta_2)} (t + 1 - i)}{f(t, \beta_2)}

    Solving this equation yields the effective degrees of freedom ρt=f(t,β2)\rho_t = f(t, \beta_2) at time step tt given decay parameter β2∈(0,1)\beta_2 \in (0, 1):

    ρt=21−β2−1−2tβ2t1−β2t\rho_t = \frac{2}{1 - \beta_2} - 1 - \frac{2t\beta_2^t}{1 - \beta_2^t}

    As t→∞t \to \infty, ρt\rho_t converges to its asymptotic maximum ρ∞=21−β2−1\rho_\infty = \frac{2}{1 - \beta_2} - 1.

  5. Knowl 5 — Variance-Driven Instability of Adaptive Learning Rates and the Warmup Mechanism

    model/method

    The early-stage failure of adaptive gradient methods (e.g., Adam) to converge, or their convergence to suspicious/suboptimal local optima in architectures like Transformers, is caused by the large and unbounded variance of the adaptive learning rate ψ(g1,…,gt)\psi(g_1, \dots, g_t) when computed with very few gradient samples. At t=1t = 1, assuming zero-mean normal gradients g1∼N(0,σ2)g_1 \sim \mathcal{N}(0, \sigma^2), the adaptive factor 1/g12\sqrt{1/g_1^2} follows a scaled inverse chi distribution with 1 degree of freedom, whose variance is divergent.

    This unbounded variance distorts the gradient distribution within the first 10 update steps into small absolute values. Learning rate warmup (e.g., linear warmup αt=tα0/Tw\alpha_t = t\alpha_0 / T_w for t<Twt < T_w) functions primarily as a variance reduction mechanism because Var[αtx]=αt2Var[x]\text{Var}[\alpha_t x] = \alpha_t^2 \text{Var}[x].

    Controlled experiments confirm that insufficient samples in estimating the adaptive rate cause this failure:

    • Adam-2k: Updating only the second moment vtv_t while keeping parameters θ\theta and momentum mtm_t frozen for the first 2,000 steps completely prevents gradient distortion and avoids convergence failure.
    • Adam-eps: Setting a large numerical stability constant ϵ=10−4\epsilon = 10^{-4} (instead of 10−810^{-8}) bounds the variance of the adaptive rate, preventing divergence but introducing significant estimation bias that degrades optimization speed and final performance.

  6. Knowl 6 — Neural Machine Translation Performance of RAdam versus Adam with Warmup

    data/table

    In neural machine translation using Transformer architectures, vanilla Adam without warmup fails to optimize properly, getting trapped in suboptimal local optima with training perplexity around 500 and BLEU scores near zero. RAdam resolves this convergence issue without requiring a warmup schedule hyperparameter TwT_w, achieving BLEU scores comparable to or slightly exceeding Adam configured with a 4,000-step linear warmup.

    Method IWSLT'14 DE-EN IWSLT'14 EN-DE WMT'16 EN-DE
    Adam with warmup 34.66±0.01434.66 \pm 0.014 28.56±0.06728.56 \pm 0.067 27.03
    RAdam 34.76±0.00334.76 \pm 0.003 28.48±0.05428.48 \pm 0.054 27.27

    IWSLT'14 evaluations report mean and standard deviation BLEU over 5 random seeds using a 6-layer Transformer encoder/decoder with 4 heads and hidden/feedforward dimensions of 512/1024. WMT'16 EN-DE uses a 6-layer Transformer with 8 heads and dimensions of 512/2048. Both setups employ label smoothing with uncertainty 0.1 and initial learning rate 3×10−43 \times 10^{-4}.

  7. Knowl 7 — Image Classification Accuracy on CIFAR-10 and ImageNet

    data/table

    On image classification benchmarks with residual networks, RAdam consistently outperforms vanilla Adam in test accuracy.

    Dataset Method Top-1 Accuracy (%)
    CIFAR-10 SGD 91.51
    CIFAR-10 Adam 90.54
    CIFAR-10 RAdam 91.38
    ImageNet SGD 69.86
    ImageNet Adam 66.54
    ImageNet RAdam 67.62

    Evaluations use ResNet-20 (batch size 128, 186 epochs, learning rate decayed by 0.1 at epochs 81 and 122) on CIFAR-10, and ResNet-18 (batch size 256, 90 epochs, learning rate decayed by 0.1 at epochs 31 and 61) on ImageNet. A decoupled weight decay of 10−410^{-4} is used. On ImageNet, while SGD achieves higher test accuracy due to the known generalization gap of adaptive methods, RAdam achieves higher training accuracy (70.30%) than both SGD (69.57%) and Adam (69.12%).

  8. Knowl 8 — Learning Rate Robustness of RAdam across Orders of Magnitude

    empirical result

    When training ResNet-20 on CIFAR-10 across four initial learning rates α0∈{0.1,0.03,0.01,0.003}\alpha_0 \in \{0.1, 0.03, 0.01, 0.003\}, RAdam exhibits high robustness: its training loss and test accuracy curves across all four learning rates nearly overlap throughout the 180-epoch training schedule. In contrast, vanilla Adam and SGD show strong sensitivity to the learning rate, suffering severe performance drops at higher learning rates. Furthermore, while Adam with a heuristic linear warmup is sensitive to the warmup length hyperparameter (e.g., Adam with 100 warmup steps achieves only 90.13% test accuracy at α0=0.1\alpha_0 = 0.1), RAdam achieves 91.06% under the same settings without requiring warmup hyperparameter tuning.

  9. Knowl 9 — Language Modeling Performance on the One Billion Word Benchmark

    empirical result

    On the One Billion Word benchmark trained with a 2-layer LSTM (2048 hidden units, 300-dimensional embeddings, adaptive softmax cutoffs at [4000, 40000, 200000], batch size 128, initial learning rate 0.001 with decay at epoch 10), RAdam achieves a test perplexity of 35.70 compared to 36.92 for vanilla Adam. Although the variance rectification factor causes RAdam to update more conservatively than vanilla Adam during the first few epochs, it prevents early parameter distortion and enables faster training loss convergence and lower perplexity throughout the remainder of training.

  10. Knowl 10 — Impact of Disabling Adaptive Updates During Divergent Variance Steps

    data/table

    An ablation study on CIFAR-10 using ResNet-20 with initial learning rate α0=0.1\alpha_0 = 0.1 evaluates the necessity of deactivating adaptive updates during the initial steps where the variance is theoretically divergent (ρt≤4\rho_t \le 4, steps 1–4) versus steps where variance is convergent (ρt>4\rho_t > 4, steps 5–8).

    Steps 1–4 Steps 5–8 Steps 8+ Test Acc (%) Train Loss Train Error (%)
    RAdam RAdam RAdam 91.08 0.021 0.74
    Adam (divergent var.) RAdam RAdam 89.98 0.060 2.12
    SGD Adam (convergent var.) RAdam 90.29 0.038 1.23

    Results averaged over 5 runs show that allowing unrectified Adam updates during steps 1–4 reduces final test accuracy from 91.08% to 89.98% and almost triples the training loss from 0.021 to 0.060. Applying unrectified Adam in steps 5–8 is significantly less damaging (90.29% test accuracy), confirming that gradient updates during the initial divergent-variance phase are the primary source of optimization degradation.

Coverage note — No substantial contributed material was omitted; all key theoretical derivations, algorithmic components, simulation checks, and experimental benchmarks across computer vision, NLP language modeling, and neural machine translation are represented.

References

  1. 1.Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  2. 2.David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In ICML, 2017.
  3. 3.Yoshua Bengio, Nicolas Boulanger-Lewandowski, and Razvan Pascanu. Advances in optimizing recurrent networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8624–8628. IEEE, 2013.
  4. 4.Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems. In ICML, 2018.
  5. 5.Augustin Cauchy. Methode generale pour la resolution des systemes d’equations simultanees. Comp. Rend. Sci. Paris, 25(1847):536–538, 1847.
  6. 6.Mauro Cettolo, Jan Niehues, Sebastian Stuker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign, iwslt 2014. In Proceedings of the International Workshop on Spoken Language Translation,, 2014.
  7. 7.Ciprian Chelba, Tomas Mikolov, Michael Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH, 2013.
  8. 8.Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.06763, 2018.
  9. 9.Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In ICML, 2009.
  10. 10.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  11. 11.Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
  12. 12.John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In COLT, 2010.
  13. 13.Carl-Friedrich Gauss. Theoria combinationis observationum erroribus minimis obnoxiae. Commentationes Societatis Regiae Scientiarum Gottingensis Recentiores, 1823.
  14. 14.Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In ICLR, 2019.
  15. 15.Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
  16. 16.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
  17. 17.Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 2012.
  18. 18.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
  19. 19.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
  20. 20.Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
  21. 21.Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, and Jiawei Han. Efficient contextualized representation: Language model pruning for sequence labeling. EMNLP, 2018.
  22. 22.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  23. 23.Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In ICLR, 2018.
  24. 24.Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In ICLR, 2019.
  25. 25.Robert Nau. Forecasting with moving averages. 2014.
  26. 26.Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL, 2019.
  27. 27.Martin Popel and Ondȇrej Bojar. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110(1):43–70, 2018.
  28. 28.Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICLR, 2018.
  29. 29.Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
  30. 30.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
  31. 31.Kirk M Wolter. Taylor series methods. In Introduction to variance estimation. 2007.
  32. 32.Lin Xiao, Adams Wei Yu, Qihang Lin, and Weizhu Chen. Dscovr: Randomized primal-dual block coordinate algorithms for asynchronous distributed optimization. J. Mach. Learn. Res., 2017.
  33. 33.Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
  34. 34.Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In ICLR, 2019.

Citation

MLA
Liu, L., et al. “On the Variance of the Adaptive Learning Rate and Beyond”. arXiv, 2019, http://arxiv.org/abs/1908.03265v4.
APA
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Han, J. (2019). On the Variance of the Adaptive Learning Rate and Beyond. arXiv. http://arxiv.org/abs/1908.03265v4
Chicago
Liu, L., H. Jiang, P. He, et al. 2019. “On the Variance of the Adaptive Learning Rate and Beyond”. arXiv. http://arxiv.org/abs/1908.03265v4.
Harvard
Liu, L. et al. (2019) “On the Variance of the Adaptive Learning Rate and Beyond”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1908.03265v4.
Vancouver
1. Liu L, Jiang H, He P, Chen W, Liu X, Gao J, Han J (2019) On the Variance of the Adaptive Learning Rate and Beyond. arXiv

BibTeX

@article{liu2019the,
  title = {On the Variance of the Adaptive Learning Rate and Beyond},
  author = {Liu, Liyuan and Jiang, Haoming and He, Pengcheng and Chen, Weizhu and Liu, Xiaodong and Gao, Jianfeng and Han, Jiawei},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1908.03265v4},
  eprint = {1908.03265}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission