Back

Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum

Zeke XieXinrui WangHuishuai ZhangIssei SatoMasashi Sugiyama

article2022ICML73 citations

Explains why Adam generalizes worse than SGD through diffusion theory and proposes Adaptive Inertia, a new optimizer that adapts momentum instead of learning rates to escape saddle points quickly while retaining SGD-like flat minima selection.

Listen

Training deep neural networks requires optimization algorithms that are both fast and capable of producing models that perform accurately on unseen real-world data. While the widely used Adam optimizer accelerates training by adjusting individual learning rates and using momentum, it often yields models that generalize worse than those trained with standard Stochastic Gradient Descent (SGD). Conversely, SGD achieves superior accuracy and generalization by finding flatter loss minima, but it suffers from slower training speeds when traversing difficult optimization obstacles, such as saddle points.

The article aims to explain the mathematical mechanisms behind this generalization gap and introduce an alternative optimization framework that combines rapid training convergence with strong model generalization.

Using continuous-time diffusion theory and physical motion equations, the analysis separates the distinct roles of adaptive learning rates and momentum during training. The authors demonstrate how these mechanisms influence the ability to escape saddle points and select flat minima. To validate the findings, extensive empirical benchmarks were conducted across major image classification datasets, including CIFAR-10, CIFAR-100, and ImageNet, as well as language modeling benchmarks using standard architectures such as ResNet, VGG, DenseNet, and Long Short-Term Memory networks.

The investigation yields four primary findings. First, momentum introduces a physical drift effect that accelerates passage through saddle points without hindering the selection of flat minima. Second, adaptive learning rates allow rapid escape from saddle points but significantly degrade generalization because they weaken the algorithm's sensitivity to loss curvature, causing models to settle in sharper, poorer-performing minima. Third, the proposed optimizer, termed Adaptive Inertia (Adai), adjusts momentum rather than learning rates on a per-parameter basis, provably preserving SGD-level flat minima selection while accelerating saddle-point traversal. Fourth, empirical benchmarks show that Adai and its weight-decay variant consistently outperform SGD, Adam, and numerous Adam variants in test accuracy, achieving top-1 error reductions of approximately 0.3% over fine-tuned SGD and nearly 4% over Adam on ImageNet ResNet50.

These findings demonstrate that the performance trade-off between fast training and strong generalization is not fundamental. Practitioners can eliminate the need to compromise between Adam's rapid convergence and SGD's superior accuracy by shifting adaptivity from step sizes to momentum parameters. Furthermore, Adai demonstrates greater tolerance across varying learning rate and regularization settings, which reduces computational tuning costs and deployment risks across machine learning workflows.

Engineering and research teams should consider adopting Adaptive Inertia methods, particularly Adai with decoupled weight decay, as an effective drop-in replacement for Adam and SGD in deep learning pipelines. When evaluating new optimization approaches, practitioners must also ensure that baseline comparisons utilize properly tuned regularization parameters to avoid misleading performance conclusions.

The theoretical derivations rely on standard physical approximations, including quasi-equilibrium and low-noise assumptions around critical points, which accurately reflect typical minibatch training regimes. Although the empirical testing covers standard computer vision and recurrent language modeling benchmarks, further large-scale validation across emerging architectures, such as modern large language models and vision transformers, is warranted to confirm broader applicability.

MicrosoftRIKEN AIPUniversity of Tokyoadaptive learningadaptive learning ratesgenetic selectionAdaptive Inertiaadaptive learning ratemomentum dynamics
Xie et al (2022).pdf
Cover for Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum

Abstract

Adaptive Moment Estimation (Adam), which combines Adaptive Learning Rate and Momentum, would be the most popular stochastic optimizer for accelerating the training of deep neural networks. However, it is empirically known that Adam often generalizes worse than Stochastic Gradient Descent (SGD). The purpose of this paper is to unveil the mystery of this behavior in the diffusion theoretical framework. Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and flat minima selection. We prove that Adaptive Learning Rate can escape saddle points efficiently, but cannot select flat minima as SGD does. In contrast, Momentum provides a drift effect to help the training process pass through saddle points, and almost does not affect flat minima selection. This partly explains why SGD (with Momentum) generalizes better, while Adam generalizes worse but converges faster. Furthermore, motivated by the analysis, we design a novel adaptive optimization framework named Adaptive Inertia, which uses parameter-wise adaptive inertia to accelerate the training and provably favors flat minima as well as SGD. Our extensive experiments demonstrate that the proposed adaptive inertia method can generalize significantly better than SGD and conventional adaptive gradient methods.

Topics

adaptive learningadaptive learning ratesgenetic selectiongradient algorithmoptimizationstochasticstochastic gradient descentonline learning

Keywords

Adaptive Inertiaadaptive learning ratemomentum dynamicsAdam dynamicsstochastic gradient descentgradient noise covarianceloss landscapesaddle-point escape

Table of Contents

  • 1. Introduction
  • 2. SGD and Diffusion
  • 2.1. Prerequisites for SGD Diffusion
  • 2.2. SGD Diffusion near Saddle Points
  • 3. Analysis of Momentum Dynamics
  • 4. Analysis of Adam Dynamics
  • 5. Adaptive Inertia
  • 6. Empirical Analysis
  • 7. Conclusion
  • Acknowledgement
  • References
  • A. Proofs
  • A.1. Proof of Theorem 1
  • A.2. Proof of Theorem 2
  • A.3. Proof of Theorem 3
  • A.4. Proof of Proposition 2
  • A.5. Proof of Proposition 4
  • A.6. Proof of Theorem 4
  • B. Classical Approximation Assumptions
  • C. Stochastic Gradient Noise Analysis
  • D. Experimental Details
  • D.1. Image Classification
  • D.2. Language Modeling
  • E. The Mean Escape Time Analysis
  • F. Supplementary Empirical Results
  • G. Adai with Stable/Decoupled Weight Decay
  • H. Expressions of Adam Dynamics

Knowls

  1. Knowl 1 — Adaptive Inertia adjusts momentum per parameter while keeping one global learning rate

    algorithm

    Adaptive Inertia (Adai) uses a single global learning rate and adapts the momentum coefficient separately for each parameter. For minibatch gradient gtg_t, parameter-wise second-moment estimate vtv_t, and momentum mtm_t, its update is:

    Input: learning rate η\eta, β0=0.1\beta_0=0.1, β2=0.99\beta_2=0.99, clipping parameter ϵ=0.001\epsilon=0.001
    Initialize m0=0m_0=0, v0=0v_0=0
    For each iteration t=1,2,…t=1,2,\ldots:
        Compute minibatch gradient gtg_t
        vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2
        v^t=vt/(1−β2t)\hat v_t = v_t/(1-\beta_2^t)
        vˉt=mean⁡(v^t)\bar v_t = \operatorname{mean}(\hat v_t), the mean over parameter coordinates
        β1,t=Clip⁡(1−β0vˉt/v^t, 0, 1−ϵ)\beta_{1,t} = \operatorname{Clip}(1-\beta_0\bar v_t/\hat v_t,\,0,\,1-\epsilon), elementwise
        mt=β1,tmt−1+(1−β1,t)gtm_t = \beta_{1,t}m_{t-1} + (1-\beta_{1,t})g_t, elementwise
        m^t=mt/(1−∏z=1tβ1,z)\hat m_t = m_t/(1-\prod_{z=1}^{t}\beta_{1,z}), with the product elementwise
        θt+1=θt−ηm^t\theta_{t+1} = \theta_t - \eta\hat m_t
    Output: updated parameters θt+1\theta_{t+1}

    The clipping range limits each inertia coefficient to [0,1−ϵ][0,1-\epsilon]; the default ϵ=0.001\epsilon=0.001 caps the maximum inertia at 1000 times the minimum. The paper uses β0=0.1\beta_0=0.1 and β2=0.99\beta_2=0.99 by default. Unlike Adam, Adai does not adapt the learning rate coordinate by coordinate. The paper establishes its theoretical saddle-escape and minima-selection properties, but does not claim that Adai must converge faster than or as fast as Adam.

  2. Knowl 2 — Adam's adaptive learning rate weakens preference for flat minima

    theoretical result

    Under the paper's local quadratic, quasi-equilibrium, and low-temperature assumptions for escape from a loss valley, Adam's mean escape time depends more weakly on curvature than the escape time for SGD or momentum. Let τ\tau be the mean escape time, BB the minibatch size, ΔL\Delta L the loss barrier, η\eta Adam's base learning rate, and Hae>0H_{ae}>0 the Hessian eigenvalue at the valley minimum along the escape direction. The paper's asymptotic curvature dependence is

    log⁡(τAdam)=O ⁣(2B ΔLηHae),log⁡(τSGD),log⁡(τMomentum)=O ⁣(2B ΔLηHae),\log(\tau_{\mathrm{Adam}})=O\!\left(\frac{2\sqrt{B}\,\Delta L}{\eta\sqrt{H_{ae}}}\right), \qquad \log(\tau_{\mathrm{SGD}}),\log(\tau_{\mathrm{Momentum}}) =O\!\left(\frac{2B\,\Delta L}{\eta H_{ae}}\right),

    where the SGD and momentum comparison uses a learning-rate-matched setting. Here O(⋅)O(\cdot) suppresses prefactors and path-dependent terms. As curvature increases, SGD and momentum escape times decrease more strongly than Adam's; accordingly, Adam is less effective at preferentially escaping sharp minima and selecting flat ones. The Adam analysis uses an idealized full-matrix adaptive learning rate, with the practical diagonal preconditioner treated as its approximation.

  3. Knowl 3 — Adai retains SGD-like curvature dependence when escaping minima

    theoretical result

    Assume a local quadratic loss landscape, quasi-equilibrium within each loss valley, and low-temperature escape over a barrier. Let aa be a valley minimum, bb its exit saddle, ΔL=L(b)−L(a)\Delta L=L(b)-L(a) the barrier height, Hae>0H_{ae}>0 the Hessian eigenvalue at aa along the escape direction ee, and Hbe<0H_{be}<0 the corresponding eigenvalue at bb. Let s∈(0,1)s\in(0,1) describe the escape path, BB be the minibatch size, η\eta the global learning rate, and HbiH_{bi} the Hessian eigenvalues at the saddle. Adai's mean escape time is

    τ=π[1+4η∑i=1n∣Hbi∣β0n+1]1∣Hbe∣exp⁡ ⁣[2BΔLη(sHae+1−s∣Hbe∣)],\tau=\pi\left[\sqrt{1+\frac{4\eta\sum_{i=1}^{n}|H_{bi}|}{\beta_0 n}}+1\right]\frac{1}{|H_{be}|} \exp\!\left[\frac{2B\Delta L}{\eta}\left(\frac{s}{H_{ae}}+\frac{1-s}{|H_{be}|}\right)\right],

    where nn is the parameter-space dimension and β0\beta_0 is Adai's inertia adaptation parameter. In particular, log⁡τ=O(2BΔL/(ηHae))\log\tau=O(2B\Delta L/(\eta H_{ae})), the same minimum-curvature scaling as SGD and momentum. The paper therefore predicts that Adai favors flat minima more effectively than Adam, whose scaling is proportional to Hae−1/2H_{ae}^{-1/2} rather than Hae−1H_{ae}^{-1}.

  4. Knowl 4 — Adam's adaptive scaling makes saddle escape approximately curvature-independent

    theoretical result

    For the paper's idealized Adam dynamics, suppose the parameters start at a critical point, the loss is locally quadratic, and the minibatch gradient-noise covariance near that point is approximately proportional to the absolute Hessian divided by batch size. In an eigendirection with Hessian eigenvalue HiH_i, the mean squared displacement after TT iterations is, for ∣Hi∣ηT≪1|H_i|\eta T\ll1,

    E[Δθi2]=η22(1−β1)[1−e−(1−β1)T]2+η2T+O ⁣(B∣Hi∣η3T2),\mathbb{E}[\Delta\theta_i^2] =\frac{\eta^2}{2(1-\beta_1)}\left[1-e^{-(1-\beta_1)T}\right]^2 +\eta^2T +O\!\left(\sqrt{B|H_i|\eta^3T^2}\right),

    where η\eta is the base learning rate, β1\beta_1 is Adam's momentum coefficient, and BB is batch size. To leading order, neither the drift contribution nor the diffusion contribution depends on HiH_i; the displacement is approximately isotropic near the critical point. This accounts for Adam's fast saddle escape, while distinguishing it from SGD's curvature-dependent diffusion.

  5. Knowl 5 — Momentum adds a drift contribution to saddle-point escape

    theoretical result

    In the continuous-time momentum model, let θ\theta be the parameter vector, MM the mass, γ\gamma the damping coefficient, and DiD_i the gradient-noise diffusion eigenvalue along a Hessian eigenvector with eigenvalue HiH_i. Starting at a critical point and considering ill-conditioned saddle directions, the mean squared displacement along that direction is

    E[Δθi2(t)]=Diγ3M2[1−e−γt]2+DiγMHi[1−e−2Hit/(γM)].\mathbb{E}[\Delta\theta_i^2(t)] =\frac{D_i}{\gamma^3M^2}\left[1-e^{-\gamma t}\right]^2 +\frac{D_i}{\gamma M H_i}\left[1-e^{-2H_i t/(\gamma M)}\right].

    Here tt is continuous dynamical time, Δθi\Delta\theta_i is displacement from the initial parameter along the eigenvector, and the first and second terms are respectively momentum drift and diffusion. For the Heavy Ball update mt=β1mt−1+β3gtm_t=\beta_1m_{t-1}+\beta_3g_t, θt+1=θt−ηmt\theta_{t+1}=\theta_t-\eta m_t, the corresponding coefficients are M=η/β3M=\eta/\beta_3 and γ=(1−β1)/η\gamma=(1-\beta_1)/\eta. Thus momentum supplies a drift mechanism in addition to stochastic diffusion for passing through saddle points.

  6. Knowl 6 — Momentum preserves SGD-like flat-minimum selection after learning-rate matching

    theoretical result

    Under a local quadratic approximation, quasi-equilibrium within loss valleys, and a low-temperature escape regime, momentum's mean escape time from a minimum over a barrier has the curvature dependence

    log⁡(τMomentum)=O ⁣(2(1−β1)BΔLβ3ηHae).\log(\tau_{\mathrm{Momentum}}) =O\!\left(\frac{2(1-\beta_1)B\Delta L}{\beta_3\eta H_{ae}}\right).

    Here τ\tau is mean escape time, BB is batch size, ΔL\Delta L is the barrier height, Hae>0H_{ae}>0 is the Hessian eigenvalue at the minimum along the escape direction, η\eta is the learning rate, and β1,β3\beta_1,\beta_3 are the momentum-update coefficients. Choosing ηMomentum=(1−β1)ηSGD/β3\eta_{\mathrm{Momentum}}=(1-\beta_1)\eta_{\mathrm{SGD}}/\beta_3 makes this curvature dependence match SGD's O(2BΔL/(ηSGDHae))O(2B\Delta L/(\eta_{\mathrm{SGD}}H_{ae})). The analysis therefore attributes momentum's principal benefit to saddle traversal, not to a substantial change in flat-minimum selection.

  7. Knowl 7 — Adai produces a coordinate-independent momentum drift at saddles

    theoretical result

    Suppose the loss is locally quadratic at a critical point, the gradient-noise covariance is approximately proportional to the absolute Hessian divided by batch size, and Adai starts at the saddle. Let HiH_i be the Hessian eigenvalues, nn the parameter dimension, BB the batch size, η\eta the global learning rate, and β0\beta_0 Adai's inertia parameter. The total momentum-drift contribution while passing through the saddle, measured along any Hessian eigendirection ii, is

    E[Δθi2]extdrift=vˉη2β0=η2β0nB∑j=1n∣Hj∣,\mathbb{E}[\Delta\theta_i^2]_{ ext{drift}} =\frac{\bar v\eta^2}{\beta_0} =\frac{\eta^2}{\beta_0 nB}\sum_{j=1}^{n}|H_j|,

    where vˉ\bar v is the mean of Adai's bias-corrected second-moment estimate and, under the stated covariance approximation, vˉ=∑j∣Hj∣/(nB)\bar v=\sum_j|H_j|/(nB). The drift is the same across eigendirections rather than shrinking with the magnitude of the individual eigenvalue. Adai's diffusion contribution remains the momentum diffusion contribution; the adaptive inertia supplies the additional saddle-crossing drift.

  8. Knowl 8 — SGD diffusion predicts slow motion in nearly flat saddle directions

    theoretical result

    Near a critical point, assume a quadratic loss and approximate minibatch gradient-noise covariance by C(θ)≈[H(θ)]+/BC(\theta)\approx[H(\theta)]^+/B, where HH is the Hessian, BB is batch size, and [H]+[H]^+ replaces each Hessian eigenvalue by its absolute value. The SGD diffusion approximation is

    dθ=−∇L(θ) dt+[ηC(θ)]1/2dWt,d\theta=-\nabla L(\theta)\,dt+[\eta C(\theta)]^{1/2}dW_t,

    where LL is the loss, η\eta is the learning rate, and WtW_t is standard Brownian motion. If DiD_i is the diffusion eigenvalue and HiH_i the Hessian eigenvalue in eigendirection ii, then starting at the critical point the parameter distribution is Gaussian with mean at the critical point and variance

    σi2(t)=DiHi[1−e−2Hit].\sigma_i^2(t)=\frac{D_i}{H_i}\left[1-e^{-2H_it}\right].

    For positive HiH_i this variance approaches a constant; for negative HiH_i it grows exponentially. With Di≈η∣Hi∣/(2B)D_i\approx\eta|H_i|/(2B) and dynamical time t=ηTt=\eta T, the small-∣Hi∣ηT|H_i|\eta T displacement is approximately η2∣Hi∣T/B\eta^2|H_i|T/B. Thus SGD diffuses slowly in nearly flat directions at a saddle, providing the baseline against which the paper analyzes momentum and adaptive learning rates.

  9. Knowl 9 — Adai has a nonconvex stationarity guarantee under bounded-gradient assumptions

    theoretical result

    Let F(θ)F(\theta) be an ℓ\ell-smooth, lower-bounded objective with lower bound F∗F^*. Assume the stochastic gradient g(θ,ξ)g(\theta,\xi) is unbiased, has variance at most δ2\delta^2, and satisfies ∥∇F(θ)∥≤G\|\nabla F(\theta)\|\le G for all θ\theta. Run Adai for t+1t+1 iterations with 0≤β1,k≤β1,max⁡<10\le\beta_{1,k}\le\beta_{1,\max}<1, and choose learning rate η≤C/(t+1)\eta\le\sqrt{C/(t+1)} for a positive constant CC. Then

    min⁡0≤k≤tE∥∇F(θk)∥2≤C1+C2+C3t+1,\min_{0\le k\le t}\mathbb{E}\|\nabla F(\theta_k)\|^2 \le\frac{C_1+C_2+C_3}{\sqrt{t+1}},

    where

    C1=F(θ0)−F∗(1−β1,max⁡)C,C2=β1,max⁡CG22(1−β1,max⁡)2,C3=ℓC(G2+δ2)2(1−β1,max⁡)2.C_1=\frac{F(\theta_0)-F^*}{(1-\beta_{1,\max})C},\qquad C_2=\frac{\beta_{1,\max}C G^2}{2(1-\beta_{1,\max})^2},\qquad C_3=\frac{\ell C(G^2+\delta^2)}{2(1-\beta_{1,\max})^2}.

    The result guarantees an O((t+1)−1/2)O((t+1)^{-1/2}) bound on the expected squared gradient norm of the best iterate, comparable to standard stochastic momentum guarantees. It does not establish a faster convergence rate than Adam.

  10. Knowl 10 — Adai improves test error across image-classification benchmarks

    empirical result

    For CIFAR-10 and CIFAR-100 image classification, the authors compared optimizers after 200 epochs, using batch size 128, weight decay 0.00050.0005, and learning-rate drops by a factor of 10 at epochs 80 and 160 for CIFAR-10 and 100 and 150 for CIFAR-100. Reported test errors are percentages, as mean ±\pm standard deviation over three runs. In the order AdaiW, Adai, SGD with momentum, and Adam, errors were: CIFAR-10 ResNet18, 4.59±0.164.59\pm0.16, 4.74±0.144.74\pm0.14, 5.01±0.035.01\pm0.03, and 6.53±0.036.53\pm0.03; CIFAR-10 VGG16, 5.81±0.075.81\pm0.07, 6.00±0.096.00\pm0.09, 6.42±0.026.42\pm0.02, and 7.31±0.257.31\pm0.25; CIFAR-100 ResNet34, 21.05±0.1021.05\pm0.10, 20.79±0.2220.79\pm0.22, 21.52±0.3721.52\pm0.37, and 27.16±0.5527.16\pm0.55; CIFAR-100 DenseNet121, 19.44±0.2119.44\pm0.21, 19.59±0.3819.59\pm0.38, 19.81±0.3319.81\pm0.33, and 25.11±0.1525.11\pm0.15; CIFAR-100 GoogLeNet, 20.50±0.2520.50\pm0.25, 20.55±0.3220.55\pm0.32, 21.21±0.2921.21\pm0.29, and 26.12±0.3326.12\pm0.33. Adai and AdaiW also outperformed the other reported adaptive-gradient variants in these comparisons. The paper reports that Adai achieved similar or faster convergence than SGD with momentum and Adam in the plotted CIFAR learning curves.

    On ImageNet with ResNet50, the reported top-1/top-5 test errors were 23.20%/6.62% for Adai, 23.51%/6.82% for SGD with momentum, and 27.13%/9.18% for Adam. On Penn TreeBank with a two-layer LSTM, the optimal test perplexities were 74.3 for Adai, 74.9 for SGD with momentum, and 74.3 for Adam. These experiments support the paper's claim that Adai can improve generalization without requiring slower training.

  11. Knowl 11 — Escape-time and Hessian measurements support Adai's flat-minimum preference

    empirical result

    The paper tests curvature-dependent escape on a 10-dimensional Styblinski–Tang loss with minibatch training and varies relative minimum sharpness by rescaling parameters so the Hessian scales by a factor kk. Escape times were estimated from 100 repetitions. Across the tested range, the plotted relation was −log⁡(Γ)=O(k−1)-\log(\Gamma)=O(k^{-1}) for Adai and SGD with momentum, while Adam followed approximately −log⁡(Γ)=O(k−1/2)-\log(\Gamma)=O(k^{-1/2}), where Γ\Gamma is the escape rate (the inverse mean escape time). This agrees with the theoretical distinction that Adai and momentum are more sensitive to sharpness than Adam. In a separate ResNet18/CIFAR-10 analysis, Adai's leading Hessian eigenvalues were reported as substantially smaller than those for SGD with momentum and Adam. Weight-perturbation measurements on the same model also showed that Adai and momentum incurred less increase in training loss under Gaussian parameter noise than Adam, indicating flatter learned minima by these empirical measures.

Coverage note — Detailed proof derivations and proof-only lemmas are omitted; supplementary optimizer-variant comparisons and implementation details were left out because they do not add independent results to the main theory and experiments.

References

  1. 1.An, G. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643–674, 1996.
  2. 2.Balakrishnan, V. Elements of nonequilibrium statistical mechanics, volume 3. Springer, 2008.
  3. 3.Berglund, N. Kramers’ law: Validity, derivations and generalisations. Markov Processes and Related Fields, 19(3):459–490, 2013.
  4. 4.Chen, J. and Gu, Q. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.06763, 2018.
  5. 5.Coffey, W. and Kalmykov, Y. P. The Langevin equation: with applications to stochastic problems in physics, chemistry and electrical engineering, volume 27. World Scientific, 2012.
  6. 6.Daneshmand, H., Kohler, J., Lucchi, A., and Hofmann, T. Escaping saddles with stochastic gradients. In International Conference on Machine Learning, pp. 1155–1164, 2018.
  7. 7.Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in Neural Information Processing Systems, 27:2933–2941, 2014.
  8. 8.Défossez, A., Bottou, L., Bach, F., and Usunier, N. On the convergence of adam and adagrad. arXiv preprint arXiv:2003.02395, 2020.
  9. 9.Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
  10. 10.Eyring, H. The activated complex in chemical reactions. The Journal of Chemical Physics, 3(2):107–115, 1935.
  11. 11.Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013.
  12. 12.Hanggi, P. Escape from a metastable state. Journal of Statistical Physics, 42(1-2):105–148, 1986.
  13. 13.Hänggi, P., Talkner, P., and Borkovec, M. Reaction-rate theory: fifty years after kramers. Reviews of modern physics, 62(2):251, 1990.
  14. 14.HaoChen, J. Z., Wei, C., Lee, J., and Ma, T. Shape matters: Understanding the implicit bias of the noise covariance. In Conference on Learning Theory, pp. 2315–2357. PMLR, 2021.
  15. 15.Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 1225–1234, 2016.
  16. 16.He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  17. 17.Hinton, G., Srivastava, N., and Swersky, K. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012.
  18. 18.Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. In Advances in neural information processing systems, pp. 529–536, 1995.
  19. 19.Hochreiter, S. and Schmidhuber, J. Flat minima. Neural Computation, 9(1):1–42, 1997a.
  20. 20.Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997b.
  21. 21.Hu, W., Li, C. J., Li, L., and Liu, J.-G. On the diffusion approximation of nonconvex stochastic gradient descent. Annals of Mathematical Sciences and Applications, 4(1):3–32, 2019.
  22. 22.Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.
  23. 23.Jastrzkebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., and Storkey, A. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017.
  24. 24.Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2019.
  25. 25.Jin, C., Ge, R., Netrapalli, P., Kakade, S. M., and Jordan, M. I. How to escape saddle points efficiently. In International Conference on Machine Learning, pp. 1724–1732. PMLR, 2017.
  26. 26.Kalinay, P. and Percus, J. K. Phase space reduction of the one-dimensional fokker-planck (kramers) equation. Journal of Statistical Physics, 148(6):1135–1155, 2012.
  27. 27.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  28. 28.Kramers, H. A. Brownian motion in a field of force and the diffusion model of chemical reactions. Physica, 7(4):284–304, 1940.
  29. 29.Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. 2009.
  30. 30.Li, Q., Tai, C., et al. Stochastic modified equations and adaptive stochastic gradient algorithms. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2101–2110. JMLR. org, 2017.
  31. 31.Li, Q., Tai, C., and Weinan, E. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. J. Mach. Learn. Res., 20:40–1, 2019.
  32. 32.Li, Z., Malladi, S., and Arora, S. On the validity of modeling sgd with stochastic differential equations (sdes). arXiv preprint arXiv:2102.12470, 2021.
  33. 33.Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2019.
  34. 34.Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
  35. 35.Luo, L., Xiong, Y., Liu, Y., and Sun, X. Adaptive gradient methods with dynamic bound of learning rate. 7th International Conference on Learning Representations, ICLR 2019, 2019.
  36. 36.Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873–4907, 2017.
  37. 37.Marcus, M., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of english: The penn treebank. 1993.
  38. 38.Neelakantan, A., Vilnis, L., Le, Q. V., Sutskever, I., Kaiser, L., Kurach, K., and Martens, J. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807, 2015.
  39. 39.Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pp. 5947–5956, 2017.
  40. 40.Nguyen, T. H., Simsekli, U., Gurbuzbalaban, M., and Richard, G. First exit time analysis of stochastic gradient descent under heavy-tailed gradient noise. In Advances in Neural Information Processing Systems, pp. 273–283, 2019.
  41. 41.Pawitan, Y. In all likelihood: statistical modelling and inference using likelihood. Oxford University Press, 2001.
  42. 42.Radpay, P. Langevin equation and fokker-planck equation. 2020. URL https://www.thphys.uni-heidelberg.de/~wolschin/statsem20_7s.pdf.
  43. 43.Reddi, S., Zaheer, M., Sra, S., Poczos, B., Bach, F., Salakhutdinov, R., and Smola, A. A generic approach for escaping saddle points. In International Conference on Artificial Intelligence and Statistics, pp. 1233–1242. PMLR, 2018.
  44. 44.Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. 6th International Conference on Learning Representations, ICLR 2018, 2019.
  45. 45.Risken, H. Fokker-planck equation. In The Fokker-Planck Equation, pp. 63–95. Springer, 1996.
  46. 46.Risken, H. and Eberly, J. The fokker-planck equation, methods of solution and applications. Journal of the Optical Society of America B Optical Physics, 2(3):508, 1985.
  47. 47.Sato, I. and Nakagawa, H. Approximation analysis of stochastic gradient langevin dynamics by using fokker-planck equation and ito process. In International Conference on Machine Learning, pp. 982–990, 2014.
  48. 48.Shi, N., Li, D., Hong, M., and Sun, R. {RMS}prop can converge with proper hyper-parameter. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=3UDSdyIcBDA.
  49. 49.Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  50. 50.Simsekli, U., Sagun, L., and Gurbuzbalaban, M. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pp. 5827–5837, 2019.
  51. 51.Staib, M., Reddi, S., Kale, S., Kumar, S., and Sra, S. Escaping saddle points with adaptive gradient methods. In International Conference on Machine Learning, pp. 5956–5965. PMLR, 2019.
  52. 52.Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9, 2015.
  53. 53.Van Kampen, N. G. Stochastic processes in physics and chemistry, volume 1. Elsevier, 1992.
  54. 54.Wang, B. and Ye, Q. Stochastic gradient descent with nonlinear conjugate gradient-style adaptive momentum. arXiv preprint arXiv:2012.02188, 2020.
  55. 55.Wang, J.-K., Lin, C.-H., and Abernethy, J. Escaping saddle points faster with stochastic momentum. In International Conference on Learning Representations, 2019.
  56. 56.Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681–688, 2011.
  57. 57.Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., and Recht, B. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4148–4158, 2017.
  58. 58.Wu, L., Ma, C., and Weinan, E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. In Advances in Neural Information Processing Systems, pp. 8279–8288, 2018.
  59. 59.Xie, Z., Sato, I., and Sugiyama, M. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2020a.
  60. 60.Xie, Z., Sato, I., and Sugiyama, M. Stable weight decay regularization. arXiv preprint arXiv:2011.11152, 2020b.
  61. 61.Xie, Z., He, F., Fu, S., Sato, I., Tao, D., and Sugiyama, M. Artificial neural variability for deep learning: On over-fitting, noise memorization, and catastrophic forgetting. Neural Computation, 2021a.
  62. 62.Xie, Z., Yuan, L., Zhu, Z., and Sugiyama, M. Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization. In International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11448–11458. PMLR, 18–24 Jul 2021b.
  63. 63.Xie, Z., Tang, Q.-Y., Cai, Y., Sun, M., and Li, P. On the power-law spectrum in deep learning: A bridge to protein science. arXiv preprint arXiv:2201.13011, 2022.
  64. 64.Xu, P., Chen, J., Zou, D., and Gu, Q. Global convergence of langevin dynamics based algorithms for nonconvex optimization. In Advances in Neural Information Processing Systems, pp. 3122–3133, 2018.
  65. 65.Yan, Y., Yang, T., Li, Z., Lin, Q., and Yang, Y. A unified analysis of stochastic momentum methods for deep learning. In IJCAI International Joint Conference on Artificial Intelligence, 2018.
  66. 66.Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pp. 9793–9803, 2018.
  67. 67.Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.
  68. 68.Zavriev, S. and Kostyuk, F. Heavy-ball method in nonconvex optimization problems. Computational Mathematics and Modeling, 4(4):336–341, 1993.
  69. 69.Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. In International Conference on Machine Learning, 2017.
  70. 70.Zhang, G., Li, L., Nado, Z., Martens, J., Sachdeva, S., Dahl, G., Shallue, C., and Grosse, R. B. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems, pp. 8196–8207, 2019a.
  71. 71.Zhang, J., He, T., Sra, S., and Jadbabaie, A. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2019b.
  72. 72.Zhou, H.-X. Rate theories for biologists. Quarterly reviews of biophysics, 43(2):219–293, 2010.
  73. 73.Zhou, M., Liu, T., Li, Y., Lin, D., Zhou, E., and Zhao, T. Toward understanding the importance of noise in training neural networks. In International Conference on Machine Learning, 2019.
  74. 74.Zhou, P., Feng, J., Ma, C., Xiong, C., Hoi, S. C. H., et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33, 2020.
  75. 75.Zhu, Z., Wu, J., Yu, B., Wu, L., and Ma, J. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In ICML, pp. 7654–7663, 2019.
  76. 76.Zhuang, J., Tang, T., Ding, Y., Tatikonda, S. C., Dvornek, N., Papademetris, X., and Duncan, J. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795–18806, 2020.
  77. 77.Zou, F., Shen, L., Jie, Z., Zhang, W., and Liu, W. A sufficient condition for convergences of adam and rmsprop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11127–11135, 2019.

Citation

MLA
Xie, Z., et al. “Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum”. International Conference on Machine Learning, vol. 162, 2022, pp. 24430–59, https://proceedings.mlr.press/v162/xie22d.html.
APA
Xie, Z., Wang, X., Zhang, H., Sato, I., & Sugiyama, M. (2022). Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum. International Conference on Machine Learning, 162, 24430–24459. https://proceedings.mlr.press/v162/xie22d.html
Chicago
Xie, Z., X. Wang, H. Zhang, I. Sato, and M. Sugiyama. 2022. “Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum”. International Conference on Machine Learning 162: 24430–59. https://proceedings.mlr.press/v162/xie22d.html.
Harvard
Xie, Z. et al. (2022) “Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum”, International Conference on Machine Learning. PMLR, pp. 24430–24459. Available at: https://proceedings.mlr.press/v162/xie22d.html.
Vancouver
1. Xie Z, Wang X, Zhang H, Sato I, Sugiyama M (2022) Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum. In: International Conference on Machine Learning. PMLR, pp 24430–24459

BibTeX

@InProceedings{pmlr-v162-xie22d,
  title = 	 {Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum},
  author =       {Xie, Zeke and Wang, Xinrui and Zhang, Huishuai and Sato, Issei and Sugiyama, Masashi},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {24430--24459},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/xie22d/xie22d.pdf},
  url = 	 {https://proceedings.mlr.press/v162/xie22d.html},
  abstract = 	 {Adaptive Moment Estimation (Adam), which combines Adaptive Learning Rate and Momentum, would be the most popular stochastic optimizer for accelerating the training of deep neural networks. However, it is empirically known that Adam often generalizes worse than Stochastic Gradient Descent (SGD). The purpose of this paper is to unveil the mystery of this behavior in the diffusion theoretical framework. Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and flat minima selection. We prove that Adaptive Learning Rate can escape saddle points efficiently, but cannot select flat minima as SGD does. In contrast, Momentum provides a drift effect to help the training process pass through saddle points, and almost does not affect flat minima selection. This partly explains why SGD (with Momentum) generalizes better, while Adam generalizes worse but converges faster. Furthermore, motivated by the analysis, we design a novel adaptive optimization framework named Adaptive Inertia, which uses parameter-wise adaptive inertia to accelerate the training and provably favors flat minima as well as SGD. Our extensive experiments demonstrate that the proposed adaptive inertia method can generalize significantly better than SGD and conventional adaptive gradient methods.}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/