A Simple Weight Decay Can Improve Generalization

Anders KroghJohn Hertz

article1991NeurIPS2,114 citations

Modern machine learning models frequently suffer from poor generalization, performing well on training data but failing when deployed on new, unseen data. This issue commonly arises when complex models overfit noisy information or when there are too few training examples relative to the number of adjustable parameters. While standard approaches often attempt to prune network connections, an alternative technique penalizes large parameter values during training. The article set out to provide a theoretical explanation and empirical demonstration of how and why applying parameter decay—penalizing the size of internal model weights—improves generalization performance in neural networks.

To evaluate this technique, the authors conducted theoretical analyses on linear network models, derived exact mathematical conditions under noisy target conditions, and extended these principles to complex, nonlinear networks through local approximations. They complemented this mathematical analysis with practical simulations on the NetTalk speech-pronunciation dataset, using a neural network containing roughly 8,400 parameters trained on varying sample sizes ranging from 400 to 5,000 words.

The findings show that parameter decay operates through two distinct mechanisms. First, it actively suppresses irrelevant model components by selecting the simplest, smallest set of parameters capable of solving the training problem. Second, when properly sized, parameter decay prevents the network from fitting static random noise in the training targets, substantially reducing test error. In the empirical NetTalk trials, penalizing large weights consistently reduced the squared prediction error across all training sample sizes and moderately lowered the classification mistake rate. The performance remained highly stable across several tested decay parameter strengths.

These results provide a clear rationale for implementing parameter penalties to lower operational risks and enhance predictive reliability in automated systems. Rather than relying solely on architectural pruning, teams can use parameter decay as a simple, computationally lightweight regularization method to prevent over-reliance on noisy data. For decision-makers and technical teams, the article supports incorporating parameter decay into standard training workflows and selecting penalty levels matched to the expected level of noise in the data.

Confidence in the linear findings is very high due to exact analytical proofs. However, users should exercise caution when extrapolating to highly nonlinear networks or datasets with extreme noise, as the theoretical guarantees for nonlinear models rely on localized approximations near the optimal solution. Further empirical tuning may be necessary to identify optimal penalty settings for specialized, highly complex operational tasks.

automated systems classification demonstrations simple weight decay generalization error neural networks

Krogh et al (1991).pdf

Paper: Optimal Brain Damage, Yann LeCun et al. (1989). Introduces Hessian-based parameter pruning to improve neural network generalization, establishing the architectural complexity reduction paradigm that weight decay complements as a continuous penalty alternative.

Paper: Keeping Neural Networks Simple by Minimizing the Description Length of the Weights, Geoffrey E. Hinton et al. (1993). Extends basic weight decay by framing weight penalization within the Minimum Description Length principle and using adaptive Gaussian noise on parameters.
Paper: Decoupled Weight Decay Regularization, Ilya Loshchilov et al. (2019). Analyzes the mathematical distinction between L2 regularization and true weight decay in modern adaptive optimizers, demonstrating how decoupled weight decay recovers intended regularization dynamics.
Paper: Feature selection, L1 vs. L2 regularization, and rotational invariance, A. Ng (2004). Expands on L2 weight penalization by comparing its sample complexity and rotational invariance properties against L1 regularization in high-dimensional settings.
Paper: Understanding deep learning requires rethinking generalization, Chiyuan Zhang et al. (2017). Re-evaluates classical generalization mechanisms, showing empirically that while explicit regularizers like weight decay boost test accuracy, overparameterized networks can still memorize arbitrary noise.

Topics

automated systems classification demonstrations neural networks ranging regularization methods training sample optimal solutions linear networks control parameters

Keywords

simple weight decay generalization error neural networks weight vector cost function linear network free parameters target function training examples

1 INTRODUCTION
2 FEED-FORWARD NETWORKS
3 THE LINEAR PERCEPTRON
4 LEARNING WITH AN UNRELIABLE TEACHER
5 NON-LINEAR NETWORKS
6 NUMERICAL EXPERIMENTS
7 CONCLUSION
Acknowledgements
References

Knowls

Knowl 1 — Universal Optimal Weight Decay for Linear Perceptrons Under Target Noise
theoretical result
For a linear perceptron $f_w(\xi) = N^{-1/2} \sum_{i=1}^N w_i \xi_i$ trained on $p$ patterns with static zero-mean target noise of variance $\sigma^2$ , minimizing the noise-averaged generalization error mode contribution $F_r = \frac{\lambda^2 u_r^2 + \Lambda_r \sigma^2}{(\lambda + \Lambda_r)^2}$ with respect to the weight decay parameter $\lambda$ gives mode-optimal value $\lambda_{\text{optimal}}^r = \sigma^2 / u_r^2$ , where $u_r$ is the $r$ -th mode component of the true teacher vector $u \in \mathbb{R}^N$ and $\Lambda_r$ is the eigenvalue of the empirical input correlation matrix $A$ .

When the teacher vector $u$ is drawn randomly and isotropically, averaging across modes produces a universal optimal weight decay parameter: $\lambda_{\text{optimal}} = \frac{\sigma^2}{u^2}$ where $u^2 = \langle N^{-1} \|u\|^2 \rangle$ is the average squared magnitude per teacher weight. This optimal value $\lambda_{\text{optimal}}$ depends exclusively on the noise variance $\sigma^2$ and the teacher weight magnitude $u^2$ , and is completely independent of the eigenvalue spectrum of the input covariance matrix $A$ .
Knowl 2 — Mode-Wise Generalization Error Under Static Target Noise
theoretical result
Consider a linear perceptron $f_w(\xi) = N^{-1/2} \sum_{i=1}^N w_i \xi_i$ trained on $p$ examples where target values generated by a teacher $u \in \mathbb{R}^N$ are corrupted by zero-mean additive static noise $\eta^\mu$ with variance $\sigma^2$ ( $f_u(\xi^\mu) \to f_u(\xi^\mu) + \eta^\mu$ ). Under isotropic input distributions satisfying $\langle \xi_i \xi_j \rangle_\xi = \delta_{ij}$ , the generalization error is $F = \langle [f_u(\xi) - f_w(\xi)]^2 \rangle_\xi = N^{-1} \sum_{r=1}^N v_r^2$ , where $v = u - w$ is the parameter error vector.

In the orthonormal coordinate system diagonalizing the empirical correlation matrix $A_{ij} = N^{-1} \sum_{\mu=1}^p \xi_i^\mu \xi_j^\mu$ with eigenvalues $\Lambda_r$ , the asymptotic steady-state parameter error in mode $r$ is: $v_r = \frac{\lambda u_r - N^{-1/2} \sum_{\mu=1}^p \eta^\mu \xi_r^\mu}{\lambda + \Lambda_r}$ Averaging over the noise distribution $\eta$ , the expected contribution of mode $r$ to the generalization error is: $F_r = \overline{v_r^2} = \frac{\lambda^2 u_r^2 + \Lambda_r \sigma^2}{(\lambda + \Lambda_r)^2}$
Knowl 3 — Minimum-Norm Weight Selection by Weight Decay in Noise-Free Linear Networks
theoretical result
In a linear perceptron with $N$ inputs trained on $p < N$ noise-free patterns, the empirical correlation matrix $A = N^{-1} \sum_{\mu=1}^p \xi^\mu (\xi^\mu)^T$ has rank at most $p$ . The weight space decomposes into the pattern subspace $V_p = \text{span}\{\xi^1, \dots, \xi^p\}$ and its orthogonal complement $V_p^\perp$ , where $A$ has zero eigenvalues ( $\Lambda_r = 0$ ). Any weight vector in the affine subspace $u + V_p^\perp$ yields zero training error.

When training without weight decay ( $\lambda = 0$ ), parameter error components $v_r = u_r - w_r$ for modes within $V_p$ decay exponentially to zero, while components in $V_p^\perp$ remain constant at their initial values. When learning with any strictly positive weight decay $\lambda > 0$ , the error components in $V_p^\perp$ decay asymptotically as $e^{-\lambda t}$ . An infinitesimal weight decay ( $\lambda \to 0^+$ ) therefore suppresses all irrelevant weight components in $V_p^\perp$ , selecting the unique zero-training-error solution with the smallest Euclidean norm $\|w\|_2$ from the solution valley $u + V_p^\perp$ .
Knowl 4 — Decoupled Learning Dynamics for the Linear Perceptron
theoretical result
Let $f_w(\xi) = N^{-1/2} \sum_{i=1}^N w_i \xi_i$ be a single-output linear perceptron with parameter vector $w \in \mathbb{R}^N$ , trained on $p$ input patterns $\{\xi^\mu\}_{\mu=1}^p$ generated by a target teacher vector $u \in \mathbb{R}^N$ with sum-of-squared errors $E_0(w) = \frac{1}{2} \sum_{\mu=1}^p [f_u(\xi^\mu) - f_w(\xi^\mu)]^2$ .

Defining the parameter discrepancy $v = u - w$ and the empirical input correlation matrix: $A_{ij} = \frac{1}{N} \sum_{\mu=1}^p \xi_i^\mu \xi_j^\mu$ the continuous-time learning dynamics in the orthonormal eigenbasis where $A$ is diagonal decouple into independent equations for each mode $r \in \{1, \dots, N\}$ : $\dot{v}_r \propto -(\Lambda_r + \lambda) v_r + \lambda u_r$ where $\Lambda_r \ge 0$ is the $r$ -th eigenvalue of $A$ , and $u_r, v_r$ are the components of $u, v$ in this eigenbasis.
Knowl 5 — Local Linearization and Solution Manifolds in Non-Linear Neural Networks
theoretical result
For a non-linear feed-forward neural network $f_w(\xi)$ with $W$ weights trained on $p$ realizable examples $f_u(\xi^\mu)$ , the condition $f_w(\xi^\mu) = f_u(\xi^\mu)$ defines a continuous manifold of zero-training-error solutions in weight space of dimension at least $W - p$ whenever $p < W$ .

Expanding the weight update dynamics around a zero-error point $\tilde{w}$ on this manifold with displacement $v = \tilde{w} - w$ yields locally linearized dynamics: $\dot{v}_i \propto -\sum_{j=1}^W \mathcal{A}_{ij}(\tilde{w}) v_j - \lambda v_i + \lambda \tilde{w}_i$ where the localized empirical Jacobian outer-product matrix $\mathcal{A}(\tilde{w})$ is: $\mathcal{A}_{ij}(\tilde{w}) = \sum_{\mu=1}^p \frac{\partial f_w(\xi^\mu)}{\partial w_i} \frac{\partial f_w(\xi^\mu)}{\partial w_j}$ Because $\mathcal{A}(\tilde{w})$ has outer-product form, its rank is bounded by $\min(p, W)$ and is rank-deficient for $p < W$ . A small weight decay $\lambda > 0$ drives gradient descent to select the point of minimal Euclidean norm within the local manifold valley. When small target errors are present, local linearization preserves the optimal weight decay relation $\lambda_{\text{optimal}} = \sigma^2 / u^2$ .
Knowl 6 — Weight Decay Regularization and Continuous Gradient Descent Dynamics
model/method
Weight decay penalizes large parameter values by modifying the empirical objective function $E_0(w)$ with an $L_2$ regularization term over the parameter vector $w \in \mathbb{R}^W$ : $E(w) = E_0(w) + \frac{1}{2} \lambda \sum_{i=1}^W w_i^2$ where $\lambda \ge 0$ is the regularization strength parameter. Under continuous-time gradient descent, the corresponding parameter update equation is: $\dot{w}_i \propto -\frac{\partial E_0(w)}{\partial w_i} - \lambda w_i$ In the absence of the task gradient force $-\frac{\partial E_0}{\partial w_i}$ , this learning rule causes an exponential decay of weights toward zero with time constant $\lambda^{-1}$ .
Knowl 7 — NetTalk Experimental Architecture and Training Setup
experimental setup
The effect of weight decay on generalization was evaluated empirically on the NetTalk English text-to-phoneme conversion task using the following protocol:
- Network Architecture: Fully connected feed-forward network with $7 \times 26 = 182$ binary input units (representing a sliding context window of 7 characters encoded via 1-of-26 vectors), 40 hidden units, and 26 output units, totaling approximately 8,400 weights.
- Loss and Optimization: Backpropagation optimizing the entropic error measure with a momentum coefficient fixed at $0.8$ , trained for 300 epochs.
- Dataset Splits: Training subsets consisting of $p \in [400, 5000]$ words randomly sampled from a 20,000-word dictionary, evaluated against a disjoint test set of 1,000 random words, with training and test partitions sampled independently per run.
- Regularization Values: Evaluated at $\lambda = 0.00008$ , with comparison checks at $\lambda = 0.00005$ , $\lambda = 0.0001$ , and unregularized training ( $\lambda = 0$ ).
Knowl 8 — Empirical Generalization Improvement on NetTalk via Weight Decay
empirical result
Across training set sizes ranging from $p \approx 400$ to $p \approx 5000$ words (corresponding to up to $4 \times 10^4$ letter patterns), training NetTalk with weight decay ( $\lambda = 0.00008$ ) consistently improves generalization compared to unregularized backpropagation ( $\lambda = 0$ ):
- Squared Generalization Error: After 300 epochs, generalization squared error is substantially reduced across all training set sizes (e.g., dropping from $>1.2$ without weight decay to $\approx 1.0$ with weight decay at $p \approx 0.5 \times 10^4$ patterns, and from $\approx 0.75$ to $\approx 0.60$ at $p \approx 2 \times 10^4$ patterns). The final error with weight decay is lower than the best validation error achieved at any point during unregularized training.
- Phoneme Error Rate: The classification error rate (fraction of incorrect phonemes selected by smallest angular distance to the output vector) decreases from $\approx 0.25$ to $\approx 0.22$ at $p \approx 0.5 \times 10^4$ patterns and from $\approx 0.17$ to $\approx 0.15$ at $p \approx 2 \times 10^4$ patterns.
- Parameter Sensitivity: Performance curves are robust and virtually identical for $\lambda \in \{0.00005, 0.00008, 0.0001\}$ .

Coverage note — No substantial contributed material from the paper was omitted.

References

1.D.B. Schwartz, V.K. Samalam, S.A. Solla, and J.S. Denker. Exhaustive learning. Neural Computation, 2:371–382, 1990.
2.N. Tishby, E. Levin, and S.A. Solla. Consistent inference of probabilities in layered networks: predictions and generalization. In International Joint Conference on Neural Networks, pages 403–410, (Washington 1989), IEEE, New York, 1989.
3.E.B. Baum and D. Haussler. What size net gives valid generalization? Neural Computation, 1:151–160, 1989.
4.Y. Le Cun, J.S. Denker, and S.A. Solla. Optimal brain damage. In D.S. Touretzky, editor, Advances in Neural Information Processing Systems, pages 598–605, (Denver 1989), Morgan Kaufmann, San Mateo, 1990.
5.H.H. Thodberg. Improving generalization of neural networks through pruning. International Journal of Neural Systems, 1:317–326, 1990.
6.D.H. Weigend, D.E. Rumelhart, and B.A. Huberman. Generalization by weight-elimination with application to forecasting. In R.P. Lippmann et al, editors, Advances in Neural Information Processing Systems, page 875–882, (Denver 1989), Morgan Kaufmann, San Mateo, 1991.
7.G.E. Hinton. Learning translation invariant recognition in a massively parallel network. In G. Goos and J. Hartmanis, editors, PARLE: Parallel Architectures and Languages Europe. Lecture Notes in Computer Science, pages 1–13, Springer-Verlag, Berlin, 1987.
8.J.Moody. Generalization, weight decay, and architecture selection for nonlinear learning systems. These proceedings.
9.D. MacKay. A practical bayesian framework for backprop networks. These proceedings.
10.A. Krogh and J.A. Hertz. Generalization in a Linear Perceptron in the Presence of Noise. To appear in Journal of Physics A 1992.
11.T.J. Sejnowski and C.R. Rosenberg. Parallel networks that learn to pronounce english text. Complex Systems, 1:145–168, 1987.
12.J.A. Hertz, A. Krogh, and R.G. Palmer. Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City, 1991.

Citation

MLA

Krogh, A., and J. Hertz. “A Simple Weight Decay Can Improve Generalization”. Advances in Neural Information Processing Systems, vol. 4, 1991, https://proceedings.neurips.cc/paper_files/paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf.

APA

Krogh, A., & Hertz, J. (1991). A Simple Weight Decay Can Improve Generalization. Advances in Neural Information Processing Systems, 4. https://proceedings.neurips.cc/paper_files/paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf

Chicago

Krogh, A., and J. Hertz. 1991. “A Simple Weight Decay Can Improve Generalization”. Advances in Neural Information Processing Systems 4. https://proceedings.neurips.cc/paper_files/paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf.

Harvard

Krogh, A. and Hertz, J. (1991) “A Simple Weight Decay Can Improve Generalization”, Advances in Neural Information Processing Systems. Curran Associates, Inc. Available at: https://proceedings.neurips.cc/paper_files/paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf.

Vancouver

1. Krogh A, Hertz J (1991) A Simple Weight Decay Can Improve Generalization. Advances in Neural Information Processing Systems 4:

BibTeX

@inproceedings{krogh1991simple,
  title = {A Simple Weight Decay Can Improve Generalization},
  author = {Krogh, Anders and Hertz, John},
  year = {1991},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {4},
  url = {https://proceedings.neurips.cc/paper_files/paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

A Simple Weight Decay Can Improve Generalization

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Universal Optimal Weight Decay for Linear Perceptrons Under Target Noise

Knowl 2 — Mode-Wise Generalization Error Under Static Target Noise

Knowl 3 — Minimum-Norm Weight Selection by Weight Decay in Noise-Free Linear Networks

Knowl 4 — Decoupled Learning Dynamics for the Linear Perceptron

Knowl 5 — Local Linearization and Solution Manifolds in Non-Linear Neural Networks

Knowl 6 — Weight Decay Regularization and Continuous Gradient Descent Dynamics

Knowl 7 — NetTalk Experimental Architecture and Training Setup

Knowl 8 — Empirical Generalization Improvement on NetTalk via Weight Decay

References

Citation

BibTeX

Access the Paper