On the Spectral Bias of Neural Networks

Nasim RahamanAristide BaratinDevansh ArpitFelix DraxlerMin LinFred A. HamprechtYoshua BengioAaron Courville

article2019ICML2,634 citations

Reveals that deep ReLU networks inherently prioritize learning low-frequency components over high-frequency variations, providing a foundational explanation for why over-parameterized neural networks generalize to simple patterns rather than memorizing noise.

Listen

Neural networks can fit arbitrary random data with perfect accuracy yet often generalize well on natural inputs. This raises questions about what implicit biases guide their learning toward simpler solutions that perform reliably.

The article set out to examine whether deep ReLU networks exhibit a spectral bias that favors low-frequency functions during training and to clarify how the geometry of the data manifold influences this behavior.

Researchers combined Fourier analysis of the piecewise-linear structure of ReLU networks with targeted experiments. They derived the frequency spectrum of such networks analytically, then trained networks on synthetic signals with controlled frequencies, on MNIST targets corrupted by noise at different frequencies, and on signals defined over manifolds of increasing shape complexity. Sample sizes ranged from hundreds to thousands of points, and results were averaged over multiple random phases and initializations.

The analysis shows that networks consistently fit lower frequencies before higher ones, regardless of amplitude. Lower-frequency components also remain more stable under random parameter changes. On manifolds with more complex shapes, higher frequencies become easier to express because the embedding map itself introduces high-frequency variation. These patterns hold on both synthetic data and MNIST.

The findings indicate that gradient descent on overparameterized networks naturally prioritizes smooth solutions that generalize across samples. This bias helps explain observed robustness to high-frequency noise and suggests that model sensitivity can be quantified directly in the Fourier domain. It also highlights that manifold geometry can ease or hinder learning of certain patterns without changing network architecture.

Practitioners may benefit from monitoring frequency content during training to detect when models begin fitting noise. Designers could explore regularizers or initializations that control the rate at which higher frequencies enter the solution. Further work is needed to test whether the same bias appears in other activation functions and larger-scale tasks before relying on it for architectural decisions.

The core spectral-bias observation rests on consistent empirical and theoretical results across multiple settings. Broader claims about generalization rest on the assumption that natural data lie on manifolds of moderate complexity, an assumption that remains only partially tested.

Heidelberg University Mila fourier frequencies relu networks Spectral Bias ReLU Network Fourier Analysis

arXiv: 1806.08734 nasimrahaman/SpectralBias

Paper: Neural Tangent Kernel: Convergence and Generalization in Neural Networks, Arthur Jacot et al. (2018). This foundational paper introduces the Neural Tangent Kernel framework, establishing the theoretical analytical tools used to study the convergence rates of neural network training across different functional components.
Paper: A Closer Look at Memorization in Deep Networks, Devansh Arpit et al. (2017). This work demonstrates empirically that neural networks learn simple, broad patterns before memorizing noisy data, providing the initial motivating puzzle that spectral bias explains mathematically.
Paper: Understanding deep learning requires rethinking generalization, Chiyuan Zhang et al. (2017). This influential paper proves that deep networks can fit arbitrary noise yet generalize well, posing the core question about implicit regularization that spectral bias directly addresses.
Paper: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, Andrew M. Saxe et al. (2014). This paper analyzes the analytical learning dynamics and mode-dependent timescales of gradient descent in deep networks, laying theoretical foundations for frequency-dependent convergence analysis.

Paper: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, Matthew Tancik et al. (2020). This paper directly builds on the spectral bias of coordinate MLPs by proposing Fourier feature mappings to circumvent low-frequency convergence bottlenecks and learn high-frequency details.
Paper: Implicit Neural Representations with Periodic Activation Functions, Vincent Sitzmann et al. (2020). This work addresses the low-frequency spectral bias of standard ReLU networks by introducing sinusoidal activation functions that effectively capture high-frequency signals and fine gradients.
Paper: Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization, Andy Zeyi Liu et al. (2026). This research utilizes Fourier-feature modeling and spectral diagnostics to analyze activation and gradient spectra in modern transformer optimization.
Paper: Shortcut learning in deep neural networks, Robert Geirhos et al. (2020). This perspective synthesizes inductive biases, such as spectral and texture biases, into a unified framework of shortcut learning in deep neural networks.

Cover for On the Spectral Bias of Neural Networks

Abstract

Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy. In this work, we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets \emph{easier} with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

Topics

fourier frequencies relu networks data manifolds network architecture recurrent neural networks

Keywords

Spectral Bias ReLU Network Fourier Analysis Low Frequency High Frequency Data Manifold Gradient Descent Lipschitz Constant Linear Region Implicit Regularization

1 Introduction
2 Fourier analysis of ReLU networks
2.1 Preliminaries
2.2 Fourier Spectrum
3 Lower Frequencies are Learned First
3.1 Synthetic Experiments
3.2 Real-Data Experiments
4 Not all Manifolds are Learned Equal
5 Related Work
6 Conclusion
References
A Experimental Details
A.1 Experiment
A.2 Experiment
A.3 Experiment
A.4 Experiment
A.4.1 Loss Curves Accompanying Figure
A.5 Qualitative Ablation over Architectures
A.6 MNIST: A Proof of Concept
A.7 Cifar-10: It’s All Connected
B The Continuous Piecewise Linear Structure of Deep ReLU Networks
C Fourier Analysis of ReLU Networks
C.1 Proof of Lemma
C.2 Fourier Transform of Polytopes
C.2.1 Theorem 1 of Diaz et al. (2016)
C.2.2 Discussion
C.3 On Theorem
C.4 Spectral Decay Rate of the Parameter Gradient
C.5 Convergence Rate of a Network Trained on Pure-Frequency Targets
C.6 Proof of the Lipschtiz bound
C.7 The Fourier Transform of a Function Composition
D Volume of High-Frequency Parameters in Parameter Space
E Kernel Machines and KNNs
E.1 Kernel Machines vs DNNs
E.2 K-NN Classifier vs. DNN classifier

Knowls

Knowl 1 — Fourier Spectrum Decay of ReLU Neural Networks
theoretical result
For a scalar ReLU network $f_\theta: \mathbb{R}^d \to \mathbb{R}$ with $L$ hidden layers, width $d_1, \dots, d_L$ , input dimension $d$ , and a single output neuron, the network represents a continuous piecewise linear (CPWL) function that partitions $\mathbb{R}^d$ into convex polytopes $P_\epsilon$ . In the sense of tempered distributions, its Fourier transform $\tilde{f}_\theta(k) = \int_{\mathbb{R}^d} f_\theta(x) e^{-i k \cdot x} \mathrm{d}x$ decomposes as:
$\tilde{f}_\theta(k) = \sum_{n=0}^d \frac{C_n(\theta, k) \mathbf{1}_{H_n^\theta}(k)}{\|k\|^{n+1}}$
where $k \in \mathbb{R}^d$ is the Fourier frequency vector, $\|k\|$ is its Euclidean norm, $H_n^\theta$ is the union of $n$ -dimensional subspaces orthogonal to $n$ -codimensional faces of the linear region polytopes $P_\epsilon$ , $\mathbf{1}_{H_n^\theta}$ is the indicator function on $H_n^\theta$ , and $C_n(\theta, k): \mathbb{R}^d \to \mathbb{C}$ is $\Theta(1)$ as $\|k\| \to \infty$ .

Key structural properties of this spectrum include:
1. Anisotropic Spectral Decay: For almost all directions in $\mathbb{R}^d$ (with respect to Lebesgue measure), the Fourier transform decays at the fast rate of $\mathcal{O}(\|k\|^{-d-1})$ . Along specific lower-dimensional subspaces orthogonal to $(d-1)$ -dimensional polytope faces bounding linear regions, the decay can be as slow as $\mathcal{O}(\|k\|^{-2})$ .
2. Lipschitz and Region Bounds on Amplitude: The coefficients are bounded by $|C_n(\theta, k)| = \mathcal{O}(L_f N_f)$ , where $N_f$ is the number of linear regions and $L_f$ is the Lipschitz constant of $f_\theta$ . The Lipschitz constant satisfies:
$L_f \le \prod_{k=1}^{L+1} \|W^{(k)}\| \le \|\theta\|_\infty^{L+1} \sqrt{d} \prod_{k=1}^L d_k$
where $\|W^{(k)}\|$ is the spectral norm of the weight matrix at layer $k$ , and $\|\theta\|_\infty$ is the maximum absolute parameter value.
Knowl 2 — Spectral Decay of Parameter Gradients and Residual Dynamics During Training
theoretical result
Let $f_\theta: \mathbb{R}^d \to \mathbb{R}$ be a ReLU network parameterized by $\theta$ . For a fixed direction $\hat{k} = k / \|k\|$ , suppose the network's Fourier spectrum decays as $\tilde{f}_\theta(k) = \mathcal{O}(\|k\|^{-\Delta - 1})$ , where $\Delta \in \{1, \dots, d\}$ is the codimension of the highest-dimensional polytope face orthogonal to $\hat{k}$ .

The spectral decay rate of the parameter gradient $\frac{\partial \tilde{f}_\theta(k)}{\partial \theta}$ satisfies:
$\left\| \frac{\partial \tilde{f}_\theta(k)}{\partial \theta} \right\| = \mathcal{O}\left(\|k\| \tilde{f}_\theta(k)\right) = \mathcal{O}\left(\|k\|^{-\Delta}\right)$
When training $f_\theta$ to fit a target function $\lambda$ under continuous-time gradient descent with learning rate $\eta$ and loss $\mathcal{L}$ , the time rate of change of the Fourier-domain residual $\tilde{h}(k, t) = \tilde{f}_{\theta(t)}(k) - \tilde{\lambda}(k)$ satisfies:
$\left\| \frac{\partial \tilde{h}(k, t)}{\partial t} \right\| = \left\| \frac{\partial \tilde{f}_\theta(k)}{\partial \theta} \frac{\mathrm{d}\theta}{\mathrm{d}t}\right\| = \mathcal{O}\left(\|k\|^{-\Delta}\right)$
For a target function with a single pure frequency component $\tilde{\lambda}(k) = A_0 \delta_{k, k_0}$ sampled at $N$ uniformly spaced points on $[0, 1]$ , the time derivative of the residual at frequency $k_0$ under Mean Squared Error loss decays with the target frequency as:
$\left| \frac{\partial \tilde{h}(k_0, t)}{\partial t} \right| = \mathcal{O}\left(k_0^{-1}\right)$
This proves that the convergence speed of gradient descent on neural network residuals decays inversely with the frequency of the target component.
Knowl 3 — Parameter Space Volume of High-Frequency ReLU Network Functions
theoretical result
Let $f_\theta: \mathbb{R}^d \to \mathbb{R}$ be a ReLU neural network whose parameter vector $\theta$ is bounded within an $L^\infty$ ball $\Theta = B_K^\infty(0) = \{\theta \in \mathbb{R}^P \mid \|\theta\|_\infty \le K\}$ of radius $K > 0$ . For a threshold $\epsilon > 0$ and a cutoff frequency $k > 0$ , define the set of parameter vectors that contribute more than $\epsilon$ to frequencies above $k$ along a direction $\hat{k}$ as:
$\Xi_\epsilon(k) = \left\{ \theta \in \Theta \;\middle|\; \exists k' > k \text{ such that } |\tilde{f}_\theta(k'\hat{k})| > \epsilon \right\}$
As the cutoff frequency $k \to \infty$ , the relative Lebesgue volume of $\Xi_\epsilon(k)$ in parameter space $\Theta$ decays as:
$\frac{\operatorname{vol}(\Xi_\epsilon(k))}{\operatorname{vol}(\Theta)} = \mathcal{O}\left(k^{-\Delta - 1}\right)$
where $\Delta \in \{1, \dots, d\}$ is the codimension of the highest-dimensional polytope face orthogonal to $\hat{k}$ .

This establishes that parameter combinations that produce non-negligible high-frequency Fourier components occupy a vanishingly small volume of the parameter space.
Knowl 4 — Frequency Coupling on Data Manifolds via Stationary Phase Approximation
theoretical result
Let data lie on an $m$ -dimensional manifold $\mathcal{M} \subset \mathbb{R}^d$ parameterized by an injective embedding $\gamma: [0, 1]^m \to \mathbb{R}^d$ . A target signal defined on the manifold latent coordinates is $\lambda: [0, 1]^m \to \mathbb{R}$ , and a network function in the ambient space is $f: \mathbb{R}^d \to \mathbb{R}$ . The Fourier transform of the pullback composite function $(f \circ \gamma)(z) = f(\gamma(z))$ on the latent domain is given by:
$\widehat{(f \circ \gamma)}(l) = \int_{\mathbb{R}^d} \tilde{f}(k) P_\gamma(l, k) \mathrm{d}k$
where $k \in \mathbb{R}^d$ is the ambient frequency, $l \in \mathbb{R}^m$ is the latent frequency, and the coupling kernel $P_\gamma(l, k)$ is:
$P_\gamma(l, k) = \int_{[0, 1]^m} \exp\left(i\left(k \cdot \gamma(z) - l \cdot z\right)\right) \mathrm{d}z$
In the stationary phase regime where $\|l\|^2 + \|k\|^2 \to \infty$ , constructive interference in $P_\gamma(l, k)$ occurs exclusively at critical points $\bar{z} \in [0, 1]^m$ satisfying:
$l = J_\gamma(\bar{z}) k$
where $J_\gamma(z) \in \mathbb{R}^{m \times d}$ is the Jacobian matrix of $\gamma$ with entries $(J_\gamma(z))_{\nu\mu} = \frac{\partial \gamma_\mu}{\partial z_\nu}$ .

When expanding periodic coordinate functions $\gamma_\mu(z) = \sum_{p \in \mathbb{Z}^m} \tilde{\gamma}_\mu[p] e^{-i p \cdot z}$ in a Fourier series, the stationary phase condition along coordinate direction $\nu$ becomes:
$l \hat{l}_\nu = -i k \sum_{p \in \mathbb{Z}^m} p_\nu \hat{k}_\mu \tilde{\gamma}_\mu[p] e^{-i p \cdot z}$
where $k = \|k\|$ , $l = \|l\|$ , and $\hat{k}, \hat{l}$ are unit vectors. Consequently, if the manifold embedding $\gamma$ contains high-frequency Fourier modes $p$ , large latent frequencies $l$ on the manifold can be matched by low ambient network frequencies $k$ .
Knowl 5 — Empirical Demonstration of Spectral Bias in Gradient Descent
empirical result
When training multi-layer ReLU neural networks on regression targets containing a superposition of sinusoids using full-batch gradient descent (Adam optimizer), networks prioritize fitting lower frequencies before higher frequencies, a phenomenon designated spectral bias.

In a 1D synthetic regression setting on $[0, 1]$ with $N = 200$ equally spaced samples, a 6-layer deep, 256-unit wide ReLU network was trained on target functions:
$\lambda(z) = \sum_{i=1}^{10} A_i \sin(2\pi k_i z + \phi_i)$
with frequencies $\kappa = (5, 10, 15, \dots, 50)$ and random phases $\phi_i \sim \mathcal{U}(0, 2\pi)$ .

Two configurations were evaluated:
1. Equal amplitudes: $A_i = 1.0$ for all $i$ .
2. Increasing amplitudes for higher frequencies: $A_i \in \{0.1, 0.2, \dots, 1.0\}$ such that the highest frequency mode $k_{10} = 50$ had an amplitude 10 times larger than $k_1 = 5$ .
In both configurations, tracking the normalized Fourier magnitudes $|\tilde{f}_\theta(k_i)| / A_i$ over training steps demonstrated that:
- Lower frequency components ( $k = 5, 10$ ) reach full target amplitude early in training.
- Higher frequency components ( $k = 45, 50$ ) are fitted progressively later, regardless of whether their target amplitude is equal to or substantially larger than the low-frequency amplitudes.
- The spectral norms of the network weight matrices increase monotonically throughout training as higher frequency modes are fitted.
Knowl 6 — Frequency-Dependent Robustness of Trained Networks to Parameter Perturbations
empirical result
For a fully-trained ReLU network converged to parameter values $\theta^*$ , lower-frequency Fourier components of the network function are significantly more robust to random parameter perturbations than higher-frequency components.

In an experiment where a 6-layer, 256-unit ReLU network was trained to fit a multiscale sinusoid $\lambda(z) = \sum_i \sin(2\pi k_i z + \phi_i)$ with frequencies $k_i \in \{10, 15, \dots, 50\}$ , isotropic parameter perturbations $\theta = \theta^* + \delta \hat{\theta}$ were applied, where $\hat{\theta}$ is a uniformly sampled unit vector in parameter space and $\delta > 0$ is the perturbation magnitude.

Evaluating the normalized expected discrete Fourier magnitude $\mathbb{E}_{\hat{\theta}, \phi}[|\tilde{f}_\theta(k_i)|] / |\tilde{f}_{\theta^*}(k_i)|$ over 100 random perturbation directions demonstrated that:
- Low-frequency components ( $k = 10, 15$ ) maintain their magnitude close to 1.0 across a broad range of perturbation magnitudes $\delta$ .
- High-frequency components ( $k = 45, 50$ ) degrade rapidly and drop near zero even under small perturbation magnitudes $\delta$ .
This empirical finding indicates that parameter combinations responsible for expressing high frequencies reside in tightly constrained, lower-dimensional volumes of parameter space compared to low-frequency representations.
Knowl 7 — Effect of Target Noise Frequency on MNIST Generalization and Validation Dynamics
empirical result
When neural networks are trained on image classification data corrupted by additive target label noise $\tau_k(x) = \tau_0(x) + \beta \psi_k(x)$ where $\tau_0$ is the ground-truth binary label and $\psi_k$ is a noise function of spatial frequency $k$ and amplitude $eta$ :
- Low-frequency label noise ( $k = 0.1$ ): Rapidly degrades validation performance from the onset of training, and increasing the noise amplitude $eta$ drastically worsens the best achievable validation loss.
- High-frequency label noise ( $k = 1.0$ or radial wave $\psi_k(x) = \sin(k \|x\|)$ on $\mathbb{R}^{784}$ ): The network fits the clean low-frequency underlying function $\tau_0$ early in training, producing an early minimum (dip) in validation loss against the clean target. The network only fits the high-frequency noise later in training, causing validation loss to rise while training loss continues to drop. Furthermore, the best validation loss at the early dip is relatively insensitive to the amplitude $eta$ of high-frequency noise.
When the target noise is decomposed using the orthonormal eigenfunctions $\phi_n$ of a Gaussian RBF kernel $k(x, y) = \exp(-\|x-y\|^2/\sigma^2)$ on MNIST data (where higher eigenfunction index $n$ corresponds to higher generalized frequency), tracking the projection $\tilde{f}[n] = \langle f, \phi_n \rangle$ confirms that the network matches low-order eigenfunctions of the true target first (around iteration 200, matching the validation minimum) before slowly fitting the high-order noise eigenfunctions.
Knowl 8 — Manifold Complexity Accelerates Learning of High Frequencies
empirical result
The geometric complexity of a lower-dimensional data manifold embedded in input space directly influences the rate at which high-frequency functions defined on the manifold are learned by neural networks.

Using the family of 2D flower-shaped curves parameterized by $\gamma_L: [0, 1] \to \mathbb{R}^2$ :
$\gamma_L(z) = \left(1 + \frac{1}{2}\sin(2\pi L z)\right) \left(\cos(2\pi z), \sin(2\pi z)\right)$
where $L \in \{0, 2, 4, \dots, 20\}$ controls the number of petals (geometric complexity, with $L=0$ being a unit circle), a 6-layer ReLU network was trained to regress or classify signals defined on the latent coordinate $z \in [0, 1]$ :
1. Multi-frequency regression: When fitting $\lambda(z) = \sum_{k \in \{20, 40, \dots, 200\}} \sin(2\pi k z + \phi)$ with $N=1000$ points, increasing $L$ from 0 to 16 substantially accelerates the learning of high-frequency components of $f \circ \gamma_L$ , attenuating the spectral bias and speeding up overall loss convergence.
2. Binary classification: When classifying binarized sinusoids $\operatorname{sgn}(\sin(2\pi k z + \phi) - 0.5)$ for frequencies $k \in [50, 400]$ across manifolds with $L \in [0, 20]$ , higher manifold complexity $L$ systematically yields higher classification accuracy for the same target frequency $k$ .
3. Contrast with $K$ -Nearest Neighbors (KNN): Unlike deep neural networks, KNN classifiers trained on the same manifold tasks do not show systematic performance improvements as $L$ increases, demonstrating that DNNs uniquely leverage the manifold embedding geometry.
Knowl 9 — Architectural Effects of Depth, Width, and Weight Clipping on High-Frequency Expressivity
empirical result
In qualitative ablations fitting ReLU neural networks to a Dirac $\delta$ -like target function (which possesses a flat Fourier spectrum across all frequencies), the capacity and speed to fit high-frequency Fourier components depend strongly on network architectural hyperparameters:
1. Depth: Increasing network depth (e.g., from 3 to 6 layers at fixed width 16) produces a substantial, superlinear improvement in the network's ability to fit high-frequency modes.
2. Width: Increasing layer width (e.g., from 16 to 128 units at fixed depth 3) also enhances high-frequency fitting, but the effect is noticeably weaker per added parameter than increasing depth.
3. Weight magnitude constraint (Weight clip / Spectral norm): Constraining the maximum parameter magnitude $\|\theta\|_\infty$ (weight clipping from 0.1 to 2.0) directly restricts the network's Lipschitz constant $L_f$ and prevents high-frequency components from being learned, empirically confirming that the theoretical upper bound $\mathcal{O}(L_f)$ on high-frequency Fourier amplitudes is tight.
Knowl 10 — Connected Decision Regions and Near-Linear Paths Between Adversarial and Clean Images in CIFAR-10
empirical result
Because deep neural networks are biased toward low-frequency, globally continuous solutions rather than isolated high-frequency decision pockets, decision regions for a given class form connected components in image space.

Using the Automated Nudged Elastic Band (AutoNEB) algorithm without springs on a ResNet-20 trained on CIFAR-10:
- Continuous paths in the $32 \times 32 \times 3$ $32 \times 32 \times 3$ image space were constructed between:
  1. Pairs of training images of the same class $c$ ,
  2. A training image of class $c$ and an adversarial image generated from class $c' \ne c$ that is classified by the network as class $c$ (with $\ge 95\%$ softmax confidence),
  3. Pairs of adversarial images originating from different true classes that are both classified as class $c$ .
- For every pair of images sharing the same predicted class label across all 10 CIFAR-10 classes, AutoNEB found a continuous path connecting them along which the network's predicted class remained constant (never crossing a decision boundary).
- The constructed paths were strikingly near-linear, with an average path length only $(3.0 \pm 0.3)\%$ longer than the straight-line Euclidean interpolation.

Coverage note — Detailed derivation of polytope Fourier transforms via recursive Stokes theorem (which adapts the framework of Diaz et al., 2016) and literature reviews on kernel machine approximation rates were omitted to focus on the paper's original theoretical and empirical contributions.

References

1.Arora, R., Basu, A., Mianjy, P., and Mukherjee, A. Understanding deep neural networks with rectified linear units. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B1J_rgWRW.
2.Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization in deep networks. arXiv preprint arXiv:1706.05394, 2017.
3.Barron, A. R. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930–945, 1993.
4.Bengio, Y. et al. Learning deep architectures for ai. Foundations and trends® in Machine Learning, 2(1):1–127, 2009.
5.Bergner, S., Möller, T., Weiskopf, D., and Muraki, D. J. A spectral analysis of function concatenations and its implications for sampling in direct volume visualization.
6.Braun, M. L., Lange, T., and Buhmann, J. M. Model selection in kernel methods based on a spectral analysis of label information. In Joint Pattern Recognition Symposium, pp. 344–353. Springer, 2006.
7.Candès, E. J. Harmonic analysis of neural networks. Applied and Computational Harmonic Analysis, 6(2):197–218, 1999.
8.Cybenko, G. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems (MCSS), 2(4):303–314, 1989.
9.Devroye, L., Györfi, L., and Lugosi, G. Consistency of the k-nearest neighbor rule. In A Probabilistic Theory of Pattern Recognition, pp. 169–185. Springer, 1996.
10.Diaz, R., Le, Q.-N., and Robins, S. Fourier transforms of polytopes, solid angle sums, and discrete volume. arXiv preprint arXiv:1602.08593, 2016.
11.Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. A. Essentially no barriers in neural network energy landscape. arXiv preprint arXiv:1803.00885, 2018.
12.Eldan, R. and Shamir, O. The power of depth for feedforward neural networks. In Conference on Learning Theory, pp. 907–940, 2016.
13.Fasshauer, G. E. Positive definite kernels: past, present and future. Dolomite Research Notes on Approximation, 4: 21–63, 2011.
14.Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
15.Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
16.Hammer, B. and Gersmann, K. A note on the universal approximation capability of support vector machines. Neural Processing Letters, 17(1):43–53, 2003.
17.Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
18.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
19.Kolsbjerg, E. L., Groves, M. N., and Hammer, B. An automated nudged elastic band method. The Journal of chemical physics, 145(9):094107, 2016.
20.Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6):861–867, 1993.
21.Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. The expressive power of neural networks: A view from the width. In Advances in Neural Information Processing Systems, pp. 6231–6239, 2017.
22.Ma, S. and Belkin, M. Diving into the shallows: a computational perspective on large-scale shallow learning. In Advances in Neural Information Processing Systems, pp. 3781–3790, 2017.
23.Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B1QRgziT-.
24.Montufar, G. F., Pascanu, R., Cho, K., and Bengio, Y. On the number of linear regions of deep neural networks. In Advances in neural information processing systems, pp. 2924–2932, 2014.
25.Neyshabur, B., Tomioka, R., and Srebro, N. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614, 2014.
26.Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pp. 5949–5958, 2017.
27.Novak, R., Bahri, Y., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J. Sensitivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=HJC2SzZCW.
28.Poggio, T., Kawaguchi, K., Liao, Q., Miranda, B., Rosasco, L., Boix, X., Hidary, J., and Mhaskar, H. Theory of deep learning iii: the non-overfitting puzzle. Technical report, Technical report, CBMM memo 073, 2018.
29.Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. Exponential expressivity in deep neural networks through transient chaos. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29, pp. 3360–3368. Curran Associates, Inc., 2016.
30.Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-Dickstein, J. On the expressive power of deep neural networks. arXiv preprint arXiv:1606.05336, 2016.
31.Rasmussen, C. E. Gaussian processes in machine learning. In Advanced lectures on machine learning, pp. 63–71. Springer, 2004.
32.Serov, V. Fourier series, Fourier transform and their applications to mathematical physics. Springer, 2017.
33.Sonoda, S. and Murata, N. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233–268, 2017.
34.Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on separable data. arXiv preprint arXiv:1710.10345, 2017.
35.Spivak, M. Calculus On Manifolds: A Modern Approach To Classical Theorems Of Advanced Calculus. CRC press, 2018.
36.Telgarsky, M. Benefits of depth in neural networks. Conference on Learning Theory (COLT), 2016, 2016.
37.Xu, Z. J. Understanding training and generalization in deep learning by fourier analysis. arXiv preprint arXiv:1808.04295, 2018.
38.Xu, Z.-Q. J., Zhang, Y., and Xiao, Y. Training behavior of deep neural network in frequency domain. arXiv preprint arXiv:1807.01251, 2018.
39.Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. International Conference on Learning Representations (ICLR), 2017a.
40.Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017b.
41.Zhang, L., Naitzat, G., and Lim, L.-H. Tropical geometry of deep neural networks. arXiv preprint arXiv:1805.07091, 2018.

Citation

MLA

Rahaman, N., et al. “On the Spectral Bias of Neural Networks”. International Conference on Machine Learning, vol. 97, 2019, pp. 5301–10, https://proceedings.mlr.press/v97/rahaman19a.html.

APA

Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., & Courville, A. (2019). On the Spectral Bias of Neural Networks. International Conference on Machine Learning, 97, 5301–5310. https://proceedings.mlr.press/v97/rahaman19a.html

Chicago

Rahaman, N., A. Baratin, D. Arpit, et al. 2019. “On the Spectral Bias of Neural Networks”. International Conference on Machine Learning 97: 5301–10. https://proceedings.mlr.press/v97/rahaman19a.html.

Harvard

Rahaman, N. et al. (2019) “On the Spectral Bias of Neural Networks”, International Conference on Machine Learning. PMLR, pp. 5301–5310. Available at: https://proceedings.mlr.press/v97/rahaman19a.html.

Vancouver

1. Rahaman N, Baratin A, Arpit D, Draxler F, Lin M, Hamprecht F, Bengio Y, Courville A (2019) On the Spectral Bias of Neural Networks. In: International Conference on Machine Learning. PMLR, pp 5301–5310

BibTeX

@InProceedings{pmlr-v97-rahaman19a,
  title = 	 {On the Spectral Bias of Neural Networks},
  author =       {Rahaman, Nasim and Baratin, Aristide and Arpit, Devansh and Draxler, Felix and Lin, Min and Hamprecht, Fred and Bengio, Yoshua and Courville, Aaron},
  booktitle = 	 {Proceedings of the 36th International Conference on Machine Learning},
  pages = 	 {5301--5310},
  year = 	 {2019},
  editor = 	 {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
  volume = 	 {97},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {09--15 Jun},
  publisher =    {PMLR},
  pdf = 	 {http://proceedings.mlr.press/v97/rahaman19a/rahaman19a.pdf},
  url = 	 {https://proceedings.mlr.press/v97/rahaman19a.html},
  abstract = 	 {Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with 100% accuracy. In this work we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we highlight a learning bias of deep networks towards low frequency functions – i.e. functions that vary globally without local fluctuations – which manifests itself as a frequency-dependent learning speed. Intuitively, this property is in line with the observation that over-parameterized networks prioritize learning simple patterns that generalize across data samples. We also investigate the role of the shape of the data manifold by presenting empirical and theoretical evidence that, somewhat counter-intuitively, learning higher frequencies gets easier with increasing manifold complexity.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

On the Spectral Bias of Neural Networks

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Fourier Spectrum Decay of ReLU Neural Networks

Knowl 2 — Spectral Decay of Parameter Gradients and Residual Dynamics During Training

Knowl 3 — Parameter Space Volume of High-Frequency ReLU Network Functions

Knowl 4 — Frequency Coupling on Data Manifolds via Stationary Phase Approximation

Knowl 5 — Empirical Demonstration of Spectral Bias in Gradient Descent

Knowl 6 — Frequency-Dependent Robustness of Trained Networks to Parameter Perturbations

Knowl 7 — Effect of Target Noise Frequency on MNIST Generalization and Validation Dynamics

Knowl 8 — Manifold Complexity Accelerates Learning of High Frequencies

Knowl 9 — Architectural Effects of Depth, Width, and Weight Clipping on High-Frequency Expressivity

Knowl 10 — Connected Decision Regions and Near-Linear Paths Between Adversarial and Clean Images in CIFAR-10

References

Citation

BibTeX

Source Code

Access the Paper