When and why PINNs fail to train: A neural tangent kernel perspective

Sifan WangXinling YuParis Perdikaris

article2020Journal of Computational Physics1,794 citations

Explains why physics-informed neural networks fail to train using neural tangent kernel analysis and develops an adaptive weighting method based on kernel eigenvalues to resolve convergence discrepancies across loss terms.

Listen

Physics-informed neural networks offer a flexible approach for modeling complex engineering and physical systems described by differential equations. Despite widespread adoption, standard fully-connected networks frequently fail during training, especially when simulating systems with multi-scale behaviors or high-frequency variations. Understanding the root causes of these training failures is critical for building trustworthy, robust machine learning tools for scientific and industrial computing.

This article investigates the training dynamics of physics-informed neural networks through neural tangent kernel theory to determine why standard training fails and how to systematically resolve these failures.

The authors analyze the theoretical behavior of neural networks in the infinite-width limit during gradient descent optimization. They derive the specific mathematical kernel that governs physics-informed networks and evaluate training dynamics across canonical benchmark problems, including a one-dimensional Poisson equation and a one-dimensional wave equation using single- and multi-layer architectures.

The analysis reveals three key findings regarding network behavior. First, as the network becomes very wide, the governing kernel converges to a fixed, deterministic matrix that remains practically constant throughout training. Second, standard models exhibit severe spectral bias, causing gradient descent to learn low-frequency features rapidly while learning high-frequency components extremely slowly. Third, a fundamental imbalance emerges during optimization because the training error rates of the physics residual and the boundary conditions are governed by vastly different kernel eigenvalues. Typically, the physics residual dominates, causing the model to satisfy the internal differential equation while failing to fit the boundary and initial conditions.

These findings explain why standard physics-informed networks often yield inaccurate results or fail completely: the optimization process prioritizes one loss component at the expense of others. To correct this pathology, the authors introduce an adaptive training algorithm that uses the trace of the kernel matrices to balance the convergence rates across all loss terms. In numerical benchmarks, this adaptive weighting scheme improved predictive accuracy by approximately two orders of magnitude, reducing relative error from over 40% down to less than 0.2% on a challenging wave equation problem without requiring costly trial-and-error hyperparameter tuning.

Organizations developing machine learning for physical modeling should incorporate eigenvalue- or trace-based adaptive weighting schemes into their training pipelines to stabilize convergence and eliminate manual weight calibration. While the formal mathematical proofs are primarily established for single-layer networks solving linear problems under infinitesimal learning rates, numerical experiments demonstrate high empirical stability across deeper architectures and modern optimizers. Future work should focus on extending formal theoretical guarantees to nonlinear equations, advanced network architectures, and inverse problem settings.

University of Pennsylvania deterministic differential equations eigenvalues neural tangent kernel PINN loss function training dynamics

arXiv: 2007.14527 PredictiveIntelligenceLab/PINNsNTK

Paper: Neural Tangent Kernel: Convergence and Generalization in Neural Networks, Arthur Jacot et al. (2018). This foundational work introduces the Neural Tangent Kernel framework and establishes how infinite-width neural network training dynamics under gradient descent reduce to linear differential equations governed by a deterministic kernel.
Paper: DeepXDE: A Deep Learning Library for Solving Differential Equations, Lu Lu et al. (2019). This paper establishes the core physics-informed neural network formulation and software implementation for enforcing partial differential equations and boundary conditions via composite loss functions.
Paper: DGM: A deep learning algorithm for solving partial differential equations, Justin Sirignano et al. (2017). This work introduces deep learning methods for high-dimensional partial differential equations trained on batches of random collocation points, serving as essential context for understanding neural PDE training mechanics.
Paper: The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems, Weinan E et al. (2017). This paper presents variational deep learning formulations for differential equations, providing foundational background on optimization challenges in neural network-based PDE solvers.

Paper: Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next, Salvatore Cuomo et al. (2022). This comprehensive review synthesizes developments across physics-informed neural networks, surveying training failure modes and modern optimization algorithms like NTK-based weighting.
Paper: Physics-informed neural networks (PINNs) for fluid mechanics: a review, Shengze Cai et al. (2021). This survey examines physics-informed neural networks applied specifically to fluid dynamics, discussing practical training difficulties and algorithmic enhancements for complex Navier-Stokes problems.
Paper: Fourier Neural Operator for Parametric Partial Differential Equations, Zongyi Li et al. (2020). This paper introduces Fourier Neural Operators as an alternative operator-learning paradigm that circumvents point-wise PINN training dynamics by mapping directly between function spaces.
Paper: Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Lu Lu et al. (2021). This work formulates DeepONet for learning general nonlinear operators, advancing beyond single-instance PINN training by building parameterized operator surrogates.
Paper: KAN: Kolmogorov-Arnold Networks, Ziming Liu et al. (2025). This text explores Kolmogorov-Arnold Networks as an architectural alternative to multilayer perceptrons for scientific machine learning and PDE solving, addressing traditional MLP optimization barriers.
Paper: Neural means and kernel corrections for operator learning, Yitzchak Shmalo (2026). This book bridges neural scientific surrogates with exact kernel methods, expanding on the interplay between deep networks and kernel dynamics in scientific machine learning.

Cover for When and why PINNs fail to train: A neural tangent kernel perspective

Abstract

Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at \url{this https URL}.

Topics

deterministic differential equations eigenvalues inverse problems neural networks numerical experiments partial differential equations numerical solution optimization problems

Keywords

neural tangent kernel PINN loss function training dynamics gradient flow kernel regression adaptive training full-batch gradient descent stochastic gradient descent Adam optimizer kernel matrix

1 Introduction
2 Infinitely Wide Neural Networks
3 Physics-informed Neural Networks (PINNs)
3.1 Neural tangent kernel theory for PINNs
4 Analyzing the training dynamics of PINNs through the lens of their NTK
5 Spectral bias in physics-informed neural networks
6 Practical insights
7 Numerical Experiments
7.1 Convergence of the NTK of PINNs
7.2 Adaptive training for PINNs
7.3 One-dimensional wave equation
8 Discussion
References
A Proof of Lemma
B Proof of Theorem
C Proof of Theorem
D Proof of Theorem

Knowls

Knowl 1 — Adaptive NTK-Based Weighting Algorithm for PINNs
algorithm
To alleviate the convergence discrepancy between different loss components in physics-informed neural networks (PINNs), the loss term weights are adaptively calibrated using the traces of their corresponding Neural Tangent Kernel (NTK) sub-matrices.

Consider a PDE $\mathcal{L}[u](x) = f(x)$ for $x \in \Omega$ with boundary condition $u(x) = g(x)$ for $x \in \partial \Omega$ , approximated by a neural network $u(x, \theta)$ with parameter vector $\theta$ . The composite loss function is: $\mathcal{L}(\theta) = \lambda_b \mathcal{L}_b(\theta) + \lambda_r \mathcal{L}_r(\theta)$ where $\mathcal{L}_b(\theta) = \frac{1}{2 N_b} \sum_{i=1}^{N_b} |u(x_b^i, \theta) - g(x_b^i)|^2$ and $\mathcal{L}_r(\theta) = \frac{1}{2 N_r} \sum_{i=1}^{N_r} |\mathcal{L}u(x_r^i, \theta) - f(x_r^i)|^2$ .
Input: Boundary training points $\{x_b^i, g(x_b^i)\}_{i=1}^{N_b}$ , residual points $\{x_r^i, f(x_r^i)\}_{i=1}^{N_r}$ , learning rate $\eta$ , total steps $S$ , update frequency $F$ .
Output: Optimized network parameters $\theta_S$ .
Initialize network parameters $\theta_0$ , set $\lambda_b = 1$ , $\lambda_r = 1$ .
for $n = 0, 1, \dots, S - 1$ do
    if $n \pmod F == 0$ then
        Compute Jacobian matrices $J_u(n) = \nabla_\theta u(x_b, \theta_n)$ and $J_r(n) = \nabla_\theta \mathcal{L}u(x_r, \theta_n)$ .
        Compute NTK sub-matrices $K_{uu}(n) = J_u(n) J_u(n)^T$ and $K_{rr}(n) = J_r(n) J_r(n)^T$ .
        Compute total NTK trace $\text{Tr}(K(n)) = \text{Tr}(K_{uu}(n)) + \text{Tr}(K_{rr}(n))$ .
        Update weights:
         $\lambda_b = \frac{\text{Tr}(K(n))}{\text{Tr}(K_{uu}(n))}$
         $\lambda_r = \frac{\text{Tr}(K(n))}{\text{Tr}(K_{rr}(n))}$
    end if
    Compute loss gradient $\nabla_\theta \mathcal{L}(\theta_n) = \lambda_b \nabla_\theta \mathcal{L}_b(\theta_n) + \lambda_r \nabla_\theta \mathcal{L}_r(\theta_n)$ .
    Update parameters: $\theta_{n+1} = \theta_n - \eta \nabla_\theta \mathcal{L}(\theta_n)$ .
end for
return $\theta_S$
For problems with $M$ separate loss components $\{\mathcal{L}_m\}_{m=1}^M$ (such as distinct initial and boundary conditions), the update generalizes to $\lambda_k = \frac{\sum_{m=1}^M \text{Tr}(K_m(n))}{\text{Tr}(K_k(n))}$ for each component $k \in \{1, \dots, M\}$ .
Knowl 2 — Continuous Gradient Flow Dynamics and Neural Tangent Kernel of PINNs
theoretical result
Consider a partial differential equation $\mathcal{L}[u](x) = f(x)$ on $\Omega \subset \mathbb{R}^d$ with boundary condition $u(x) = g(x)$ on $\partial \Omega$ , parameterized by a neural network $u(x, \theta)$ with parameter vector $\theta$ . Given boundary data $\{x_b^i, g(x_b^i)\}_{i=1}^{N_b}$ and collocation points $\{x_r^i, f(x_r^i)\}_{i=1}^{N_r}$ , the composite loss is: $\mathcal{L}(\theta) = \frac{1}{2}\sum_{i=1}^{N_b} |u(x_b^i, \theta) - g(x_b^i)|^2 + \frac{1}{2}\sum_{i=1}^{N_r} |\mathcal{L}u(x_r^i, \theta) - f(x_r^i)|^2$ Under continuous-time gradient descent flow $\frac{d\theta}{dt} = -\nabla_\theta \mathcal{L}(\theta)$ , the time evolution of the network outputs $u(x_b, \theta(t))$ and $\mathcal{L}u(x_r, \theta(t))$ satisfies: $\begin{bmatrix} \frac{du(x_b, \theta(t))}{dt} \\[6pt] \frac{d\mathcal{L}u(x_r, \theta(t))}{dt} \end{bmatrix} = -\mathbf{K}(t) \begin{bmatrix} u(x_b, \theta(t)) - g(x_b) \\[6pt] \mathcal{L}u(x_r, \theta(t)) - f(x_r) \end{bmatrix}$ where the PINN Neural Tangent Kernel $\mathbf{K}(t)$ is defined blockwise by: $\mathbf{K}(t) = \begin{bmatrix} \mathbf{K}_{uu}(t) & \mathbf{K}_{ur}(t) \\[4pt] \mathbf{K}_{ru}(t) & \mathbf{K}_{rr}(t) \end{bmatrix} = \begin{bmatrix} J_u(t) \\[4pt] J_r(t) \end{bmatrix} \begin{bmatrix} J_u(t)^T & J_r(t)^T \end{bmatrix}$ with entries given by the parameter inner products: $(\mathbf{K}_{uu})_{ij}(t) = \left\langle \frac{\partial u(x_b^i, \theta(t))}{\partial \theta}, \frac{\partial u(x_b^j, \theta(t))}{\partial \theta} \right\rangle$ $(\mathbf{K}_{ur})_{ij}(t) = \left\langle \frac{\partial u(x_b^i, \theta(t))}{\partial \theta}, \frac{\partial \mathcal{L}u(x_r^j, \theta(t))}{\partial \theta} \right\rangle$ $(\mathbf{K}_{rr})_{ij}(t) = \left\langle \frac{\partial \mathcal{L}u(x_r^i, \theta(t))}{\partial \theta}, \frac{\partial \mathcal{L}u(x_r^j, \theta(t))}{\partial \theta} \right\rangle$ and $\mathbf{K}_{ru}(t) = \mathbf{K}_{ur}(t)^T$ . The matrix $\mathbf{K}(t)$ is symmetric and positive semi-definite.
Knowl 3 — Constancy of PINN Neural Tangent Kernel During Training
theoretical result
For a one-dimensional Poisson problem $u_{xx}(x) = f(x)$ on $\Omega$ with $u(x) = g(x)$ on $\partial \Omega$ , approximated by a fully-connected single-hidden-layer neural network $u(x, \theta) = \frac{1}{\sqrt{N}} W^{(1)} \sigma(W^{(0)} x + b^{(0)}) + b^{(1)}$ trained via continuous gradient descent flow with infinitesimal learning rate, suppose for any fixed time horizon $T > 0$ :
1. Parameters remain uniformly bounded: $\sup_{t \in [0, T]} \|\theta(t)\|_\infty \le C$ for a constant $C$ independent of width $N$ .
2. Cumulative training errors remain bounded: $\int_0^T \left| \sum_{i=1}^{N_b} (u(x_b^i, \theta(\tau)) - g(x_b^i)) \right| d\tau \le C, \quad \int_0^T \left| \sum_{i=1}^{N_r} (u_{xx}(x_r^i, \theta(\tau)) - f(x_r^i)) \right| d\tau \le C$
3. The activation function $\sigma$ is smooth and satisfies $|\sigma^{(k)}| \le C$ for all derivatives $k \in \{0, 1, 2, 3, 4\}$ .
Then, in the infinite-width limit, the PINN Neural Tangent Kernel $\mathbf{K}(t)$ remains constant over the entire training trajectory: $\lim_{N \to \infty} \sup_{t \in [0, T]} \|\mathbf{K}(t) - \mathbf{K}(0)\|_2 = 0$
Knowl 4 — Deterministic Limiting Neural Tangent Kernel of PINNs at Initialization
theoretical result
For a single-hidden-layer fully-connected network $u(x, \theta) = \frac{1}{\sqrt{N}} W^{(1)} \sigma(W^{(0)} x + b^{(0)}) + b^{(1)}$ with independent standard normal parameter initialization $\mathcal{N}(0,1)$ , the PINN Neural Tangent Kernel at initialization $\mathbf{K}(0)$ converges in probability as the width $N \to \infty$ to a deterministic limiting kernel matrix: $\mathbf{K}(0) = \begin{bmatrix} \mathbf{K}_{uu}(0) & \mathbf{K}_{ur}(0) \\[4pt] \mathbf{K}_{ru}(0) & \mathbf{K}_{rr}(0) \end{bmatrix} \xrightarrow{\mathcal{P}} \begin{bmatrix} \Theta_{uu}^{(1)} & \Theta_{ur}^{(1)} \\[4pt] \Theta_{ru}^{(1)} & \Theta_{rr}^{(1)} \end{bmatrix} =: \mathbf{K}^*$ where for inputs $x, x'$ : $\Theta_{uu}^{(1)}(x, x') = \dot{\Sigma}^{(1)}(x, x')(x x') + \Sigma^{(1)}(x, x') + \dot{\Sigma}^{(1)}(x, x') + 1$ $\Theta_{rr}^{(1)}(x, x') = A_{rr}(x, x') + B_{rr}(x, x') + C_{rr}(x, x')$ $\Theta_{ur}^{(1)}(x, x') = A_{ur}(x, x') + B_{ur}(x, x') + C_{ur}(x, x')$ with covariance $\Sigma^{(1)}(x, x') = \mathbb{E}_{(u,v) \sim \mathcal{N}(0, \Lambda^{(1)})}[\sigma(u)\sigma(v)] + 1$ , derivative covariance $\dot{\Sigma}^{(1)}(x, x') = \mathbb{E}_{(u,v) \sim \mathcal{N}(0, \Lambda^{(1)})}[\dot{\sigma}(u)\dot{\sigma}(v)]$ , and components defined via expectations over standard Gaussians $W_k^{(0)} \sim \mathcal{N}(0,1)$ : $A_{rr} = \mathbb{E}[(W_k^{(0)})^4 \dddot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \dddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})] x x' + 2\mathbb{E}[(W_k^{(0)})^3 \dddot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \ddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})] x + 2\mathbb{E}[(W_k^{(0)})^3 \dddot{\sigma}(W_k^{(0)}x'+b_k^{(0)}) \ddot{\sigma}(W_k^{(0)}x+b_k^{(0)})] x' + 4\mathbb{E}[(W_k^{(0)})^2 \ddot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \ddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})]$ $B_{rr} = \mathbb{E}[(W_k^{(0)})^4 \ddot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \ddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})]$ $C_{rr} = \mathbb{E}[(W_k^{(0)})^4 \dddot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \dddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})]$ $A_{ur} = \mathbb{E}[(W_k^{(0)})^2 \dot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \dddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})] x x' + 2\mathbb{E}[W_k^{(0)} \dot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \ddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})] x$ $B_{ur} = \mathbb{E}[(W_k^{(0)})^2 \sigma(W_k^{(0)}x+b_k^{(0)}) \ddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})]$ $C_{ur} = \mathbb{E}[(W_k^{(0)})^2 \dot{\sigma}(W_k^{(0)}x+b_k^{(0)}) \dddot{\sigma}(W_k^{(0)}x'+b_k^{(0)})]$
Knowl 5 — Gaussian Process Correspondence for Infinitely Wide PINNs on Linear PDEs
theoretical result
For a single-hidden-layer fully-connected network $u(x, \theta) = \frac{1}{\sqrt{N}} W^{(1)} \sigma(W^{(0)} x + b^{(0)}) + b^{(1)}$ with independent standard normal parameter initialization $\mathcal{N}(0, 1)$ and a smooth activation function $\sigma$ with bounded second derivative $\ddot{\sigma}$ , the network output $u(x, \theta)$ and its second derivative $u_{xx}(x, \theta)$ asymptotically converge in distribution as width $N \to \infty$ to centered Gaussian processes: $u(x, \theta) \xrightarrow{\mathcal{D}} \mathcal{GP}\left(0, \Sigma^{(1)}(x, x')\right)$ $u_{xx}(x, \theta) \xrightarrow{\mathcal{D}} \mathcal{GP}\left(0, \Sigma_{xx}^{(1)}(x, x')\right)$ where: $\Sigma^{(1)}(x, x') = \mathbb{E}_{(u,v) \sim \mathcal{N}(0, \Lambda^{(1)})}[\sigma(u)\sigma(v)] + 1, \quad \Lambda^{(1)}(x, x') = \begin{bmatrix} x^T x + 1 & x^T x' + 1 \\[4pt] x'^T x + 1 & x'^T x' + 1 \end{bmatrix}$ $\Sigma_{xx}^{(1)}(x, x') = \mathbb{E}_{u,v \sim \mathcal{N}(0,1)}\left[ u^4 \ddot{\sigma}(ux + v) \ddot{\sigma}(ux' + v) \right]$ Because linear combinations and differential operations on Gaussian processes preserve Gaussianity, this PINN-GP correspondence generalizes to any linear partial differential operator $\mathcal{L}$ under suitable regularity conditions.
Knowl 6 — Convergence Rate Discrepancy and Spectral Bias in PINNs
theoretical result
Given the positive semi-definite PINN NTK $\mathbf{K}(0) = \mathbf{Q}^T \mathbf{\Lambda} \mathbf{Q}$ with orthonormal eigenvectors $\mathbf{Q}$ and non-negative eigenvalues $\mathbf{\Lambda} = \text{diag}(\lambda_1, \dots, \lambda_n)$ , the training error under gradient flow evolves as: $\mathbf{Q} \left( \begin{bmatrix} u(x_b, \theta(t)) \\[4pt] u_{xx}(x_r, \theta(t)) \end{bmatrix} - \begin{bmatrix} g(x_b) \\[4pt] f(x_r) \end{bmatrix} \right) \approx -e^{-\mathbf{\Lambda} t} \mathbf{Q} \begin{bmatrix} g(x_b) \\[4pt] f(x_r) \end{bmatrix}$ Each error component along eigenvector $i$ decays at exponential rate $e^{-\lambda_i t}$ . Higher eigenvalues correspond to lower-frequency components, causing rapid decay for low frequencies and severe slowing for high frequencies (spectral bias).

The average convergence rate of an $n \times n$ kernel matrix $\mathbf{K}$ is defined as: $c = \frac{\sum_{i=1}^n \lambda_i}{n} = \frac{\text{Tr}(\mathbf{K})}{n}$ Kernel $\mathbf{K}_1$ dominates $\mathbf{K}_2$ if $c_1 \gg c_2$ . In standard PINNs, the residual kernel $\mathbf{K}_{rr}$ typically dominates the boundary kernel $\mathbf{K}_{uu}$ ( $c_{rr} \gg c_{uu}$ ). As a consequence, the PDE residual loss converges significantly faster than the boundary condition loss, preventing the network from learning the correct boundary-constrained PDE solution.
Knowl 7 — Gradient Flow Dynamics and Forward Euler Stability Limit for Weighted PINN Losses
theoretical result
For a weighted PINN loss function with component weights $\lambda_b$ and $\lambda_r$ : $\mathcal{L}(\theta) = \frac{\lambda_b}{2 N_b} \sum_{i=1}^{N_b} |u(x_b^i, \theta) - g(x_b^i)|^2 + \frac{\lambda_r}{2 N_r} \sum_{i=1}^{N_r} |\mathcal{L}u(x_r^i, \theta) - f(x_r^i)|^2$ the continuous gradient descent dynamics of the predictions follow: $\begin{bmatrix} \frac{du(x_b, \theta(t))}{dt} \\[6pt] \frac{d\mathcal{L}u(x_r, \theta(t))}{dt} \end{bmatrix} = -\tilde{\mathbf{K}}(t) \begin{bmatrix} u(x_b, \theta(t)) - g(x_b) \\[6pt] \mathcal{L}u(x_r, \theta(t)) - f(x_r) \end{bmatrix}$ where the effective weighted NTK is: $\tilde{\mathbf{K}}(t) = \begin{bmatrix} \frac{\lambda_b}{N_b}\mathbf{K}_{uu}(t) & \frac{\lambda_r}{N_r}\mathbf{K}_{ur}(t) \\[6pt] \frac{\lambda_b}{N_b}\mathbf{K}_{ru}(t) & \frac{\lambda_r}{N_r}\mathbf{K}_{rr}(t) \end{bmatrix}$ When discretized with forward Euler (gradient descent with learning rate $\eta$ ), numerical stability requires: $\eta \le \frac{2}{\lambda_{\max}(\tilde{\mathbf{K}}(t))}$ In terms of convergence rate effects, scaling the weight $\lambda_b$ or $\lambda_r$ is mathematically equivalent to inverse-scaling the corresponding batch size $N_b$ or $N_r$ .
Knowl 8 — Empirical Validation of Adaptive Weighting on 1D Poisson Equation
empirical result
The adaptive NTK weighting method was evaluated on a 1D Poisson boundary value problem: $u_{xx}(x) = -16\pi^2 \sin(4\pi x), \quad x \in [0, 1], \quad u(0) = u(1) = 0$ with exact solution $u(x) = \sin(4\pi x)$ . A fully-connected neural network with 1 hidden layer of width 100 and $\tanh$ activations was trained using full-batch gradient descent (batch sizes $N_b = N_r = 100$ ) with learning rate $10^{-5}$ for 40,000 iterations.
- Standard PINN (unweighted baseline with $\lambda_b = \lambda_r = 1$ ) achieved a relative $L^2$ error of $2.40 \times 10^{-1}$ .
- NTK-calibrated PINN (fixing $\lambda_r = 1$ and setting $\lambda_b = \frac{\text{Tr}(\mathbf{K}_{rr}(0))}{\text{Tr}(\mathbf{K}_{uu}(0))} \approx 100$ ) achieved a relative $L^2$ error of $1.63 \times 10^{-3}$ , improving predictive accuracy by over two orders of magnitude.
- Manual hyper-parameter sweeps across $\lambda_b \in [1, 500]$ confirmed that the relative $L^2$ error reaches its global minimum near $\lambda_b \approx 100$ , verifying that the NTK trace ratio directly identifies the optimal weighting.
Knowl 9 — Resolution of Training Failure on 1D Wave Equation via Adaptive NTK Weighting
empirical result
The adaptive NTK weighting method was tested on a 1D wave equation exhibiting high stiffness: $u_{tt}(x, t) - 4u_{xx}(x, t) = 0, \quad (x,t) \in (0, 1) \times (0, 1)$ with boundary conditions $u(0, t) = u(1, t) = 0$ , initial position $u(x, 0) = \sin(\pi x) + 0.5\sin(4\pi x)$ , and initial velocity $u_t(x, 0) = 0$ . The exact analytical solution is $u(x, t) = \sin(\pi x)\cos(2\pi t) + 0.5\sin(4\pi x)\cos(8\pi t)$ .

A 5-layer fully-connected network with 500 neurons per layer was trained using the Adam optimizer for 80,000 steps with mini-batch sizes $N_u = N_{u_t} = N_r = 300$ .
- Standard PINN (equal loss weights $\lambda_u = \lambda_{u_t} = \lambda_r = 1$ ) completely failed to learn the wave solution, yielding a relative $L^2$ error of $4.518 \times 10^{-1}$ (over $45\%$ error), caused by $\mathbf{K}_r$ and $\mathbf{K}_{u_t}$ dominating $\mathbf{K}_u$ .
- Adaptive NTK-weighted PINN (updating weights $\lambda_u, \lambda_{u_t}, \lambda_r$ every 1,000 steps via trace ratios $\lambda_k = \frac{\text{Tr}(\mathbf{K}_u) + \text{Tr}(\mathbf{K}_{u_t}) + \text{Tr}(\mathbf{K}_r)}{\text{Tr}(\mathbf{K}_k)}$ ) resolved the eigenvalue discrepancy and achieved a relative $L^2$ error of $1.728 \times 10^{-3}$ .
Knowl 10 — Methodological and Theoretical Limitations of NTK Analysis for PINNs
limitation
The NTK formulation for PINNs possesses three primary limitations:
1. Loss weighting adjusts relative convergence rates across tasks but does not alter the underlying eigenvalue spectrum within individual kernel blocks; consequently, adaptive loss weighting cannot resolve the fundamental spectral bias of neural networks against high frequencies.
2. The weighted continuous dynamics matrix $\tilde{\mathbf{K}}$ is generally asymmetric and indefinite, which can introduce complex eigenvalues and lead to numerical instability during training if weights $\lambda_b$ or $\lambda_r$ become excessively large.
3. The theoretical proofs of NTK constancy during training and Gaussian process convergence are rigorously derived only for single-hidden-layer networks applied to linear Poisson equations under continuous gradient flow. Extensions to deep multi-layer architectures, non-linear partial differential equations, and momentum-based optimizers (such as Adam) remain empirical observations and conjectures.

Coverage note — None omitted; all primary theoretical derivations, theorems, algorithms, empirical validations, and limitations contributed by the paper have been extracted as self-contained knowls.

References

1.Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481):1026–1030, 2020.
2.Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:112732, 2020.
3.Maziar Raissi, Hessam Babaee, and Peyman Givi. Deep learning of turbulent scalar mixing. Physical Review Fluids, 4(12):124501, 2019.
4.Maziar Raissi, Zhicheng Wang, Michael S Triantafyllou, and George Em Karniadakis. Deep learning of vortex-induced vibrations. Journal of Fluid Mechanics, 861:119–137, 2019.
5.Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. arXiv preprint arXiv:2003.06496, 2020.
6.Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics, 8:42, 2020.
7.Georgios Kissas, Yibo Yang, Eileen Hwuang, Walter R Witschey, John A Detre, and Paris Perdikaris. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4d flow mri data using physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 358:112623, 2020.
8.Zhiwei Fang and Justin Zhan. Deep physical informed neural networks for metamaterial design. IEEE Access, 8:24506–24513, 2019.
9.Dehao Liu and Yan Wang. Multi-fidelity physics-constrained neural network and its application in materials modeling. Journal of Mechanical Design, 141(12), 2019.
10.Yuyao Chen, Lu Lu, George Em Karniadakis, and Luca Dal Negro. Physics-informed neural networks for inverse problems in nano-optics and metamaterials. Optics Express, 28(8):11618–11633, 2020.
11.Sifan Wang and Paris Perdikaris. Deep learning of free boundary and stefan problems. arXiv preprint arXiv:2006.05311, 2020.
12.Yibo Yang and Paris Perdikaris. Adversarial uncertainty quantification in physics-informed neural networks. Journal of Computational Physics, 394:136–152, 2019.
13.Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56–81, 2019.
14.Yibo Yang and Paris Perdikaris. Physics-informed deep generative models. arXiv preprint arXiv:1812.03511, 2018.
15.Luning Sun and Jian-Xun Wang. Physics-constrained bayesian neural network for fluid flow reconstruction with sparse and noisy data. arXiv preprint arXiv:2001.05542, 2020.
16.Liu Yang, Xuhui Meng, and George Em Karniadakis. B-pinns: Bayesian physics-informed neural networks for forward and inverse pde problems with noisy data. arXiv preprint arXiv:2003.06097, 2020.
17.Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339–1364, 2018.
18.Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
19.Dongkun Zhang, Ling Guo, and George Em Karniadakis. Learning in modal space: Solving time-dependent stochastic PDEs using physics-informed neural networks. SIAM Journal on Scientific Computing, 42(2):A639–A665, 2020.
20.Guofei Pang, Lu Lu, and George Em Karniadakis. fpinns: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 41(4):A2603–A2626, 2019.
21.Guofei Pang, Marta D’Elia, Michael Parks, and George E Karniadakis. npinns: nonlocal physics-informed neural networks for a parametrized nonlocal universal laplacian operator. algorithms and applications. arXiv preprint arXiv:2004.04276, 2020.
22.Alexandre M Tartakovsky, Carlos Ortiz Marrero, Paris Perdikaris, Guzel D Tartakovsky, and David Barajas-Solano. Learning parameters and constitutive relationships with physics informed deep neural networks. arXiv preprint arXiv:1808.03398, 2018.
23.Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
24.AM Tartakovsky, C Ortiz Marrero, Paris Perdikaris, GD Tartakovsky, and D Barajas-Solano. Physics-informed deep neural networks for learning parameters and constitutive relationships in subsurface flow problems. Water Resources Research, 56(5):e2019WR026731, 2020.
25.Yeonjong Shin, Jerome Darbon, and George Em Karniadakis. On the convergence and generalization of physics informed neural networks. arXiv preprint arXiv:2004.01806, 2020.
26.Hamdi A Tchelepi and Olga Fuks. Limitations of physics informed machine learning for nonlinear two-phase transport in porous media. Journal of Machine Learning for Modeling and Computing, 1(1), 2020.
27.Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1):932–955, 2018.
28.Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:2001.04536, 2020.
29.Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301–5310, 2019.
30.Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv:1912.01198, 2019.
31.Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.10739, 2020.
32.Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. arXiv preprint arXiv:2003.04560, 2020.
33.Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571–8580, 2018.
34.Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760, 2019.
35.Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv:1804.11271, 2018.
36.Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165, 2017.
37.David JC MacKay. Introduction to gaussian processes. NATO ASI Series F Computer and Systems Sciences, 168:133–166, 1998.
38.Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, pages 8141–8150, 2019.
39.Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.
40.Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in neural information processing systems, pages 8572–8583, 2019.
41.Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523, 2019.
42.Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Advances in Neural Information Processing Systems, pages 4761–4771, 2019.
43.Lu Lu, Xuhui Meng, Zhiping Mao, and George E Karniadakis. Deepxde: A deep learning library for solving differential equations. arXiv preprint arXiv:1907.04502, 2019.
44.Parviz Moin. Fundamentals of engineering numerical analysis. Cambridge University Press, 2010.
45.L.C. Evans and American Mathematical Society. Partial Differential Equations. Graduate studies in mathematics. American Mathematical Society, 1998.
46.Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256, 2010.
47.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

Citation

MLA

Wang, S., et al. “When and Why PINNs Fail to Train: A Neural Tangent Kernel Perspective”. arXiv, 2020, http://arxiv.org/abs/2007.14527v1.

APA

Wang, S., Yu, X., & Perdikaris, P. (2020). When and why PINNs fail to train: A neural tangent kernel perspective. arXiv. http://arxiv.org/abs/2007.14527v1

Chicago

Wang, S., X. Yu, and P. Perdikaris. 2020. “When and Why PINNs Fail to Train: A Neural Tangent Kernel Perspective”. arXiv. http://arxiv.org/abs/2007.14527v1.

Harvard

Wang, S., Yu, X. and Perdikaris, P. (2020) “When and why PINNs fail to train: A neural tangent kernel perspective”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2007.14527v1.

Vancouver

1. Wang S, Yu X, Perdikaris P (2020) When and why PINNs fail to train: A neural tangent kernel perspective. arXiv

BibTeX

@article{wang2020when,
  title = {When and why PINNs fail to train: A neural tangent kernel perspective},
  author = {Wang, Sifan and Yu, Xinling and Perdikaris, Paris},
  year = {2020},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2007.14527v1},
  eprint = {2007.14527}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

When and why PINNs fail to train: A neural tangent kernel perspective

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Adaptive NTK-Based Weighting Algorithm for PINNs

Knowl 2 — Continuous Gradient Flow Dynamics and Neural Tangent Kernel of PINNs

Knowl 3 — Constancy of PINN Neural Tangent Kernel During Training

Knowl 4 — Deterministic Limiting Neural Tangent Kernel of PINNs at Initialization

Knowl 5 — Gaussian Process Correspondence for Infinitely Wide PINNs on Linear PDEs

Knowl 6 — Convergence Rate Discrepancy and Spectral Bias in PINNs

Knowl 7 — Gradient Flow Dynamics and Forward Euler Stability Limit for Weighted PINN Losses

Knowl 8 — Empirical Validation of Adaptive Weighting on 1D Poisson Equation

Knowl 9 — Resolution of Training Failure on 1D Wave Equation via Adaptive NTK Weighting

Knowl 10 — Methodological and Theoretical Limitations of NTK Analysis for PINNs

References

Citation

BibTeX

Source Code

Access the Paper