Exact learning dynamics of deep linear networks with prior knowledge

Lukas BraunClémentine C. J. DominéJames FitzgeraldAndrew M. Saxe

article2022NeurIPS63 citations

Derives exact analytical solutions for the learning dynamics of deep linear networks from structured initial weights, revealing how prior knowledge changes training speed and weight alignment across tasks such as continual and reversal learning.

Listen

Deep neural networks are central to modern artificial intelligence, yet how initial network states and pre-existing knowledge influence training trajectories remains poorly understood. Machine learning practitioners frequently rely on pre-training, fine-tuning, and transfer learning, but the mathematical principles linking prior knowledge to learning speed, representation structure, and failure modes have remained elusive. This gap makes it difficult to reliably forecast model training behavior, prevent catastrophic forgetting in sequential tasks, or design optimal initialization strategies.

The main objective of the article is to provide an exact, closed-form mathematical description of gradient-based learning dynamics in deep linear neural networks trained from structured initial weights. It demonstrates how diverse initializations alter learning speed, internal representations, and model performance across sequential learning, reversal learning, and hierarchical task revision.

To achieve this, the article extends a mathematical framework based on matrix Riccati differential equations to describe two-layer networks under full-batch gradient descent. The approach accommodates both equal and unequal input-output dimensions and derives closed-form expressions that remain numerically stable by relying entirely on decaying exponential terms. The analytical solutions were validated against numerical simulations across dozens of network architectures, learning rates, and structured benchmark tasks, including hierarchical semantic categories.

The findings establish several fundamental insights into neural network training. First, the derived analytical solution exactly tracks the evolution of training loss, network functions, internal representational similarity, and kernel metrics over time. Second, the analysis reveals that network initialization variance can decouple training speed from representation richness: large zero-balanced initial weights produce fast, exponential learning trajectories while still converging to the exact, rich internal representations traditionally associated only with slow, step-like learning. Third, network weights dynamically decouple and realign with task structure during training, explaining why tabula rasa networks with small weights succeed through a fast, silent alignment phase. Fourth, in sequential task learning, catastrophic forgetting is complete for linear networks and can be calculated before training begins purely from the geometric similarity between tasks. Finally, when revising existing knowledge or reversing task associations, deep networks encounter mathematical saddle points that cause catastrophic slowing, an impediment not present in shallow networks.

These insights demonstrate that fast learning does not require sacrificing rich, generalizable internal representations, providing a theoretical foundation to optimize initialization schemes and reduce compute time. However, the findings also show that possessing pre-existing structured knowledge offers no inherent learning speedup when the task structure must be revised, and in cases of association reversals, it severely stalls gradient-based optimization. Furthermore, the exact predictability of catastrophic forgetting highlights the fundamental limitation of standard gradient descent in retaining historical task knowledge without explicit preservation mechanisms.

For practical implementation, engineering teams should explore structured, zero-balanced initialization schemes that accelerate convergence while preserving representational quality. When designing continual learning pipelines, practitioners should utilize task-similarity metrics to forecast forgetting risks and avoid relying on prior training when task mappings undergo structural reversals. Researchers should prioritize extending these analytical formulations to fully nonlinear architectures and non-whitened data distributions.

These conclusions are established with high theoretical and numerical confidence within the scope of two-layer deep linear networks with whitened inputs and zero-balanced initial states. Readers should exercise caution when extrapolating specific quantitative predictions to highly nonlinear architectures, such as networks using non-linear activations, where weight balancing breaks down and empirically results in even greater catastrophic forgetting.

CIFAR Howard Hughes Medical Institute University College London University of Oxford linear networks network architecture deep linear networks exact learning dynamics prior knowledge

Braun et al (2022).pdf

Paper: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, Andrew M. Saxe et al. (2014). It derives the original exact analytical solutions for gradient descent learning dynamics in deep linear networks under small, decoupled initializations that the source explicitly generalizes to rich prior knowledge.
Paper: Neural Tangent Kernel: Convergence and Generalization in Neural Networks, Arthur Jacot et al. (2018). It introduces the Neural Tangent Kernel framework governing network evolution in function space, providing the theoretical context for the analytical NTK dynamics tracked in the source.
Paper: Wide neural networks of any depth evolve as linear models under gradient descent, Jaehoon Lee et al. (2019). It formulates how infinitely wide deep networks evolve as linear models under gradient descent, establishing the foundational link between linear dynamics and representation learning.
Paper: Continual Learning Through Synaptic Intelligence, Friedemann Zenke et al. (2017). It develops the theory of parameter trajectories and consolidation in continual learning, providing the baseline context for understanding the source's findings on continual and reversal learning.

Paper: Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks, Blake Bordelon et al. (2022). It extends the analysis of dynamic kernel evolution to infinite-width feature learning regimes using dynamical field theory, analyzing kernel trajectories beyond exact deep linear settings.
Paper: High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation, Jimmy Ba et al. (2022). It characterizes high-dimensional representation alignment and feature learning in nonlinear settings from initial gradient updates, building on the initial weight dynamics analyzed in the source.
Paper: Probing Representation Forgetting in Supervised and Unsupervised Continual Learning, MohammadReza Davari et al. (2022). It investigates how internal representations evolve and survive across sequential tasks using linear probing, connecting directly to the source's theoretical implications for continual learning dynamics.
Paper: Continual Learning via Sequential Function-Space Variational Inference, Tim G. J. Rudner et al. (2022). It formulates continual learning directly in function space via variational inference, offering a Bayesian counterpart to the functional learning trajectories analyzed in the source.

Cover for Exact learning dynamics of deep linear networks with prior knowledge

Abstract

Learning in deep neural networks is known to depend critically on the knowledge embedded in the initial network weights. However, few theoretical results have precisely linked prior knowledge to learning dynamics. Here we derive exact solutions to the dynamics of learning with rich prior knowledge in deep linear networks by generalising Fukumizu's matrix Riccati solution [1]. We obtain explicit expressions for the evolving network function, hidden representational similarity, and neural tangent kernel over training for a broad class of initialisations and tasks. The expressions reveal a class of task-independent initialisations that radically alter learning dynamics from slow non-linear dynamics to fast exponential trajectories while converging to a global optimum with identical representational similarity, dissociating learning trajectories from the structure of initial internal representations. We characterise how network weights dynamically align with task structure, rigorously justifying why previous solutions successfully described learning from small initial weights without incorporating their fine-scale structure. Finally, we discuss the implications of these findings for continual learning, reversal learning and learning of structured knowledge. Taken together, our results provide a mathematical toolkit for understanding the impact of prior knowledge on deep learning.

Topics

linear networks network architecture

Keywords

deep linear networks exact learning dynamics prior knowledge catastrophic forgetting continual learning finite-width neural tangent kernel gradient flow transfer learning reversal learning lazy learning

1 Introduction
1.1 Contributions
1.2 Related work
2 Preliminaries and setting
3 Exact learning dynamics with prior knowledge
4 Rich and lazy learning regimes and generalisation
5 Decoupling dynamics
6 Applications
7 Discussion
Acknowledgments and Disclosure of Funding
References
Checklist

Knowls

Knowl 1 — Exact Gradient Flow Dynamics of Two-Layer Linear Networks with Arbitrary Initialisations
theoretical result
Consider a two-layer linear network with weight matrices $W_1 \in \mathbb{R}^{N_h \times N_i}$ and $W_2 \in \mathbb{R}^{N_o \times N_h}$ that maps inputs $x \in \mathbb{R}^{N_i}$ to predictions $\hat{y} = W_2 W_1 x \in \mathbb{R}^{N_o}$ , trained under continuous gradient flow on the mean squared error loss $\mathcal{L} = \frac{1}{2}\langle \|\hat{y} - y\|^2 \rangle$ with learning rate time constant $\tau = 1/\eta$ .

Assume:
1. Whitened inputs: $\tilde{\Sigma}^{xx} = \frac{1}{P}\sum_{n=1}^P x_n x_n^T = I_{N_i}$ .
2. Zero-balanced weights at initialization: $W_1(0) W_1(0)^T = W_2(0)^T W_2(0)$ .
3. Full rank: $\text{rank}(\tilde{\Sigma}^{yx}) = \text{rank}(W_2(0)W_1(0)) = N_m = \min(N_i, N_o)$ with $N_h \ge N_m$ , where $\tilde{\Sigma}^{yx} = \frac{1}{P}\sum_{n=1}^P y_n x_n^T$ .
Let the compact singular value decompositions (SVD) of the initial network function and task cross-correlation be $W_2(0)W_1(0) = U S V^T$ and $\tilde{\Sigma}^{yx} = \tilde{U}\tilde{S}\tilde{V}^T$ , where $U, \tilde{U} \in \mathbb{R}^{N_o \times N_m}$ , $V, \tilde{V} \in \mathbb{R}^{N_i \times N_m}$ , and $S, \tilde{S} \in \mathbb{R}^{N_m \times N_m}$ contain positive singular values on their diagonals. When $N_o > N_i$ , let $\tilde{U}_\perp \in \mathbb{R}^{N_o \times (N_o - N_i)}$ complete the orthonormal basis $[\tilde{U}, \tilde{U}_\perp]$ ; when $N_i > N_o$ , let $\tilde{V}_\perp \in \mathbb{R}^{N_i \times (N_i - N_o)}$ complete $[\tilde{V}, \tilde{V}_\perp]$ .

Define alignment matrices $B = U^T \tilde{U} + V^T \tilde{V}$ and $C = U^T \tilde{U} - V^T \tilde{V}$ , and assume $B$ is non-singular. For the composite weight matrix $Q(t) = \begin{bmatrix} W_1(t)^T \\ W_2(t) \end{bmatrix} \in \mathbb{R}^{(N_i + N_o) \times N_h}$ , the exact temporal evolution of $QQ^T(t)$ is given by:

$QQ^T(t) = Z(t) M(t)^{-1} Z(t)^T$

where

$M(t) = 4 e^{-\frac{\tilde{S}t}{\tau}} B^{-1} S^{-1} (B^T)^{-1} e^{-\frac{\tilde{S}t}{\tau}} + \left(I - e^{-\frac{2\tilde{S}t}{\tau}}\right)\tilde{S}^{-1} - e^{-\frac{\tilde{S}t}{\tau}} B^{-1} C \left(e^{-\frac{2\tilde{S}t}{\tau}} - I\right)\tilde{S}^{-1} C^T (B^T)^{-1} e^{-\frac{\tilde{S}t}{\tau}} + \frac{4t}{\tau} e^{-\frac{\tilde{S}t}{\tau}} B^{-1} \left(V^T \tilde{V}_\perp \tilde{V}_\perp^T V + U^T \tilde{U}_\perp \tilde{U}_\perp^T U\right) (B^T)^{-1} e^{-\frac{\tilde{S}t}{\tau}}$

and

$Z(t) = \begin{bmatrix} \tilde{V}\left(I - e^{-\frac{\tilde{S}t}{\tau}} C^T (B^T)^{-1} e^{-\frac{\tilde{S}t}{\tau}}\right) + 2 \tilde{V}_\perp \tilde{V}_\perp^T V (B^T)^{-1} e^{-\frac{\tilde{S}t}{\tau}} \\ \tilde{U}\left(I + e^{-\frac{\tilde{S}t}{\tau}} C^T (B^T)^{-1} e^{-\frac{\tilde{S}t}{\tau}}\right) + 2 \tilde{U}_\perp \tilde{U}_\perp^T U (B^T)^{-1} e^{-\frac{\tilde{S}t}{\tau}} \end{bmatrix}$

Because this formulation contains only negative matrix exponentials $e^{-\tilde{S}t/\tau}$ , it is numerically stable and holds for both equal and unequal input-output dimensions.
Knowl 2 — Convergence to Global Minimum and Rich Task-Specific Representations
theoretical result
For a two-layer linear network $\hat{y} = W_2 W_1 x$ trained under gradient flow on whitened input data ( $\tilde{\Sigma}^{xx} = I$ ) with zero-balanced initial weights ( $W_1(0)W_1(0)^T = W_2(0)^T W_2(0)$ ), full rank initialization, and non-singular alignment matrix $B = U^T \tilde{U} + V^T \tilde{V}$ , the network function and internal representations asymptotically approach the unique global optimum determined by the task singular value decomposition $\tilde{\Sigma}^{yx} = \tilde{U}\tilde{S}\tilde{V}^T$ :

$\lim_{t \to \infty} W_2(t) W_1(t) = \tilde{U} \tilde{S} \tilde{V}^T$ $\lim_{t \to \infty} W_1(t)^T W_1(t) = \tilde{V} \tilde{S} \tilde{V}^T$ $\lim_{t \to \infty} W_2(t) W_2(t)^T = \tilde{U} \tilde{S} \tilde{U}^T$

Consequently, training from any zero-balanced initialization guarantees that the network converges to a rich, task-specific internal representation whose input and output correlation structures align exactly with the task's singular vectors and singular values.
Knowl 3 — Representational Similarity Matrices and Neural Tangent Kernel from the Weight Correlation Matrix
definition
In a two-layer linear network with weight matrices $W_1(t) \in \mathbb{R}^{N_h \times N_i}$ and $W_2(t) \in \mathbb{R}^{N_o \times N_h}$ , defining the composite matrix $Q(t) = \begin{bmatrix} W_1(t)^T \\ W_2(t) \end{bmatrix}$ yields the block weight correlation matrix:

$QQ^T(t) = \begin{bmatrix} W_1(t)^T W_1(t) & W_1(t)^T W_2(t)^T \\ W_2(t) W_1(t) & W_2(t) W_2(t)^T \end{bmatrix}$

The four quadrants determine three key observables of network behavior over training time $t$ :
1. Network Function: The off-diagonal block $W_2(t) W_1(t)$ gives the input-output mapping $\hat{Y}(t) = W_2(t) W_1(t) X$ , where $X$ is the dataset input matrix.
2. Hidden Representational Similarity Matrices (RSM): The on-diagonal blocks give the kernel matrices $\phi(x)^T \phi(x')$ of the hidden representations for inputs and outputs:
$\text{RSM}_I(t) = X^T W_1(t)^T W_1(t) X$ $\text{RSM}_O(t) = Y^T \left(W_2(t) W_2(t)^T\right)^+ Y$

where $Y$ is the target matrix and $+$ denotes the Moore-Penrose pseudoinverse.
1. Finite-Width Neural Tangent Kernel (NTK): The finite-width NTK governing functional gradient flow is:
$\text{NTK}(t) = I_{N_o} \otimes \left(X^T W_1(t)^T W_1(t) X\right) + \left(W_2(t) W_2(t)^T\right) \otimes \left(X^T X\right)$

where $I_{N_o}$ is the identity matrix and $\otimes$ denotes the Kronecker product.
Knowl 4 — Exact Decoupling Dynamics in the Task Singular Vector Basis
theoretical result
Let a two-layer linear network with equal input and output dimensions ( $N_i = N_o$ ) be trained under gradient flow on whitened data ( $\tilde{\Sigma}^{xx} = I$ ) from zero-balanced initial weights ( $W_1(0) W_1(0)^T = W_2(0)^T W_2(0)$ ).

Suppose the initial weights are expressed as $W_1(0) = A(0) \tilde{V}^T$ and $W_2(0) = \tilde{U} A(0)^T$ , where $\tilde{U}\tilde{S}\tilde{V}^T$ is the compact SVD of the task cross-correlation matrix $\tilde{\Sigma}^{yx}$ , and $A(0) \in \mathbb{R}^{N_h \times N_i}$ is an arbitrary invertible matrix satisfying $U V = \tilde{U}\tilde{V}$ .

The temporal evolution of the internal representation matrix $A(t)^T A(t)$ directly in the task SVD basis is given by:

$A(t)^T A(t) = \left[ e^{-\frac{\tilde{S}t}{\tau}} \left(A(0)^T A(0)\right)^{-1} e^{-\frac{\tilde{S}t}{\tau}} + \left(I - e^{-\frac{2\tilde{S}t}{\tau}}\right) \tilde{S}^{-1} \right]^{-1}$

where $\tau = 1/\eta$ is the learning time constant. In this basis, on-diagonal elements $[A(t)^T A(t)]_{ii}$ describe the strength of individual task singular modes converging to $\tilde{s}_i$ , while off-diagonal elements $[A(t)^T A(t)]_{ij}$ ( $i \ne j$ ) quantify cross-modal coupling (misalignment) that dynamically decouples toward zero during learning.
Knowl 5 — Analytical Characterization of Mode Decoupling and Peak Misalignment in Two-Dimensional Linear Networks
theoretical result
In a two-dimensional two-layer linear network ( $N_i = N_o = 2$ ) trained under gradient flow with time constant $\tau = 1/\eta$ and equal task singular values $\tilde{S} = \text{diag}(s, s)$ , let the initial representation matrix in the task SVD basis be:

$A(0)^T A(0) = \begin{bmatrix} a_1(0) & b(0) \\ b(0) & a_2(0) \end{bmatrix}$

where $a_1(0), a_2(0)$ are task-aligned mode strengths and $b(0)$ represents cross-modal coupling.
1. Non-monotonic Decoupling Peak: The off-diagonal coupling $b(t)$ exhibits a transient amplification peak before decaying to zero. The time at which the magnitude of misalignment peaks is:
$t_{\text{peak}} = \frac{\tau}{4s} \ln \left[ \frac{s(s - a_1(0) - a_2(0))}{a_1(0)a_2(0) - b(0)^2} \right]$

which coincides approximately with the time at which the diagonal mode reaches half its asymptotic strength.
1. Shift from Sigmoidal to Exponential Trajectories: When initial weights are small ( $|a_1(0)|, |a_2(0)|, |b(0)| \ll s$ ), the on-diagonal mode trajectory is sigmoidal (step-like):
$a_1(t) = \frac{s a_1(0)}{e^{-\frac{2st}{\tau}} [s - a_1(0) - a_2(0)] + a_1(0) + a_2(0)}$

When initial weights are large ( $|a_1(0)|, |a_2(0)|, |b(0)| \gg 0$ ), the mode trajectory transitions into a fast exponential decay toward the task target.
Knowl 6 — Dissociation of Learning Trajectory Speed from Representational Richness via Large Zero-Balanced Initialisations
theoretical result
In deep learning, small random initial weights are conventionally associated with slow, sigmoidal dynamics that converge to rich task-aligned representations, whereas large random initial weights yield fast exponential dynamics that remain lazy (trapped in the initial task-agnostic random feature kernel).

For two-layer linear networks, zero-balanced initial weights ( $W_1(0)W_1(0)^T = W_2(0)^T W_2(0)$ ) dissociate the speed of learning trajectories from the asymptotic richness of internal representations:
1. Task-agnostic large zero-balanced initial weights produce fast exponential loss trajectories that match the learning speed of the lazy regime.
2. Despite having exponential learning dynamics from large initial weights, gradient flow provably converges to the exact rich global minimum where representations perfectly match task structure ( $W_1^T W_1 = \tilde{V}\tilde{S}\tilde{V}^T$ and $W_2 W_2^T = \tilde{U}\tilde{S}\tilde{U}^T$ ).
Thus, fast exponential loss decay does not imply lazy representation learning.
Knowl 7 — Exact Prediction of Catastrophic Forgetting in Continual Learning of Deep Linear Networks
theoretical result
When a two-layer linear network is trained sequentially with gradient flow across tasks with input-output cross-correlation matrices $T_1 = \tilde{\Sigma}^{yx}_1, T_2 = \tilde{\Sigma}^{yx}_2, \dots$ from any zero-balanced initialization:
1. Convergence on task $T_j$ completely overwrites the network function to $W_2 W_1 = \tilde{\Sigma}^{yx}_j$ , causing true catastrophic forgetting of previous tasks.
2. The post-training evaluation mean squared error loss on any prior task $T_i$ after converging on task $T_j$ is given exactly by:
$\mathcal{L}_i(T_j) = \frac{1}{2} \|\tilde{\Sigma}^{yx}_j - \tilde{\Sigma}^{yx}_i\|_F^2 + c_i$

where $\|\cdot\|_F$ is the Frobenius norm and $c_i = \frac{1}{2}\langle \|y_i\|^2 \rangle - \frac{1}{2}\|\tilde{\Sigma}^{yx}_i\|_F^2$ is an invariant dataset constant.
1. The exact catastrophic forgetting on task $T_a$ when switching training from task $T_b$ to task $T_c$ is:
$\Delta \mathcal{L}_a = \mathcal{L}_a(T_c) - \mathcal{L}_a(T_b) = \frac{1}{2} \|\tilde{\Sigma}^{yx}_c - \tilde{\Sigma}^{yx}_a\|_F^2 - \frac{1}{2} \|\tilde{\Sigma}^{yx}_b - \tilde{\Sigma}^{yx}_a\|_F^2$

This relative change in loss is fully determined prior to training by the similarity structure of the tasks' cross-correlation matrices. In nonlinear networks (e.g., tanh or ReLU activations), weights rapidly unbalance, leading to greater forgetting that exceeds this linear prediction.
Knowl 8 — Saddle-Point Trapping and Catastrophic Slowing in Reversal Learning
theoretical result
In reversal learning, a network is trained on a task where input-output associations are reversed relative to a previously learned task (such that the new target cross-correlation is negatively aligned with prior weights).

In deep linear networks trained with gradient flow:
1. The prior network weights lie exactly on the separatrix of a saddle point in the loss landscape. Under deterministic continuous gradient flow, the alignment matrix $B = U^T \tilde{U} + V^T \tilde{V}$ is singular, causing the analytical network function to collapse to zero rather than converging to the reversed target.
2. In numerical gradient descent with finite precision, floating-point or stochastic noise perturbs the trajectory off the separatrix, eventually allowing convergence; however, the dynamics exhibit catastrophic slowing due to prolonged stagnation near the saddle point.
3. In contrast, shallow (single-layer) linear networks possess a strictly convex loss landscape without saddle points and converge rapidly during reversal learning without catastrophic slowing.
Knowl 9 — Impact of Structured Pre-training on Knowledge Revision in Deep Linear Networks
empirical result
Pre-training a two-layer deep linear network on a structured hierarchical task (e.g., an 8-item living kingdom taxonomy) prior to learning a revised hierarchy yields the following dynamics:
1. Revisions that swap neighboring leaf nodes within the hierarchy correspond to reversal learning on the swapped subspace, placing the network near a saddle point and causing catastrophically slowed learning dynamics.
2. Revisions involving an entirely new hierarchy with identical items but different features (e.g., a color taxonomy) yield convergence times no faster than starting from a random initialization with matched variance.
3. Initializing from large zero-balanced weights accelerates learning on the revised task by orders of magnitude compared to structured pre-training, while converging to a global optimum with identical representational similarity.
Knowl 10 — Assumptions and Scope of the Generalized Matrix Riccati Framework
limitation
The exact closed-form solutions for gradient flow learning dynamics in two-layer linear networks rely on four fundamental conditions:
1. Input Whitening: The input data correlation matrix must satisfy $\tilde{\Sigma}^{xx} = I$ .
2. Zero-Balanced Weights: The weights must satisfy $W_1(0)W_1(0)^T = W_2(0)^T W_2(0)$ at initialization, which ensures balanced layer dynamics throughout training.
3. Full-Rank Task and Initialisation: Both the task cross-correlation $\tilde{\Sigma}^{yx}$ and the initial network product $W_2(0)W_1(0)$ must have rank equal to $\min(N_i, N_o) \le N_h$ , excluding bottlenecked hidden layers.
4. Alignment Matrix Invertibility: The alignment matrix $B = U^T \tilde{U} + V^T \tilde{V}$ must be non-singular, which fails in deterministic reversal learning.
Extending exact analytical dynamics to deep linear networks with unwhitened data or unbalanced weights remains outside the scope of this matrix Riccati formulation.

Coverage note — Deliberately omitted were intermediate mathematical proof steps and derivations detailed in the appendices (e.g., the step-by-step matrix Riccati reduction and coordinate transformations for Q(t)), as well as standard introductory reviews of prior literature.

References

1.Kenji Fukumizu. Effect of batch learning in multilayer neural networks. Gen, 1(04):1E–03, 1998.
2.S.E. Carey. Conceptual Change In Childhood. MIT Press, Cambridge, MA, 1985.
3.James L McClelland, Bruce L McNaughton, and Randall C O’Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
4.Gregory Murphy. The big book of concepts. MIT press, 2004.
5.James L McClelland. Incorporating rapid neocortical learning of new schema-consistent information into complementary learning systems theory. Journal of Experimental Psychology: General, 142(4):1190, 2013.
6.Timo Flesch, Jan Balaguer, Ronald Dekker, Hamed Nili, and Christopher Summerfield. Comparing continual task learning in minds and machines. Proceedings of the National Academy of Sciences, 115(44):E10313–E10322, 2018.
7.Burak Erdeniz and Nart Bedin Atalay. Simulating probability learning and probabilistic reversal learning using the attention-gated reinforcement learning (agrel) model. In The 2010 International Joint Conference on Neural Networks (IJCNN), pages 1–6. IEEE, 2010.
8.Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
9.Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
10.Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprint arXiv:1809.10374, 2018.
11.Federica Gerace, Luca Saglietti, Stefano Sarao Mannelli, Andrew Saxe, and Lenka Zdeborová. Probing transfer learning with a model of synthetic correlated datasets. Machine Learning: Science and Technology, 2022.
12.German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019.
13.James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
14.Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017.
15.Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
16.Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, pages 1820–1830, 2019.
17.Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
18.Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
19.Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11:501–528, 2020.
20.Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
21.Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
22.Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
23.Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53–58, 1989.
24.Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pages 244–253. PMLR, 2018.
25.Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 (23):11537–11546, 2019.
26.Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022.
27.Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pages 10153–10161. PMLR, 2021.
28.Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. arXiv preprint arXiv:1810.02281, 2018.
29.Simon Du and Wei Hu. Width provably matters in optimization for deep linear neural networks. In International Conference on Machine Learning, pages 1655–1664. PMLR, 2019.
30.Tomaso Poggio, Qianli Liao, Brando Miranda, Andrzej Banburski, Xavier Boix, and Jack Hidary. Theory iiib: Generalization in deep networks. arXiv preprint arXiv:1806.11379, 2018.
31.Dongsung Huh. Curvature-corrected learning dynamics in deep neural networks. In International Conference on Machine Learning, pages 4552–4560. PMLR, 2020.
32.Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pages 2902–2907. PMLR, 2018.
33.Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv:1810.02032, 2018.
34.Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems, 31, 2018.
35.Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
36.Michael Biehl and Holm Schwarze. Learning by on-line gradient descent. Journal of Physics A: Mathematical and general, 28(3):643, 1995.
37.David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337, 1995.
38.Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019.
39.Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
40.Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
41.Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
42.Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33): E7665–E7671, 2018.
43.Grant Rotskoff and Eric Vanden-Eijnden. Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. Advances in neural information processing systems, 31, 2018.
44.Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130(3):1820–1852, 2020.
45.Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
46.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
47.Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pages 5393–5402. PMLR, 2018.
48.Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Reviews of Modern Physics, 91(4):045002, 2019.
49.Raman Arora, Sanjeev Arora, Joan Bruna, Nadav Cohen, Simon Du, Rong Ge, Suriya Gunasekar, Chi Jin, Jason Lee, Tengyu Ma, Benhnam Neyshabur, and Zhao Song. Theory of deep learning, 2020.
50.Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008.
51.Timo Flesch, Keno Juechems, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Orthogonal representations for robust context-dependent task performance in brains and neural networks. Neuron, 2022.
52.Dmytro Mishkin and Jiri Matas. All you need is a good init. arXiv preprint arXiv:1511.06422, 2015.
53.Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
54.Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999.
55.Haruka Asanuma, Shiro Takagi, Yoshihiro Nagano, Yuki Yoshida, Yasuhiko Igarashi, and Masato Okada. Statistical mechanical analysis of catastrophic forgetting in continual learning with teacher and student networks. Journal of the Physical Society of Japan, 90(10):104001, 2021.
56.Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, pages 1072–1080. PMLR, 2021.
57.Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of task similarity. In International Conference on Machine Learning, pages 6109–6119. PMLR, 2021.
58.Gal Shachaf, Alon Brutzkus, and Amir Globerson. A theoretical analysis of fine-tuning with linear teachers. Advances in Neural Information Processing Systems, 34, 2021.
59.Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in Neural Information Processing Systems, 33:7852–7862, 2020.
60.Sebastian Lee, Stefano Sarao Mannelli, Claudia Clopath, Sebastian Goldt, and Andrew Saxe. Maslow’s hammer for catastrophic forgetting: Node re-use vs node activation. arXiv preprint arXiv:2205.09029, 2022.
61.Wei-Yong Yan, Uwe Helmke, and John B Moore. Global analysis of oja’s flow for neural networks. IEEE Transactions on Neural Networks, 5(5):674–683, 1994.

Citation

MLA

Braun, L., et al. “Exact Learning Dynamics of Deep Linear Networks with Prior Knowledge”. Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 6615–29, https://proceedings.neurips.cc/paper_files/paper/2022/file/2b3bb2c95195130977a51b3bb251c40a-Paper-Conference.pdf.

APA

Braun, L., Dominé, C., Fitzgerald, J., & Saxe, A. (2022). Exact learning dynamics of deep linear networks with prior knowledge. Advances in Neural Information Processing Systems, 35, 6615–6629. https://proceedings.neurips.cc/paper_files/paper/2022/file/2b3bb2c95195130977a51b3bb251c40a-Paper-Conference.pdf

Chicago

Braun, L., C. Dominé, J. Fitzgerald, and A. Saxe. 2022. “Exact Learning Dynamics of Deep Linear Networks with Prior Knowledge”. Advances in Neural Information Processing Systems 35: 6615–29. https://proceedings.neurips.cc/paper_files/paper/2022/file/2b3bb2c95195130977a51b3bb251c40a-Paper-Conference.pdf.

Harvard

Braun, L. et al. (2022) “Exact learning dynamics of deep linear networks with prior knowledge”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 6615–6629. Available at: https://proceedings.neurips.cc/paper_files/paper/2022/file/2b3bb2c95195130977a51b3bb251c40a-Paper-Conference.pdf.

Vancouver

1. Braun L, Dominé C, Fitzgerald J, Saxe A (2022) Exact learning dynamics of deep linear networks with prior knowledge. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 6615–6629

BibTeX

@inproceedings{braun2022exact,
  title = {Exact learning dynamics of deep linear networks with prior knowledge},
  author = {Braun, Lukas and Dominé, Clémentine and Fitzgerald, James and Saxe, Andrew},
  year = {2022},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {35},
  pages = {6615-6629},
  url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/2b3bb2c95195130977a51b3bb251c40a-Paper-Conference.pdf}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Exact learning dynamics of deep linear networks with prior knowledge

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Exact Gradient Flow Dynamics of Two-Layer Linear Networks with Arbitrary Initialisations

Knowl 2 — Convergence to Global Minimum and Rich Task-Specific Representations

Knowl 3 — Representational Similarity Matrices and Neural Tangent Kernel from the Weight Correlation Matrix

Knowl 4 — Exact Decoupling Dynamics in the Task Singular Vector Basis

Knowl 5 — Analytical Characterization of Mode Decoupling and Peak Misalignment in Two-Dimensional Linear Networks

Knowl 6 — Dissociation of Learning Trajectory Speed from Representational Richness via Large Zero-Balanced Initialisations

Knowl 7 — Exact Prediction of Catastrophic Forgetting in Continual Learning of Deep Linear Networks

Knowl 8 — Saddle-Point Trapping and Catastrophic Slowing in Reversal Learning

Knowl 9 — Impact of Structured Pre-training on Knowledge Revision in Deep Linear Networks

Knowl 10 — Assumptions and Scope of the Generalized Matrix Riccati Framework

References

Citation

BibTeX

Access the Paper