The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

Kazuki IrieRóbert CsordásJürgen Schmidhuber

article2022ICML57 citations

Reformulates linear layers in gradient-trained neural networks as key-value attention over the entire training history, providing a direct method to trace and interpret how specific training samples drive test-time predictions.

Listen

Modern deep neural networks deliver powerful predictions across many domains, but their internal decision-making processes remain largely opaque black boxes. Because models compress vast amounts of training data into static weight matrices, it is difficult to identify which specific training examples drive individual predictions at test time. This lack of transparency presents a growing challenge for accountability, auditability, and trust as artificial intelligence systems are deployed into high-stakes environments.

The article demonstrates how every linear layer in a neural network trained by gradient descent can be exactly reformulated as a key-value memory system using dot-product attention over its entire training history. By adopting this dual formulation, the authors evaluate whether examining attention scores across stored training examples provides a direct, interpretable link between training data and test-time outputs.

To test this approach empirically, the authors recorded layer inputs during standard gradient descent training and evaluated attention weights at test time across multiple setups. They studied small-scale feedforward networks with two hidden layers on MNIST and Fashion-MNIST image classification across single-task, joint multi-task, and sequential continual learning environments. They also evaluated one-layer recurrent language models on small text corpora, including WikiText-2 and Aesop's Fables.

The investigation produced four primary findings. First, while individual top-ranked training examples do not always match the target class, the aggregate sum of attention weights across classes strongly correlates with correct model outputs, with accuracy reaching 84.7% in the final layer for correct predictions compared to roughly 20.6% for incorrect predictions. Second, in joint multi-task training, representations in deeper layers exhibit cross-task attention, drawing on relevant structural features across datasets. Third, the dual view explains catastrophic forgetting in continual learning as a retrieval interference failure; stored memory of earlier tasks remains intact within the network history but is overwhelmed by later training patterns, causing test accuracy on the initial task to plummet from 97% to 45%. Fourth, in language modeling, attention retrieves training passages sharing context, grammatical structure, and semantic concepts rather than superficial token matches.

These findings provide fundamental insights into how neural networks retain and retrieve information. They show that neural networks do not truly lose previous data in their historical formulation, but rather suffer from retrieval bottlenecks. This understanding opens concrete pathways to analyze safety risks, diagnose misclassifications, and explain algorithmic bias by pinpointing influential training examples.

Organizations developing or auditing machine learning models should consider capturing layer activations during training for pilot interpretability and diagnostic workflows where storage allows. In continual learning scenarios, exploring selective retrieval mechanisms or attention masks represents a promising option to mitigate interference without architectural redesigns. Future research should prioritize developing memory-efficient approximations, such as re-computation pipelines during testing, to scale this diagnostic analysis to industrial foundation models.

The findings are derived from exact mathematical formulations, providing high confidence in the underlying duality. However, practical application is constrained by high time and space complexity, as storage requirements scale linearly with the total number of training iterations. In addition, the method requires logging activations during training and cannot be applied retroactively to already-trained models.

Dalle Molle Institute for Artificial Intelligence Research King Abdullah University of Science and Technology classification classification methods data points dual form of neural networks test time predictions spotlights of attention

arXiv: 2202.05798 robertcsordas/linear_layer_as_attention

Paper: Transformer Feed-Forward Layers Are Key-Value Memories, Mor Geva et al. (2020). This paper establishes that feed-forward layers in deep models operate as key-value memory retrieval mechanisms, providing the foundational conceptual framing for viewing network layers as associative memories.
Paper: Understanding Black-box Predictions via Influence Functions, Pang Wei Koh et al. (2017). This work introduces the classic influence function paradigm for attributing test-time predictions to individual training examples, providing essential context for alternative data-attribution formulations.
Paper: End-To-End Memory Networks, Sainbayar Sukhbaatar et al. (2015). This paper introduces end-to-end memory networks with attention-based addressing, foundational to understanding how neural networks store and retrieve historical data through soft attention.
Paper: Attention is not Explanation, Sarthak Jain et al. (2019). This work critically analyzes the fidelity of attention mechanisms as interpretability tools, which motivates the source's empirical evaluation of attention-based dual formulations.
Paper: Understanding intermediate layers using linear classifier probes, Guillaume Alain et al. (2016). This text details the use of linear probes on intermediate representations to track representation evolution, establishing key diagnostic methods used to evaluate internal network states.
Paper: Overcoming catastrophic forgetting with hard attention to the task, Joan Serrà et al. (2018). This study analyzes catastrophic forgetting in continual learning through attention masking mechanisms, offering crucial background for the source's dual-retrieval interpretation of forgetting.

Paper: Test-time regression: a unifying framework for designing sequence models with associative memory, Ke Alexander Wang et al. (2025). This work formalizes test-time retrieval and associative recall into a unified test-time regression framework, expanding the view of sequence models and attention as key-value memory systems.
Paper: Test-Time Training with KV Binding Is Secretly Linear Attention, Junchen Liu et al. (2026). This research proves that test-time training with key-value binding mathematically reduces to linear attention operators, extending the duality between gradient-based training dynamics and attention-based retrieval.
Paper: Simple linear attention language models balance the recall-throughput tradeoff, Simran Arora et al. (2024). This paper explores the fundamental tradeoffs between inference memory state and associative recall capacity in linear attention models, continuing the investigation of memory bottlenecks in attention-driven systems.
Paper: Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks, Zhiwei Deng et al. (2022). This paper leverages addressable memory architectures to distill datasets and overcome catastrophic forgetting, directly applying memory-retrieval concepts to lifelong learning problems.
Paper: Probing Representation Forgetting in Supervised and Unsupervised Continual Learning, MohammadReza Davari et al. (2022). This empirical study proves that intermediate representations survive catastrophic forgetting during continual learning, reinforcing and extending the source's finding that forgetting is primarily a retrieval failure.

Cover for The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

Abstract

Linear layers in neural networks (NNs) trained by gradient descent can be expressed as a key-value memory system which stores all training datapoints and the initial weights, and produces outputs using unnormalised dot attention over the entire training experience. While this has been technically known since the 1960s, no prior work has effectively studied the operations of NNs in such a form, presumably due to prohibitive time and space complexities and impractical model sizes, all of them growing linearly with the number of training patterns which may get very large. However, this dual formulation offers a possibility of directly visualising how an NN makes use of training patterns at test time, by examining the corresponding attention weights. We conduct experiments on small scale supervised image classification tasks in single-task, multi-task, and continual learning settings, as well as language modelling, and discuss potentials and limits of this view for better understanding and interpreting how NNs exploit training patterns. Our code is public†.

Topics

classification classification methods data points feedforward neural networks image classification language model memory systems random access memory

Keywords

dual form of neural networks test time predictions spotlights of attention training patterns attention weights memory networks multi-task joint training gradient descent

1. Introduction
2. Preliminaries
3. The Dual Form of Linear Layers in NNs Trained by Gradient Descent
4. Related Work
5. Experiments
5.1. Common Settings
5.2. Single Task Case
5.3. Multi-Task Case
5.4. Continual Learning Case
5.5. Language Modelling
6. Discussion and Limitations
7. Conclusion
Acknowledgements
References
A. Top Matching Examples
B. More Examples/Visualisation
B.1. Single Task Case
B.2. Joint Training Case
B.3. Continual Learning Case
C. Language Modelling Experiments
C.1. Basic Settings
C.2. Character-Level Experiments on 'Aesop's Fables'
C.3. Word-Level Experiments on WikiText-2
D. Further Discussion on Scalability

Knowls

Knowl 1 — Dual Form of a Linear Layer Trained by Gradient Descent
theoretical result
Consider a linear layer in a neural network parameterized by initial weight matrix $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ , trained with gradient descent over $T$ update events on layer inputs $(x_1, \dots, x_T)$ with $x_t \in \mathbb{R}^{d_{\text{in}}}$ . Let $(e_1, \dots, e_T)$ with $e_t \in \mathbb{R}^{d_{\text{out}}}$ denote the corresponding backpropagated error signals, defined as $e_t = -\eta_t (\nabla_y \mathcal{L})_t$ for learning rate $\eta_t \in \mathbb{R}$ , loss $\mathcal{L}$ , and layer output $y_t = W_t x_t$ .

The trained weight matrix in primal form is: $W = W_0 + \sum_{t=1}^T e_t \otimes x_t = W_0 + E X^\top$ where $X = (x_1, \dots, x_T) \in \mathbb{R}^{d_{\text{in}} \times T}$ is the key matrix and $E = (e_1, \dots, e_T) \in \mathbb{R}^{d_{\text{out}} \times T}$ is the value matrix.

For any test query input $x \in \mathbb{R}^{d_{\text{in}}}$ , the primal forward operation $S_1(x) = Wx$ is mathematically equivalent to the dual form key-value memory system $S_2(x)$ : $S_2(x) = W_0 x + \text{Attention}(X, E, x) = W_0 x + \sum_{t=1}^T (x_t^\top x) e_t$ where $\text{Attention}(X, E, x) = E X^\top x$ is unnormalised linear dot-product attention with attention weights $\alpha_t = x_t^\top x \in \mathbb{R}$ .
Knowl 2 — Attribution of Test Predictions to Training Patterns via Dual-Form Attention
model/method
The dual formulation of linear layers allows direct attribution of test predictions to specific training events by recording intermediate layer activations during training:
1. During standard backpropagation training, input vectors $x_t \in \mathbb{R}^{d_{\text{in}}}$ to each linear layer are recorded for all $T$ training updates across all mini-batches.
2. At test time, an evaluation example is passed through the network to generate the layer input query $x \in \mathbb{R}^{d_{\text{in}}}$ at each layer.
3. Unnormalised attention weights across all training points are computed as the dot products $\alpha_t = x_t^\top x$ .
4. To quantify the total attention directed toward a specific class $c$ , cumulative class scores are calculated by summing $\alpha_t$ over all training points $t$ belonging to class $c$ . For the input layer, the sum of absolute values $\sum_{t \in c} |\alpha_t|$ is used to account for potentially negative raw dot products; for subsequent hidden layers with ReLU activations, the sum $\sum_{t \in c} \alpha_t$ is strictly non-negative.

Knowl 3 — Correlation Between Per-Class Cumulative Attention Scores and Network Predictions

data/table

In a feedforward network with two hidden layers (768-800-800-10 units with ReLU activations) trained on MNIST to 97% test accuracy, the argmax of per-class cumulative attention scores correlates strongly with both the true target class and the network's predicted output, with the correlation increasing in deeper layers.

		Is Model Prediction Correct?
Layer	Evaluated Metric	No	Yes
0 (Input layer)	Target Accuracy (%)	$17.2 \pm 2.7$	$75.1 \pm 0.0$
0 (Input layer)	Output Accuracy (%)	$49.5 \pm 1.5$	—
1 (Hidden layer 1)	Target Accuracy (%)	$18.0 \pm 2.9$	$78.8 \pm 0.8$
1 (Hidden layer 1)	Output Accuracy (%)	$52.9 \pm 2.9$	—
2 (Output layer)	Target Accuracy (%)	$20.6 \pm 2.6$	$84.7 \pm 1.1$
2 (Output layer)	Output Accuracy (%)	$60.1 \pm 4.5$	—

The values represent the mean and standard deviation over 5 independent runs. When the network predicts correctly, the argmax of the total attention scores identifies the true label with $75.1\%$ accuracy at layer 0 and rises to $84.7\%$ at layer 2. When the network misclassifies, the argmax of cumulative attention still matches the model's erroneous output $60.1\%$ of the time at layer 2, confirming that cumulative dual attention reflects network decisions rather than pixel-level similarity alone.

Knowl 4 — Cross-Task Representation Sharing in Jointly Trained Deep Linear Layers
empirical result
When a feedforward network (768-800-800-10 with ReLU activations) is trained jointly on MNIST and Fashion-MNIST (with 10 shared output classes), dual-form attention weights demonstrate layer-dependent task disentanglement:
1. In the input layer (layer 0), top-matching training points and per-class total attention scores are dominated by training patterns from the query's own dataset (e.g., a Fashion-MNIST class 9 "ankle boot" query only attends strongly to Fashion-MNIST keys).
2. In intermediate and final layers (layer 1 and layer 2), top attention weights become distributed across both tasks. For a Fashion-MNIST class 9 test query, the top matching training keys in layer 2 include MNIST digit 9 samples.
This indicates that while early layers operate on domain-specific raw inputs, representations in deeper linear layers become task-independent and attend to training examples across tasks that share semantic class indices.
Knowl 5 — Catastrophic Forgetting as Retrieval Interference in Dual Form
empirical result
In a continual learning experiment where a two-hidden-layer feedforward network is trained first on MNIST for 3,000 steps (achieving 97% accuracy) and subsequently on Fashion-MNIST for 3,000 steps (reducing MNIST accuracy to 45%), the dual form reveals that catastrophic forgetting is caused by retrieval interference rather than memory erasure:
1. The network's key-value memory retains all training patterns and error signals from the initial MNIST task.
2. In the absence of an explicit task mask, a test query from MNIST (e.g., digit "1") activates strong unnormalised dot-attention on training keys from the subsequent task (e.g., Fashion-MNIST class 3 "dress").
3. Across successive layers, cross-task interference accumulates: in layer 2, attention weights for Fashion-MNIST class 3 exceed those for MNIST class 1, causing the network to misclassify the MNIST digit "1" as class 3.
Knowl 6 — Contextual and Structural Retrieval of Training Passages in LSTM Language Models
empirical result
In recurrent language models (such as a 1-layer LSTM with hidden dimension 200 trained on WikiText-2 or hidden dimension 1024 trained on Aesop's Fables), unnormalised attention over training inputs to the LSTM's linear projection layer retrieves contextually and syntactically aligned training text:
1. For a test prompt describing a warship (e.g., "... Her principal role was for combat in the English Channel ..."), the training tokens with the highest attention weights originate from other Wikipedia pages about warships with matching syntactic patterns (e.g., "... Her main battery guns were replaced ...", "... Her secondary armament consisted of ...").
2. Passages with identical syntactic structures but unrelated topics (e.g., "... Her first hit song in America was ..." from a biography of a singer) receive negligible attention weights, demonstrating that the dual form performs contextual rather than purely superficial pattern retrieval.
Knowl 7 — Self-Attention Linear Projections as Hierarchical Two-Level Retrieval
theoretical result
Applying the dual form of linear layers to the projection operations in Transformer self-attention casts the mechanism as a hierarchical two-level retrieval architecture:
1. First-level retrieval (dual linear projection): Each linear transformation projecting an input vector $x$ into query, key, or value vectors ( $Q = W_Q x$ , $K = W_K x$ , $V = W_V x$ ) can be expressed in dual form as unnormalised dot-attention over the entire historical training set $X \in \mathbb{R}^{d_{\text{in}} \times T}$ : $W x = W_0 x + \sum_{t=1}^T (x_t^\top x) e_t$ This performs a first level of retrieval over stored training experiences.
2. Second-level retrieval (softmax self-attention): The subsequent softmax dot-product operation $\text{softmax}(Q K^\top / \sqrt{d}) V$ executes a second level of retrieval across the contextual sequence representations produced by the first level.
Knowl 8 — Storage Overhead and Observational Constraints of Dual-Form Neural Network Analysis
limitation
Analyzing deep neural networks through their dual key-value representation has several operational constraints:
1. Linear memory growth: Storing layer input activations across $T$ training steps requires $\mathcal{O}((d_{\text{in}} + d_{\text{out}}) T)$ storage. For a model with hidden state dimension 12,000 trained on 300 billion tokens (such as GPT-3), dual-form analysis of a single linear layer would require approximately 14 petabytes of memory, unless the model is deterministically retrained to recompute activations on the fly.
2. Pre-training incompatibility: The dual form cannot be applied post-hoc to already trained models whose intermediate layer inputs were not logged during training.
3. Disregard of error vector dynamics: Visualizing test queries strictly through dot-product attention weights $\alpha_t = x_t^\top x$ neglects the magnitude and sign of value error vectors $e_t$ and subsequent nonlinear layers, restricting the analysis to which training points are combined rather than explaining how combined representations produce complex generative outputs.

Coverage note — No substantial contributed material was omitted; intermediate algebraic proofs for Lemma 2.2 and Proposition 1 were omitted in accordance with the rules to keep the standalone dual-form theoretical result and its empirical applications self-contained.

References

1.Aizerman, M. A., Braverman, E. M., and Rozonoer, L. I. Theoretical foundations of potential function method in pattern recognition. Automation and Remote Control, 25 (6):917–936, 1964.
2.Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z., and Ionescu, C. Using fast weights to attend to the recent past. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 4331–4339, Barcelona, Spain, December 2016.
3.Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, May 2015.
4.Balestriero, R., Pesenti, J., and LeCun, Y. Learning in high dimension always amounts to extrapolation. Preprint arXiv:2110.09485, 2021.
5.Bishop, C. M. Pattern Recognition and Machine Learning. Springer, 2006.
6.Boser, B. E., Guyon, I., and Vapnik, V. A training algorithm for optimal margin classifiers. In Proc. Annual ACM Conference on Computational Learning Theory (COLT), pp. 144–152, Pittsburgh, PA, USA, July 1992. ACM.
7.Brown, T. B. et al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, December 2020.
8.Burges, C. J. A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery, 2(2):121–167, 1998.
9.Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. Rethinking attention with performers. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021.
10.Domingos, P. Every model learned by gradient descent is approximately a kernel machine. Preprint arXiv:2012.00152, 2020.
11.French, R. M. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999.
12.Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 5484–5495, Online and Punta Cana, Dominican Republic, November 2021.
13.Graves, A. Supervised sequence labelling with recurrent neural networks. PhD thesis, Technical University Munich, 2008.
14.Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. Preprint arXiv:1410.5401, 2014.
15.Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
16.Irie, K., Zeyer, A., Schlüter, R., and Ney, H. Language modeling with deep Transformers. In Proc. Interspeech, pp. 3905–3909, Graz, Austria, September 2019.
17.Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2020.
18.Krizhevsky, A. Learning multiple layers of features from tiny images. Master’s thesis, Computer Science Department, University of Toronto, 2009.
19.LeCun, Y., Cortes, C., and Burges, C. J. The MNIST database of handwritten digits. URL http://yann. lecun. com/exdb/mnist, 1998.
20.Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 1412–1421, Lisbon, Portugal, September 2015.
21.Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
22.Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 3111–3119, Lake Tahoe, NV, USA, September 2013.
23.Miller, A. H., Fisch, A., Dodge, J., Karimi, A., Bordes, A., and Weston, J. Key-value memory networks for directly reading documents. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 1400–1409, Austin, TX, USA, November 2016.
24.Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., and Kong, L. Random feature attention. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021.
25.Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-toimage generation. In Proc. Int. Conf. on Machine Learning (ICML), pp. 8821–8831, Virtual only, July 2021.
26.Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
27.Schlag, I., Irie, K., and Schmidhuber, J. Linear Transformers are secretly fast weight programmers. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2021.
28.Schmidhuber, J. Learning to control fast-weight memories: An alternative to recurrent nets. Technical Report FKI147-91, Institut für Informatik, Technische Universität München, March 1991.
29.Schmidhuber, J. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In International Conference on Artificial Neural Networks (ICANN), pp. 460–463, Amsterdam, Netherlands, September 1993.
30.Schölkopf, B. and Smola, A. J. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
31.Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R. Endto-end memory networks. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 2440–2448, Montréal, Canada, December 2015.
32.Sukhbaatar, S., Grave, E., Lample, G., Jegou, H., and Joulin, A. Augmenting self-attention with persistent memory. Preprint arXiv:1907.01470, 2019.
33.Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 4344–4353, Hong Kong, China, November 2019.
34.Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 5998–6008, Long Beach, CA, USA, December 2017.
35.Xiao, H., Rasul, K., and Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint arXiv:1708.07747, 2017.
36.Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In Proc. European Conf. on Computer Vision (ECCV), volume 8689, pp. 818–833, Zurich, Switzerland, September 2014.

Citation

MLA

Irie, K., et al. “The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention”. International Conference on Machine Learning, vol. 162, 2022, pp. 9639–59, https://proceedings.mlr.press/v162/irie22a.html.

APA

Irie, K., Csordás, R., & Schmidhuber, J. (2022). The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention. International Conference on Machine Learning, 162, 9639–9659. https://proceedings.mlr.press/v162/irie22a.html

Chicago

Irie, K., R. Csordás, and J. Schmidhuber. 2022. “The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention”. International Conference on Machine Learning 162: 9639–59. https://proceedings.mlr.press/v162/irie22a.html.

Harvard

Irie, K., Csordás, R. and Schmidhuber, J. (2022) “The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention”, International Conference on Machine Learning. PMLR, pp. 9639–9659. Available at: https://proceedings.mlr.press/v162/irie22a.html.

Vancouver

1. Irie K, Csordás R, Schmidhuber J (2022) The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention. In: International Conference on Machine Learning. PMLR, pp 9639–9659

BibTeX

@InProceedings{pmlr-v162-irie22a,
  title = 	 {The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention},
  author =       {Irie, Kazuki and Csord{\'a}s, R{\'o}bert and Schmidhuber, J{\"u}rgen},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {9639--9659},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/irie22a/irie22a.pdf},
  url = 	 {https://proceedings.mlr.press/v162/irie22a.html},
  abstract = 	 {Linear layers in neural networks (NNs) trained by gradient descent can be expressed as a key-value memory system which stores all training datapoints and the initial weights, and produces outputs using unnormalised dot attention over the entire training experience. While this has been technically known since the 1960s, no prior work has effectively studied the operations of NNs in such a form, presumably due to prohibitive time and space complexities and impractical model sizes, all of them growing linearly with the number of training patterns which may get very large. However, this dual formulation offers a possibility of directly visualising how an NN makes use of training patterns at test time, by examining the corresponding attention weights. We conduct experiments on small scale supervised image classification tasks in single-task, multi-task, and continual learning settings, as well as language modelling, and discuss potentials and limits of this view for better understanding and interpreting how NNs exploit training patterns. Our code is public.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Dual Form of a Linear Layer Trained by Gradient Descent

Knowl 2 — Attribution of Test Predictions to Training Patterns via Dual-Form Attention

Knowl 3 — Correlation Between Per-Class Cumulative Attention Scores and Network Predictions

Knowl 4 — Cross-Task Representation Sharing in Jointly Trained Deep Linear Layers

Knowl 5 — Catastrophic Forgetting as Retrieval Interference in Dual Form

Knowl 6 — Contextual and Structural Retrieval of Training Passages in LSTM Language Models

Knowl 7 — Self-Attention Linear Projections as Hierarchical Two-Level Retrieval

Knowl 8 — Storage Overhead and Observational Constraints of Dual-Form Neural Network Analysis

References

Citation

BibTeX

Source Code

Access the Paper