Back

A Closer Look at Memorization in Deep Networks

Devansh ArpitStanisław JastrzębskiNicolas BallasDavid KruegerEmmanuel BengioMaxinder S. KanwalTegan MaharajAsja FischerAaron CourvilleYoshua Bengio

article2017ICML2,264 citations

Demonstrates that deep neural networks naturally learn simple patterns before memorizing noise, proving that training data directly controls effective capacity and generalization during gradient-based optimization.

Listen

Modern deep neural networks are massive mathematical models that possess enough capacity to memorize entire datasets by brute force, including completely random noise. Despite this immense capacity, these models reliably learn meaningful rules and generalize well when trained on real-world data, creating a fundamental puzzle for understanding artificial intelligence systems. The article sets out to evaluate whether deep neural networks rely on brute-force memorization when trained on real datasets and to demonstrate how their internal learning dynamics differ between genuine data and random noise.

To examine this question, the researchers conducted controlled experiments comparing standard neural network architectures trained on standard image benchmarks (MNIST handwritten digits and CIFAR-10 natural images) against versions of these datasets containing synthetic noise, such as randomized image pixels or completely randomized category labels. They tracked optimization dynamics across various network capacities, training durations, and dataset sizes, measuring model complexity through gradient sensitivity and the density of decision boundaries using adversarial sample search methods.

Across multiple experiments, the article established several key findings. First, gradient-based optimization behaves in fundamentally different ways on real data compared to noise. On real data, models quickly discover shared patterns, creating substantial variations where simple examples are learned in a single training epoch while difficult examples take longer; in contrast, noise examples are learned at a uniform, independent rate. Second, deep networks systematically prioritize learning simple, broad patterns before they begin memorizing individual data points or noisy labels. When trained on corrupted datasets, validation accuracy peaks early before declining as the model begins fitting the noise. Third, increasing the model capacity on noisy datasets actually improves validation performance on real examples, showing that larger networks can absorb noise without disrupting genuine pattern learning. Finally, explicit regularization methods—most notably dropout, especially when combined with adversarial training—can substantially slow down or halt memorization of noise without degrading performance on real data.

These findings indicate that traditional theories of machine learning, which argue that model capacity should be restricted to prevent memorization, are incomplete because they ignore how training data directly guides optimization. In practical terms, developers do not need to arbitrarily shrink model capacity to avoid overfitting; instead, standard training methods naturally capture broad, cost-effective patterns first. However, because networks will eventually memorize corruptions and noise if trained too long, relying solely on training loss creates severe performance risks.

For engineering and operational practice, teams developing deep learning systems should combine early stopping with targeted regularization techniques, specifically dropout and adversarial training, to prevent the memorization of noisy or mislabeled inputs. Moving forward, researchers and organizations should focus on developing data-dependent measures of model capacity and testing how dataset characteristics influence learning efficiency across different real-world operational domains.

The findings are supported by consistent empirical demonstrations across standard image benchmarks and network architectures. However, confidence should be tempered when applying these conclusions to radically different settings, such as extreme class imbalance, complex language models, or highly non-standard loss functions, which may exhibit different optimization behaviors.

Jagiellonian UniversityMcGill UniversityMila – Québec Artificial Intelligence InstitutePolytechnique MontréalUniversité de MontréalUniversity of BonnUniversity of California Berkeleynetwork architectureoptimizationregularizationDataReal DataTraining
arXiv: 1706.05394
  • Paper: Overcoming catastrophic forgetting in neural networks, James Kirkpatrick et al. (2017). This paper builds directly upon the memorization and generalization dynamics discussed in the source by introducing elastic weight consolidation to prevent catastrophic forgetting.
  • Paper: Continual Learning Through Synaptic Intelligence, Friedemann Zenke et al. (2017). This work extends the source's exploration of network training trajectories by examining how synaptic intelligence protects critical parameters during sequential learning.
Cover for A Closer Look at Memorization in Deep Networks

Abstract

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

Topics

network architectureoptimizationregularizationdeep belief networksconvolutional neural networkscommunication networksmesh networkintegrated data

Keywords

DataReal DataTrainingDeepMemorizationNoiseNetworksDeep NetworksLearningCloser Look

Table of Contents

  • 1 Introduction
  • 2 Experiment Details
  • 3 Qualitative Differences of DNNs Trained on Random vs. Real Data
  • 3.1 Easy Examples as Evidence of Patterns in Real Data
  • 3.2 Loss-Sensitivity in Real vs. Random Data
  • 3.3 Capacity and Effective Capacity
  • 3.3.1 Effects of capacity and dataset size on validation performances
  • 3.3.2 Effects of capacity and dataset size on training time
  • 4 DNNs Learn Patterns First
  • 4.1 Critical Sample Ratio (CSR)
  • 4.2 Critical Samples Throughout Training
  • 5 Effect of Regularization on Learning
  • 6 Related Work
  • 7 Conclusion
  • References

Knowls

  1. Knowl 1 — Effective Capacity of a Learning Algorithm

    definition

    The effective capacity EC(A)\text{EC}(\mathcal{A}) of a learning algorithm A\mathcal{A}—defined jointly by a model architecture and its optimization procedure (such as training a specific neural network architecture for a set number of epochs using stochastic gradient descent with a given learning rate)—is the set of all hypotheses that can be produced by applying A\mathcal{A} to some dataset D\mathcal{D}. Formally:

    EC(A)={h∣∃D such that h∈A(D)}\text{EC}(\mathcal{A}) = \{h \mid \exists \mathcal{D} \text{ such that } h \in \mathcal{A}(\mathcal{D})\}

    where A(D)\mathcal{A}(\mathcal{D}) denotes the set of hypotheses reachable by A\mathcal{A} on dataset D\mathcal{D} (which is a set when A\mathcal{A} is stochastic). This notion distinguishes reachable hypotheses from representational capacity, which encompasses all hypotheses expressible by any parameter assignment in the architecture.

  2. Knowl 2 — Critical Sample and Critical Sample Ratio (CSR)

    definition

    For a classification network with output vector f(x)=(f1(x),…,fk(x))∈Rkf(x) = (f_1(x), \dots, f_k(x)) \in \mathbb{R}^k for an input x∈Rnx \in \mathbb{R}^n drawn from a data distribution, a data point xx is defined as a critical sample if there exists an adversarial sample x^\hat{x} within an L∞L_\infty-ball of radius rr centered at xx such that the network's predicted label changes:

    arg⁡max⁡ifi(x)≠arg⁡max⁡jfj(x^)subject to ∥x−x^∥∞≤r\arg\max_i f_i(x) \neq \arg\max_j f_j(\hat{x}) \quad \text{subject to } \|x - \hat{x}\|_{\infty} \le r

    Here r>0r > 0 is a fixed scalar bound, and the condition depends on the network's own prediction on xx rather than the ground-truth label.

    The Critical Sample Ratio (CSR) on a dataset D\mathcal{D} is the proportion of samples in D\mathcal{D} that are critical samples:

    CSR(D)=∣{x∈D∣x is a critical sample}∣∣D∣\text{CSR}(\mathcal{D}) = \frac{|\{x \in \mathcal{D} \mid x \text{ is a critical sample}\}|}{|\mathcal{D}|}

    CSR serves as a geometric proxy for the complexity of the learned decision surface: a higher CSR indicates a higher density of decision boundaries in the vicinity of the data manifold.

  3. Knowl 3 — Langevin Adversarial Sample Search (LASS)

    algorithm

    Langevin Adversarial Sample Search (LASS) searches for an adversarial sample x^\hat{x} within an L∞L_\infty-box of radius rr around an input x∈Rnx \in \mathbb{R}^n for a classifier f:Rn→Rkf: \mathbb{R}^n \to \mathbb{R}^k. It extends the Fast Gradient Sign Method (FGSM) with Langevin dynamics noise, allowing the search to escape points where the gradient is zero but higher-order curvature leads to label changes nearby.

    Input: Input point x∈Rnx \in \mathbb{R}^n, step size α>0\alpha > 0, noise coefficient β>0\beta > 0, radius bound r>0r > 0, noise distribution η∼N(0,I)\eta \sim \mathcal{N}(0, I), maximum iterations MM
    Output: Adversarial point x^\hat{x}, or ∅\emptyset if no critical sample is found
    converged ←\leftarrow false
    x~←x\tilde{x} \leftarrow x
    x^←∅\hat{x} \leftarrow \emptyset
    iter ←\leftarrow 0
    while not converged and iter <M< M do
        iter ←\leftarrow iter + 1
        Δ←α⋅sign(∇xfk(x))+β⋅η\Delta \leftarrow \alpha \cdot \text{sign}(\nabla_x f_k(x)) + \beta \cdot \eta
        x~←x~+Δ\tilde{x} \leftarrow \tilde{x} + \Delta
        for each feature dimension i∈{1,…,n}i \in \{1, \dots, n\} do
            if ∣x~i−xi∣>r|\tilde{x}_i - x_i| > r then
                x~i←xi+r⋅sign(x~i−xi)\tilde{x}_i \leftarrow x_i + r \cdot \text{sign}(\tilde{x}_i - x_i)
            end if
        end for
        if arg⁡max⁡ifi(x)≠arg⁡max⁡jfj(x~)\arg\max_i f_i(x) \neq \arg\max_j f_j(\tilde{x}) then
            converged ←\leftarrow true
            $\hat{x} \leftarrow \tilde{x}
        end if
    end while
    return x^\hat{x}

    In standard evaluation on pixel inputs scaled to [0,255][0, 255], hyperparameters are configured with α=0.25\alpha = 0.25, β=0.2\beta = 0.2, and r=0.3r = 0.3, where η\eta is sampled from a standard normal distribution.

  4. Knowl 4 — Priority of Simple Patterns Over Noise Memorization in Training Dynamics

    empirical result

    When deep neural networks are trained with stochastic gradient descent on datasets containing partially or fully randomized labels (randY), they prioritize learning generalizable patterns before memorizing noise.

    Across datasets like MNIST and CIFAR-10 with label noise levels from 20% to 80%:

    1. The network achieves its maximum validation accuracy on clean holdout data early in training, well before reaching high training accuracy on the noisy dataset.
    2. The Critical Sample Ratio (CSR) remains low during the early pattern-learning stage, indicating smooth and simple decision boundaries.
    3. As training continues and the network fits the random labels, training loss continues to decrease and training accuracy increases, but validation accuracy declines and CSR rises steeply, reflecting the transition from simple hypothesis fitting to complex decision boundary formation necessary for memorization.

  5. Knowl 5 — Selective Suppression of Noise Memorization via Explicit Regularization

    empirical result

    Explicit regularization techniques differentially impede a deep network's ability to memorize random labels while preserving its capacity to learn generalizable features on real data.

    Evaluating convolutional neural networks on CIFAR-10 with clean labels versus 100% random labels (randY) across various regularizers reveals distinct behaviors:

    1. Dropout and Adversarial Training: Dropout (rates 0.0 to 0.9), particularly when combined with adversarial training on LASS-generated critical samples (weighting factor 0.2 to 0.7, dropout 0.03 to 0.5), is the most effective. It flattens the trade-off curve, dramatically suppressing the final training accuracy on random labels (capping memorization) without reducing validation accuracy on real data.
    2. Weight Decay and Gaussian Noise: Weight decay (range 0 to 1), input Gaussian noise (standard deviation 0 to 5), and hidden Gaussian noise (standard deviation 0 to 0.3) slow down optimization but are substantially less effective at preventing memorization without compromising clean validation performance.

  6. Knowl 6 — Loss-Sensitivity and Gini Measure of Memorization

    model/method

    Loss-sensitivity quantifies the influence of each training sample xx on subsequent optimization loss updates. For loss Lt\mathcal{L}_t after tt SGD steps, the sensitivity of the loss to an input xx is:

    gxt=∥∂Lt∂x∥1g_x^t = \left\| \frac{\partial \mathcal{L}_t}{\partial x} \right\|_1

    computed by backpropagating through the unrolled computation graph of the tt SGD update steps. The average loss-sensitivity over TT steps is gˉx=1T∑t=1Tgxt\bar{g}_x = \frac{1}{T} \sum_{t=1}^T g_x^t.

    The inequality of gˉx\bar{g}_x across all training samples is measured by the Gini coefficient:

    G=∑i=1N∑j=1N∣gˉxi−gˉxj∣2N∑i=1NgˉxiG = \frac{\sum_{i=1}^N \sum_{j=1}^N |\bar{g}_{x_i} - \bar{g}_{x_j}|}{2N \sum_{i=1}^N \bar{g}_{x_i}}

    where NN is the dataset size. A Gini coefficient of 0 indicates equal sensitivity across all samples, while 1 indicates maximum concentration.

    When trained on real datasets, the Gini coefficient rises as training progresses, demonstrating that loss-sensitivity concentrates on a small subset of influential samples. On random data (noise inputs or random labels), the Gini coefficient remains low throughout training, showing that the model remains uniformly sensitive to virtually every sample.

  7. Knowl 7 — Example Difficulty Disparity Between Real and Random Data

    empirical result

    When a 2-layer MLP (4096 hidden units per layer) is trained for a single epoch across 100 random weight initializations and data shufflings, individual examples exhibit distinct difficulty characteristics depending on the data type:

    1. Real Data: The distribution of per-example misclassification rates exhibits extreme variance. A substantial subset of examples is consistently classified correctly after just one epoch across all initializations ('easy examples'), while another subset is consistently misclassified ('hard examples'), indicating shared underlying patterns learned early.
    2. Random Noise Inputs (randX): The distribution of per-example classification accuracies closely matches a Binomial distribution Bin(n=100,p)\text{Bin}(n=100, p), where pp is the mean accuracy across the dataset. This demonstrates that random input examples are fit independently and with uniform difficulty, lacking shared features.

  8. Knowl 8 — Scaling Dynamics of Convergence Time and Model Capacity under Data Noise

    empirical result

    The interaction between network capacity, training set size, and time-to-convergence (epochs to achieve 100% training accuracy) differs fundamentally between real and noise data:

    1. Convergence Time Scaling: Training time increases much more sharply as a function of dataset size when training on Gaussian noise inputs (randX) than on real data, because real data contains regularities that allow unseen or additional examples to be predicted from existing representations.
    2. Capacity Requirements with Noise: In contrast to traditional statistical learning theory—which suggests capacity must be restricted to prevent overfitting on noisy data—optimal validation performance on datasets corrupted with label or input noise requires higher representational capacity (e.g., more hidden units in MLPs on MNIST) than on clean data. This extra capacity allows the network to memorize noisy samples in parameter space without distorting the representations of clean patterns.

Coverage note — None was omitted; all key contributions—including the conceptual framing of effective capacity, example difficulty dynamics, gradient-based loss-sensitivity, critical sample ratio, the LASS algorithm, pattern-first dynamics, and regularizer comparison—are covered in the knowls.

References

  1. 1.An, Guozhong. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643–674, 1996.
  2. 2.Bartlett, Peter L, Bousquet, Olivier, Mendelson, Shahar, et al. Local rademacher complexities. The Annals of Statistics, 33(4):1497–1537, 2005.
  3. 3.Bengio, Yoshua et al. Learning deep architectures for ai. Foundations and trends® in Machine Learning, 2(1):1–127, 2009.
  4. 4.Bishop, Chris M. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116, 1995.
  5. 5.Bojanowski, P. and Joulin, A. Unsupervised Learning by Predicting Noise. ArXiv e-prints, April 2017.
  6. 6.Bottou, Léon. Online learning and stochastic approximations. On-line learning in neural networks, 17(9):142, 1998.
  7. 7.Chaudhari, Pratik, Choromanska, Anna, Soatto, Stefano, and LeCun, Yann. Entropy-sgd: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838, 2016.
  8. 8.Chollet, François et al. Keras. https://github.com/fchollet/keras, 2015.
  9. 9.Cybenko, George. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems (MCSS), 2(4):303–314, 1989.
  10. 10.Fix, Evelyn and Hodges Jr, Joseph L. Discriminatory analysis-nonparametric discrimination: consistency properties. Technical report, DTIC Document, 1951.
  11. 11.Gini, Corrado. Variabilita e mutabilita. Journal of the Royal Statistical Society, 76(3), 1913.
  12. 12.Goodfellow, Ian, Bengio, Yoshua, and Courville, Aaron. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
  13. 13.Goodfellow, Ian J, Mirza, Mehdi, Xiao, Da, Courville, Aaron, and Bengio, Yoshua. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
  14. 14.Goodfellow, Ian J, Shlens, Jonathon, and Szegedy, Christian. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  15. 15.Hardt, Moritz, Recht, Benjamin, and Singer, Yoram. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.01240, 2015.
  16. 16.Hornik, Kurt, Stinchcombe, Maxwell, and White, Halbert. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  17. 17.Im, Daniel Jiwoong, Tao, Michael, and Branson, Kristin. An empirical analysis of deep network loss surfaces. arXiv preprint arXiv:1612.04010, 2016.
  18. 18.Keskar, Nitish Shirish, Mudigere, Dheevatsa, Nocedal, Jorge, Smelyanskiy, Mikhail, and Tang, Ping Tak Peter. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.
  19. 19.Koh, Pang Wei and Liang, Percy. Understanding blackbox predictions via influence functions. arXiv preprint arXiv:1703.04730, 2017.
  20. 20.Krizhevsky, Alex, Nair, Vinod, and Hinton, Geoffrey. Cifar-10 (canadian institute for advanced research). URL http://www.cs.toronto.edu/~kriz/cifar.html.
  21. 21.Kurakin, Alexey, Goodfellow, Ian, and Bengio, Samy. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.
  22. 22.LeCun, Yann, Cortes, Corinna, and Burges, Christopher JC. The mnist database of handwritten digits, 1998.
  23. 23.Lin, Henry W and Tegmark, Max. Why does deep and cheap learning work so well? arXiv preprint arXiv:1608.08225, 2016.
  24. 24.Maclaurin, Dougal, Duvenaud, David K, and Adams, Ryan P. Gradient-based hyperparameter optimization through reversible learning. In ICML, pp. 2113–2122, 2015.
  25. 25.Miyato, Takeru, Maeda, Shin-ichi, Koyama, Masanori, Nakae, Ken, and Ishii, Shin. Distributional smoothing with virtual adversarial training. stat, 1050:25, 2015.
  26. 26.Montavon, Grégoire, Braun, Mikio L., and Müller, Klaus-Robert. Kernel analysis of deep networks. Journal of Machine Learning Research, 12, 2011.
  27. 27.Montufar, Guido F, Pascanu, Razvan, Cho, Kyunghyun, and Bengio, Yoshua. On the number of linear regions of deep neural networks. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q. (eds.), Advances in Neural Information Processing Systems 27, pp. 2924–2932. Curran Associates, Inc., 2014.
  28. 28.Neyshabur, Behnam, Tomioka, Ryota, and Srebro, Nathan. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614, 2014.
  29. 29.Poole, Ben, Lahiri, Subhaneil, Raghu, Maithreyi, Sohl-Dickstein, Jascha, and Ganguli, Surya. Exponential expressivity in deep neural networks through transient chaos. In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29, pp. 3360–3368. Curran Associates, Inc., 2016.
  30. 30.Raghu, Maithra, Poole, Ben, Kleinberg, Jon, Ganguli, Surya, and Sohl-Dickstein, Jascha. On the expressive power of deep neural networks. arXiv preprint arXiv:1606.05336, 2016.
  31. 31.Saxe, Andrew M, McClelland, James L, and Ganguli, Surya. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
  32. 32.Sjoberg, J., Sjoeberg, J., Sjöberg, J., and Ljung, L. Overtraining, regularization and searching for a minimum, with application to neural networks. International Journal of Control, 62:1391–1407, 1995.
  33. 33.Sokolic, Jure, Giryes, Raja, Sapiro, Guillermo, and Rodrigues, Miguel RD. Robust large margin deep neural networks. arXiv preprint arXiv:1605.08254, 2016.
  34. 34.Szegedy, Christian, Zaremba, Wojciech, Sutskever, Ilya, Bruna, Joan, Erhan, Dumitru, Goodfellow, Ian J., and Fergus, Rob. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013. URL http://arxiv.org/abs/1312.6199.
  35. 35.Theano Development Team, and others. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016.
  36. 36.Vapnik, Vladimir Naumovich and Vapnik, Vlamimir. Statistical learning theory, volume 1. Wiley New York, 1998.
  37. 37.Wang, Shengjie. Analysis of deep neural networks with the extended data jacobian matrix.
  38. 38.Wilson, D Randall and Martinez, Tony R. The general inefficiency of batch training for gradient descent learning. Neural Networks, 16(10):1429–1451, 2003.
  39. 39.Yao, Yuan, Rosasco, Lorenzo, and Caponnetto, Andrea. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289–315, 2007.
  40. 40.Zhang, Chiyuan, Bengio, Samy, Hardt, Moritz, Recht, Benjamin, and Vinyals, Oriol. Understanding deep learning requires rethinking generalization. International Conference on Learning Representations (ICLR), 2017.

Citation

MLA
Arpit, D., et al. “A Closer Look at Memorization in Deep Networks”. arXiv, 2017, http://arxiv.org/abs/1706.05394v2.
APA
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., & Lacoste-Julien, S. (2017). A Closer Look at Memorization in Deep Networks. arXiv. http://arxiv.org/abs/1706.05394v2
Chicago
Arpit, D., S. Jastrzębski, N. Ballas, et al. 2017. “A Closer Look at Memorization in Deep Networks”. arXiv. http://arxiv.org/abs/1706.05394v2.
Harvard
Arpit, D. et al. (2017) “A Closer Look at Memorization in Deep Networks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1706.05394v2.
Vancouver
1. Arpit D, Jastrzębski S, Ballas N, et al (2017) A Closer Look at Memorization in Deep Networks. arXiv

BibTeX

@article{arpit2017closer,
  title = {A Closer Look at Memorization in Deep Networks},
  author = {Arpit, Devansh and Jastrzębski, Stanisław and Ballas, Nicolas and Krueger, David and Bengio, Emmanuel and Kanwal, Maxinder S. and Maharaj, Tegan and Fischer, Asja and Courville, Aaron and Bengio, Yoshua and Lacoste-Julien, Simon},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1706.05394v2},
  eprint = {1706.05394}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/