Understanding Neural Networks Through Deep Visualization

Jason YosinskiJeff CluneAnh NguyenThomas FuchsHod Lipson

article2015arXiv1,969 citations

Presents open-source visualization methods that explain intermediate layer computations in deep convolutional networks by tracking real-time activations and synthesizing clear feature images via regularized optimization.

Listen

Modern computer vision relies heavily on deep neural networks that achieve near-human or superhuman accuracy in image classification tasks. However, these complex architectures operate as opaque systems with tens of millions of parameters, making it difficult for researchers and practitioners to understand how intermediate computational steps function. This lack of transparency impedes systematic debugging, slows model optimization, and poses operational risks when models are deployed into dynamic, real-world environments. The article sets out to develop, evaluate, and demonstrate open-source visualization tools and optimization techniques that clearly interpret what individual neurons and intermediate layers represent within trained deep networks.

The approach introduces two complementary software mechanisms evaluated on an eight-layer vision architecture trained on standard object classification benchmarks. The first tool renders real-time activations across all computational layers as the network processes live webcam video and static imagery. The second tool synthesizes preferred synthetic input images for individual neurons using gradient-based optimization paired with a set of four regularizers: weight decay, periodic Gaussian blurring, clipping of pixels with small norm, and clipping of pixels with low contribution. To assess these methods, the evaluation tested individual parameter sweeps and executed a random search across 300 hyperparameter combinations to identify optimal configurations for human interpretability.

The findings provide key insights into internal network representations and stability. First, intermediate and late convolutional layers develop localized detectors for specific high-level concepts, such as faces, text, and shoulders, even when the model is never explicitly trained on those categories. Second, synthetic feature visualization drastically improves in clarity when multiple regularizers are combined, suppressing high-frequency noise and isolating recognizable object structures across all layers. Third, the network displays distinct layer-by-layer stability profiles: lower and intermediate layers remain robust and invariant to changes in scale, pose, and lighting, whereas the final fully connected classification layers show extreme sensitivity to minor webcam noise and non-standard inputs when trained target categories are absent. Finally, the analysis explains this high-frequency noise bias by observing that low-frequency base filters average activation values roughly 17 times higher than high-frequency filters, which inadvertently forces subsequent network layers to assign larger multipliers to high-frequency signals during backpropagation.

These results demonstrate that discriminatively trained networks capture rich structural representations of the physical world rather than simple, superficial statistical shortcuts. For organizations developing or adopting computer vision, these tools reduce deployment risk and operational failure by providing transparency into failure modes caused by out-of-distribution inputs. Based on these insights, practitioners should adopt regularized visualization suites to inspect trained models, explore sparse connectivity during transfer learning to take advantage of localized intermediate representations, and integrate stronger data priors into generative pipelines. Users must note that the synthetic visualizations rely on qualitative interpretability and hand-crafted regularization constraints, and that out-of-distribution inputs require caution due to high sensitivity in the final classification layers.

Cover for Understanding Neural Networks Through Deep Visualization

Abstract

Recent years have produced great advances in training large, deep neural networks (DNNs), including notable successes in training convolutional neural networks (convnets) to recognize natural images. However, our understanding of how these models work, especially what computations they perform at intermediate layers, has lagged behind. Progress in the field will be further accelerated by the development of better tools for visualizing and interpreting neural nets. We introduce two such tools here. The first is a tool that visualizes the activations produced on each layer of a trained convnet as it processes an image or video (e.g. a live webcam stream). We have found that looking at live activations that change in response to user input helps build valuable intuitions about how convnets work. The second tool enables visualizing features at each layer of a DNN via regularized optimization in image space. Because previous versions of this idea produced less recognizable images, here we introduce several new regularization methods that combine to produce qualitatively clearer, more interpretable visualizations. Both tools are open source and work on a pre-trained convnet with minimal setup.

Table of Contents

  • 1 Introduction
  • 2 Visualizing Live Convnet Activations
  • 3 Visualizing via Regularized Optimization
  • 4 Discussion and Conclusion
  • References
  • S1 Why are gradient optimized images dominated by high frequencies?
  • S2 Conv Layer Montages

Knowls

  1. Knowl 1 — Regularized Gradient Ascent for Unit Feature Visualization

    model/method

    To visualize the preferred stimulus or feature encoded by an individual neuron ii in a convolutional neural network (CNN), an input image x∈RC×H×Wx \in \mathbb{R}^{C \times H \times W} (where C=3C=3 color channels and H=W=227H=W=227 pixels) is synthesized from a random initialization x0x_0 using regularized gradient ascent in input space. The objective is formulated as finding an image x∗x^* maximizing the activation ai(x)a_i(x) of unit ii subject to a regularization operator rθ(⋅)r_\theta(\cdot):

    x∗=arg⁡max⁡x(ai(x)−Rθ(x))x^* = \arg\max_x \left( a_i(x) - R_\theta(x) \right)

    In an iterative gradient descent/ascent framework with step size η\eta, the optimization alternates between moving along the gradient of unit activation ∂ai(x)/∂x\partial a_i(x) / \partial x and applying the regularizing transformation rθr_\theta:

    x←rθ(x+η∂ai(x)∂x)x \leftarrow r_\theta\left( x + \eta \frac{\partial a_i(x)}{\partial x} \right)

    Defining regularization via the operator rθ(⋅)r_\theta(\cdot) rather than solely as the analytical gradient of an objective penalty Rθ(x)R_\theta(x) enables incorporating non-differentiable transformations (such as periodic blurring or percentile-based pixel clipping) that constrain the generated image to visually interpretable natural priors.

  2. Knowl 2 — Regularization Operators for Input-Space Feature Synthesis

    model/method

    Unconstrained gradient ascent produces high-frequency, non-interpretable artifacts and extreme pixel values. Four distinct regularization operators rθ(x)r_\theta(x) can be applied iteratively to bias the synthesized image toward natural, interpretable features:

    1. L2L_2 Decay: Penalizes large pixel values to prevent isolated pixels from dominating the activation. It is defined by: rθ(x)=(1−θdecay)⋅xr_\theta(x) = (1 - \theta_{\text{decay}}) \cdot x

    2. Periodic Gaussian Blur: Suppresses high-frequency noise by convolving the image with a Gaussian kernel of width θb_width\theta_{b\_\text{width}} every θb_every\theta_{b\_\text{every}} gradient ascent steps: rθ(x)=GaussianBlur(x,θb_width)r_\theta(x) = \text{GaussianBlur}(x, \theta_{b\_\text{width}})

    3. Small-Norm Pixel Clipping: Suppresses background non-zero pixel drift by computing the L2L_2 norm of each pixel across the red, green, and blue channels (∑c=1Cxc,u,v2\sqrt{\sum_{c=1}^C x_{c,u,v}^2}) and setting all pixels whose norm falls below the percentile threshold θn_pct\theta_{n\_\text{pct}} to zero.

    4. Small-Contribution Pixel Clipping: Sets pixels to zero if their estimated contribution to unit activation ai(x)a_i(x) falls below a percentile threshold θc_pct\theta_{c\_\text{pct}}, leaving only salient regions active.

  3. Knowl 3 — First-Order Approximation for Pixel Contribution Clipping

    equation

    The exact contribution of a pixel jj at spatial coordinate (u,v)(u, v) to the activation ai(x)a_i(x) of unit ii is defined by the difference ∣ai(x)−ai(x−j)∣|a_i(x) - a_i(x_{-j})|, where x−jx_{-j} is the image xx with the jthj^{\text{th}} pixel set to zero. Because computing this directly across all pixels requires a prohibitively expensive forward pass per pixel, the activation function ai(x)a_i(x) is linearized around xx using a first-order Taylor expansion.

    The contribution at spatial location (u,v)(u, v) across all CC channels is approximated as:

    Contribution(u,v)=∣∑c=1Cxc,u,v⋅[∇xai(x)]c,u,v∣\text{Contribution}(u, v) = \left| \sum_{c=1}^C x_{c, u, v} \cdot \left[ \nabla_x a_i(x) \right]_{c, u, v} \right|

    where ∇xai(x)=∂ai(x)∂x\nabla_x a_i(x) = \frac{\partial a_i(x)}{\partial x} is the spatial gradient of the activation with respect to the input image, and xc,u,vx_{c, u, v} is the pixel intensity for channel cc. The regularization operator rθ(x)r_\theta(x) sets any pixel whose estimated contribution is below the θc_pct\theta_{c\_\text{pct}} percentile across all spatial coordinates to zero.

  4. Knowl 4 — Hyperparameter Configurations for Interpretable Feature Synthesis

    data/table

    Evaluating 300 random combinations of regularization hyperparameters identified four distinct, complementary parameter configurations that produce recognizable visualization styles for AlexNet-style convolutional networks:

    θdecay\theta_{\text{decay}} θb_width\theta_{b\_\text{width}} θb_every\theta_{b\_\text{every}} θn_pct\theta_{n\_\text{pct}} θc_pct\theta_{c\_\text{pct}}
    0 0.5 4 50 0
    0.3 0 0 20 0
    0.0001 1.0 4 0 0
    0 0.5 4 0 90

    Each configuration produces distinct visual characteristics:

    • Row 1 combines moderate blurring with aggressive norm clipping (50th percentile), isolating sparse, high-contrast salient outlines.
    • Row 2 uses strong L2L_2 decay (0.3) with lower norm clipping (20th percentile) and no blur, capturing sharp, high-frequency motifs.
    • Row 3 combines very low L2L_2 decay (10−410^{-4}) with a wider Gaussian blur kernel (θb_width=1.0\theta_{b\_\text{width}}=1.0) every 4 steps, serving as the most consistently interpretable standalone hyperparameter set for synthesizing smooth, lower-frequency global object features across all layers.
    • Row 4 pairs Gaussian blurring with strict small-contribution clipping (90th percentile), highlighting only the pixels driving the gradient.
  5. Knowl 5 — Emergence of Localized Semantic Part Detectors in Intermediate Convolutional Layers

    empirical result

    In deep convolutional neural networks (such as AlexNet) trained on the 1,000-class ImageNet classification task, units in intermediate convolutional layers (specifically conv4 and conv5) self-organize into localized, highly specialized detectors for distinct object parts and concepts—such as human and animal faces, text, wheels, shoulders, flowers, and fruit—even though none of these concepts exist as explicit target classes in the training data.

    For example, channel 151 on the conv5 layer (conv5_151) functions as a generalized face detector that fires selectively for human, lion, and cat faces while remaining invariant to changes in scale, pose, lighting, and scene context. This indicates that intermediate representations in supervised discriminative CNNs are partially local rather than entirely distributed across channels.

  6. Knowl 6 — Robustness Disparity Between Convolutional Layers and Fully Connected Outputs

    empirical result

    When a deep convolutional neural network processes dynamic live video inputs (such as an arbitrary webcam stream containing common scenes not represented among the 1,000 ImageNet categories), feature representations across lower and intermediate convolutional layers (conv1 through conv5) exhibit robust invariance to physical shifts, scale changes, and environmental noise.

    In contrast, the top fully connected layers (fc6 and fc7) and the final softmax class probability vector display extreme volatility. Rather than outputting steady, uniform, low-confidence distributions for out-of-domain scenes, the output probability distributions fluctuate rapidly and unpredictably in response to minute frame-to-frame sensor noise.

  7. Knowl 7 — Layer-wise Progression of Feature Complexity and Geometric Invariance

    empirical result

    Visualizing preferred stimuli across the eight successive layers of an AlexNet-style network reveals a monotonic increase in feature complexity, abstraction, and invariance:

    • Layers 1 and 2 synthesize localized, low-level oriented Gabor-like edge filters and color blobs.
    • Layers 3, 4, and 5 synthesize increasingly complex compositional structures and distinct object parts (e.g., eyes, wheels, corners, animal faces, handles).
    • Fully connected layers 6, 7, and 8 synthesize holistic, multi-part objects and class prototypes.

    Across multiple random optimization initializations for the same unit, the structural variation of the synthesized images increases with layer depth, reflecting greater learned transformation invariance. The transition from the final convolutional layer (conv5) to the first fully connected layer (fc6) produces the largest step increase in spatial invariance and visual pattern diversity.

  8. Knowl 8 — Mechanism for High-Frequency Gradient Dominance in Feature Optimization

    theoretical result

    The tendency of unregularized gradient ascent in pixel space to generate high-frequency noise patterns rather than natural images can be explained by the interaction between the 1/f1/f power spectrum of natural images and network connection weights:

    1. In the first layer (conv1), low-frequency edge filters receive substantially higher mean activation values than high-frequency edge filters (empirically observed to differ by an average factor of approximately 17, with mean activations of 90 for low-frequency filters versus 5.4 for high-frequency filters on ImageNet).
    2. To combine signals from both low- and high-frequency conv1 channels into comparable ranges, second-layer (conv2) units develop systematically larger weight magnitudes for connections coming from high-frequency conv1 units.
    3. During backpropagation, the activation gradient propagated backward through conv2 is multiplied by these larger weights toward high-frequency conv1 channels, causing high-frequency components to disproportionately dominate the gradient update vector in pixel space.
  9. Knowl 9 — Interactive Multi-View Architecture for ConvNet Inspection

    model/method

    An interactive neural network visualization architecture combines multiple complementary dataset-centric and network-centric views to resolve representational ambiguities in individual units:

    1. Live Forward Activations: Plots 2D spatial feature activation maps across every channel and layer (including pooling and normalization layers) in response to dynamic video feeds or static images.
    2. Regularized Optimization: Generates synthetic preferred stimuli via gradient ascent for selected individual units.
    3. Top Dataset Image Patches: Displays the image crops from the training set that elicited the highest activation for a selected channel.
    4. Deconvolution / Backpropagation Highlighting: Computes backward diffs via guided backpropagation or deconvolution starting from individual units to reveal the specific input pixels driving unit activation.

    Combining these views disambiguates unit function; for instance, a unit responding primarily to dog faces but secondarily to floral patterns can be identified through regularized optimization and live responses even when top dataset exemplars only contain dogs.

Coverage note — None was omitted; all contributed visualization methods, regularization operators, the empirical findings on layer-wise invariance and stability, and the theoretical explanation of high-frequency gradient dominance are included.

References

  1. 1.Bergstra, James, Breuleux, Olivier, Bastien, Fred´eric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Presentation.
  2. 2.Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clement. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011.
  3. 3.Dai, Jifeng, Lu, Yang, and Wu, Ying Nian. Generative modeling of convolutional neural networks. In International Conference on Learning Representations (ICLR), 2015.
  4. 4.Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248–255. IEEE, 2009.
  5. 5.Erhan, Dumitru, Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Visualizing higher-layer features of a deep network. Technical report, Technical report, University of Montreal, 2009.
  6. 6.Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. JMLR W&CP Volume, volume 15, pp. 315–323, 2011.
  7. 7.Goodfellow, Ian J, Warde-Farley, David, Lamblin, Pascal, Dumoulin, Vincent, Mirza, Mehdi, Pascanu, Razvan, Bergstra, James, Bastien, Fred´eric, and Bengio, Yoshua. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013.
  8. 8.Goodfellow, Ian J, Shlens, Jonathon, and Szegedy, Christian. Explaining and Harnessing Adversarial Examples. ArXiv e-prints, December 2014.
  9. 9.Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., and Ng, A. Y. Deep Speech: Scaling up end-to-end speech recognition. ArXiv e-prints, December 2014.
  10. 10.Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
  11. 11.Jia, Yangqing, Shelhamer, Evan, Donahue, Jeff, Karayev, Sergey, Long, Jonathan, Girshick, Ross, Guadarrama, Sergio, and Darrell, Trevor. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
  12. 12.Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoff. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1106–1114, 2012.
  13. 13.Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollar, Piotr, and Zitnick, C. Lawrence. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014. URL http://arxiv.org/abs/1405.0312.
  14. 14.Mahendran, A. and Vedaldi, A. Understanding Deep Image Representations by Inverting Them. ArXiv e-prints, November 2014.
  15. 15.Nguyen, Anh, Yosinski, Jason, and Clune, Jeff. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. ArXiv e-prints, December 2014.
  16. 16.Schroff, F., Kalenichenko, D., and Philbin, J. FaceNet: A Unified Embedding for Face Recognition and Clustering. ArXiv e-prints, March 2015.
  17. 17.Simonyan, Karen, Vedaldi, Andrea, and Zisserman, Andrew. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, presented at ICLR Workshop 2014, 2013.
  18. 18.Szegedy, Christian, Zaremba, Wojciech, Sutskever, Ilya, Bruna, Joan, Erhan, Dumitru, Goodfellow, Ian J., and Fergus, Rob. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013.
  19. 19.Taigman, Yaniv, Yang, Ming, Ranzato, Marc’Aurelio, and Wolf, Lior. Deepface: Closing the gap to human-level performance in face verification. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pp. 1701–1708. IEEE, 2014.
  20. 20.Torralba, Antonio and Oliva, Aude. Statistics of natural image categories. Network: computation in neural systems, 14(3): 391–412, 2003.
  21. 21.Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 27, pp. 3320–3328. Curran Associates, Inc., December 2014.
  22. 22.Zeiler, Matthew D and Fergus, Rob. Visualizing and understanding convolutional neural networks. arXiv preprint arXiv:1311.2901, 2013.
  23. 23.Zhou, Bolei, Khosla, Aditya, Lapedriza, Agata, Oliva, Aude, and Torralba, Antonio. Object detectors emerge in deep scene cnns. CoRR, abs/1412.6856, 2014. URL http://arxiv.org/abs/1412.6856.

Citation

MLA
Yosinski, J., et al. “Understanding Neural Networks Through Deep Visualization”. arXiv, 2015, http://arxiv.org/abs/1506.06579v1.
APA
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., & Lipson, H. (2015). Understanding Neural Networks Through Deep Visualization. arXiv. http://arxiv.org/abs/1506.06579v1
Chicago
Yosinski, J., J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. 2015. “Understanding Neural Networks Through Deep Visualization”. arXiv. http://arxiv.org/abs/1506.06579v1.
Harvard
Yosinski, J. et al. (2015) “Understanding Neural Networks Through Deep Visualization”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1506.06579v1.
Vancouver
1. Yosinski J, Clune J, Nguyen A, Fuchs T, Lipson H (2015) Understanding Neural Networks Through Deep Visualization. arXiv

BibTeX

@article{yosinski2015understanding,
  title = {Understanding Neural Networks Through Deep Visualization},
  author = {Yosinski, Jason and Clune, Jeff and Nguyen, Anh and Fuchs, Thomas and Lipson, Hod},
  year = {2015},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1506.06579v1},
  eprint = {1506.06579}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors