Deep learning and the information bottleneck principle

Naftali TishbyNoga Zaslavsky

article2015Information Theory Workshop2,131 citations

Establishes an information-theoretic framework for deep learning by applying the Information Bottleneck principle to explain how successive layers compress input data while preserving target information to achieve generalization.

Listen

Deep neural networks are the leading technology behind modern artificial intelligence breakthroughs, yet the core principles governing how they work, how many layers they require, and how well they generalize to new data remain poorly understood. This lack of theoretical foundation creates significant uncertainty when engineering network architectures and evaluating real-world model risk. The article aims to establish a rigorous theoretical framework for deep neural networks by evaluating them through the information bottleneck principle, which treats learning as a fundamental trade-off between compressing input data and preserving accurate predictions.

The authors conducted a theoretical and statistical analysis evaluating feedforward deep neural networks through information-theoretic measures. By modeling a network's layers as a sequential processing chain, the analysis quantified the shared information between hidden representations, raw inputs, and target outputs. The approach evaluated optimal mathematical boundaries and established sample complexity limits for learning from finite training data without relying on ad hoc error metrics.

The analysis produced several key findings. First, optimal learning fundamentally requires continuous data compression; relying solely on raw input representations leads to severe overfitting and poor generalization on unseen data. Second, calculating shared information enables direct performance benchmarking of individual hidden layers, revealing measurable generalization and complexity gaps relative to theoretical performance limits. Third, the study showed that the generalization error bound depends directly on the effective complexity of the learned internal representations rather than the high dimensionality of the raw input. Finally, structural phase transitions along the optimal information-compression curve mathematically align with the points where individual artificial neurons lose the ability to linearly separate data, explaining why hierarchical multi-layer architectures are necessary.

These findings suggest that deep learning success is driven by successive information compression rather than mere pattern matching. This shifts the engineering perspective from heuristic network tuning toward mathematically grounded optimization, offering pathways to reduce computational costs, eliminate redundant network parameters, and better manage generalization risks. The analysis also suggests that introducing stochastic, or probabilistic, mapping between layers can help networks approach theoretical efficiency limits more closely than purely deterministic methods.

The article recommends developing new training algorithms explicitly designed around information bottleneck optimality criteria to systematically drive network layers closer to the theoretical limit. Practitioners and researchers should also explore using structural phase transitions to systematically determine the optimal number of hidden layers and internal unit configurations instead of relying on trial-and-error design.

These insights are primarily theoretical and conceptual, meaning caution is warranted before altering existing commercial training pipelines without further empirical validation. Future work requires testing these information-theoretic bounds across diverse large-scale deep learning models and complex production datasets to confirm practical feasibility and performance trade-offs.

The Hebrew University of Jerusalem layered mutual informations output layer information bottleneck principle deep learning information plane

arXiv: 1503.02406

Paper: The information bottleneck method, Naftali Tishby et al. (2000). This foundational paper introduces the information bottleneck principle and its mutual-information trade-off curves that the source directly adapts to analyze deep neural network representations.
Paper: Keeping Neural Networks Simple by Minimizing the Description Length of the Weights, Geoffrey E. Hinton et al. (1993). This classic work establishes the connection between network simplicity, generalization, and information-theoretic compression of representations in neural networks.
Paper: A Fast Learning Algorithm for Deep Belief Nets, Geoffrey E. Hinton et al. (2006). This work introduces layer-wise hierarchical representation learning in deep networks, providing the conceptual multi-layer architecture analyzed through the information bottleneck.
Paper: Greedy Layer-Wise Training of Deep Networks, Yoshua Bengio et al. (2007). This paper demonstrates how successive layers capture increasingly abstract representations, motivating the source's analysis of layer-by-layer mutual information transitions.
Paper: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, Andrew M. Saxe et al. (2014). This paper examines the theoretical learning dynamics and phase-like transitions in deep architectures, establishing background for the source's bifurcation analysis.

Paper: Deep Variational Information Bottleneck, Alexander A. Alemi et al. (2017). This paper translates the theoretical information bottleneck framework into an efficient, scalable variational optimization objective for modern deep neural networks.
Paper: Learning deep representations by mutual information estimation and maximization, R. Devon Hjelm et al. (2018). This work develops practical deep learning objectives that explicitly estimate and optimize local and global mutual information across network layers.
Paper: Fixing a Broken ELBO, Alexander A. Alemi et al. (2018). This work investigates rate-distortion trade-offs and mutual information bounds in deep latent-variable models, directly extending the information-theoretic analysis of representations.
Paper: A Closer Look at Memorization in Deep Networks, Devansh Arpit et al. (2017). This study empirically examines representation compression versus memorization dynamics during deep network training, building on the learning phases proposed by the bottleneck principle.
Paper: Understanding deep learning requires rethinking generalization, Chiyuan Zhang et al. (2017). This work critically challenges conventional complexity bounds and generalization theories in deep neural networks, continuing the inquiry into what governs deep learning generalization.
Paper: Deep Graph Infomax, Petar Veličković et al. (2019). This work applies mutual information maximization principles between local and global representations to graph neural network architectures.
Paper: InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets, Xi Chen et al. (2016). This paper utilizes mutual information maximization in deep generative networks to learn compact, interpretable representations.

Cover for Deep learning and the information bottleneck principle

Abstract

Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.

Topics

layered mutual informations output layer deep belief networks convolutional neural networks network architecture user information architecture designs

Keywords

information bottleneck principle deep learning information plane mutual information DNN architectures representational complexity generalization bounds information curve optimal representation deterministic annealing

I Introduction
II Background
II-A Deep Neural Networks
II-B The Information Bottleneck Principle
III A new Information Theoretic Learning Principle for DNNs
III-A Information characteristics of the layers
III-B Finite Samples and Generalization Bounds
IV IB Phase Transitions and the Breakdown of Linear Separability
V Discussion
References

Knowls

Knowl 1 — Markov Chain Structure and Data Processing Inequality for Feedforward DNNs
theoretical result
In a feedforward deep neural network (DNN) with $m$ hidden layers $h_1, h_2, \dots, h_m$ , an input variable $X$ , and a predicted output variable $\hat{Y}$ trained on a target label $Y$ , the representations across successive layers form a Markov chain:

$Y \to X \to h_1 \to h_2 \to \dots \to h_m \to \hat{Y}$

Applying the Data Processing Inequality (DPI) to this chain implies that the mutual information about the target variable $Y$ decreases monotonically (or at best remains constant) with layer depth:

$I(Y; X) \ge I(Y; h_1) \ge I(Y; h_2) \ge \dots \ge I(Y; h_m) \ge I(Y; \hat{Y})$

As a consequence, relevant information about $Y$ that is lost at any given hidden layer cannot be recovered by subsequent layers. Exact equality $I(Y; h_i) = I(Y; h_{i-1})$ holds if and only if layer $h_i$ forms a sufficient statistic of $h_{i-1}$ with respect to $Y$ . The fraction of target information captured by the complete network is quantified by $\frac{I(Y; \hat{Y})}{I(X; Y)}$ .
Knowl 2 — Layer-Wise Information Bottleneck Optimality Criterion
theoretical result
Under the Information Bottleneck (IB) framework, each layer $h_i$ of a feedforward neural network (with $h_0 = X$ as the input and $h_{m+1} = \hat{Y}$ as the network prediction) is evaluated by trading off its representational complexity against its residual distortion regarding the target label $Y$ . The optimal mapping $p(h_i \mid h_{i-1})$ for layer $h_i$ minimizes the layer-wise IB Lagrangian:

$\mathcal{L}[p(h_i \mid h_{i-1})] = I(h_{i-1}; h_i) + \beta_i I(Y; h_{i-1} \mid h_i)$

which is equivalent up to an additive constant to:

$\mathcal{L}[p(h_i \mid h_{i-1})] = I(h_{i-1}; h_i) - \beta_i I(Y; h_i)$

where $\beta_i > 0$ is a Lagrange multiplier determining the trade-off, $I(h_{i-1}; h_i)$ represents the description length (complexity) of layer $h_i$ given its input $h_{i-1}$ , and $I(Y; h_{i-1} \mid h_i) = I(Y; h_{i-1}) - I(Y; h_i)$ is the IB distortion quantifying the relevant information lost at layer $h_i$ . Moving deeper into the network hierarchy corresponds to successively smaller values of $\beta_i$ , yielding increasingly compressed representations.
Knowl 3 — Information Plane Representation of Deep Neural Network Layers
model/method
A deep neural network (DNN) can be quantified and visualized in the information plane, a two-dimensional coordinate system where:
- The horizontal axis is the representational complexity $R = I(X; h_i)$ , measuring the mutual information between the input $X$ and layer representation $h_i$ .
- The vertical axis is either the preserved relevant information $I(Y; h_i)$ or the Information Bottleneck (IB) distortion $D_{\text{IB}} = I(X; Y \mid h_i) = I(X; Y) - I(Y; h_i)$ .
The theoretical upper limit on achievable representation quality is given by the optimal IB rate-distortion curve for the joint distribution $p(X, Y)$ , where the negative inverse slope is the tradeoff parameter $\beta$ . The layers of a trained DNN form a discrete trajectory across this plane:
1. The input layer $X$ lies at maximal complexity $R = H(X)$ and zero distortion $D_{\text{IB}} = 0$ .
2. Successive hidden layers $h_1, h_2, \dots, h_m$ trade off small increments in distortion for large reductions in complexity $R$ .
3. The final output $\hat{Y}$ achieves the lowest complexity among the layers, retaining relevant predictive information about $Y$ while compressing away irrelevant variations in $X$ .
Knowl 4 — Finite-Sample Generalization Bounds and the Compression Requirement for Deep Neural Networks
theoretical result
When learning from a finite training sample of size $n$ drawn from a joint distribution $p(X, Y)$ with empirical distribution $\hat{p}(X, Y)$ and empirical mutual information $\hat{I}$ , the generalization performance of an extracted discrete representation $\hat{X}$ with cardinality $K = |\hat{\mathcal{X}}|$ is bounded by:

$I(\hat{X}; Y) \le \hat{I}(\hat{X}; Y) + \mathcal{O}\left(\frac{K |\mathcal{Y}|}{\sqrt{n}}\right)$

and

$I(X; \hat{X}) \le \hat{I}(X; \hat{X}) + \mathcal{O}\left(\frac{K}{\sqrt{n}}\right)$

where $\mathcal{Y}$ is the support of the target variable $Y$ . The effective cardinality $K$ is governed by the representation's description length, $K \approx 2^{I(X; \hat{X})}$ .

Because the generalization error bound degrades exponentially with $I(X; \hat{X})$ , the uncompressed input layer $X$ (for which $I(X; X) = H(X)$ is large) cannot ensure generalization from finite samples. Hidden layers must compress the input representation (reducing $I(X; h_i)$ ) to bring the network to an operating point where the worst-case finite-sample generalization error is controlled.
Knowl 5 — Generalization Gap and Complexity Gap Metrics for Deep Neural Networks
definition
For a deep neural network $N$ with output layer $\hat{Y}$ , trained on a finite sample of size $n$ , let $D_N = I(X; Y \mid \hat{Y})$ denote the empirical Information Bottleneck (IB) distortion and $R_N = I(X; \hat{Y})$ denote the representational complexity of the network's output. Let $(R^*(n), D_{\text{IB}}^*(n))$ be the optimal point minimizing the worst-case true generalization bound on the finite-sample empirical information curve. The network's performance is evaluated via two gaps:
1. Generalization gap ( $\Delta G$ ): $\Delta G = D_N - D_{\text{IB}}^*(n)$ which bounds the amount of relevant information about $Y$ that the network failed to capture relative to the optimal finite-sample achievable limit.
2. Complexity gap ( $\Delta C$ ): $\Delta C = R_N - R^*(n)$ which bounds the amount of unnecessary representational complexity retained by the network beyond the optimal finite-sample representation.
Knowl 6 — Linear Separability and Exact Posterior Computation in Sigmoidal Neurons
theoretical result
A single sigmoidal neuron computing $\sigma(\mathbf{w} \cdot \mathbf{h} + b)$ , with standard sigmoid $\sigma(u) = \frac{1}{1 + \exp(-u)}$ , weight vector $\mathbf{w}$ , and bias $b$ , computes the exact class posterior $p(y \mid \mathbf{x})$ for binary classes $\{y, y_0\}$ if and only if the input features are conditionally independent given the class label.

By Bayes' theorem, the binary class posterior is:

$p(y \mid \mathbf{x}) = \frac{1}{1 + \exp\left(-\log \frac{p(\mathbf{x} \mid y)}{p(\mathbf{x} \mid y_0)} - \log \frac{p(y)}{p(y_0)}\right)}$

When the class-conditional probability distribution factorizes over $N$ input features as:

$\frac{p(\mathbf{x} \mid y)}{p(\mathbf{x} \mid y_0)} = \prod_{j=1}^N \left(\frac{p(x_j \mid y)}{p(x_j \mid y_0)}\right)^{n p(x_j)}$

the posterior matches the sigmoidal activation with weights $w_j = \log \frac{p(x_j \mid y)}{p(x_j \mid y_0)}$ , bias $b = \log \frac{p(y)}{p(y_0)}$ , and inputs $h_j = n p(x_j)$ . When conditional independence does not hold due to higher-order dependencies in $p(X, Y)$ , single-layer linear separability breaks down, necessitating intermediate hidden layers to perform representational transformations that statistically decouple the features.
Knowl 7 — Correspondence Between Information Bottleneck Phase Transitions and Hidden Layer Emergence
theoretical result
Along the optimal Information Bottleneck (IB) curve, structural phase transitions (bifurcations into sub-optimal branches, changes in topological representation, cluster splits, or dimensional expansions) occur at critical values of the tradeoff parameter $\beta$ . A critical value $\beta_c$ is determined by the largest eigenvalue of the second-order correlations of the conditional distribution $p(X, Y \mid \hat{X}(\beta))$ .

Concurrently, the linear separability condition for sigmoidal hidden layers breaks down when the conditional second-order correlations of the data cannot be neglected—specifically, at the values of $\beta$ where the second-order (first non-linear) term of the conditional log-likelihood ratio becomes significant under the identical eigenvalues governing the IB bifurcations.

It is conjectured that the optimal placement of successive hidden layers in a deep neural network corresponds to values of $\beta$ immediately following bifurcation transitions along the optimal IB curve. When multiple phase transitions are linearly independent, they can be captured simultaneously within a single layer.

Coverage note — No substantial contributed material was omitted; all core theoretical formulations, optimality criteria, generalization bounds, evaluation metrics, and architectural conjectures from the paper are included.

References

1.Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798–828, Aug. 2013.
2.G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504– 507, July 2006.
3.A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems (NIPS), 2012, pp. 1106–1114.
4.P. Mehta and D. J. Schwab, “An exact mapping between the variational renormalization group and deep learning,” CoRR, vol. abs/1410.3831, 2014.
5.T. Cover and J. Thomas, Elements of information theory. Wiley New York, 1991.
6.N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” in Proceedings of the 37-th Annual Allerton Conference on Communication, Control and Computing, 1999, pp. 368–377.
7.W. H. R. Equitz and T. M. Cover, “Successive refinement of information,” IEEE Transactions on Information Theory, vol. 37, no. 2, pp. 269–275, 1991.
8.O. Shamir, S. Sabato, and N. Tishby, “Learning and generalization with the information bottleneck,” Theor. Comput. Sci., vol. 411, no. 29-30, pp. 2696–2711, 2010.
9.Y. Bengio, “Learning Deep Architectures for AI,” Foundations and Trends in Machine Learning, vol. 2, no. 1, pp. 1–127, 2009.
10.Y. LeCun and Y. Bengio, “Convolutional networks for images, speech, and time series,” The handbook of brain theory and neural networks, vol. 3361, p. 310, 1995.
11.R. Gilad-Bachrach, A. Navot, and N. Tishby, “An information theoretic tradeoff between complexity and accuracy,” in Proceedings of the COLT, 2003.
12.K. Rose, “Deterministic annealing for clustering, compression, classification, regression, and related optimization problems,” in Proceedings of the IEEE, 1998, pp. 2210–2239.
13.G. Chechik, A. Globerson, N. Tishby, and Y. Weiss, “Information bottleneck for gaussian variables,” Journal of Machine Learning Research, vol. 6, pp. 165–188, 2005.
14.K. Rose, E. Gurewitz, and G. C. Fox, “Statistical mechanics and phase transitions in clustering,” Phys. Rev. Lett., vol. 65, pp. 945–948, 1990.

Citation

MLA

Tishby, N., and N. Zaslavsky. “Deep Learning and the Information Bottleneck Principle”. arXiv, 2015, http://arxiv.org/abs/1503.02406v1.

APA

Tishby, N., & Zaslavsky, N. (2015). Deep Learning and the Information Bottleneck Principle. arXiv. http://arxiv.org/abs/1503.02406v1

Chicago

Tishby, N., and N. Zaslavsky. 2015. “Deep Learning and the Information Bottleneck Principle”. arXiv. http://arxiv.org/abs/1503.02406v1.

Harvard

Tishby, N. and Zaslavsky, N. (2015) “Deep Learning and the Information Bottleneck Principle”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1503.02406v1.

Vancouver

1. Tishby N, Zaslavsky N (2015) Deep Learning and the Information Bottleneck Principle. arXiv

BibTeX

@article{tishby2015deep,
  title = {Deep Learning and the Information Bottleneck Principle},
  author = {Tishby, Naftali and Zaslavsky, Noga},
  year = {2015},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1503.02406v1},
  eprint = {1503.02406}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Deep learning and the information bottleneck principle

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Markov Chain Structure and Data Processing Inequality for Feedforward DNNs

Knowl 2 — Layer-Wise Information Bottleneck Optimality Criterion

Knowl 3 — Information Plane Representation of Deep Neural Network Layers

Knowl 4 — Finite-Sample Generalization Bounds and the Compression Requirement for Deep Neural Networks

Knowl 5 — Generalization Gap and Complexity Gap Metrics for Deep Neural Networks

Knowl 6 — Linear Separability and Exact Posterior Computation in Sigmoidal Neurons

Knowl 7 — Correspondence Between Information Bottleneck Phase Transitions and Hidden Layer Emergence

References

Citation

BibTeX

Access the Paper