Back

How Does Information Bottleneck Help Deep Learning?

Kenji KawaguchiZhun DengXu JiJiaoyang Huang

article2023ICML143 citations

Establishes the first rigorous theoretical foundation linking the information bottleneck principle to generalization in deep neural networks by deriving novel sample complexity bounds that depend directly on intermediate layer compression rather than parameter counts.

Listen

Deep learning models are widely used across modern applications, yet understanding why they generalize well to unseen data rather than merely memorizing training data remains a fundamental challenge. A popular conceptual framework is the information bottleneck principle, which posits that optimal performance comes from eliminating task-irrelevant information while preserving predictive signals. However, existing theories lacked a rigorous mathematical proof demonstrating that controlling this representation bottleneck directly controls the generalization gap in practical end-to-end learning.

The article provides the first formal learning theory that bounds generalization errors using the information bottleneck framework for learned intermediate representations. It evaluates how representation compression works in tandem with model compression to govern overall model generalization.

The authors established their mathematical bounds using information-theoretic tools, including typical set analysis, multinomial concentration inequalities, and union bounding across parameter hypothesis spaces. They also relaxed standard assumptions to accommodate deterministic continuous networks and infinite mutual information scenarios via feature binning and noise injection. To empirically validate the theory, the authors trained hundreds of neural networks across synthetic clustering benchmarks, MNIST, Fashion MNIST, and CIFAR-10 image classification, using techniques such as stochastic weight averaging to estimate model posterior distributions.

The analysis produced three primary findings. First, the article establishes a formal sample complexity bound showing that controlling representation complexity alongside model complexity guarantees a controlled generalization error. Second, empirical evaluation confirms that representation compression alone is insufficient to predict generalization accurately; metrics combining representation compression and model parameter mutual information achieved the highest correlation with the generalization gap (e.g., reaching Pearson correlations of approximately 0.85 on CIFAR-10). Third, summarizing across network depth by taking the minimum bound over intermediate layers significantly outperformed taking averages or selecting specific boundary layers.

These findings demonstrate that effective generalization requires simplicity in both the internal representations and the mapping functions that generate them. For practitioners and decision-makers, this clarifies that solely regularizing hidden features cannot prevent parameter overfitting. Relying on joint model-and-representation compression provides a more reliable indicator of model reliability and deployment risk than traditional parameter-count metrics.

Organizations designing machine learning systems should prioritize regularization strategies that jointly constrain representation expressivity and parameter-data dependence rather than focusing solely on hidden feature bottlenecks. For model monitoring and performance auditing, evaluation pipelines should estimate joint compression metrics minimized across layers to better forecast generalization risks.

The theoretical bounds depend on standard finite-sample or discrete-domain assumptions, though the authors demonstrate that extensions via discretization or noise injection remain practically valid. While the empirical correlations strongly support the theory across vision and benchmark tasks, confidence is highest in standard supervised architectures; further validation is warranted for large-scale architectures such as foundation models and generative frameworks before applying these bounds as strict operational performance guarantees.

Columbia UniversityMilaNational University of SingaporeUniversity of Pennsylvaniauser informationmathematicsconvolutional neural networksinformation bottleneckdeep learningbottleneck regularization
arXiv: 2305.18887xu-ji/information-bottleneck
  • Paper: The information bottleneck method, Naftali Tishby et al. (2000). The original information bottleneck method defines the compression–relevance trade-off that the source uses to analyze deep-learning generalization.
  • Paper: Deep learning and the information bottleneck principle, Naftali Tishby et al. (2015). This earlier deep-learning application develops the information-bottleneck framework for neural representations, giving context for the source’s theoretical account of its generalization benefits.
  • Paper: Deep Variational Information Bottleneck, Alexander A. Alemi et al. (2017). Its variational formulation makes the information-bottleneck objective tractable in deep networks, a concrete method that the source’s learning-theoretic justification helps explain.
Cover for How Does Information Bottleneck Help Deep Learning?

Abstract

Numerous deep learning algorithms have been inspired by and understood via the notion of information bottleneck, where unnecessary information is (often implicitly) minimized while task-relevant information is maximized. However, a rigorous argument for justifying why it is desirable to control information bottlenecks has been elusive. In this paper, we provide the first rigorous learning theory for justifying the benefit of information bottleneck in deep learning by mathematically relating information bottleneck to generalization errors. Our theory proves that controlling information bottleneck is one way to control generalization errors in deep learning, although it is not the only or necessary way. We investigate the merit of our new mathematical findings with experiments across a range of architectures and learning settings. In many cases, generalization errors are shown to correlate with the degree of information bottleneck: i.e., the amount of the unnecessary information at hidden layers. This paper provides a theoretical foundation for current and future methods through the lens of information bottleneck. Our new generalization bounds scale with the degree of information bottleneck, unlike the previous bounds that scale with the number of parameters, VC dimension, Rademacher complexity, stability or robustness. Our code is publicly available at: https://github.com/xu-ji/information-bottleneck

Topics

user informationmathematicsconvolutional neural networksonline learningdeep belief networkstransfer learningoutput layersystem theory

Keywords

information bottleneckdeep learningbottleneck regularizationmutual informationrepresentation compressioninformation-theoretic boundsRademacher complexityinformation processing inequalitysample complexity bounds

Table of Contents

  • 1. Introduction
  • 2. Preliminaries
  • 3. Main Results
  • 3.1. Encoder independent with the training data
  • 3.2. Encoder learned with the training data
  • 4. Extensions
  • 4.1. Neural networks with ReLU activation functions
  • 4.2. Modification for valid bounds in the case of infinite mutual information
  • 5. Experiments
  • 5.1. On the Representation Compression Bound
  • 5.2. Image Classification with DNNs
  • 5.3. Feature binning experiments
  • 6. Proof Sketch
  • 6.1. Proof Sketch of Theorem 1
  • 6.2. Proof Sketch of Theorem 2
  • 7. Related Works
  • 8. Conclusion
  • References
  • REPRODUCIBILITY STATEMENT
  • A. Experimental details for section 5.1
  • A.1. Training
  • A.2. Metrics
  • A.3. Additional results
  • B. MNIST and Fashion MNIST
  • C. Feature binning
  • D. Experimental details for section 5.2
  • D.1. Training
  • D.2. Metrics
  • D.3. Kernel Density Estimation
  • D.4. Further results
  • E. Additional Results and Explanations for Theory
  • E.1. On Theorems 1-2
  • E.1.1. ADDITIONAL NOTATION
  • E.1.2. DETAILS OF OTHER TERMS
  • E.2. On the application to the case of infinite mutual information
  • E.3. On comparisons with previous information-theoretic bounds
  • E.4. On the standard arguments for proving the conjecture
  • F. Proofs
  • F.1. Overview of Proofs of Theorems
  • F.2. Proofs of Key Lemmas
  • F.2.2. SIZE OF THE TYPICAL SUBSET
  • F.2.5. BOUNDING THE FIRST AND SECOND TERM IN THE DECOMPOSITION
  • F.2.6. COMBINE LEMMAS
  • F.3. Completing the Proof of Theorem 1 with Key Lemmas
  • F.4. Completing the Proof of Theorem 2 with Key Lemmas
  • F.4.1. FINDING A LIKELY SPACE OF ENCODER
  • F.4.2. RESULT WITH FIXED LAYER INDEX
  • F.4.3. COMPLETING THE PROOF
  • F.5. Proof of Remark 1
  • F.6. Proof of Corollary 1
  • F.7. Proof of Proposition 1
  • F.8. Proof of Proposition 2
  • F.9. Proof of Proposition 3

Knowls

  1. Knowl 1 — Generalization bound for a learned representation

    theoretical result

    Let SS be a training dataset of nn i.i.d. pairs from a distribution PP on (X,Y)(X,Y), and let ss be its realized value. A deep network trained on ss is fs=gls∘ϕlsf^s=g_l^s\circ\phi_l^s, where ϕls\phi_l^s is the encoder through layer ll and glsg_l^s is the remaining network. Write Zls=ϕls(X)Z_l^s=\phi_l^s(X) for the representation of a fresh input, and define the generalization gap by Δ(s)=E(X,Y)∼P[ℓ(fs(X),Y)]−n−1∑i=1nℓ(fs(xi),yi)\Delta(s)=\mathbb E_{(X,Y)\sim P}[\ell(f^s(X),Y)]-n^{-1}\sum_{i=1}^n\ell(f^s(x_i),y_i). Under the paper’s finite-input and finite-encoder-family assumptions and bounded loss, for any confidence level δ>0\delta>0 and any nonempty selected layer set L⊆{1,…,D+1}\mathcal L\subseteq\{1,\ldots,D+1\}, with probability at least 1−δ1-\delta over SS, Δ(s)≤min⁡l∈LQl\Delta(s)\leq\min_{l\in\mathcal L}Q_l. For hidden or input layers l≤Dl\leq D, the bound has the form Ql=A3,l[I(X;Zls∣Y)+I(ϕlS;S)]ln⁡2+A2,ln+A1,lnQ_l=A_{3,l}\sqrt{\frac{[I(X;Z_l^s\mid Y)+I(\phi_l^S;S)]\ln 2+A_{2,l}}{n}}+\frac{A_{1,l}}{\sqrt n}. For the output layer l=D+1l=D+1, where ϕD+1S=fS\phi_{D+1}^S=f^S, it has the form QD+1=R(fs)I(ϕD+1S;S)ln⁡2+A2,D+12nQ_{D+1}=R(f^s)\sqrt{\frac{I(\phi_{D+1}^S;S)\ln 2+A_{2,D+1}}{2n}}, with R(fs)=sup⁡x,yℓ(fs(x),y)R(f^s)=\sup_{x,y}\ell(f^s(x),y). The paper’s coefficient terms satisfy A2,l,A3,l,A2,D+1=O~(1)A_{2,l},A_{3,l},A_{2,D+1}=\widetilde O(1) and A1,l=O~(I(ϕlS;S)+1)A_{1,l}=\widetilde O(\sqrt{I(\phi_l^S;S)+1}) as nn grows; the bound’s detailed coefficients also depend on quantities such as loss and representation sensitivity. The two information terms have distinct randomness: I(X;Zls∣Y)I(X;Z_l^s\mid Y) is evaluated for the realized trained encoder and fresh inputs, while I(ϕlS;S)I(\phi_l^S;S) measures the dependence of the learned encoder function on the random training dataset. The theorem also holds with the learned encoder’s parameter vector in place of the encoder function.

  2. Knowl 2 — Improved bound when the encoder is independent of training data

    theoretical result

    Let SS consist of nn i.i.d. training examples, let fs=gls∘ϕlf^s=g_l^s\circ\phi_l be the trained network, and suppose the layer-ll encoder ϕl\phi_l is fixed independently of SS (it may, for example, have been learned using separate data). For l∈{1,…,D}l\in\{1,\ldots,D\}, let Zl=ϕl(X)Z_l=\phi_l(X) for a fresh input and let Δ(s)\Delta(s) be expected loss minus training loss for a bounded per-example loss. Under the paper’s finite-domain setting, for every δ>0\delta>0, with probability at least 1−δ1-\delta over SS, Δ(s)≤G3,lI(X;Zl∣Y)ln⁡2+G2,ln+G1,l(0)n\Delta(s)\leq G_{3,l}\sqrt{\frac{I(X;Z_l\mid Y)\ln 2+G_{2,l}}{n}}+\frac{G_{1,l}(0)}{\sqrt n}. The coefficients are O~(1)\widetilde O(1) as nn grows (their detailed forms depend on the loss and encoder-related quantities). Thus, up to logarithmic factors, the bound scales as O~((I(X;Zl∣Y)+1)/n)\widetilde O(\sqrt{(I(X;Z_l\mid Y)+1)/n}): it depends linearly, rather than exponentially, on the conditional mutual information. This result applies when the representation is fixed relative to the target training sample, including the paper’s motivating fixed-feature transfer-learning setting.

  3. Knowl 3 — Conditional mutual information measures retained within-class input detail

    definition

    For an encoder whose representation obeys the Markov relation Y ⁣→ ⁣X ⁣→ ⁣ZY\!\to\!X\!\to\!Z, the paper identifies I(X;Z∣Y)I(X;Z\mid Y) as the amount of input information retained after conditioning on the target label. It satisfies I(X;Z)=I(X;Z∣Y)+I(Y;Z)I(X;Z)=I(X;Z\mid Y)+I(Y;Z). The term I(Y;Z)I(Y;Z) is label-related information, whereas I(X;Z∣Y)I(X;Z\mid Y) measures residual input detail within a class and can be zero without requiring the representation to discard all label information. Consequently, minimizing the usual I(X;Z)I(X;Z) while preserving predictive information can indirectly reduce this residual term. For a learned encoder, however, representation compression alone does not account for how strongly the encoder function was selected using the training sample; the main bound therefore pairs I(X;Zls∣Y)I(X;Z_l^s\mid Y) with I(ϕlS;S)I(\phi_l^S;S).

  4. Knowl 4 — CIFAR-10: combined representation and model compression best predicted generalization

    data/table

    The study trained 540 PreResNet models on CIFAR-10, varying architecture (PreResNet56/83/110), weight decay, batch size, five training-set draws of 15,000 examples, and four random seeds; models were trained for 200 epochs. Representation information was estimated across five layers using Gaussian noise with variance selected by maximum likelihood subject to estimated information being non-increasing through layers. Model–dataset information was estimated with SWAG, also averaging over seeds for the reported Iˉ\bar I metric. The table reports correlations with generalization gaps in loss and error; positive values indicate positive correlation. The minimum across layers of combined model and conditional representation information was the strongest overall metric among those compared. In contrast, the average conditional representation-information estimate had a much weaker Pearson correlation.

    Metric Spearman loss Spearman error Pearson loss Pearson error Kendall loss Kendall error
    min⁡lIˉ(S;θlS)+Iˇ(X;Zls∣Y)\min_l\bar I(S;\theta_l^S)+\check I(X;Z_l^s\mid Y) 0.8632 0.7576 0.8511 0.7562 0.6626 0.5664
    D−1∑lIˇ(X;Zls∣Y)D^{-1}\sum_l\check I(X;Z_l^s\mid Y) 0.8481 0.7406 0.2140 0.1853 0.6427 0.5435
    Iˉ(S;θD+1S)\bar I(S;\theta_{D+1}^S) 0.5370 0.3800 0.2924 0.1218 0.2442 0.1526

    Here θlS\theta_l^S denotes parameters through layer ll, and Iˉ\bar I and Iˇ\check I are the paper’s empirical estimators. Across the tested layer-summarization methods, taking the minimum of the combined metric outperformed taking its mean or maximum. The measured model–dataset information increased with layer depth while representation–input information decreased, revealing a trade-off between the two compression measures.

  5. Knowl 5 — Constrained 2D classification: combining the two information measures improved prediction

    data/table

    In a five-class clustered 2D classification task, the authors trained 216 five-layer MLP configurations and evaluated the penultimate-layer representation. The learned stochastic feature model was trained with an estimated representation-information constraint of 1.51.5, approximately half the unconstrained value; 36 runs with training accuracy below 85% were excluded, leaving 180 models for the reported analysis. The Pearson correlations below are between each metric and the loss generalization gap. The rescaled sum of model–dataset information and conditional representation information was the strongest listed predictor; parameter count and model–dataset information alone had little correlation.

    Metric Pearson correlation
    Number of parameters -0.0294
    Penultimate-layer parameter Frobenius norm -0.0871
    Iˇ(X;Zls)\check I(X;Z_l^s) 0.3712
    Iˇ(X;Zls∣Y)\check I(X;Z_l^s\mid Y) 0.3842
    Iˇ(S;θD+1S)\check I(S;\theta_{D+1}^S) 0.0091
    Iˇ(S;θlS)\check I(S;\theta_l^S) 0.0211
    I~(S;θlS)+Iˇ(X;Zls)\widetilde I(S;\theta_l^S)+\check I(X;Z_l^s) 0.3928
    I~(S;θlS)+Iˇ(X;Zls∣Y)\widetilde I(S;\theta_l^S)+\check I(X;Z_l^s\mid Y) 0.4130

    The results support the paper’s claim that information about the learned representation function complements information about the representation itself when assessing generalization.

  6. Knowl 6 — Finite-bin approximations extend the bounds to infinite mutual information

    theoretical result

    Let ϕ~ls\widetilde\phi_l^s be an encoder whose output may have infinite mutual information with the input, and let ElE_l map its outputs into a finite set. Define the finite-output encoder ϕls=El∘ϕ~ls\phi_l^s=E_l\circ\widetilde\phi_l^s and compare the original predictor f~s=gls∘ϕ~ls\widetilde f^s=g_l^s\circ\widetilde\phi_l^s with its binned version fs=gls∘El∘ϕ~lsf^s=g_l^s\circ E_l\circ\widetilde\phi_l^s. If their per-example losses differ by at most a finite ClC_l almost surely, then the information-bottleneck bounds apply to the original predictor using the finite mutual information of the binned representation, with an additive loss-discrepancy penalty 2Cl2C_l. Specifically, the fixed-encoder bound becomes Δf~(s)≤Q^l+2Cl\Delta_{\widetilde f}(s)\leq\widehat Q_l+2C_l, and the learned-encoder bound acquires the corresponding 2Cl2C_l penalty on its right-hand side. A finite discrepancy is available, for example, when the decoder-composed loss is Lipschitz and each bin has sufficiently small radius. Increasing bin size can lower estimated mutual information but increases the discrepancy penalty, preventing arbitrary improvement of the bound by changing the binning scheme.

  7. Knowl 7 — ReLU networks can have finite conditional information on continuous inputs

    theoretical result

    For any fixed deterministic neural network with ReLU activations, there are infinitely many continuous input distributions for which the representation ZZ has finite conditional mutual information I(X;Z∣Y)I(X;Z\mid Y). The paper’s construction uses distributions whose within-class input support has separable components and a ReLU network that maps those components to a finite set of representation values. The resulting representation has finite entropy, and therefore finite conditional mutual information. This establishes that the finite-information regime of the bounds is not restricted to discrete input domains, although it does not assert finite mutual information for every continuous distribution or every activation.

  8. Knowl 8 — MNIST and Fashion-MNIST experiments corroborated the combined-metric result

    empirical result

    The authors trained 144 stochastic-feature models across MNIST and Fashion-MNIST, using convolutional/linear architectures, varying weight decay, batch size, dataset draw, and random seed. Each training dataset contained 8,000 examples and each test set used the original 10,000-example test split; the penultimate layer parameterized a distribution over latent features. In both datasets, metrics combining model–dataset information with representation compression were reported to predict the loss generalization gap better than representation-compression metrics alone. This is corroborating empirical evidence in an unconstrained stochastic-feature setting, rather than a claim that every individual combined metric outperformed every alternative.

  9. Knowl 9 — Feature-binning experiments show a weakness of feature-only metrics under bottleneck regularization

    empirical result

    For deterministic-feature MLPs on the clustered classification task, the authors discretized each node’s activation into 10 buckets and trained 216 models, with or without an information-bottleneck regularizer. In the regularized runs, feature-only estimates had weak correlations with the loss generalization gap: I^(X;Zls)\hat I(X;Z_l^s) had Spearman/Pearson correlations −0.0403/−0.0440-0.0403/-0.0440, and I^(X;Zls∣Y)\hat I(X;Z_l^s\mid Y) had 0.0252/0.03900.0252/0.0390. A combined model- and representation-information metric, Iˇ(S;θlS)+Iˇ(X;Zls∣Y)\check I(S;\theta_l^S)+\check I(X;Z_l^s\mid Y), had correlations 0.0794/0.08150.0794/0.0815 on the same measures. The correlations were modest, but the comparison illustrates that imposing bottleneck regularization can reduce the usefulness of feature-compression metrics alone, while including model information improves their predictive signal in this experiment.

  10. Knowl 10 — The result is a sufficient route to generalization control, not a necessity claim

    limitation

    The paper proves upper bounds showing that controlling representation information together with encoder–training-data dependence can control generalization under the stated assumptions. It does not prove the converse: good generalization need not require a small information bottleneck, and the results do not establish that bottleneck control is the only or a necessary explanation for deep-learning generalization. In particular, the information of a learned representation alone is not asserted to capture all overfitting of the representation function.

Coverage note — Omitted proof-only lemmas and detailed coefficient-tail analyses used to establish the bounds, as well as supplementary estimator formulas and repetitive experimental tables; these support the stated results but do not constitute separate standalone findings.

References

  1. 1.Achille, A. and Soatto, S. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):1947–1980, 2018.
  2. 2.Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R. A., and Murphy, K. Fixing a broken elbo. In International Conference on Machine Learning, pp. 159–168. PMLR, 2018.
  3. 3.Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
  4. 4.Amjad, R. A. and Geiger, B. C. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence, 42(9):2225–2239, 2019.
  5. 5.Aziznejad, S., Gupta, H., Campos, J., and Unser, M. Deep neural networks with trainable activations and controlled lipschitz constant. IEEE Transactions on Signal Processing, 68:4688–4699, 2020.
  6. 6.Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463–482, 2002.
  7. 7.Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285–2301, 2019.
  8. 8.Bassily, R., Moran, S., Nachum, I., Shafer, J., and Yehudayoff, A. Learners that use little information. In Algorithmic Learning Theory, pp. 25–55. PMLR, 2018.
  9. 9.Bertsekas, D. P. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
  10. 10.Bousquet, O. and Elisseeff, A. Stability and generalization. Journal of Machine Learning Research, 2(Mar):499–526, 2002.
  11. 11.Burhanpurkar, M., Deng, Z., Dwork, C., and Zhang, L. Scaffolding sets. arXiv preprint arXiv:2111.03135, 2021.
  12. 12.Chelombiev, I., Houghton, C., and O’Donnell, C. Adaptive estimators show information compression in deep neural networks. In International Conference on Learning Representations, 2019.
  13. 13.Deng, Z., He, H., and Su, W. Toward better generalization bounds with locally elastic stability. In International Conference on Machine Learning, pp. 2590–2600. PMLR, 2021.
  14. 14.Dziugaite, G. K., Hsu, K., Gharbieh, W., Arpino, G., and Roy, D. On the role of data in pac-bayes bounds. In International Conference on Artificial Intelligence and Statistics, pp. 604–612. PMLR, 2021.
  15. 15.Eysenbach, B., Salakhutdinov, R. R., and Levine, S. Robust predictable control. Advances in Neural Information Processing Systems, 34:27813–27825, 2021.
  16. 16.Fazlyab, M., Robey, A., Hassani, H., Morari, M., and Pappas, G. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  17. 17.Federici, M., Dutta, A., Forre, P., Kushman, N., and Akata, Z. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2020.
  18. 18.Fischer, I. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
  19. 19.Galloway, A., Golubeva, A., Salem, M., Nica, M., Ioannou, Y., and Taylor, G. W. Bounding generalization error with input compression: An empirical study with infinite-width networks. arXiv preprint arXiv:2207.09408, 2022.
  20. 20.Goldfeld, Z., Van Den Berg, E., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., and Polyanskiy, Y. Estimating information flow in deep neural networks. In International Conference on Machine Learning, pp. 2299–2308. PMLR, 2019.
  21. 21.Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. Proc. A, Royal Soc., arXiv:2011.15091, 2022.
  22. 22.Hafez-Kolahi, H., Kasaei, S., and Soleymani-Baghshah, M. Sample complexity of classification with compressed input. Neurocomputing, 415:286–294, 2020.
  23. 23.He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In European Conference on Computer Vision, pp. 630–645. Springer, 2016.
  24. 24.Hellstrom, F. and Durisi, G. Generalization bounds via information density and conditional information density. IEEE Journal on Selected Areas in Information Theory, 1(3):824–839, 2020.
  25. 25.Hromkovic, J. Randomized algorithms. In Algorithmics for Hard Problems, pp. 341–429. Springer, 2004.
  26. 26.Hu, Z., Jagtap, A. D., Karniadakis, G. E., and Kawaguchi, K. When do extended physics-informed neural networks (xpinns) improve generalization? SIAM Journal on Scientific Computing, 44(5):A3158–A3182, 2022.
  27. 27.Ji, W., Deng, Z., Nakada, R., Zou, J., and Zhang, L. The power of contrast for feature learning: A theoretical analysis. arXiv, 2110.02473, 2021a.
  28. 28.Ji, W., Lu, Y., Zhang, Y., Deng, Z., and Su, W. J. An unconstrained layer-peeled perspective on neural collapse. arXiv, 2110.02796, 2021b.
  29. 29.Kawaguchi, K., Deng, Z., Luh, K., and Huang, J. Robustness implies generalization via data-dependent generalization bounds. In International Conference on Machine Learning (ICML), pp. 10866–10894. PMLR, 2022a.
  30. 30.Kawaguchi, K., Kaelbling, L., and Bengio, Y. Generalization in Deep Learning. Cambridge University Press, 2022b. URL https://arxiv.org/abs/1710.05468.
  31. 31.Kingma, D. P., Salimans, T., and Welling, M. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015.
  32. 32.Kirsch, A., Lyle, C., and Gal, Y. Unpacking information bottlenecks: Surrogate objectives for deep learning. 2020.
  33. 33.Kolchinsky, A. and Tracey, B. D. Estimating mixture entropy with pairwise distances. Entropy, 19(7):361, 2017.
  34. 34.Latorre, F., Rolland, P., and Cevher, V. Lipschitz constant estimation of neural networks via sparse polynomial optimization. In International Conference on Learning Representations, 2019.
  35. 35.Lee, K.-H., Arnab, A., Guadarrama, S., Canny, J., and Fischer, I. Compressive visual representations. Advances in Neural Information Processing Systems, 34:19538–19552, 2021.
  36. 36.Levine, A. and Feizi, S. Robustness certificates for sparse adversarial attacks by randomized ablation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4585–4593, 2020.
  37. 37.Li, B., Shen, Y., Wang, Y., Zhu, W., Li, D., Keutzer, K., and Zhao, H. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7399–7407, 2022a.
  38. 38.Li, Q., Wu, Z., Kong, L., and Bi, W. Explanation regeneration via information bottleneck. arXiv preprint arXiv:2212.09603, 2022b.
  39. 39.Liu, D., Lamb, A., Kawaguchi, K., Goyal, A., Sun, C., Mozer, M. C., and Bengio, Y. Discrete-valued neural communication. In Advances in Neural Information Processing Systems, 2021.
  40. 40.Liu, D., Lamb, A., Ji, X., Notsawo, P., Mozer, M., Bengio, Y., and Kawaguchi, K. Adaptive discrete communication bottlenecks with dynamic vector quantization. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
  41. 41.Lotfi, S., Finzi, M., Kapoor, S., Potapczynski, A., Goldblum, M., and Wilson, A. G. Pac-bayes compression bounds so tight that they can explain generalization. Advances in Neural Information Processing Systems, 35:31459–31473, 2022.
  42. 42.Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
  43. 43.Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference. arXiv preprint arXiv:1704.04289, 2017.
  44. 44.Nagarajan, V. and Kolter, J. Z. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
  45. 45.Papyan, V., Han, X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
  46. 46.Pauli, P., Koch, A., Berberich, J., Kohler, P., and Allgower, F. Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters, 6:121–126, 2021.
  47. 47.Pinot, R., Meunier, L., Araujo, A., Kashima, H., Yger, F., Gouy-Pailler, C., and Atif, J. Theoretical evidence for adversarial robustness through randomization. Advances in Neural Information Processing Systems, 32, 2019.
  48. 48.Pinot, R., Ettedgui, R., Rizk, G., Chevaleyre, Y., and Atif, J. Randomization matters how to defend against strong adversarial attacks. In International Conference on Machine Learning, pp. 7717–7727. PMLR, 2020.
  49. 49.Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., and Cox, D. D. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
  50. 50.Shalev-Shwartz, S. and Ben-David, S. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
  51. 51.Shamir, O., Sabato, S., and Tishby, N. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411(29-30):2696–2711, 2010.
  52. 52.Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.
  53. 53.Shwartz-Ziv, R., Painsky, A., and Tishby, N. Representation compression and generalization in deep neural networks, 2019. URL https://openreview.net/forum?id=SkeL6sCqK7.
  54. 54.Slonim, N. and Tishby, N. Document clustering using word clusters via the information bottleneck method. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 208–215, 2000.
  55. 55.Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
  56. 56.Steinke, T. and Zakynthinou, L. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory, pp. 3437–3452. PMLR, 2020.
  57. 57.Su, T., Song, C., and Cheng, J. Vision transformer with information bottleneck for fine-grained visual classification. In Advances in Guidance, Navigation and Control: Proceedings of 2022 International Conference on Guidance, Navigation and Control, pp. 4010–4019. Springer, 2023.
  58. 58.Sun, Q., Li, J., Peng, H., Wu, J., Fu, X., Ji, C., and Philip, S. Y. Graph structure learning with variational information bottleneck. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4165–4174, 2022.
  59. 59.Tishby, N. and Zaslavsky, N. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1–5. IEEE, 2015.
  60. 60.Tishby, N., Pereira, F. C., and Bialek, W. The information bottleneck method. In Proc. 37th Annual Allerton Conference on Communications, Control and Computing, 1999, pp. 368–377, 1999.
  61. 61.Trauble, F., Goyal, A., Rahaman, N., Mozer, M., Kawaguchi, K., Bengio, Y., and Scholkopf, B. Discrete key-value bottleneck. In International Conference on Machine Learning (ICML), 2023.
  62. 62.Truong, L. V. On rademacher complexity-based generalization bounds for deep learning. arXiv preprint arXiv:2208.04284, 2022.
  63. 63.Vapnik, V. The nature of statistical learning theory. Springer science & business media, 1999.
  64. 64.Vera, M., Piantanida, P., and Vega, L. R. The role of the information bottleneck in representation learning. In 2018 IEEE International Symposium on Information Theory (ISIT), pp. 1580–1584. IEEE, 2018.
  65. 65.Xie, C., Wang, J., Zhang, Z., Ren, Z., and Yuille, A. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.
  66. 66.Xu, A. and Raginsky, M. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, 30, 2017.
  67. 67.Xu, H. and Mannor, S. Robustness and generalization. Machine learning, 86(3):391–423, 2012.
  68. 68.Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021a.
  69. 69.Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., and Zou, J. How does mixup help with robustness and generalization? In International Conference on Learning Representations (ICLR), 2021b.

Citation

MLA
Kawaguchi, K., et al. “How Does Information Bottleneck Help Deep Learning?”. International Conference on Machine Learning, vol. 202, 2023, pp. 16049–96, https://proceedings.mlr.press/v202/kawaguchi23a.html.
APA
Kawaguchi, K., Deng, Z., Ji, X., & Huang, J. (2023). How Does Information Bottleneck Help Deep Learning?. International Conference on Machine Learning, 202, 16049–16096. https://proceedings.mlr.press/v202/kawaguchi23a.html
Chicago
Kawaguchi, K., Z. Deng, X. Ji, and J. Huang. 2023. “How Does Information Bottleneck Help Deep Learning?”. International Conference on Machine Learning 202: 16049–96. https://proceedings.mlr.press/v202/kawaguchi23a.html.
Harvard
Kawaguchi, K. et al. (2023) “How Does Information Bottleneck Help Deep Learning?”, International Conference on Machine Learning. PMLR, pp. 16049–16096. Available at: https://proceedings.mlr.press/v202/kawaguchi23a.html.
Vancouver
1. Kawaguchi K, Deng Z, Ji X, Huang J (2023) How Does Information Bottleneck Help Deep Learning?. In: International Conference on Machine Learning. PMLR, pp 16049–16096

BibTeX

@InProceedings{pmlr-v202-kawaguchi23a,
  title = 	 {How Does Information Bottleneck Help Deep Learning?},
  author =       {Kawaguchi, Kenji and Deng, Zhun and Ji, Xu and Huang, Jiaoyang},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {16049--16096},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/kawaguchi23a/kawaguchi23a.pdf},
  url = 	 {https://proceedings.mlr.press/v202/kawaguchi23a.html},
  abstract = 	 {Numerous deep learning algorithms have been inspired by and understood via the notion of information bottleneck, where unnecessary information is (often implicitly) minimized while task-relevant information is maximized. However, a rigorous argument for justifying why it is desirable to control information bottlenecks has been elusive. In this paper, we provide the first rigorous learning theory for justifying the benefit of information bottleneck in deep learning by mathematically relating information bottleneck to generalization errors. Our theory proves that controlling information bottleneck is one way to control generalization errors in deep learning, although it is not the only or necessary way. We investigate the merit of our new mathematical findings with experiments across a range of architectures and learning settings. In many cases, generalization errors are shown to correlate with the degree of information bottleneck: i.e., the amount of the unnecessary information at hidden layers. This paper provides a theoretical foundation for current and future methods through the lens of information bottleneck. Our new generalization bounds scale with the degree of information bottleneck, unlike the previous bounds that scale with the number of parameters, VC dimension, Rademacher complexity, stability or robustness. Our code is publicly available at: https://github.com/xu-ji/information-bottleneck}
}
Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/