Back

Why Does Unsupervised Pre-training Help Deep Learning?

D. ErhanAaron C. CourvilleYoshua BengioPascal Vincent

article2010AISTATS2,516 citations

Demonstrates through extensive empirical analysis and functional trajectory visualization that unsupervised pre-training acts primarily as a data-dependent regularizer, initializing network parameters inside basins of attraction that yield substantially better generalization.

Listen

The article addresses the challenge of training deep neural network architectures, which are powerful for complex tasks like vision and language processing but suffer from non-convex optimization landscapes that often lead to poor generalization when initialized randomly. Standard supervised training frequently settles into suboptimal regions of parameter space. The work investigates why adding an unsupervised pre-training phase before supervised fine-tuning substantially improves results, a technique that had shown strong empirical success but lacked clear mechanistic understanding.

The authors set out to evaluate competing explanations—such as pre-training acting as a preconditioner for better optimization, an aid to reaching lower training error, or a form of regularization—through controlled experiments on MNIST and related large-scale datasets. They compared deep belief networks and stacked denoising auto-encoders against ordinary networks, varying network depth, layer size, training data volume, and initialization, while tracking generalization error, training trajectories, and learned features via visualizations and dimensionality reduction.

The experiments reveal that unsupervised pre-training consistently yields lower test error, with the benefit growing for deeper networks; it guides parameters into basins of attraction that support better generalization even when training error is not lower. Pre-training functions as a data-dependent regularizer by restricting the starting point in parameter space to regions aligned with input structure, an effect that persists with very large datasets unlike classical weight penalties. Early training examples exert disproportionate influence due to increasing non-linearity during stochastic gradient descent, and pre-training reduces variance across random seeds.

These findings indicate that the advantage stems from initialization dynamics in non-convex landscapes rather than simple optimization improvements or marginal weight scaling. The approach helps deep models avoid poor apparent local minima and maintains gains as data scales, with implications for robustness in high-capacity settings. Limitations include reliance on specific vision datasets and the need for further tests on diverse domains.

Next steps include developing training methods that lessen dependence on early examples for large-scale non-convex optimization and exploring hybrid semi-supervised techniques that integrate unsupervised signals more continuously.

GoogleUniversité de Montréaldeep belief networksdimensionality reductionencodersUnsupervised Pre‑TrainingDeep ArchitectureRegularization
Erhan et al (2010).pdf
Cover for Why Does Unsupervised Pre-training Help Deep Learning?

Abstract

Much recent research has been devoted to learning algorithms for deep architectures such as Deep Belief Networks and stacks of auto-encoder variants, with impressive results obtained in several areas, mostly on vision and language data sets. The best results obtained on supervised learning tasks involve an unsupervised learning component, usually in an unsupervised pre-training phase. Even though these new algorithms have enabled training deep models, many questions remain as to the nature of this difficult learning problem. The main question investigated here is the following: how does unsupervised pre-training work? Answering this questions is important if learning in deep architectures is to be further improved. We propose several explanatory hypotheses and test them through extensive simulations. We empirically show the influence of pre-training with respect to architecture depth, model capacity, and number of training examples. The experiments confirm and clarify the advantage of unsupervised pre-training. The results suggest that unsupervised pre-training guides the learning towards basins of attraction of minima that support better generalization from the training data set; the evidence from these results supports a regularization explanation for the effect of pre-training.

Topics

deep belief networksdimensionality reductionencodershigh capacitymarginalsnetwork architectureoptimizationpreconditionersregularizationstochastic

Keywords

Unsupervised Pre‑TrainingDeep ArchitectureRegularizationDeep Belief NetworkStacked Denoising Auto‑EncoderStochastic Gradient DescentBasin Of AttractionOnline LearningLayer‑Wise TrainingGeneralization Error

Table of Contents

  • 1. Introduction
  • 2. The Challenges of Deep Learning
  • 3. Unsupervised Pre-training Acts as a Regularizer
  • 4. Previous Relevant Work
  • 4.1 Related Semi-Supervised Methods
  • 4.2 Early Stopping as a Form of Regularization
  • 5. Experimental Setup and Methodology
  • 5.1 Models
  • 5.1.1 DEEP BELIEF NETWORKS
  • 5.1.2 STACKED DENOISING AUTO-ENCODERS
  • 5.2 Data Sets
  • 5.3 Setup
  • 6. The Effect of Unsupervised Pre-training
  • 6.1 Better Generalization
  • 6.2 Visualization of Features
  • 6.3 Visualization of Model Trajectories During Learning
  • 6.4 Implications
  • 7. The Role of Unsupervised Pre-training
  • 7.1 Experiment 1: Does Pre-training Provide a Better Conditioning Process for Supervised Learning?
  • 7.2 Experiment 2: The Effect of Pre-training on Training Error
  • 7.3 Experiment 3: The Influence of the Layer Size
  • 7.4 Experiment 4: Challenging the Optimization Hypothesis
  • 7.5 Experiment 5: Comparing pre-training to L1L_1 and L2L_2 regularization
  • 7.6 Summary of Findings: Experiments 1-5
  • 8. The Online Learning Setting
  • 8.1 Experiment 6: Effect of Pre-training with Very Large Data Sets
  • 8.2 Experiment 7: The Effect of Example Ordering
  • 8.3 Experiment 8: Pre-training only kk layers
  • 9. Discussion and Conclusions
  • Acknowledgments
  • References

Knowls

  1. Knowl 1 — Unsupervised Pre-training as an Implicit Prior over Optimization Basins

    theoretical result

    Unsupervised pre-training can be formalized as inducing an implicit prior distribution over parameter space that restricts subsequent supervised fine-tuning to specific basins of attraction. Let parameter space be a bounded region S⊂RdS \subset \mathbb{R}^d, partitioned into disjoint basins of attraction {Rk}\{R_k\} of gradient descent procedures on the empirical training error such that ⋃kRk=S\bigcup_k R_k = S and Ri∩Rj=∅R_i \cap R_j = \emptyset for i≠ji \neq j. Let vk=∫θ∈Rkdθv_k = \int_{\theta \in R_k} d\theta be the volume of basin RkR_k.

    Under a factorized uniform random initialization procedure, let rkr_k be the probability of landing in basin RkR_k. Under unsupervised pre-training initialized randomly, let πk\pi_k be the probability of landing in basin RkR_k, where ∑krk=∑kπk=1\sum_k r_k = \sum_k \pi_k = 1. The initialization procedure corresponds to a regularizer −log⁡P(θ)-\log P(\theta), where the implicit prior for pre-trained models is:

    Ppre-training(θ)=∑k1θ∈RkπkvkP_{\text{pre-training}}(\theta) = \sum_k \mathbf{1}_{\theta \in R_k} \frac{\pi_k}{v_k}

    and for models trained without unsupervised pre-training:

    Pno-pre-training(θ)=∑k1θ∈RkrkvkP_{\text{no-pre-training}}(\theta) = \sum_k \mathbf{1}_{\theta \in R_k} \frac{r_k}{v_k}

    When πk\pi_k is near zero for a basin RkR_k, the implicit regularizer penalty −log⁡P(θ)-\log P(\theta) is high. Unsupervised pre-training acts as a data-dependent regularizer by placing the parameter initialization within basins of attraction that capture dominant statistical variations of the input distribution P(X)P(X), keeping supervised gradient descent confined within those basins.

  2. Knowl 2 — Marginal Weight Pre-conditioning vs Joint Unsupervised Pre-training

    data/table

    To evaluate whether the advantage of unsupervised pre-training is merely due to better scaling or marginal distributions of initial weights (the pre-conditioning hypothesis), multi-layer feed-forward networks were evaluated on the MNIST classification benchmark under three distinct initialization schemes:

    1. Uniform random initialization from uniform[−1/k,1/k]\text{uniform}[-1/\sqrt{k}, 1/\sqrt{k}], where kk is the fan-in connection count of a unit.
    2. Independent sampling from empirical marginal histograms of the weights and biases obtained after unsupervised pre-training (matching marginal distributions per layer while breaking joint inter-parameter dependencies).
    3. Full unsupervised pre-training with stacked denoising auto-encoders (preserving joint parameter dependencies).
    Architecture Uniform Initialization Marginal Histogram Initialization Unsupervised Pre-training
    1 hidden layer 1.81±0.07%1.81 \pm 0.07\% 1.94±0.09%1.94 \pm 0.09\% 1.41±0.07%1.41 \pm 0.07\%
    2 hidden layers 1.77±0.10%1.77 \pm 0.10\% 1.69±0.11%1.69 \pm 0.11\% 1.37±0.09%1.37 \pm 0.09\%

    The entries report the mean and standard deviation of test classification error percentages across multiple random initialization seeds. Matching only the independent marginal weight distributions of pre-trained networks fails to replicate the performance gains of unsupervised pre-training. This demonstrates that pre-training does not act simply as a marginal pre-conditioner; rather, the coordinated joint parameter dependencies learned during unsupervised pre-training are essential.

  3. Knowl 3 — Regularization Signature of Pre-training in Training vs Test Negative Log-Likelihood

    empirical result

    Tracking the trajectory of training negative log-likelihood (NLL) against test NLL during supervised gradient descent on MNIST reveals a fundamental signature of regularization. For networks with 2 and 3 hidden layers, unsupervised pre-trained models (using stacked denoising auto-encoders or deep belief networks) do not achieve lower training NLL than randomly initialized networks; in fact, randomly initialized networks often reach equal or lower training error. However, at every level of training cost, pre-trained networks systematically achieve a lower test cost and exhibit significantly reduced overfitting as training converges. This empirical divergence—higher or equal training error paired with lower test error—demonstrates that the primary effect of unsupervised pre-training in finite datasets is data-dependent regularization rather than improved empirical cost optimization.

  4. Knowl 4 — Interaction Between Layer Capacity and Unsupervised Pre-training

    empirical result

    The effect of unsupervised pre-training depends strongly on hidden layer width. When evaluating deep belief networks and stacked denoising auto-encoders on MNIST and Shapeset across layer sizes ranging from 25 to 800 hidden units per layer:

    • For small layer sizes (e.g., 25 and 50 units per layer), unsupervised pre-training yields higher test classification error than training from standard random initialization.
    • For large layer sizes (e.g., 400 and 800 units per layer), unsupervised pre-training consistently outperforms random initialization across 1, 2, and 3 hidden layers.

    This occurs because unsupervised pre-training acts as an additional regularizer that biases the representations toward capturing the dominant factors of variation in the input distribution P(X)P(X). When layer capacity is constrained, extracting dominant unsupervised variations can displace features necessary for predicting the target class label YY, whereas wide layers possess sufficient capacity to retain both unsupervised structure and label-relevant features.

  5. Knowl 5 — Depth Degradation and Initialization Variance Mitigation by Pre-training

    empirical result

    Evaluating feed-forward neural networks across 400 random initialization seeds on the MNIST benchmark reveals two distinct effects of network depth (from 1 to 5 hidden layers):

    • Without unsupervised pre-training, classification test error decreases from 1 to 2 hidden layers, but degrades sharply for 3 and 4 hidden layers, while 5-layer networks fail to converge to acceptable solutions. Concurrently, the variance of the test error across different initialization seeds increases drastically with depth, reflecting an increased probability of gradient descent getting trapped in poor apparent local minima.
    • With unsupervised pre-training (using stacked denoising auto-encoders or deep belief networks), test error decreases steadily from 1 to 4 hidden layers, and the variance across initialization seeds remains uniformly small across all depths (with test errors concentrated tightly between 1.2%1.2\% and 1.6%1.6\%).

    Unsupervised pre-training thus stabilizes optimization across random seeds and enables effective supervised learning in deeper architectures.

  6. Knowl 6 — Function Space Separation and Trajectory Divergence into Local Minima

    empirical result

    Visualizing learning trajectories in function space—represented by concatenating a model's class probability predictions across all test examples at each training iteration and projecting them to two dimensions using t-SNE and ISOMAP—demonstrates that:

    1. Models trained with unsupervised pre-training and models trained from random initialization occupy completely disjoint regions of function space throughout supervised training.
    2. For both pre-trained and un-pretrained models, trajectories corresponding to different random seeds initially move together during early epochs, but diverge after approximately 7 epochs into distinct, non-converging trajectories corresponding to different apparent local minima.
    3. The functional variance across seeds is substantially smaller for pre-trained models than for randomly initialized models, with pre-trained models converging to a compact cluster of functions with higher mutual similarity.

    This indicates that unsupervised pre-training restricts the optimization path to a small, localized sub-region of function space associated with superior generalization.

  7. Knowl 7 — Disproportionate Influence of Early Training Samples on Optimization Basins

    empirical result

    In online stochastic gradient descent on 10 million examples from the InfiniteMNIST dataset, the sensitivity of the final model to training data at different stages was measured by resampling subsets of 1 million consecutive examples across 10 random draws while keeping all other examples fixed.

    The variance of the final model's predictions on a test set is highest when resampling the earliest training examples (the first 10% to 20% of the stream) and decreases substantially for subsequent intermediate blocks. Furthermore, models initialized with unsupervised pre-training exhibit significantly lower functional output variance with respect to perturbations in early training data than models trained purely with supervised stochastic gradient descent. This confirms that early stochastic gradient descent steps disproportionately determine the basin of attraction in parameter space from which the network cannot escape, and unsupervised pre-training acts as a variance reduction mechanism for this trajectory selection.

  8. Knowl 8 — Persistence of Pre-training Advantage in Large-Scale Online Learning

    empirical result

    In large-scale online learning on the InfiniteMNIST benchmark (where models receive 10 million non-repeating stochastically deformed digit samples), the performance advantage of unsupervised pre-training does not vanish as sample size grows:

    • For 3-layer networks, both Stacked Denoising Auto-Encoders (SDAE) and Deep Belief Networks (DBN) maintain a substantial generalization advantage over 3-layer networks trained without pre-training throughout the entire stream of 10 million examples.
    • For 1-layer networks, pre-trained networks achieve lower online classification error and lower error across all seen training examples compared to non-pre-trained networks.
    • Classical L1L_1 and L2L_2 regularizers provide zero optimal weight penalty in this asymptotic data regime, whereas unsupervised pre-training remains beneficial.

    Because stochastic gradient descent on an infinite stream optimizes the true expected loss, this persistent advantage indicates that pre-training serves an essential optimization role in guiding non-convex gradient dynamics away from suboptimal basins of attraction.

  9. Knowl 9 — Effect of Progressively Pre-training Bottom Layers

    empirical result

    When varying the number of bottom layers k∈{0,1,2,3}k \in \{0, 1, 2, 3\} initialized with unsupervised pre-training in a 3-layer network on MNIST (with the top 3−k3-k layers randomly initialized):

    • Test classification error decreases monotonically as more bottom layers are pre-trained (k=3k=3 achieves the lowest test error, followed by k=2k=2, k=1k=1, and k=0k=0).
    • Final training negative log-likelihood on finite MNIST becomes progressively higher (worse) as more layers are pre-trained (k=3k=3 reaches the highest training error, while k=0k=0 achieves the lowest training error).

    On the InfiniteMNIST dataset, pre-training the bottom two or all three layers yields substantial online error improvements over pre-training only the first layer or none. The combination of worsening training error and improving test error on finite datasets confirms that layer-wise unsupervised initialization acts progressively as a regularizer across the depth of the network.

  10. Knowl 10 — Unsupervised Pre-training in Top-Layer Bottleneck Architectures Without Early Stopping

    empirical result

    When a 3-layer neural network on MNIST is constrained with a narrow 20-unit top hidden layer to prevent the output classifier from easily memorizing the training data, and trained using stochastic gradient descent to complete convergence without early stopping:

    • The unsupervised pre-trained network achieves a higher final training negative log-likelihood (NLL) than the network trained from random initialization without pre-training.
    • The pre-trained network achieves a lower test NLL and better test classification error than the randomly initialized network.

    This contrasts with earlier studies that used early stopping and found lower training error for pre-trained networks. Running optimization to convergence shows that early stopping acted as a confounder, and the underlying effect of unsupervised pre-training in this constrained setting remains consistent with regularization rather than improved training cost optimization.

Coverage note — Qualitative activation-maximization feature visualization images were omitted as standalone knowls because their core conceptual finding—that pre-training fixes early-layer feature representations and prevents fine-tuning from leaving the initial basin—is subsumed by the trajectory and optimization knowls.

References

  1. 1.Shun-ichi Amari, Noboru Murata, Klaus-Robert Müller, Michael Finke, and Howard Hua Yang. Asymptotic statistical theory of overtraining and cross-validation. IEEE Transactions on Neural Networks, 8(5):985–996, 1997.
  2. 2.Lalit Bahl, Peter Brown, Peter deSouza, and Robert Mercer. Maximum mutual information estimation of hidden markov parameters for speech recognition. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 49–52, Tokyo, Japan, 1986.
  3. 3.Andrew E. Barron. Complexity regularization with application to artificial neural networks. In G. Roussas, editor, Nonparametric Functional Estimation and Related Topics, pages 561–576. Kluwer Academic Publishers, 1991.
  4. 4.Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In T.G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14 (NIPS’01), Cambridge, MA, 2002. MIT Press.
  5. 5.Yoshua Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1–127, 2009. Also published as a book. Now Publishers, 2009.
  6. 6.Yoshua Bengio and Olivier Delalleau. Justifying and generalizing contrastive divergence. Neural Computation, 21(6):1601–1621, June 2009.
  7. 7.Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. DeCoste, and J. Weston, editors, Large Scale Kernel Machines, pages 321–360. MIT Press, 2007.
  8. 8.Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of highly variable functions for local kernel machines. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems 18 (NIPS’05), pages 107–114. MIT Press, Cambridge, MA, 2006.
  9. 9.Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Bernhard Schölkopf, John Platt, and Thomas Hoffman, editors, Advances in Neural Information Processing Systems 19 (NIPS’06), pages 153–160. MIT Press, 2007.
  10. 10.Marc H. Bornstein. Sensitive periods in development : interdisciplinary perspectives / edited by Marc H. Bornstein. Lawrence Erlbaum Associates, Hillsdale, N.J. :, 1987.
  11. 11.Olivier Chapelle, Jason Weston, and Bernhard Schölkopf. Cluster kernels for semi-supervised learning. In S. Becker, S. Thrun, and K. Obermayer, editors, Advances in Neural Information Processing Systems 15 (NIPS’02), pages 585–592, Cambridge, MA, 2003. MIT Press.
  12. 12.Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. MIT Press, 2006.
  13. 13.Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML’08), pages 160–167. ACM, 2008.
  14. 14.Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report 1341, Université de Montréal, 2009.
  15. 15.Patrick Gallinari, Yann LeCun, Sylvie Thiria, and Francoise Fogelman-Soulie. Memoires associatives distribuees. In Proceedings of COGNITIVA 87, Paris, La Villette, 1987.
  16. 16.Ian Goodfellow, Quoc Le, Andrew Saxe, and Andrew Ng. Measuring invariances in deep networks. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 646–654. 2009.
  17. 17.Raia Hadsell, Ayse Erkan, Pierre Sermanet, Marco Scoffier, Urs Muller, and Yann LeCun. Deep belief net learning in a long-range vision system for autonomous off-road driving. In Proc. Intelligent Robots and Systems (IROS’08), pages 628–633, 2008.
  18. 18.Johan Håstad. Almost optimal lower bounds for small depth circuits. In Proceedings of the 18th annual ACM Symposium on Theory of Computing, pages 6–20, Berkeley, California, 1986. ACM Press.
  19. 19.Johan Håstad and Mikael Goldmann. On the power of small-depth threshold circuits. Computational Complexity, 1:113–129, 1991.
  20. 20.Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14:1771–1800, 2002.
  21. 21.Geoffrey E. Hinton. To recognize shapes, first learn to generate images. In Paul Cisek, Trevor Drew, and John Kalaska, editors, Computational Neuroscience: Theoretical Insights into Brain Function. Elsevier, 2007.
  22. 22.Geoffrey E. Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, July 2006.
  23. 23.Goeffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006.
  24. 24.Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted Boltzmann machines. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML’08), pages 536–543. ACM, 2008.
  25. 25.Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Int. Conf. Mach. Learn., pages 473–480, 2007.
  26. 26.Hugo Larochelle, Yoshua Bengio, Jerome Louradour, and Pascal Lamblin. Exploring strategies for training deep neural networks. The Journal of Machine Learning Research, 10:1–40, January 2009.
  27. 27.Julia A. Lasserre, Christopher M. Bishop, and Thomas P. Minka. Principled hybrids of generative and discriminative models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR’06), pages 87–94, Washington, DC, USA, 2006. IEEE Computer Society.
  28. 28.Yann LeCun. Modèles connexionistes de l’apprentissage. PhD thesis, Université de Paris VI, 1987.
  29. 29.Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  30. 30.Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area V2. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20 (NIPS’07), pages 873–880. MIT Press, Cambridge, MA, 2008.
  31. 31.Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Léon Bottou and Michael Littman, editors, Proceedings of the Twenty-sixth International Conference on Machine Learning (ICML’09). ACM, Montreal (Qc), Canada, 2009.
  32. 32.Gaëlle Loosli, Stéphane Canu, and Léon Bottou. Training invariant support vector machines using selective sampling. In Léon Bottou, Olivier Chapelle, Dennis DeCoste, and Jason Weston, editors, Large Scale Kernel Machines, pages 301–320. MIT Press, Cambridge, MA., 2007.
  33. 33.Hossein Mobahi, Ronan Collobert, and Jason Weston. Deep learning from temporal coherence in video. In Léon Bottou and Michael Littman, editors, Proceedings of the 26th International Conference on Machine Learning, pages 737–744, Montreal, June 2009. Omnipress.
  34. 34.Andrew Y. Ng and Michael I. Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In T.G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14 (NIPS’01), pages 841–848, 2002.
  35. 35.Simon Osindero and Geoffrey E. Hinton. Modeling image patches with a directed hierarchy of markov random field. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20 (NIPS’07), pages 1121–1128, Cambridge, MA, 2008. MIT Press.
  36. 36.Dan Povey and Philip C. Woodland. Minimum phone error and i-smoothing for improved discriminative training. In Acoustics, Speech, and Signal Processing, 2002. Proceedings. (ICASSP ’02). IEEE International Conference on, volume 1, pages I–105–I–108 vol.1, 2002.
  37. 37.Marc’Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann LeCun. Efficient learning of sparse representations with an energy-based model. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19 (NIPS’06), pages 1137–1144. MIT Press, 2007.
  38. 38.Marc’Aurelio Ranzato, Y-Lan Boureau, and Yann LeCun. Sparse feature learning for deep belief networks. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20 (NIPS’07), pages 1185–1192, Cambridge, MA, 2008. MIT Press.
  39. 39.Ruslan Salakhutdinov and Geoffrey E. Hinton. Using deep belief nets to learn covariance kernels for Gaussian processes. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20 (NIPS’07), pages 1249–1256, Cambridge, MA, 2008. MIT Press.
  40. 40.Ruslan Salakhutdinov and Geoffrey E. Hinton. Semantic hashing. In Proceedings of the 2007 Workshop on Information Retrieval and applications of Graphical Models (SIGIR 2007), Amsterdam, 2007. Elsevier.
  41. 41.Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey E. Hinton. Restricted Boltzmann machines for collaborative filtering. In Zoubin Ghahramani, editor, Proceedings of the Twenty-fourth International Conference on Machine Learning (ICML’07), pages 791–798, New York, NY, USA, 2007. ACM.
  42. 42.Sebastian H. Seung. Learning continuous attractors in recurrent networks. In M.I. Jordan, M.J. Kearns, and S.A. Solla, editors, Advances in Neural Information Processing Systems 10 (NIPS’97), pages 654–660. MIT Press, 1998.
  43. 43.Jonas Sjöberg and Lennart Ljung. Overtraining, regularization and searching for a minimum, with application to neural networks. International Journal of Control, 62(6):1391–1407, 1995.
  44. 44.Joshua M. Susskind, Geoffrey E., Javier R. Movellan, and Adam K. Anderson. Generating facial expressions with deep belief nets. In V. Kordic, editor, Affective Computing, Emotion Modelling, Synthesis and Recognition, pages 421–440. ARS Publishers, 2008.
  45. 45.Joshua Tenenbaum, Vin de Silva, and John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319–2323, December 2000.
  46. 46.Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579–2605, November 2008.
  47. 47.Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Andrew McCallum and Sam Roweis, editors, Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008), pages 1096–1103. Omnipress, 2008.
  48. 48.Max Welling, Michal Rosen-Zvi, and Geoffrey E. Hinton. Exponential family harmoniums with an application to information retrieval. In L.K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 17 (NIPS’04), pages 1481–1488, Cambridge, MA, 2005. MIT Press.
  49. 49.Jason Weston, Frédéric Ratle, and Ronan Collobert. Deep learning via semi-supervised embedding. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Proceedings of the Twenty-fifth International Conference on Machine Learning (ICML’08), pages 1168–1175, New York, NY, USA, 2008. ACM.
  50. 50.Andrew Yao. Separating the polynomial-time hierarchy by oracles. In Proceedings of the 26th Annual IEEE Symposium on Foundations of Computer Science, pages 1–10, 1985.
  51. 51.Long Zhu, Yuanhao Chen, and Alan Yuille. Unsupervised learning of probabilistic grammar-markov models for object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1):114–128, 2009.

Citation

MLA
Erhan, D., et al. “Why Does Unsupervised Pre-training Help Deep Learning?”. Journal of Machine Learning Research, vol. 11, no. 19, 2010, pp. 201–08, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.207.1102.
APA
Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010). Why Does Unsupervised Pre-training Help Deep Learning?. Journal of Machine Learning Research, 11(19), 201–208. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.207.1102
Chicago
Erhan, D., Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio. 2010. “Why Does Unsupervised Pre-training Help Deep Learning?”. Journal of Machine Learning Research 11 (19): 201–8. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.207.1102.
Harvard
Erhan, D. et al. (2010) “Why Does Unsupervised Pre-training Help Deep Learning?”, Journal of Machine Learning Research, 11(19), pp. 201–208. Available at: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.207.1102.
Vancouver
1. Erhan D, Bengio Y, Courville A, Manzagol P-A, Vincent P, Bengio S (2010) Why Does Unsupervised Pre-training Help Deep Learning?. Journal of Machine Learning Research 11:201–208

BibTeX

@article{erhan2010why,
  title = {Why Does Unsupervised Pre-training Help Deep Learning?},
  author = {Erhan, Dumitru and Bengio, Yoshua and Courville, Aaron and Manzagol, Pierre-Antoine and Vincent, Pascal and Bengio, Samy},
  year = {2010},
  journal = {Journal of Machine Learning Research},
  volume = {11},
  number = {19},
  pages = {201-208},
  url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.207.1102}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission