Practical Recommendations for Gradient-Based Training of Deep Architectures

article2012Neural Networks2,419 citations

Presents essential practical heuristics and actionable guidelines for tuning hyperparameters, diagnosing optimization failures, and successfully training deep neural networks with gradient descent.

Listen

Training deep neural networks requires tuning numerous configuration choices, known as hyper-parameters, which directly control computational cost, optimization speed, and predictive accuracy. As modern machine learning systems scale up, inefficient search methods and poorly chosen training settings lead to wasted compute budgets and degraded model accuracy. The article synthesizes empirical findings, mathematical foundations, and established engineering practices to establish actionable principles for gradient-based deep learning optimization.

To establish these guidelines, the article reviews extensive experimental literature and theoretical analyses covering iterative optimization methods, automatic differentiation frameworks, parameter initialization strategies, and hyper-parameter search protocols across diverse deep learning architectures.

The findings establish that the initial learning rate is the single most critical hyper-parameter in stochastic gradient descent and must be tuned, with optimal values typically within a factor of two of divergence thresholds. For hyper-parameter exploration, random search is exponentially more efficient than exhaustive grid search because it allocates search budget to the few high-impact dimensions rather than repeatedly testing identical values across full factorial grids. For model architecture and optimization stability, scaling initial weights inversely with the square root of layer connectivity and selecting non-saturating activation functions, such as rectifiers, substantially improves gradient flow and model convergence in deeper networks. Additionally, mini-batch training provides a balanced sweet spot, capturing hardware speedups through parallel matrix computations while preserving the regularization and fast exploration advantages of stochastic updates.

These insights demonstrate that automated, systematic optimization practices drastically reduce compute expenditure and accelerate model delivery timelines compared to manual trial-and-error. Early stopping based on out-of-sample validation error serves as a computationally inexpensive safeguard against overfitting, reducing the need for heavy explicit weight regularization. Adopting automatic differentiation frameworks with symbolic graph simplifications further lowers software defect rates while optimizing memory allocations.

Organizations developing deep learning systems should implement random hyper-parameter search, standardize weight initializations by network connectivity, and enforce rigorous gradient-checking sanity tests during software development. However, because many practical rules remain heuristics derived from empirical benchmarks rather than universally proven theorems, teams should treat these recommendations as starting points, testing their robustness across new application domains and tracking emerging adaptive learning rate techniques.

deep learning optimization recommendation Learning Training One

arXiv: 1206.5533

Book: Deep Learning Tuning Playbook, Varun Godbole et al.. This playbook provides foundational empirical methodologies for hyperparameter tuning and model optimization that directly inform the practical training strategies discussed in the source.
Paper: An overview of gradient descent optimization algorithms, Sebastian Ruder (2016). This survey details the core gradient descent optimization algorithms and hyperparameters evaluated in the source.

Paper: Adam: A Method for Stochastic Optimization, Diederik P. Kingma et al. (2015). This paper introduces Adam, an adaptive stochastic optimization method that extends the gradient-based training strategies detailed in the source.
Paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Sergey Ioffe et al. (2015). This paper builds directly upon the training recommendations of the source by introducing batch normalization to address internal covariate shift.

Cover for Practical Recommendations for Gradient-Based Training of Deep Architectures

Abstract

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.

Topics

deep learning optimization recommendation network architecture optimization problems recurrent neural networks

Keywords

Learning Training One Deep Gradient Bengio Neural Hyper-parameters Neural Networks Ing

1 Introduction
1.1 Deep Learning and Greedy Layer-Wise Pretraining
1.2 Denoising and Contractive Auto-Encoders
1.3 Online Learning and Optimization of Generalization Error
2 Gradients
2.1 Gradient Descent and Learning Rate
2.2 Gradient Computation and Automatic Differentiation
3 Hyper-Parameters
3.1 Neural Network Hyper-Parameters
3.1.1 Hyper-Parameters of the Approximate Optimization
3.2 Hyper-Parameters of the Model and Training Criterion
3.3 Manual Search and Grid Search
3.3.1 General guidance for the exploration of hyper-parameters
3.3.2 Coordinate Descent and Multi-Resolution Search
3.3.3 Automated and Semi-automated Grid Search
3.3.4 Layer-wise optimization of hyper-parameters
3.4 Random Sampling of Hyper-Parameters
4 Debugging and Analysis
4.1 Gradient Checking and Controlled Overfitting
4.2 Visualizations and Statistics
5 Other Recommendations
5.1 Multi-core machines, BLAS and GPUs
5.2 Sparse High-Dimensional Inputs
5.3 Symbolic Variables, Embeddings, Multi-Task Learning and Multi-Relational Learning
6 Open Questions
6.1 On the Added Difficulty of Training Deeper Architectures
6.2 Adaptive Learning Rates and Second-Order Methods
6.3 Conclusion
References

Knowls

Knowl 1 — Greedy Layer-Wise Hyper-Parameter Optimization
algorithm
In deep architectures trained with greedy layer-wise unsupervised pre-training, joint optimization of all hyper-parameters across layers is computationally prohibitive. A greedy strategy optimizes hyper-parameters layer by layer from the input upward, using a cheap surrogate evaluation on the intermediate representations.
Input: K, number of top configurations to retain at each stage
Input: Nlevels, total number of layers in the deep network
Input: LevelSettings, list of candidate hyper-parameter configurations for pre-training a layer
Input: SFTSettings, list of candidate hyper-parameter configurations for supervised fine-tuning
Output: S, set of K best-performing full network configurations
Initialize best configurations set S = {empty}
for L = 1 to Nlevels do
    Initialize candidate set Scandy = {empty}
    for each setting C in LevelSettings do
        for each configuration H in (if S is empty then {empty} else S) do
            Pretrain layer L using hyper-parameters C with lower layers fixed to H
            Train a linear predictor on the depth-L representation
            Evaluate target validation error Lval
            Add candidate (C union H, Lval) to Scandy
        end for
    end for
    Set S = K configurations from Scandy with lowest Lval
end for
Initialize fine-tuned set Sfine = {empty}
for each setting C in SFTSettings do
    for each configuration H in S do
        Perform supervised fine-tuning on pre-trained network H using setting C
        Evaluate validation error Lval of the fine-tuned network
        Add candidate (C union H, Lval) to Sfine
    end for
end for
Set S = K configurations from Sfine with lowest Lval
return S
Retaining the $K$ best configurations at each level scales computational cost by a factor of $K$ per layer beyond the first, avoiding the exponential combinatorial explosion of full multi-layer hyper-parameter searches.
Knowl 2 — Superiority of Random Search Over Grid Search in High-Dimensional Hyper-Parameter Spaces
model/method
When optimizing hyper-parameters for deep neural networks, random search uniformly sampled across a pre-defined search space (typically uniform in the logarithmic domain for positive scale parameters) is significantly more efficient than grid search whenever the number of hyper-parameters exceeds 2 or 3.

This efficiency stems from the low effective dimensionality of hyper-parameter response surfaces: for any given task, only a small subset of hyper-parameters critically governs performance (with learning rate almost always among them). In a grid search testing $V$ values across $D$ dimensions ( $V^D$ total evaluations), each individual parameter is tested at only $V$ distinct values, wasting trials on variations of unimportant dimensions. In contrast, $N$ random search trials test $N$ distinct values for every single hyper-parameter, offering exponentially denser coverage of the low-dimensional subspace of critical parameters.

Practically, random search also decouples trials, allowing asynchronous parallel execution, resilience to individual node failures without invalidating grid coverage, and adaptive addition of further trials without discarding existing samples.
Knowl 3 — Normalized Weight Initialization for Deep Neural Networks
model/method
To prevent signals and back-propagated gradients from exponentially vanishing or exploding across deep layers, weights must be initialized to preserve activation and gradient variances. For a fully connected layer where each neuron has $\text{fan-in}$ incoming connections and $\text{fan-out}$ outgoing connections, weights $W_{ij}$ should be drawn independently from a uniform distribution:

$W_{ij} \sim U[-r, r]$

where the interval half-width $r$ is:

$r = \sqrt{\frac{6}{\text{fan-in} + \text{fan-out}}}$

for hyperbolic tangent ( $anh$ ) activation functions, and

$r = 4\sqrt{\frac{6}{\text{fan-in} + \text{fan-out}}}$

for standard logistic sigmoid activation functions $s(a) = 1 / (1 + e^{-a})$ .

Biases can generally be initialized to 0. Output layer weights can also be initialized to 0 because distinct target signals break symmetry across output units, but hidden layer weights must use random initialization to break unit symmetry within the same layer.
Knowl 4 — Proper Regularizer Scaling in Mini-Batch Stochastic Gradient Descent
equation
When optimizing a penalized empirical loss criterion consisting of per-example losses $L(z_i, \theta)$ over $T$ total training examples with a global parameter regularization penalty $\Omega(\theta)$ (such as $L_2$ weight decay $\Omega(\theta) = \frac{1}{2}\lambda \|\theta\|^2_2$ or $L_1$ penalty $\Omega(\theta) = \lambda \|\theta\|_1$ ):

$J(\theta) = \sum_{i=1}^T L(z_i, \theta) + \Omega(\theta)$

the mini-batch parameter update for a batch of size $B$ must scale the regularizer gradient by $B/T$ to remain an unbiased estimator of the total objective gradient:

$\theta^{(t)} \leftarrow \theta^{(t-1)} - \epsilon_t \left( \frac{1}{B} \sum_{t'=Bt+1}^{B(t+1)} \frac{\partial L(z_{t'}, \theta)}{\partial \theta} + \frac{B}{T} \frac{\partial \Omega(\theta)}{\partial \theta} \right)$

where $\epsilon_t$ is the learning rate at iteration $t$ . If the final mini-batch in an epoch has size $B' < B$ , the regularizer scale factor is adjusted to $B'/T$ . In an online streaming setting without a fixed dataset size, $B/T$ is replaced by $B/t$ , where $t$ is the number of samples processed up to that step.
Knowl 5 — Early Stopping with Dynamic Patience Window
algorithm
Early stopping optimizes the effective number of training updates $T$ while mitigating overfitting without requiring manual capacity tuning. A dynamic patience heuristic ensures training continues sufficiently long beyond the current lowest validation error to confirm convergence.
Input: N, evaluation interval in number of processed training examples
Input: P0, initial patience (minimum additional examples to observe)
Input: gamma, multiplicative patience expansion factor (or additive constant delta_P)
Output: theta_star, model parameter vector at minimal validation error
Initialize t = 0 (examples seen)
Initialize L_star = infinity
Initialize t_star = 0
Initialize T_limit = P0
while t < T_limit do
    Train network on next mini-batch and update parameters theta
    Increment t by the mini-batch size
    if t is an integer multiple of N then
        Compute validation loss L_val
        if L_val < L_star then
            Set L_star = L_val
            Set t_star = t
            Save current parameters theta_star = theta
            Set T_limit = max(T_limit, t + P0) // or t * (1 + gamma)
        end if
    end if
end while
return theta_star
The evaluation interval $N$ should be at least as large as the validation set size to maintain low computational overhead relative to training.
Knowl 6 — Symmetrical Two-Sided Finite-Difference Gradient Checking
equation
To verify the analytic gradient $\frac{\partial f(x)}{\partial x}$ of a scalar loss function $f$ with respect to a scalar variable or parameter $x$ , the central two-sided finite-difference approximation provides second-order accuracy $O(\varepsilon^2)$ :

$\frac{\partial f(x)}{\partial x} = \frac{f(x + \varepsilon) - f(x - \varepsilon)}{2\varepsilon} + O(\varepsilon^2)$

This formulation cancels out even-order Taylor expansion terms, achieving quadratically higher precision than the one-sided difference quotient $\frac{f(x + \varepsilon) - f(x)}{\varepsilon} = \frac{\partial f(x)}{\partial x} + O(\varepsilon)$ .

In floating-point neural network implementations, setting $\varepsilon \approx 10^{-4}$ provides an optimal operating point that minimizes truncation error while avoiding numerical instability and machine-precision rounding noise.
Knowl 7 — Importance-Sampled Reconstruction for Sparse High-Dimensional Auto-Encoders
model/method
For high-dimensional sparse inputs $x \in \mathbb{R}^D$ where $D$ is large and most entries are 0, reconstructing the complete vector $r(x) \in \mathbb{R}^D$ in an auto-encoder or denoising auto-encoder is computationally expensive ( $O(D \cdot n_h)$ ). Sampled reconstruction reduces this complexity linearly with input sparsity while preserving an unbiased loss gradient.

On each example, a subset of coordinate indices $S \subset \{1, \dots, D\}$ is sampled:
1. All non-zero coordinates in the input $x$ (or corrupted input $\tilde{x}$ for denoising auto-encoders) are included deterministically ( $p_j = 1$ ).
2. An equal number of zero-valued coordinates are sampled uniformly at random with probability $p_j = p_{\text{zero}} < 1$ .
Each sampled coordinate $j \in S$ is weighted by the inverse sampling probability $w_j = 1 / p_j$ in the reconstruction loss:

$L_{\text{sampled}}(x, r(x)) = \sum_{j \in S} w_j L_j(x_j, r_j(x))$

Back-propagation is computed strictly through the sampled output units $j \in S$ , speeding up training proportionally to input sparsity without degrading convergence or representation quality.
Knowl 8 — Learning Rate Selection and Hyperbolic Decay Schedule
model/method
The initial learning rate $\epsilon_0$ is the single most critical optimization hyper-parameter in stochastic gradient descent. A reliable tuning heuristic is to start from a large learning rate and divide by 3 iteratively until divergence disappears; the optimal rate is typically within a factor of 2 of the maximum non-diverging rate.

To ensure asymptotic convergence while maintaining fast initial exploration, the learning rate can be scheduled according to:

$\epsilon_t = \frac{\epsilon_0 \tau}{\max(t, \tau)}$

where $t$ is the update index and $\tau$ is a time-constant hyper-parameter. The learning rate remains constant at $\epsilon_0$ for the first $\tau$ updates, and subsequently follows an $O(1/t)$ decay. An adaptive heuristic sets $\tau$ dynamically by keeping $\epsilon_t = \epsilon_0$ until the relative epoch-to-epoch reduction in training loss falls below a predefined threshold.
Knowl 9 — Activation Sparsity Regularization Mechanisms
model/method
Constraining hidden unit activations $h_j = s(w_j^T x + b_j)$ to be sparse (values near zero) encourages the discovery of disentangled underlying factors and prevents capacity saturation in overcomplete layers. The primary mechanisms to enforce activation sparsity include:
1. Kullback-Leibler Divergence Penalty on Mean Activations: For sigmoid units where activations $h_j \in (0, 1)$ behave as probabilities, the batch-averaged activation $\bar{h}_j = \frac{1}{B} \sum_{k=1}^B h_j^{(k)}$ is penalized against a low target activation level $\rho$ (e.g., $\rho = 0.05$ ): $D_{\mathrm{KL}}(\rho \,\|\, \bar{h}_j) = -\rho \log \bar{h}_j - (1 - \rho) \log(1 - \bar{h}_j) + \text{constant}$
2. Direct Activation Penalties: Adding an $L_1$ penalty $\alpha \sum_j |h_j|$ or Student- $t$ penalty $\alpha \sum_j \log(1 + h_j^2)$ to the training loss.
3. Rectifying Non-Linearities: Employing rectifier activation functions $s(a) = \max(0, a)$ , which output exact zeros for negative pre-activations without vanishing gradients along active units ( $a > 0$ ).
Knowl 10 — Architectural Layer Width and Overcompleteness Heuristics
empirical result
Empirical evaluations across diverse benchmark tasks establish three structural heuristics for setting hidden layer widths ( $n_h$ ) in deep networks:
1. Uniform Layer Width: Maintaining a constant width across all hidden layers ( $n_{h, 1} = n_{h, 2} = \dots = n_{h, L}$ ) consistently performs as well as or better than geometric pyramid (decreasing width) or inverted pyramid (increasing width) designs.
2. Overcomplete First Hidden Layer: Setting the number of units in the first hidden layer to be larger than the input dimension ( $n_{h, 1} > D$ ) yields superior representations and classification accuracy compared to undercomplete first layers ( $n_{h, 1} < D$ ).
3. Capacity Scaling with Unsupervised Pre-Training: The optimal number of hidden units $n_h$ is substantially larger (often an order of magnitude larger, moving from hundreds to thousands of units) when layers undergo unsupervised pre-training than when trained purely supervised from random initialization. Because unsupervised pre-training captures task-irrelevant data variations alongside relevant ones, larger hidden layers are necessary to ensure that task-salient information is fully preserved.

Coverage note — Deliberately omitted general introductory reviews of deep learning history, standard unregularized backpropagation equations, high-level overviews of Restricted Boltzmann Machines (deferred to external chapters), and general software descriptions of Theano/Torch.

References

1.Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251–276.
2.Bach, F. and Moulines, E. (2011). Non-asymptotic analysis of stochastic approximation algorithms. In NIPS’2011.
3.Bagnell, J. A. and Bradley, D. M. (2009). Differentiable sparse coding. In NIPS’2009, pages 113–120.
4.Baxter, J. (1995). Learning internal representations. In COLT’95, pages 311–320.
5.Baxter, J. (1997). A Bayesian/information theoretic model of learning via multiple task sampling. Machine Learning, 28, 7–40.
6.Bengio, Y. (2008). Neural net language models. Scholarpedia, 3(1), 3881.
7.Bengio, Y. (2009). Learning deep architectures for AI. Now Publishers.
8.Bengio, Y. (2011). Deep learning of representations for unsupervised and transfer learning. In JMLR W&CP: Proc. Unsupervised and Transfer Learning.
9.Bengio, Y. and Delalleau, O. (2011). On the expressive power of deep architectures. In ALT’2011.
10.Bengio, Y. and LeCun, Y. (2007). Scaling learning algorithms towards AI. In Large Scale Kernel Machines.
11.Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003). A neural probabilistic language model. JMLR, 3, 1137–1155.
12.Bengio, Y., Le Roux, N., Vincent, P., Delalleau, O., and Marcotte, P. (2006a). Convex neural networks. In NIPS’2005, pages 123–130.
13.Bengio, Y., Delalleau, O., and Le Roux, N. (2006b). The curse of highly variable functions for local kernel machines. In NIPS’2005, pages 107–114.
14.Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H. (2007). Greedy layer-wise training of deep networks. In NIPS’2006.
15.Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In ICML’09.
16.Bengio, Y., Alain, G., and Rifai, S. (2012). Implicit density estimation by local moment matching to sample from auto-encoders. Technical report, arXiv:1207.0057.
17.Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. J. Machine Learning Res., 13, 281–305.
18.Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proc. Python for Scientific Comp. Conf. (SciPy).
19.Bergstra, J., Bardenet, R., Bengio, Y., and K´egl, B. (2011). Algorithms for hyper-parameter optimization. In NIPS’2011.
20.Berkes, P. and Wiskott, L. (2002). Applying slow feature analysis to image sequences yields a rich repertoire of complex cell properties. In ICANN’02, pages 81–86.
21.Bertsekas, D. P. (2010). Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. Technical Report 2848, LIDS.
22.Bordes, A., Bottou, L., and Gallinari, P. (2009). Sgd-qn: Careful quasi-newton stochastic gradient descent. Journal of Machine Learning Research, 10, 1737–1754.
23.Bordes, A., Weston, J., Collobert, R., and Bengio, Y. (2011). Learning structured embeddings of knowledge bases. In AAAI 2011.
24.Bordes, A., Glorot, X., Weston, J., and Bengio, Y. (2012). Joint learning of words and meaning representations for open-text semantic parsing. AISTATS’2012.
25.Bottou, L. (2011). From machine learning to machine reasoning. Technical report, arXiv.1102.1808.
26.Bottou, L. (2013). Large-scale learning with stochastic gradient descent. In K.-R. M¨uller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade, Reloaded. Springer.
27.Bottou, L. and Bousquet, O. (2008). The tradeoffs of large scale learning. In NIPS’2008.
28.Bottou, L. and LeCun, Y. (2004). Large-scale on-line learning. In NIPS’2003.
29.Breiman, L. (1994). Bagging predictors. Machine Learning, 24(2), 123–140.
30.Breuleux, O., Bengio, Y., and Vincent, P. (2011). Quickly generating representative samples from an rbm-derived process. Neural Computation, 23(8), 2053–2073.
31.Caruana, R. (1993). Multitask connectionist learning. In Proceedings of the 1993 Connectionist Models Summer School, pages 372–379.
32.Cho, K., Raiko, T., and Ilin, A. (2011). Enhanced gradient and adaptive learning rate for training restricted boltzmann machines. In ICML’2011, pages 105–112.
33.Coates, A. and Ng, A. Y. (2011). The importance of encoding versus training with sparse coding and vector quantization. In ICML’2011.
34.Collobert, R. and Bengio, S. (2004a). Links between perceptrons, MLPs and SVMs. In ICML’2004.
35.Collobert, R. and Bengio, S. (2004b). Links between perceptrons, MLPs and SVMs. In International Conference on Machine Learning, ICML.
36.Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011a). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12, 2493–2537.
37.Collobert, R., Kavukcuoglu, K., and Farabet, C. (2011b). Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop.
38.Courville, A., Bergstra, J., and Bengio, Y. (2011). Unsupervised models of images by spike-and-slab RBMs. In ICML’2011.
39.Dauphin, Y., Glorot, X., and Bengio, Y. (2011). Sampled reconstruction for large-scale learning of embeddings. In Proc. ICML’2011.
40.Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. (1990). Indexing by latent semantic analysis. J. Am. Soc. Information Science, 41(6), 391–407.
41.Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research.
42.Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48, 781–799.
43.Erhan, D., Courville, A., and Bengio, Y. (2010a). Understanding representations learned in deep architectures. Technical Report 1355, Universit´e de Montr´eal/DIRO.
44.Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., and Bengio, S. (2010b). Why does unsupervised pre-training help deep learning? J. Machine Learning Res., 11, 625–660.
45.Frasconi, P., Gori, M., and Sperduti, A. (1998). A general framework for adaptive processing of data structures. IEEE Transactions on Neural Networks, 9(5), 768–786.
46.Geman, S., Bienenstock, E., and Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4(1), 1–58.
47.Getoor, L. and Taskar, B. (2006). Introduction to Statistical Relational Learning. MIT Press.
48.Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In AISTATS’2010, pages 249–256.
49.Glorot, X., Bordes, A., and Bengio, Y. (2011a). Deep sparse rectifier neural networks. In AISTATS’2011.
50.Glorot, X., Bordes, A., and Bengio, Y. (2011b). Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML’2011.
51.Goodfellow, I., Le, Q., Saxe, A., and Ng, A. (2009). Measuring invariances in deep networks. In NIPS’2009, pages 646–654.
52.Goodfellow, I., Courville, A., and Bengio, Y. (2011). Spike-and-slab sparse coding for unsupervised feature discovery. In NIPS Workshop on Challenges in Learning Hierarchical Models.
53.Graepel, T., Candela, J. Q., Borchert, T., and Herbrich, R. (2010). Web-scale Bayesian click-through rate prediction for sponsored search advertising in microsoft’s bing search engine. In ICML’2010.
54.H˚astad, J. (1986). Almost optimal lower bounds for small depth circuits. In STOC’86, pages 6–20.
55.H˚astad, J. and Goldmann, M. (1991). On the power of small-depth threshold circuits. Computational Complexity, 1, 113–129.
56.Hinton, G. E. (1978). Relaxation and its role in vision. Ph.D. thesis, University of Edinburgh.
57.Hinton, G. E. (1986). Learning distributed representations of concepts. In Proc. 8th Annual Conf. Cog. Sc. Society, pages 1–12.
58.Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, 40, 185–234.
59.Hinton, G. E. (2010). A practical guide to training restricted Boltzmann machines. Technical Report UTML TR 2010-003, Department of Computer Science, University of Toronto.
60.Hinton, G. E. (2013). A practical guide to training restricted boltzmann machines. In K.-R. M¨uller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade, Reloaded. Springer.
61.Hinton, G. E., Osindero, S., and Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554.
62.Hutter, F. (2009). Automated Configuration of Algorithms for Solving Hard Computational Problems. Ph.D. thesis, University of British Columbia.
63.Hutter, F., Hoos, H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In LION-5.
64.Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2009). What is the best multi-stage architecture for object recognition? In ICCV’09.
65.Kavukcuoglu, K., Ranzato, M.-A., Fergus, R., and LeCun, Y. (2009). Learning invariant features through topographic filter maps. In CVPR’2009.
66.Krueger, K. A. and Dayan, P. (2009). Flexible shaping: how learning in small steps helps. Cognition, 110, 380–394.
67.Lamblin, P. and Bengio, Y. (2010). Important gains from supervised fine-tuning of deep architectures on large labeled sets. NIPS*2010 Deep Learning and Unsupervised Feature Learning Workshop.
68.Lang, K. J. and Hinton, G. E. (1988). The development of the time-delay neural network architecture for speech recognition. Technical Report CMU-CS-88-152, Carnegie-Mellon University.
69.Larochelle, H. and Bengio, Y. (2008). Classification using discriminative restricted Boltzmann machines. In ICML’2008.
70.Larochelle, H., Bengio, Y., Louradour, J., and Lamblin, P. (2009). Exploring strategies for training deep neural networks. J. Machine Learning Res., 10, 1–40.
71.Le, Q., Ngiam, J., Chen, Z., hao Chia, D. J., Koh, P. W., and Ng, A. (2010). Tiled convolutional neural networks. In NIPS’2010.
72.Le, Q., Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., and Ng, A. (2011). On optimization methods for deep learning. In ICML’2011.
73.Le Roux, N., Manzagol, P.-A., and Bengio, Y. (2008). Topmoumoute online natural gradient algorithm. In NIPS’07.
74.Le Roux, N., Bengio, Y., and Fitzgibbon, A. (2011). Improving first and second-order methods by modeling uncertainty. In Optimization for Machine Learning. MIT Press.
75.Le Roux, N., Schmidt, M., and Bach, F. (2012). A stochastic gradient method with an exponential convergence rate for strongly-convex optimization with finite training sets. Technical report, arXiv:1202.6258.
76.LeCun, Y. (1987). Mod`eles connexionistes de l’apprentissage. Ph.D. thesis, Universit´e de Paris VI.
77.LeCun, Y. (1989). Generalization and network design strategies. Technical Report CRG-TR-89-4, University of Toronto.
78.LeCun, Y. (2013). to appear. In K.-R. M¨uller, G. Montavon, and G. B. Orr, editors, Neural Networks: Tricks of the Trade, Reloaded. Springer.
79.LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4), 541–551.
80.LeCun, Y., Bottou, L., Orr, G. B., and M¨uller, K. (1998a). Efficient backprop. In Neural Networks, Tricks of the Trade.
81.LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998b). Gradient based learning applied to document recognition. IEEE, 86(11), 2278–2324.
82.Lee, H., Ekanadham, C., and Ng, A. (2008). Sparse deep belief net model for visual area V2. In NIPS’07.
83.Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. (2009). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In ICML’2009.
84.Martens, J. (2010). Deep learning via Hessian-free optimization. In ICML’2010, pages 735–742.
85.Mesnil, G., Dauphin, Y., Glorot, X., Rifai, S., Bengio, Y., Goodfellow, I., Lavoie, E., Muller, X., Desjardins, G., Warde-Farley, D., Vincent, P., Courville, A., and Bergstra, J. (2011). Unsupervised and transfer learning challenge: a deep learning approach. In JMLR W&CP: Proc. Unsupervised and Transfer Learning, volume 7.
86.Montavon, G., Braun, M. L., and Muller, K.-R. (2012). Deep boltzmann machines as feed-forward hierarchies. In AISTATS’2012.
87.Nair, V. and Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In ICML’2010.
88.Nemirovski, A. and Yudin, D. (1983). Problem complexity and method efficiency in optimization. Wiley.
89.Nesterov, Y. (2009). Primal-dual subgradient methods for convex problems. Mathematical programming, 120(1), 221–259.
90.Olshausen, B. A. and Field, D. J. (1997). Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Research, 37, 3311–3325.
91.Pearlmutter, B. (1994). Fast exact multiplication by the Hessian. Neural Computation, 6(1), 147–160.
92.Pinto, N., Doukhan, D., DiCarlo, J. J., and Cox, D. D. (2009). A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS Comput Biol, 5(11), e1000579.
93.Pollack, J. B. (1990). Recursive distributed representations. Artificial Intelligence, 46(1), 77–105.
94.Polyak, B. and Juditsky, A. (1992). Acceleration of stochastic approximation by averaging. SIAM J. Control and Optimization, 30(4), 838–855.
95.Raiko, T., Valpola, H., and LeCun, Y. (2012). Deep learning made easier by linear transformations in perceptrons. In AISTATS’2012.
96.Ranzato, M., Poultney, C., Chopra, S., and LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. In NIPS’06.
97.Ranzato, M., Boureau, Y.-L., and LeCun, Y. (2008a). Sparse feature learning for deep belief networks. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20 (NIPS’07), pages 1185–1192, Cambridge, MA. MIT Press.
98.Ranzato, M., Boureau, Y., and LeCun, Y. (2008b). Sparse feature learning for deep belief networks. In NIPS’2007.
99.Richardson, M. and Domingos, P. (2006). Markov logic networks. Machine Learning, 62, 107–136.
100.Rifai, S., Vincent, P., Muller, X., Glorot, X., and Bengio, Y. (2011a). Contracting auto-encoders: Explicit invariance during feature extraction. In ICML’2011.
101.Rifai, S., Dauphin, Y., Vincent, P., Bengio, Y., and Muller, X. (2011b). The manifold tangent classifier. In NIPS’2011.
102.Rifai, S., Bengio, Y., Dauphin, Y., and Vincent, P. (2012). A generative process for sampling contractive auto-encoders. In ICML’2012.
103.Robbins, H. and Monro, S. (1951). A stochastic approximation method. Annals of Mathematical Statistics, 22, 400–407.
104.Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by backpropagating errors. Nature, 323, 533–536.
105.Salakhutdinov, R. and Hinton, G. (2009). Deep Boltzmann machines. In AISTATS’2009.
106.Saxe, A. M., Koh, P. W., Chen, Z., Bhand, M., Suresh, B., and Ng, A. (2011). On random weights and unsupervised feature learning. In ICML’2011.
107.Schaul, T., Zhang, S., and LeCun, Y. (2012). No More Pesky Learning Rates. Technical report.
108.Schraudolph, N. N. (1998). Centering neural network gradient factors. In G. B. Orr and K.-R. Muller, editors, Neural Networks: Tricks of he Trade, pages 548–548. Springer.
109.Socher, R., Manning, C., and Ng, A. Y. (2011). Parsing natural scenes and natural language with recursive neural networks. In ICML’2011.
110.Srinivasan, A. and Ramakrishnan, G. (2011). Parameter screening and optimisation for ILP using designed experiments. Journal of Machine Learning Research, 12, 627–662.
111.Swersky, K., Chen, B., Marlin, B., and de Freitas, N. (2010). A tutorial on stochastic approximation algorithms for training restricted boltzmann machines and deep belief nets. In Information Theory and Applications Workshop.
112.Tenenbaum, J., de Silva, V., and Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500), 2319–2323.
113.Tieleman, T. and Hinton, G. (2009). Using fast weights to improve persistent contrastive divergence. In ICML’2009.
114.van der Maaten, L. and Hinton, G. E. (2008). Visualizing data using t-sne. J. Machine Learning Res., 9.
115.Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7).
116.Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. In ICML 2008.
117.Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Manzagol, P.-A. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Machine Learning Res., 11.
118.Weston, J., Ratle, F., and Collobert, R. (2008). Deep learning via semi-supervised embedding. In ICML 2008.
119.Weston, J., Bengio, S., and Usunier, N. (2011). Wsabie: Scaling up to large vocabulary image annotation. In Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI.
120.Wiskott, L. and Sejnowski, T. J. (2002). Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 14(4), 715–770.
121.Zou, W. Y., Ng, A. Y., and Yu, K. (2011). Unsupervised learning of visual invariance with temporal coherence. In NIPS 2011 Workshop on Deep Learning and Unsupervised Feature Learning.

Citation

MLA

Bengio, Y. “Practical Recommendations for Gradient-Based Training of Deep Architectures”. Lecture Notes in Computer Science, Springer Berlin Heidelberg, 2012, pp. 437–78, https://doi.org/10.1007/978-3-642-35289-8_26.

APA

Bengio, Y. (2012). Practical Recommendations for Gradient-Based Training of Deep Architectures. In Lecture Notes in Computer Science (pp. 437–478). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-35289-8_26

Chicago

Bengio, Y. 2012. “Practical Recommendations for Gradient-Based Training of Deep Architectures”. In Lecture Notes in Computer Science. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-35289-8_26.

Harvard

Bengio, Y. (2012) “Practical Recommendations for Gradient-Based Training of Deep Architectures”, Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 437–478. Available at: https://doi.org/10.1007/978-3-642-35289-8_26.

Vancouver

1. Bengio Y (2012) Practical Recommendations for Gradient-Based Training of Deep Architectures. In: Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp 437–478

BibTeX

@inbook{Bengio_2012, title={Practical Recommendations for Gradient-Based Training of Deep Architectures}, ISBN={9783642352898}, ISSN={1611-3349}, url={http://dx.doi.org/10.1007/978-3-642-35289-8_26}, DOI={10.1007/978-3-642-35289-8_26}, booktitle={Neural Networks: Tricks of the Trade}, publisher={Springer Berlin Heidelberg}, author={Bengio, Yoshua}, year={2012}, pages={437–478} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Practical Recommendations for Gradient-Based Training of Deep Architectures

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Greedy Layer-Wise Hyper-Parameter Optimization

Knowl 2 — Superiority of Random Search Over Grid Search in High-Dimensional Hyper-Parameter Spaces

Knowl 3 — Normalized Weight Initialization for Deep Neural Networks

Knowl 4 — Proper Regularizer Scaling in Mini-Batch Stochastic Gradient Descent

Knowl 5 — Early Stopping with Dynamic Patience Window

Knowl 6 — Symmetrical Two-Sided Finite-Difference Gradient Checking

Knowl 7 — Importance-Sampled Reconstruction for Sparse High-Dimensional Auto-Encoders

Knowl 8 — Learning Rate Selection and Hyperbolic Decay Schedule

Knowl 9 — Activation Sparsity Regularization Mechanisms

Knowl 10 — Architectural Layer Width and Overcompleteness Heuristics

References

Citation

BibTeX

Access the Paper