Back

Weight Uncertainty in Neural Network

Charles BlundellJulien CornebiseKoray KavukcuogluDaan Wierstra

article2015ICML2,467 citations

Introduces Bayes by Backprop, a variational learning algorithm that learns probability distributions over neural network weights using unbiased Monte Carlo gradients, matching dropout regularisation performance while providing uncertainty estimates to guide reinforcement learning exploration.

Listen

Standard artificial neural networks are prone to fitting training noise too closely and often produce overly confident predictions when encountering unfamiliar situations. In mission-critical applications and automated decision systems, this overconfidence creates operational and safety risks. While existing techniques help prevent overfitting, they typically yield point estimates rather than quantified uncertainty, making it difficult to assess prediction reliability or guide autonomous exploration. The article introduces a practical, backpropagation-compatible method called Bayes by Backprop to learn a probability distribution over the network parameters rather than fixed values.

The article aims to demonstrate that treating network parameters probabilistically can effectively regularize deep models, provide well-calibrated confidence estimates in data-sparse regions, and naturally drive exploration in reinforcement learning scenarios. The authors evaluate this approach across three distinct benchmarks: handwritten digit classification using standard image datasets, a synthetic nonlinear regression task, and an online sequential decision-making bandit task. Rather than tracking single static values, the method optimizes both the center and spread of each parameter's distribution by combining standard gradient calculations with Monte Carlo sampling and a two-component mixture distribution prior.

The empirical findings demonstrate strong practical utility across multiple domains. First, on digit classification benchmarks, the method achieved a test error rate of approximately 1.32% to 1.36%, matching the competitive performance of standard dropout regularization. Second, the learned uncertainty enabled extensive model compression: up to 95% of the parameters could be pruned using a signal-to-noise heuristic with virtually no drop in classification performance (maintaining a 1.29% error rate). Third, on nonlinear regression tasks, the network appropriately showed wider uncertainty intervals in regions with little or no training data, avoiding the unwarranted confidence of standard architectures. Finally, in the sequential bandit task, the model achieved rapid convergence and low cumulative regret by using parameter uncertainty to balance exploration and exploitation from the start.

These findings suggest that incorporating parameter uncertainty can improve system reliability, reduce computational memory requirements through effective model compression, and lower operational risk in autonomous environments. Organizations deploying automated decision systems can avoid brittle, overconfident failures while maintaining high task performance. The method seamlessly integrates with standard hardware accelerators and existing parallel gradient descent pipelines, keeping implementation barriers low.

For technical teams considering deployment, the authors recommend adopting mixture priors and tuning hyperparameters through validation rather than empirical optimization during training. Where runtime efficiency is paramount, teams should leverage post-training parameter pruning to shrink model footprints without extra training stages. Although the per-iteration training cost is roughly twice that of standard approaches, the confidence in the results remains high across the evaluated settings. Further research is recommended to assess performance on large-scale production architectures and complex, stateful reinforcement learning environments.

Googlebackpropagation algorithmclassificationfeedforward neural networksweight uncertaintyvariational Bayesian learningvariational inference
arXiv: 1505.05424
Cover for Weight Uncertainty in Neural Network

Abstract

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

Topics

backpropagation algorithmclassificationfeedforward neural networksmarginalsreinforcement learningtraderegularization

Keywords

weight uncertaintyvariational Bayesian learningvariational inferencestochastic gradient descentThompson samplingcontextual bandit problemsMNIST classificationweight decaymodel averaging

Table of Contents

  • 1. Introduction
  • 2. Point Estimates of Neural Networks
  • 3. Being Bayesian by Backpropagation
  • 3.1. Unbiased Monte Carlo gradients
  • 3.2. Gaussian variational posterior
  • 3.3. Scale mixture prior
  • 3.4. Minibatches and KL re-weighting
  • 4. Contextual Bandits
  • 4.1. Thompson Sampling for Neural Networks
  • 5. Experiments
  • 5.1. Classification on MNIST
  • 5.2. Regression curves
  • 5.3. Bandits on Mushroom Task
  • 6. Discussion
  • References

Knowls

  1. Knowl 1 — Unbiased Gradient Estimator for Expected Functions via Reparameterization

    theoretical result

    Let ϵ\epsilon be a random variable distributed according to a parameter-free probability density q(ϵ)q(\epsilon), and let w=t(θ,ϵ)w = t(\theta, \epsilon) be a deterministic, differentiable transformation parameterized by θ\theta. If the induced marginal probability density q(w∣θ)q(w \mid \theta) satisfies the conservation of probability measure q(ϵ)dϵ=q(w∣θ)dwq(\epsilon) d\epsilon = q(w \mid \theta) dw, then for any function f(w,θ)f(w, \theta) differentiable with respect to ww and θ\theta, the derivative of the expectation can be expressed as the expectation of the derivative:

    ∂∂θEq(w∣θ)[f(w,θ)]=Eq(ϵ)[∂f(w,θ)∂w∂w∂θ+∂f(w,θ)∂θ]\frac{\partial}{\partial \theta} \mathbb{E}_{q(w \mid \theta)}[f(w, \theta)] = \mathbb{E}_{q(\epsilon)} \left[ \frac{\partial f(w, \theta)}{\partial w} \frac{\partial w}{\partial \theta} + \frac{\partial f(w, \theta)}{\partial \theta} \right]

    Applying this result to the variational objective f(w,θ)=log⁡q(w∣θ)−log⁡P(w)−log⁡P(D∣w)f(w, \theta) = \log q(w \mid \theta) - \log P(w) - \log P(\mathcal{D} \mid w) enables unbiased Monte Carlo gradient estimation for learning probability distributions over the weight parameters of neural networks via backpropagation.

  2. Knowl 2 — Variational Free Energy Objective and Monte Carlo Approximation for Bayesian Neural Networks

    equation

    In variational Bayesian learning of neural networks, the intractable true posterior distribution P(w∣D)P(w \mid \mathcal{D}) over network weights ww given dataset D\mathcal{D} is approximated by a tractable distribution q(w∣θ)q(w \mid \theta) parameterized by θ\theta. The variational parameters θ\theta are learned by minimizing the Kullback-Leibler (KL) divergence KL[q(w∣θ)∥P(w∣D)]\text{KL}[q(w \mid \theta) \parallel P(w \mid \mathcal{D})], which is equivalent to minimizing the variational free energy (expected lower bound):

    F(D,θ)=KL[q(w∣θ)∥P(w)]−Eq(w∣θ)[log⁡P(D∣w)]\mathcal{F}(\mathcal{D}, \theta) = \text{KL}[q(w \mid \theta) \parallel P(w)] - \mathbb{E}_{q(w \mid \theta)}[\log P(\mathcal{D} \mid w)]

    where P(w)P(w) is the prior distribution over weights and P(D∣w)P(\mathcal{D} \mid w) is the likelihood of the data given the weights.

    Rather than computing the KL term in closed form, the objective is approximated using nn Monte Carlo samples w(i)∼q(w∣θ)w^{(i)} \sim q(w \mid \theta) via the common random numbers technique:

    F(D,θ)≈∑i=1n(log⁡q(w(i)∣θ)−log⁡P(w(i))−log⁡P(D∣w(i)))\mathcal{F}(\mathcal{D}, \theta) \approx \sum_{i=1}^n \left( \log q(w^{(i)} \mid \theta) - \log P(w^{(i)}) - \log P(\mathcal{D} \mid w^{(i)}) \right)

    Sampling all terms simultaneously avoids closed-form restrictions on P(w)P(w) and q(w∣θ)q(w \mid \theta), ensuring that gradient updates depend consistently on the specific weight configurations sampled.

  3. Knowl 3 — Bayes by Backprop with Gaussian Variational Posterior

    algorithm

    Bayes by Backprop parameterizes a diagonal Gaussian variational posterior for each weight ww using mean μ\mu and standard deviation σ=log⁡(1+exp⁡(ρ))\sigma = \log(1 + \exp(\rho)), where ρ∈R\rho \in \mathbb{R} ensures non-negativity of σ\sigma. The variational parameters are θ=(μ,ρ)\theta = (\mu, \rho).

    Input: Training dataset D, learning rate alpha, prior P(w)
    Output: Variational parameters theta = (mu, rho) for all weights
    for each training iteration do
        Sample standard Gaussian noise epsilon ~ N(0, I)
        Compute weight sample w = mu + log(1 + exp(rho)) * epsilon
        Compute loss function f(w, theta) = log q(w | theta) - log P(w) - log P(D | w)
        Compute gradient with respect to sample weights via backpropagation: g_w = df(w, theta) / dw
        Compute mean gradient Delta_mu = g_w + df(w, theta) / dmu
        Compute rho gradient Delta_rho = g_w * (epsilon / (1 + exp(-rho))) + df(w, theta) / drho
        Update variational mean mu = mu - alpha * Delta_mu
        Update variational parameter rho = rho - alpha * Delta_rho
    end for

    The gradient computation reuses ∂f(w,θ)∂w\frac{\partial f(w, \theta)}{\partial w} computed by standard backpropagation through the network, only requiring scaling and shifting to update the distribution parameters μ\mu and ρ\rho.

  4. Knowl 4 — Scale Mixture Gaussian Prior for Neural Network Weights

    model/method

    To provide heavy tails and induce sparsity without suffering from empirical Bayes instability during stochastic gradient training, the prior distribution over network weights w={wj}w = \{w_j\} is defined as a fixed scale mixture of two zero-mean Gaussians:

    P(w)=∏j(πN(wj∣0,σ12)+(1−π)N(wj∣0,σ22))P(w) = \prod_{j} \left( \pi \mathcal{N}(w_j \mid 0, \sigma_1^2) + (1 - \pi) \mathcal{N}(w_j \mid 0, \sigma_2^2) \right)

    where π∈(0,1)\pi \in (0, 1) is the mixture weight, σ1\sigma_1 is the standard deviation of the wide component, and σ2≪1\sigma_2 \ll 1 is the standard deviation of the narrow spike component with σ1>σ2\sigma_1 > \sigma_2.

    Hyperparameters (π,σ1,σ2)(\pi, \sigma_1, \sigma_2) are kept fixed during training and selected via validation. Optimizing prior parameters via gradients (empirical Bayes) causes the prior to fit poor early posterior states, whereas a fixed scale mixture acts as a continuous spike-and-slab prior that concentrates non-essential weights tightly around zero while accommodating large weights in the heavier tail.

  5. Knowl 5 — Minibatch KL Cost Re-weighting Schedule

    model/method

    When partitioning a dataset D\mathcal{D} into MM equal minibatches D1,D2,…,DM\mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_M, the exact variational cost is preserved in expectation across minibatches using assigned weights πi∈[0,1]\pi_i \in [0, 1] satisfying ∑i=1Mπi=1\sum_{i=1}^M \pi_i = 1:

    Fiπ(Di,θ)=πiKL[q(w∣θ)∥P(w)]−Eq(w∣θ)[log⁡P(Di∣w)]\mathcal{F}_i^{\pi}(\mathcal{D}_i, \theta) = \pi_i \text{KL}[q(w \mid \theta) \parallel P(w)] - \mathbb{E}_{q(w \mid \theta)}[\log P(\mathcal{D}_i \mid w)]

    Under uniformly random batch partitioning, the expected sum satisfies EM[∑i=1MFiπ(Di,θ)]=F(D,θ)\mathbb{E}_M \left[ \sum_{i=1}^M \mathcal{F}_i^{\pi}(\mathcal{D}_i, \theta) \right] = \mathcal{F}(\mathcal{D}, \theta).

    The decay weighting schedule:

    πi=2M−i2M−1\pi_i = \frac{2^{M-i}}{2^M - 1}

    weights the prior complexity cost heavily during the initial minibatches of each epoch (where data updates are small and exploratory) and allows the data likelihood to dominate in later minibatches.

  6. Knowl 6 — Thompson Sampling for Contextual Bandits using Variational Neural Networks

    algorithm

    Bayes by Backprop enables Thompson sampling for contextual bandits without persistent state by sampling parameter configurations from the learned variational posterior distribution q(w∣θ)q(w \mid \theta) to drive exploration.

    Input: Variational posterior parameters theta
    Output: Action selection and updated model parameters
    for each bandit interaction step do
        Sample network weights from variational posterior: w ~ q(w | theta)
        Receive environment context vector x
        Predict expected reward for each available action a: r_pred(a) = E_{P(r | x, a, w)}[r]
        Select action with the highest predicted reward: a* = argmax_a r_pred(a)
        Execute action a* and receive scalar reward r
        Store experience tuple (x, a*, r) into replay buffer
        Sample minibatch from buffer and update variational parameters theta via Bayes by Backprop
    end for

    Early in learning, wide posterior distributions cause stochastic action exploration; as observations accumulate, posterior uncertainty narrows on high-reward actions, naturally transitioning the policy from exploration to exploitation.

  7. Knowl 7 — Signal-to-Noise Ratio Weight Pruning in Variational Neural Networks

    data/table

    Trained Bayes by Backprop models allow structured network compression by ranking individual weights according to their Signal-to-Noise Ratio (SNR), defined as SNRi=∣μi∣σi\text{SNR}_i = \frac{|\mu_i|}{\sigma_i}, and setting weights below a threshold to zero without retraining.

    Proportion removed # Weights Test Error
    0% 2.4m 1.24%
    50% 1.2m 1.24%
    75% 600k 1.24%
    95% 120k 1.29%
    98% 48k 1.39%

    On an MNIST classification network with two hidden layers of 1200 rectified linear units (2.4 million total weights), up to 75% of the weights can be pruned with zero increase in test error (1.24%), and up to 95% of weights can be pruned with only a minor increase in test error to 1.29%. The empirical density of weight SNR exhibits a bimodal distribution where the lower 75% constitutes redundant parameters concentrated at low SNR values.

  8. Knowl 8 — MNIST Classification Error Rates Across Model Configurations

    data/table

    Classification error rates on the standard MNIST benchmark (50,000 training, 10,000 validation, 10,000 test digits) evaluated for feedforward networks of two hidden layers of rectified linear units across layer sizes (400, 800, and 1200 units per hidden layer):

    Method # Units/Layer # Weights Test Error
    SGD, no regularisation (Simard et al., 2003) 800 1.3m 1.6%
    SGD, dropout (Hinton et al., 2012) – – ≈1.3%\approx 1.3\%
    SGD, dropconnect (Wan et al., 2013)∗^* 800 1.3m 1.2%
    SGD (unregularised) 400 500k 1.83%
    SGD (unregularised) 800 1.3m 1.84%
    SGD (unregularised) 1200 2.4m 1.88%
    SGD, dropout 400 500k 1.51%
    SGD, dropout 800 1.3m 1.33%
    SGD, dropout 1200 2.4m 1.36%
    Bayes by Backprop, Gaussian prior 400 500k 1.82%
    Bayes by Backprop, Gaussian prior 800 1.3m 1.99%
    Bayes by Backprop, Gaussian prior 1200 2.4m 2.04%
    Bayes by Backprop, Scale mixture prior 400 500k 1.36%
    Bayes by Backprop, Scale mixture prior 800 1.3m 1.34%
    Bayes by Backprop, Scale mixture prior 1200 2.4m 1.32%

    (∗^*indicates an ensemble of 5 networks).

    Bayes by Backprop with a scale mixture prior outperforms a standard Gaussian prior across all capacities and matches or improves upon standard dropout regularisation (reaching 1.32% test error for a 1200-unit layer network).

  9. Knowl 9 — Epistemic Uncertainty Quantification in Non-Linear 1D Regression

    empirical result

    In non-linear regression on synthetic data generated from y=x+0.3sin⁡(2π(x+ϵ))+0.3sin⁡(4π(x+ϵ))+ϵy = x + 0.3\sin(2\pi(x+\epsilon)) + 0.3\sin(4\pi(x+\epsilon)) + \epsilon with ϵ∼N(0,0.02)\epsilon \sim \mathcal{N}(0, 0.02), a standard neural network trained by maximum likelihood minimizes conditional Gaussian loss by reducing predictive variance to zero and fitting an arbitrary deterministic extrapolation in regions devoid of training samples.

    In contrast, a neural network trained with Bayes by Backprop performs Bayesian model averaging over sampled weight configurations. In regions containing training data, predictive interquartile intervals remain tight around the median curve; in regions lacking training observations, the interquartile confidence intervals diverge broadly, capturing epistemic model uncertainty across possible function extrapolations.

  10. Knowl 10 — Exploration-Exploitation Efficiency in Mushroom Contextual Bandit Benchmark

    empirical result

    On the UCI Mushroom contextual bandit task—where eating an edible mushroom gives a reward of +5+5, eating a poisonous mushroom gives +5+5 (probability 0.50.5) or −35-35 (probability 0.50.5) (expected reward −15-15), and not eating gives 00—agents using a two-hidden-layer (100 ReLU units each) neural network were evaluated on cumulative regret against an oracle.

    Thompson sampling with Bayes by Backprop (sampling weights twice per interaction to compute expected rewards) actively explores edible and poisonous actions from the start and quickly converges to optimal eating decisions, achieving an almost flat cumulative regret curve. In comparison, ϵ\epsilon-greedy with ϵ=5%\epsilon = 5\% continuously accumulates linear regret due to fixed random exploration, while pure greedy (ϵ=0%\epsilon = 0\%) initially avoids all mushrooms for roughly 1,000 steps before non-local function updates trigger exploration.

Coverage note — None was omitted; all core theoretical contributions, algorithmic derivations, training regularizers, pruning procedures, and empirical evaluations across classification, regression, and bandit tasks have been extracted into self-contained knowls.

References

  1. 1.Shipra Agrawal and Navin Goyal. Analysis of Thompson sampling for the multi-armed bandit problem. In Proceedings of the 25th Annual Conference On Learning Theory (COLT), volume 23, pages 39.1–39.26, 2012.
  2. 2.Shipra Agrawal and Navin Goyal. Further optimal regret bounds for Thompson sampling. In Proceedings of the 16th International Conference on Artificial Intelligence and Statistics Learning (AISTATS), pages 99–107, 2013.
  3. 3.Kevin Bache and Moshe Lichman. UCI Machine Learning Repository. University of California, Irvine, School of Information and Computer Sciences, 2013. URL http://archive.ics.uci.edu/ml.
  4. 4.Christopher M Bishop. Section 10.1: variational inference. In Pattern Recognition and Machine Learning. Springer, 2006. ISBN 9780387310732.
  5. 5.Wray L Buntine and Andreas S Weigend. Bayesian backpropagation. Complex systems, 5(6):603–643, 1991.
  6. 6.Olivier Chapelle and Lihong Li. An empirical evaluation of Thompson sampling. In Advances in Neural Information Processing Systems (NIPS), pages 2249–2257, 2011.
  7. 7.Hugh Chipman. Bayesian variable selection with related predictors. Canadian Journal of Statistics, 24(1):17–36, 1996.
  8. 8.Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems (NIPS), pages 1223–1231, 2012.
  9. 9.Sarah Filippi, Olivier Cappe, Aurlien Garivier, and Csaba Szepesvri. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems, pages 586–594, 2010.
  10. 10.Karl Friston, Jérémie Mattout, Nelson Trujillo-Barreto, John Ashburner, and Will Penny. Variational free energy and the Laplace approximation. Neuroimage, 34(1):220–234, 2007.
  11. 11.Andrew Gelman. Objections to Bayesian statistics. Bayesian Analysis, 3:445–450, 2008. ISSN 1931-6690. doi: 11.1214/08-BA318.
  12. 12.Edward I George and Robert E McCulloch. Variable selection via gibbs sampling. Journal of the American Statistical Association, 88(423):881–889, 1993.
  13. 13.Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier networks. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics Learning (AISTATS), volume 15, pages 315–323, 2011.
  14. 14.Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 2348–2356, 2011.
  15. 15.Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep AutoRegressive networks. In Proceedings of the 31st International Conference on Machine Learning (ICML), pages 1242–1250, 2014.
  16. 16.Arthur Guez. Sample-Based Search Methods For Bayes-Adaptive Planning. PhD thesis, University College London, 2015.
  17. 17.Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS 2014 Deep Learning and Representation Learning Workshop, 2014.
  18. 18.Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the 16th Annual Conference On Learning Theory (COLT), pages 5–13. ACM, 1993.
  19. 19.Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, July 2012.
  20. 20.Tommi S. Jaakkola and Michael I. Jordan. Bayesian parameter estimation via variational methods. Statistics and Computing, 10(1):25–37, 2000.
  21. 21.Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In Proceedings of the 23rd Annual Conference on Algorithmic Learning Theory (ALT), pages 199–213. Springer, 2012.
  22. 22.Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014. arXiv: 1312.6114.
  23. 23.Yann LeCun. Une procédure d’apprentissage pour réseau à seuil asymétrique (a learning scheme for asymmetric threshold networks). In Proceedings of Cognitiva 85, Paris, France, pages 599–604, 1985.
  24. 24.Yann LeCun and Corinna Cortes. The MNIST database of handwritten digits. 1998. URL http://yann.lecun.com/exdb/mnist/.
  25. 25.Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW ’10, pages 661–670, New York, NY, USA, 2010. ACM. ISBN 978-1-60558-799-8. doi: 10.1145/1772690.1772758.
  26. 26.David JC MacKay. A practical Bayesian framework for backpropagation networks. Neural computation, 4(3):448–472, 1992.
  27. 27.David JC MacKay. Probable networks and plausible predictions-a review of practical Bayesian methods for supervised neural networks. Network: Computation in Neural Systems, 6(3):469–505, 1995.
  28. 28.Benedict C May, Nathan Korda, Anthony Lee, and David S. Leslie. Optimistic Bayesian sampling in contextual-bandit problems. The Journal of Machine Learning Research, 13(1):2069–2106, 2012.
  29. 29.Thomas P Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001.
  30. 30.Thomas P Minka. Divergence measures and message passing. Technical report, Microsoft Research, 2005.
  31. 31.Toby J Mitchell and John J Beauchamp. Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83(404):1023–1032, 1988.
  32. 32.Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML), pages 807–814, 2010.
  33. 33.Radford M Neal and Geoffrey E Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pages 355–368. Springer, 1998.
  34. 34.Manfred Opper and Cédric Archambeau. The variational Gaussian approximation revisited. Neural computation, 21(3):786–792, 2009.
  35. 35.Art B. Owen. Monte Carlo theory, methods and examples. 2013.
  36. 36.Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning (ICML), pages 1278–1286, 2014.
  37. 37.David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Cognitive modeling, 5, 1988.
  38. 38.Lawrence K Saul, Tommi Jaakkola, and Michael I Jordan. Mean field theory for sigmoid belief networks. Journal of artificial intelligence research, 4(1):61–76, 1996.
  39. 39.Patrice Y Simard, Dave Steinkraus, and John C Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR), volume 2, pages 958–958. IEEE Computer Society, 2003.
  40. 40.William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, pages 285–294, 1933.
  41. 41.Michalis Titsias and Miguel Lázaro-Gredilla. Doubly stochastic variational bayes for non-conjugate inference. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1971–1979, 2014.
  42. 42.Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), pages 1058–1066, 2013.
  43. 43.Jonathan S Yedidia, William T Freeman, and Yair Weiss. Generalized belief propagation. In Advances in Neural Information Processing Systems (NIPS), volume 13, pages 689–695, 2000.

Citation

MLA
Blundell, C., et al. “Weight Uncertainty in Neural Networks”. arXiv, 2015, http://arxiv.org/abs/1505.05424v2.
APA
Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight Uncertainty in Neural Networks. arXiv. http://arxiv.org/abs/1505.05424v2
Chicago
Blundell, C., J. Cornebise, K. Kavukcuoglu, and D. Wierstra. 2015. “Weight Uncertainty in Neural Networks”. arXiv. http://arxiv.org/abs/1505.05424v2.
Harvard
Blundell, C. et al. (2015) “Weight Uncertainty in Neural Networks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1505.05424v2.
Vancouver
1. Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight Uncertainty in Neural Networks. arXiv

BibTeX

@article{blundell2015weight,
  title = {Weight Uncertainty in Neural Networks},
  author = {Blundell, Charles and Cornebise, Julien and Kavukcuoglu, Koray and Wierstra, Daan},
  year = {2015},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1505.05424v2},
  eprint = {1505.05424}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission