BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Matthieu CourbariauxYoshua BengioJean-Pierre David

article2015NeurIPS3,267 citations

Proposes BinaryConnect, a method for training deep neural networks with binary weights during forward and backward propagation while accumulating gradients in high-precision weights, replacing expensive multiplications with simple additions for efficient hardware implementation without sacrificing accuracy.

Listen

Deep neural networks achieve strong results on image and speech tasks but require heavy computation from repeated multiplications of real-valued weights and activations, limiting deployment on low-power hardware. Researchers therefore seek ways to simplify arithmetic during both training and inference while preserving accuracy.

The paper introduces BinaryConnect, a training procedure that forces network weights to take only the values +1 or −1 during forward and backward passes, while retaining full-precision accumulators for the gradient updates themselves.

The method was evaluated on three standard benchmarks—permutation-invariant MNIST, CIFAR-10 without data augmentation, and SVHN—using multilayer perceptrons and convolutional networks of moderate size. Training employed either deterministic sign binarization or stochastic sampling from a hard-sigmoid distribution, combined with weight clipping to [−1, 1], batch normalization, and scaled learning rates. Results were averaged over multiple random initializations and compared against unregularized baselines and established regularizers such as Dropout.

Stochastic BinaryConnect reduced test error relative to the unregularized networks on all three tasks, reaching 1.18 % on MNIST, 8.27 % on CIFAR-10, and 2.15 % on SVHN—figures competitive with prior state-of-the-art methods that retain full-precision weights throughout. The deterministic variant performed slightly worse yet still improved over the baseline, confirming that the binarization noise functions as a regularizer. Histograms of learned weights and training curves further showed that the binary constraint increases training cost while lowering validation error, consistent with dropout-like behavior.

These outcomes indicate that roughly two-thirds of the multiplications normally required for back-propagation can be replaced by additions without sacrificing final accuracy. At inference time the deterministic version eliminates multiplications entirely and reduces memory footprint by at least a factor of sixteen, directly lowering silicon area, power, and bandwidth demands for specialized accelerators.

The reported experiments cover only three datasets and relatively compact architectures; larger models and additional tasks will be needed to establish broader applicability. Extending the approach to remove multiplications from the weight-update step itself remains an open direction that could further increase hardware efficiency.

CIFAR Polytechnique Montréal Université de Montréal deep learning gpus low power BinaryConnect Deep Neural Network binary weight

arXiv: 1511.00363 MatthieuCourbariaux/BinaryConnect

Paper: Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, Yoshua Bengio et al. (2013). This paper establishes foundational gradient propagation methods for discrete and stochastic neural network units, directly motivating BinaryConnect's handling of non-differentiable binary weights.

Paper: XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks, Mohammad Rastegari et al. (2016). Building directly upon BinaryConnect, XNOR-Net extends binary weight binarization to layer inputs as well, achieving dramatic compute and memory savings on full-scale architectures.
Paper: Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding, Song Han et al. (2015). This work continues the investigation of model compression started by BinaryConnect by combining weight pruning, trained quantization, and Huffman coding to drastically shrink deep neural networks.

Cover for BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Abstract

Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.

Topics

deep learning gpus low power multiplier training sample convolutional neural networks

Keywords

BinaryConnect Deep Neural Network binary weight Stochastic Gradient Descent DropConnect hard sigmoid Batch Normalization MNIST CIFAR-10 SVHN

1 Introduction
2 BinaryConnect
2.1 +1 or -1
2.2 Deterministic vs stochastic binarization
2.3 Propagations vs updates
2.4 Clipping
2.5 A few more tricks
2.6 Test-Time Inference
3 Benchmark results
3.1 Permutation-invariant MNIST
3.2 CIFAR-10
3.3 SVHN
4 Related works
5 Conclusion and future works
6 Acknowledgments
References

Knowls

Knowl 1 — Training Deep Neural Networks with BinaryConnect
algorithm
BinaryConnect trains deep neural networks by constraining weights to binary values ( $\pm 1$ ) during forward and backward propagations, while accumulating stochastic gradient descent (SGD) updates into high-precision real-valued continuous weights.
Input: Minibatch of training pairs (inputs, targets), initial real-valued weights $w_{t-1}$ , initial biases $b_{t-1}$ , learning rate $\eta$ , cost function $C$ , number of layers $L$ .
Output: Updated real-valued weights $w_t$ and biases $b_t$ .
1. Forward propagation:
     $w_b \leftarrow \text{binarize}(w_{t-1})$
    For $k = 1$ to $L$ :
        Compute activation $a_k$ from $a_{k-1}$ , binarized weights $w_b$ , and bias $b_{t-1}$
2. Backward propagation:
    Initialize output activation gradient $\frac{\partial C}{\partial a_L}$
    For $k = L$ down to 2:
        Compute activation gradient $\frac{\partial C}{\partial a_{k-1}}$ using $\frac{\partial C}{\partial a_k}$ and $w_b$
3. Parameter update:
    For $k = 1$ to $L$ :
        Compute gradients $\frac{\partial C}{\partial w_b}$ and $\frac{\partial C}{\partial b_{t-1}}$ using $\frac{\partial C}{\partial a_k}$ and $a_{k-1}$
     $w_t \leftarrow \text{clip}(w_{t-1} - \eta \frac{\partial C}{\partial w_b}, -1, 1)$
     $b_t \leftarrow b_{t-1} - \eta \frac{\partial C}{\partial b_{t-1}}$
High-precision continuous accumulators are essential for SGD because each parameter update step is small and noisy; binarizing only the propagations allows the expected value of the weights to track the gradient descent trajectory.
Knowl 2 — Deterministic and Stochastic Binarization Schemes in BinaryConnect
model/method
BinaryConnect transforms continuous real-valued weights $w \in \mathbb{R}$ into binary weights $w_b \in \{-1, +1\}$ using either a deterministic or a stochastic formulation.

Deterministic Binarization: Uses the sign function: $w_b = \begin{cases} +1 & \text{if } w \ge 0, \\ -1 & \text{otherwise.} \end{cases}$

Stochastic Binarization: Samples binary weights according to a probability parameterized by the real-valued weight: $w_b = \begin{cases} +1 & \text{with probability } p = \sigma(w), \\ -1 & \text{with probability } 1 - p, \end{cases}$ where $\sigma(x)$ is a computationally efficient piecewise-linear hard sigmoid function: $\sigma(x) = \text{clip}\left(\frac{x + 1}{2}, 0, 1\right) = \max\left(0, \min\left(1, \frac{x + 1}{2}\right)\right)$

Stochastic binarization guarantees that the expectation of the binary weight matches the continuous weight for $w \in [-1, 1]$ (i.e., $\mathbb{E}[w_b] = w$ ), turning discretization error into zero-mean unbiased noise across hidden unit inputs.
Knowl 3 — Weight Clipping and Continuous Accumulation in BinaryConnect
model/method
Because the binarization operator $\text{binarize}(w)$ is insensitive to changes in the magnitude of the continuous weight $w$ beyond $[-1, +1]$ , unconstrained gradient updates would cause real-valued parameters to grow arbitrarily large without altering the binarized weights during propagations. Such growth severely hinders the network's ability to adapt when a weight sign needs to change.

To prevent this saturation and regularize the network, the continuous weights are clipped to $[-1, 1]$ immediately after each gradient step: $w_t = \text{clip}\left(w_{t-1} - \eta \frac{\partial C}{\partial w_b}, -1, 1\right) = \max\left(-1, \min\left(1, w_{t-1} - \eta \frac{\partial C}{\partial w_b}\right)\right)$ where $w_{t-1}$ is the continuous parameter prior to the step, $\eta$ is the learning rate, and $\frac{\partial C}{\partial w_b}$ is the loss gradient computed with respect to the binarized weights.
Knowl 4 — Test-Time Inference Strategies for BinaryConnect Networks
model/method
Once a deep neural network is trained using BinaryConnect, inference on new test samples can be performed using one of three methods:
1. Binarized Weights ( $w_b$ ): The network uses fixed binary weights $w_b = \text{sign}(w) \in \{-1, +1\}$ . This is the standard strategy for deterministic BinaryConnect, eliminating all multiplication operations at test time and reducing memory bandwidth.
2. Real-Valued Weights ( $w$ ): The network directly uses the learned continuous parameters $w \in [-1, 1]$ . Analogous to Dropout where stochastic noise is removed at test time, this uses the expected weights. This is the primary strategy for stochastic BinaryConnect.
3. Stochastic Ensemble Averaging: Multiple binarized networks are instantiated by independently sampling $w_b \sim \text{Bernoulli}(\sigma(w))$ for each weight, and their output predictions are averaged together.
Knowl 5 — Learning Rate Scaling and Optimization Techniques for BinaryConnect
model/method
Effective training of deep neural networks with BinaryConnect relies on Batch Normalization (BN) and layer-dependent learning rate scaling:
- Batch Normalization: Applied to all hidden layers to reduce internal covariate shift and minimize sensitivity to weight scaling.
- Learning Rate Scaling with Initialization Factors: Let $c = \sqrt{2 / (n_{\text{in}} + n_{\text{out}})}$ $c = 2/ (n_{in} + n_{out})$ denote the Glorot-Bengio weight initialization coefficient for a given layer. The layer-specific learning rate $\eta$ $η$ is scaled as:
  - $\eta_{\text{layer}} = c \cdot \eta$ when using the ADAM optimizer.
  - $\eta_{\text{layer}} = c^2 \cdot \eta$ when using standard SGD or Nesterov momentum.

Knowl 6 — Effect of Learning Rate Scaling across Optimizers on CIFAR-10

data/table

The choice of optimizer and the application of initialization-coefficient learning rate scaling markedly affect the performance of convolutional neural networks trained with BinaryConnect.

The table lists test error rates for a small CNN on CIFAR-10 (without data augmentation). Scaling the learning rate by initialization coefficients consistently reduces test error across all optimization algorithms, with ADAM achieving the best result at 10.47%.

Knowl 7 — Classification Benchmark Results of BinaryConnect on MNIST, CIFAR-10, and SVHN

data/table

BinaryConnect acts as an effective regularizer, outperforming unregularized baselines and achieving near state-of-the-art results on permutation-invariant MNIST, CIFAR-10, and SVHN benchmarks.

Method	MNIST	CIFAR-10	SVHN
No regularizer	$1.30 \pm 0.04\%$	10.64%	2.44%
BinaryConnect (deterministic)	$1.29 \pm 0.08\%$	9.90%	2.30%
BinaryConnect (stochastic)	$1.18 \pm 0.04\%$	8.27%	2.15%
50% Dropout	$1.01 \pm 0.04\%$	—	—
Maxout Networks	0.94%	11.68%	2.47%
Deep L2-SVM	0.87%	—	—
Network in Network	—	10.41%	2.35%
DropConnect	—	—	1.94%
Deeply-Supervised Nets	—	9.78%	1.92%

Benchmark configurations:

MNIST: Permutation-invariant setting (no convolutions, no pretraining, no data augmentation). Model: MLP with 3 hidden layers of 1024 ReLU units and an L2-SVM output layer trained with SGD and square hinge loss.
CIFAR-10: Global contrast normalization and ZCA whitening applied without data augmentation. Model: $(2\times 128\text{C3})-\text{MP2}-(2\times 256\text{C3})-\text{MP2}-(2\times 512\text{C3})-\text{MP2}-(2\times 1024\text{FC})-10\text{SVM}$ trained with ADAM and square hinge loss.
SVHN: Same architecture as CIFAR-10 with half the number of hidden units, trained for 200 epochs.

Stochastic BinaryConnect consistently provides stronger regularization than deterministic BinaryConnect and the unregularized network across all datasets.

Knowl 8 — Hardware and Computational Efficiency Gains of BinaryConnect
theoretical result
Standard deep network training involves arithmetic multiplications in three distinct phases: forward propagation ( $a_{k-1} \cdot w$ ), backward propagation ( $\frac{\partial C}{\partial a_k} \cdot w$ ), and parameter gradient accumulation ( $\frac{\partial C}{\partial a_k} \cdot a_{k-1}$ ).

By constraining weights to binary values $w_b \in \{-1, +1\}$ during forward and backward propagations:
- Multiply-accumulate (MAC) operations in forward and backward passes are converted into simple additions and subtractions on fixed-point adders.
- Approximately two-thirds ( $\approx 2/3$ ) of all multiplication operations during training are removed, enabling an estimated $3\times$ training speedup on specialized hardware.
- During deterministic test-time inference, multiplications are completely eliminated, and weight memory storage is reduced by at least a factor of 16 (compared to 16-bit float) or 32 (compared to 32-bit float).

Coverage note — Visual plots (weight histograms in Figure 2, training/validation loss curves in Figure 3, and qualitative first-layer filter images in Figure 1) were omitted as standalone knowls because their core quantitative conclusions are fully represented in the benchmark tables and method descriptions.

References

1.Geoffrey Hinton, Li Deng, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, Nov. 2012.
2.Tara Sainath, Abdel rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep convolutional neural networks for LVCSR. In ICASSP 2013, 2013.
3.A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS’2012. 2012.
4.Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Technical report, arXiv:1409.4842, 2014.
5.Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. Fast and robust neural network joint models for statistical machine translation. In Proc. ACL’2014, 2014.
6.Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS’2014, 2014.
7.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR’2015, arXiv:1409.0473, 2015.
8.Rajat Raina, Anand Madhavan, and Andrew Y. Ng. Large-scale deep unsupervised learning using graphics processors. In ICML’2009, 2009.
9.Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, 2003.
10.J. Dean, G.S Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le, M.Z. Mao, M.A. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS’2012, 2012.
11.Sang Kyun Kim, Lawrence C McAfee, Peter Leonard McMahon, and Kunle Olukotun. A highly scalable restricted Boltzmann machine FPGA implementation. In Field Programmable Logic and Applications, 2009. FPL 2009. International Conference on, pages 367–372. IEEE, 2009.
12.Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and operating systems, pages 269–284. ACM, 2014.
13.Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, et al. Dadiannao: A machine-learning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on, pages 609–622. IEEE, 2014.
14.Lorenz K Muller and Giacomo Indiveri. Rounding methods for neural networks with low resolution synaptic weights. arXiv preprint arXiv:1504.05767, 2015.
15.Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML’2015, 2015.
16.Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Low precision arithmetic for deep learning. In Arxiv:1412.7024, ICLR’2015 Workshop, 2015.
17.Thomas M Bartol, Cailey Bromer, Justin P Kinney, Michael A Chirillo, Jennifer N Bourne, Kristen M Harris, and Terrence J Sejnowski. Hippocampal spine head sizes are highly precise. bioRxiv, 2015.
18.Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348–2356. Curran Associates, Inc., 2011.
19.Nitish Srivastava. Improving neural networks with dropout. Master’s thesis, U. Toronto, 2013.
20.Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.
21.Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML’2013, 2013.
22.J.P. David, K. Kalach, and N. Tittley. Hardware complexity of modular multiplication and exponentiation. Computers, IEEE Transactions on, 56(10):1308–1319, Oct 2007.
23.R. Collobert. Large Scale Machine Learning. PhD thesis, Universite de Paris VI, LIP6, 2004.
24.X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In AISTATS’2011, 2011.
25.Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS’2010, 2010.
26.Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015.
27.Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
28.Yu Nesterov. A method for unconstrained convex minimization problem with the rate of convergence o(1/k2 ). Doklady AN SSSR (translated as Soviet. Math. Docl.), 269:543–547, 1983.
29.Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. Technical Report Arxiv report 1302.4389, Universite de Montreal, February 2013.
30.Yichuan Tang. Deep learning using linear support vector machines. Workshop on Challenges in Representation Learning, ICML, 2013.
31.Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.
32.Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. arXiv preprint arXiv:1409.5185, 2014.
33.Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998.
34.V. Nair and G.E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML’2010, 2010.
35.Benjamin Graham. Spatially-sparse convolutional neural networks. arXiv preprint arXiv:1409.6070, 2014.
36.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
37.Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In NIPS’2014, 2014.
38.Zhiyong Cheng, Daniel Soudry, Zexi Mao, and Zhenzhong Lan. Training binary multilayer neural networks for image classification using expectation backpropgation. arXiv preprint arXiv:1503.03562, 2015.
39.Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pages 1–6. IEEE, 2014.
40.Jonghong Kim, Kyuyeon Hwang, and Wonyong Sung. X1000 real-time phoneme recognition vlsi using feed-forward deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pages 7510–7514. IEEE, 2014.
41.Thomas P Minka. Expectation propagation for approximate bayesian inference. In UAI’2001, 2001.
42.James Bergstra, Olivier Breuleux, Frederic Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Des-jardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Presentation.
43.Frederic Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
44.Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu, James Bergstra, Frederic Bastien, and Yoshua Bengio. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013.
45.Sander Dieleman, Jan Schlter, Colin Raffel, Eben Olson, Sren Kaae Snderby, Daniel Nouri, Daniel Maturana, Martin Thoma, Eric Battenberg, Jack Kelly, Jeffrey De Fauw, Michael Heilman, diogo149, Brian McFee, Hendrik Weideman, takacsg84, peterderivaz, Jon, instagibbs, Dr. Kashif Rasul, CongLiu, Britefury, and Jonas Degrave. Lasagne: First release., August 2015.

Citation

MLA

Courbariaux, M., et al. “BinaryConnect: Training Deep Neural Networks with Binary Weights During Propagations”. arXiv, 2015, https://doi.org/10.48550/arxiv.1511.00363.

APA

Courbariaux, M., Bengio, Y., & David, J.-P. (2015). BinaryConnect: Training Deep Neural Networks with binary weights during propagations. arXiv. https://doi.org/10.48550/arxiv.1511.00363

Chicago

Courbariaux, M., Y. Bengio, and J.-P. David. 2015. “BinaryConnect: Training Deep Neural Networks with Binary Weights During Propagations”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1511.00363.

Harvard

Courbariaux, M., Bengio, Y. and David, J.-P. (2015) “BinaryConnect: Training Deep Neural Networks with binary weights during propagations”. arXiv. Available at: https://doi.org/10.48550/arxiv.1511.00363.

Vancouver

1. Courbariaux M, Bengio Y, David J-P (2015) BinaryConnect: Training Deep Neural Networks with binary weights during propagations. https://doi.org/10.48550/arxiv.1511.00363

BibTeX

@misc{https://doi.org/10.48550/arxiv.1511.00363,
  doi = {10.48550/ARXIV.1511.00363},
  url = {https://arxiv.org/abs/1511.00363},
  author = {Courbariaux, Matthieu and Bengio, Yoshua and David, Jean-Pierre},
  keywords = {Machine Learning (cs.LG), Computer Vision and Pattern Recognition (cs.CV), Neural and Evolutionary Computing (cs.NE), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {BinaryConnect: Training Deep Neural Networks with binary weights during propagations},
  publisher = {arXiv},
  year = {2015},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Published with permission

Optimization Method	No learning rate scaling	Learning rate scaling
SGD	—	11.45%
Nesterov momentum	15.65%	11.30%
ADAM	12.81%	10.47%

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Training Deep Neural Networks with BinaryConnect

Knowl 2 — Deterministic and Stochastic Binarization Schemes in BinaryConnect

Knowl 3 — Weight Clipping and Continuous Accumulation in BinaryConnect

Knowl 4 — Test-Time Inference Strategies for BinaryConnect Networks

Knowl 5 — Learning Rate Scaling and Optimization Techniques for BinaryConnect