Back

Deep Learning with Limited Numerical Precision

Suyog GuptaAnkur AgrawalKailash GopalakrishnanPritish Narayanan

article2015ICML2,205 citations

Demonstrates that stochastic rounding enables deep neural networks to be trained using 16-bit fixed-point arithmetic with virtually no accuracy loss, opening the door to highly energy-efficient hardware acceleration.

classificationcomputational resourcesdeep learningPrecisionDeepLearning
arXiv: 1502.02551
Cover for Deep Learning with Limited Numerical Precision

Abstract

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

Topics

classificationcomputational resourcesdeep learningenergy-efficientfixed point arithmetichardware acceleratorsnumerical solutionstochasticnetwork architectureconvolutional neural networks

Keywords

PrecisionDeepLearningXed-pointRoundingUsingTrainingDeep LearningLimited Numerical PrecisionNetwork

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Limited Precision Arithmetic
  • 3.1 Rounding Modes
  • 3.2 Multiply and accumulate (MACC) operation
  • 4 Training Deep Networks
  • 4.1 MNIST
  • 4.1.1 Fully connected DNN
  • 4.1.2 CNN
  • 4.2 CIFAR10
  • 5 Hardware Prototyping
  • 5.1 System Description
  • 5.2 Systolic Array Architecture
  • 5.3 Results
  • 6 Conclusion
  • References

Knowls

  1. Knowl 1 — Stochastic Rounding Scheme for Fixed-Point Arithmetic

    definition

    In fixed-point arithmetic with integer word length IL\text{IL} and fractional word length FL\text{FL} (denoted ⟨IL,FL⟩\langle \text{IL}, \text{FL} \rangle), the smallest positive representable quantity is ϵ=2−FL\epsilon = 2^{-\text{FL}}. For any real number x∈Rx \in \mathbb{R}, let ⌊x⌋\lfloor x \rfloor denote the largest integer multiple of ϵ\epsilon less than or equal to xx. The stochastic rounding operation Round(x,⟨IL,FL⟩)\text{Round}(x, \langle \text{IL}, \text{FL} \rangle) is defined probabilistically based on the proximity of xx to ⌊x⌋\lfloor x \rfloor:

    Round(x,⟨IL,FL⟩)={⌊x⌋with probability 1−x−⌊x⌋ϵ⌊x⌋+ϵwith probability x−⌊x⌋ϵ\text{Round}(x, \langle \text{IL}, \text{FL} \rangle) = \begin{cases} \lfloor x \rfloor & \text{with probability } 1 - \frac{x - \lfloor x \rfloor}{\epsilon} \\[6pt] \lfloor x \rfloor + \epsilon & \text{with probability } \frac{x - \lfloor x \rfloor}{\epsilon} \end{cases}

    Stochastic rounding is an unbiased rounding scheme whose expected rounding error is zero:

    E[Round(x,⟨IL,FL⟩)]=x\mathbb{E}[\text{Round}(x, \langle \text{IL}, \text{FL} \rangle)] = x

    Unlike deterministic round-to-nearest rounding, where parameter updates smaller in magnitude than ϵ2\frac{\epsilon}{2} are rounded to zero, stochastic rounding preserves a non-zero probability for small gradient updates to round to ±ϵ\pm \epsilon, thereby statistically preserving gradient information during low-precision neural network training.

  2. Knowl 2 — Fixed-Point Representation and Conversion with Saturation

    equation

    For a target fixed-point format ⟨IL,FL⟩\langle \text{IL}, \text{FL} \rangle, where IL\text{IL} is the number of integer bits (including the sign bit) and FL\text{FL} is the number of fractional bits, the total word length is WL=IL+FL\text{WL} = \text{IL} + \text{FL}, the precision is bounded by ϵ=2−FL\epsilon = 2^{-\text{FL}}, and the representable range is [−2IL−1,2IL−1−2−FL][-2^{\text{IL}-1}, 2^{\text{IL}-1} - 2^{-\text{FL}}].

    The conversion function Convert(x,⟨IL,FL⟩)\text{Convert}(x, \langle \text{IL}, \text{FL} \rangle) maps an arbitrary value x∈Rx \in \mathbb{R} into this fixed-point format by saturating values outside the representable interval and applying a chosen rounding function Round(x,⟨IL,FL⟩)\text{Round}(x, \langle \text{IL}, \text{FL} \rangle) to in-range values:

    Convert(x,⟨IL,FL⟩)={−2IL−1if x≤−2IL−12IL−1−2−FLif x≥2IL−1−2−FLRound(x,⟨IL,FL⟩)otherwise\text{Convert}(x, \langle \text{IL}, \text{FL} \rangle) = \begin{cases} -2^{\text{IL}-1} & \text{if } x \le -2^{\text{IL}-1} \\[6pt] 2^{\text{IL}-1} - 2^{-\text{FL}} & \text{if } x \ge 2^{\text{IL}-1} - 2^{-\text{FL}} \\[6pt] \text{Round}(x, \langle \text{IL}, \text{FL} \rangle) & \text{otherwise} \end{cases}

    Saturation prevents catastrophic wrap-around errors when activations or parameter accumulations exceed the dynamic range allowed by IL\text{IL}.

  3. Knowl 3 — Two-Step Multiply and Accumulate Procedure

    algorithm

    To compute the inner product c0=a⋅bc_0 = \mathbf{a} \cdot \mathbf{b} of two dd-dimensional fixed-point vectors a,b∈Rd\mathbf{a}, \mathbf{b} \in \mathbb{R}^d in representation ⟨IL,FL⟩\langle \text{IL}, \text{FL} \rangle into a target fixed-point representation ⟨IL~,FL~⟩\langle \widetilde{\text{IL}}, \widetilde{\text{FL}} \rangle, computation is separated into full-precision accumulation followed by a single conversion step:

    Input: Vectors a,b∈Rd\mathbf{a}, \mathbf{b} \in \mathbb{R}^d formatted as ⟨IL,FL⟩\langle \text{IL}, \text{FL} \rangle, target format ⟨IL~,FL~⟩\langle \widetilde{\text{IL}}, \widetilde{\text{FL}} \rangle
    Output: Scaled inner product c0c_0 formatted as ⟨IL~,FL~⟩\langle \widetilde{\text{IL}}, \widetilde{\text{FL}} \rangle
    z←0z \leftarrow 0
    for i←1i \leftarrow 1 to dd do
        z←z+ai⋅biz \leftarrow z + a_i \cdot b_i
    end for
    c0←Convert(z,⟨IL~,FL~⟩)c_0 \leftarrow \text{Convert}(z, \langle \widetilde{\text{IL}}, \widetilde{\text{FL}} \rangle)
    return c0c_0

    Each product ai⋅bia_i \cdot b_i resides in format ⟨2⋅IL,2⋅FL⟩\langle 2 \cdot \text{IL}, 2 \cdot \text{FL} \rangle. Accumulator register zz has bit-width ⌈log⁡2d⌉+2WL\lceil \log_2 d \rceil + 2\text{WL} (where WL=IL+FL\text{WL} = \text{IL} + \text{FL}) to prevent overflow and precision loss during summation. Performing stochastic rounding via Convert(⋅)\text{Convert}(\cdot) only after accumulating all dd partial products minimizes hardware rounding overhead and allows fixed-point matrix multiplication to be simulated on CPUs/GPUs via single-precision floating-point matrix multiplication (SGEMM) followed by element-wise conversion.

  4. Knowl 4 — Impact of Stochastic Rounding on MNIST Classification Training

    empirical result

    Training neural networks on the MNIST dataset using 16-bit fixed-point arithmetic (word length WL=16\text{WL} = 16) yields the following comparative behaviors across rounding modes:

    • Fully Connected Deep Neural Network (two hidden layers with 1000 ReLU units each, minibatch stochastic gradient descent with batch size 100):

      • The 32-bit floating-point baseline achieves a test error of 1.4%1.4\%.
      • Under round-to-nearest rounding, the network achieves comparable accuracy at 14 fractional bits (format ⟨2,14⟩\langle 2, 14 \rangle), but training degrades significantly when fractional precision is reduced to 10 bits (⟨6,10⟩\langle 6, 10 \rangle) or 8 bits (⟨8,8⟩\langle 8, 8 \rangle) because parameter updates round down to zero.
      • Under stochastic rounding, the network achieves test errors matching the 32-bit floating-point baseline (1.4%1.4\%) with as few as 8 fractional bits (format ⟨8,8⟩\langle 8, 8 \rangle). Degradation only occurs when precision drops below 8 fractional bits.

    • Convolutional Neural Network (LeNet-5 architecture with two 5×55 \times 5 conv layers with 8 and 16 feature maps, 2×22 \times 2 max pooling, 128 ReLU units, and 10-way softmax; layer activations in ⟨6,10⟩\langle 6, 10 \rangle format):

      • The 32-bit floating-point baseline achieves a test error of 0.77%0.77\%.
      • Training with round-to-nearest rounding fails to converge.
      • Training with stochastic rounding converges reliably, achieving test errors of 0.83%0.83\% with format ⟨2,14⟩\langle 2, 14 \rangle and 0.90%0.90\% with format ⟨4,12⟩\langle 4, 12 \rangle for weights and weight updates.

  5. Knowl 5 — CIFAR-10 Classification and Mixed-Precision Fine-Tuning Strategy

    empirical result

    On the CIFAR-10 dataset using a convolutional neural network (three convolutional layers with 64 5×55 \times 5 filters each, 3×33 \times 3 max pooling with stride 2, 10-way softmax output, and layer outputs in format ⟨4,12⟩\langle 4, 12 \rangle):

    • The 32-bit floating-point baseline achieves a test error of 24.6%24.6\%.
    • Training using 16-bit fixed-point arithmetic (word length WL=16\text{WL} = 16) with round-to-nearest rounding collapses within a few epochs and fails to converge.
    • Training using 16-bit fixed-point arithmetic with stochastic rounding:
      • Format ⟨2,14⟩\langle 2, 14 \rangle for weights and updates achieves a test error of 25.4%25.4\%, close to the floating-point baseline.
      • Format ⟨4,12⟩\langle 4, 12 \rangle causes learning to stagnate around a test error of 28.8%28.8\% due to sparse parameter updates caused by diminishing learning rates combined with smaller gradients.
    • Mixed-Precision Recovery: When training with format ⟨4,12⟩\langle 4, 12 \rangle stagnates at epoch 100, increasing the word length by 4 bits to format ⟨4,16⟩\langle 4, 16 \rangle (20 bits total) for an additional 15–20 epochs rapidly recovers performance, with test error approaching the 32-bit floating-point baseline. This demonstrates that models can be trained primarily with low-precision fixed-point arithmetic and fine-tuned in higher precision for a few epochs.

  6. Knowl 6 — Wavefront 2D Systolic Array Architecture for Matrix Multiplication

    model/method

    A hardware accelerator implements matrix multiplication (C=A⋅BC = A \cdot B) using a two-dimensional n×nn \times n wavefront systolic array of Digital Signal Processing (DSP) multiply-accumulate (MACC) blocks:

    • Wavefront Execution: Elements of matrix AA cascade horizontally across rows while elements of matrix BB cascade vertically down columns. For an inner dimension kk, the upper-left MACC unit (1,1) completes accumulating all its partial products after kk cycles, writes its sum to a local register, resets to accept data for the next GEMM operation, and forwards output data to stochastic rounding blocks at the array perimeter. The complete n×nn \times n matrix multiplication finishes in k+2n−2k + 2n - 2 clock cycles.
    • On-Chip Tiling and Data Reuse: To operate within off-chip memory bandwidth limits (6.4 GB/s6.4\text{ GB/s} DDR3 interface), input matrices are partitioned and stored in an on-chip L2 Block RAM cache in blocks of pnpn rows of AA and nn columns of BB. Each loaded element of AA is reused mm times and each loaded element of BB is reused pnpn times, with double buffering used to overlap off-chip data fetch latencies with array computation.

  7. Knowl 7 — Single-DSP Stochastic Rounding and Saturation Circuit

    model/method

    To convert wide 48-bit fixed-point accumulator outputs from an FPGA systolic array to lower bit-width output formats without explicit multi-bit comparators, a combined stochastic rounding and saturation module is implemented using a single DSP unit per column:

    1. Stochastic Rounding: A Linear Feedback Shift Register (LFSR) generates an unbiased pseudo-random number whose width matches the number of least significant bits (LSBs) being trimmed. The DSP unit adds this random value directly to the incoming 48-bit accumulator output and drops the trimmed LSBs.
    2. Saturation: Built-in pattern-detector circuitry in the DSP monitors the excess most significant bits (MSBs). If the excess MSBs are neither all zeros nor all ones, an overflow or underflow condition is signaled, and the DSP unit clamps the output to the maximum or minimum two's complement value for the target format.

    For a 28×2828 \times 28 systolic array, 28 DSP rounding blocks are required, adding less than 4%4\% hardware overhead relative to the 784 DSP MACC computation units.

  8. Knowl 8 — FPGA Resource Utilization and Energy Efficiency of Fixed-Point Systolic Accelerator

    data/table

    A 28×2828 \times 28 systolic array matrix multiplier with stochastic rounding was implemented on a Xilinx Kintex-325T FPGA (XCVK325T). Synthesized and placed-and-routed via Xilinx Vivado, the accelerator operates at a maximum clock frequency of 166 MHz166\text{ MHz}, consumes 7 W7\text{ W} of power, achieves a computational throughput of 260 G-ops/s260\text{ G-ops/s}, and reaches an energy efficiency of 37 G-ops/s/W37\text{ G-ops/s/W} (substantially exceeding the 1–5 G-ops/s/W1\text{--}5\text{ G-ops/s/W} typical of CPUs and GPUs).

    Resource Usage Available on XCVK325T Utilization Ratio
    LUTs 62,922 203,800 31%
    Flip-flops 146,510 407,600 36%
    DSP 812 840 97%
    Block RAM 334 445 75%

    The 812 allocated DSP units consist of 784 DSP units for the 28×2828 \times 28 MACC array and 28 DSP units for column-wise stochastic rounding and saturation.

Coverage note — No substantial contributed material was omitted; all theoretical definitions, empirical evaluation benchmarks, hardware systolic array designs, and FPGA implementation results are represented.

References

  1. 1.Audhkhasi, Kartik, Osoba, Osonde, and Kosko, Bart. Noise benefits in backpropagation and deep bidirectional pre-training. In Neural Networks (IJCNN), The 2013 International Joint Conference on, pp. 1–8. IEEE, 2013.
  2. 2.Baboulin, Marc, Buttari, Alfredo, Dongarra, Jack, Kurzak, Jakub, Langou, Julie, Langou, Julien, Luszczek, Piotr, and Tomov, Stanimire. Accelerating scientific computations with mixed precision algorithms. Computer Physics Communications, 180(12):2526–2533, 2009.
  3. 3.Bishop, Chris M. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116, 1995.
  4. 4.Bottou, Léon and Bousquet, Olivier. The tradeoffs of large scale learning. In NIPS, volume 4, pp. 2, 2007.
  5. 5.Chen, Yunji, Luo, Tao, Liu, Shaoli, Zhang, Shijin, He, Liqiang, Wang, Jia, Li, Ling, Chen, Tianshi, Xu, Zhiwei, Sun, Ninghui, et al. Dadiannao: A machine-learning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on, pp. 609–622. IEEE, 2014.
  6. 6.Chilimbi, Trishul, Suzue, Yutaka, Apacible, Johnson, and Kalyanaraman, Karthik. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pp. 571–582, Broomfield, CO, October 2014.
  7. 7.Coates, Adam, Huval, Brody, Wang, Tao, Wu, David, Catanzaro, Bryan, and Andrew, Ng. Deep learning with cots hpc systems. In Proceedings of The 30th International Conference on Machine Learning, pp. 1337–1345, 2013.
  8. 8.Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. Low precision arithmetic for deep learning. arXiv preprint arXiv:1412.7024, 2014.
  9. 9.Dean, Jeffrey, Corrado, Greg, Monga, Rajat, Chen, Kai, Devin, Matthieu, Mao, Mark, Senior, Andrew, Tucker, Paul, Yang, Ke, Le, Quoc V, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, pp. 1223–1231, 2012.
  10. 10.Farabet, Clément, Martini, Berin, Corda, Benoit, Akselrod, Polina, Culurciello, Eugenio, and LeCun, Yann. Neuflow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on, pp. 109–116. IEEE, 2011.
  11. 11.Gokhale, Vinayak, Jin, Jonghoon, Dundar, Aysegul, Martini, Berin, and Culurciello, Eugenio. A 240 g-ops/s mobile coprocessor for deep neural networks. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on, pp. 696–701. IEEE, 2014.
  12. 12.Hammerstrom, Dan. A vlsi architecture for high-performance, low-cost, on-chip learning. In Neural Networks, 1990., 1990 IJCNN International Joint Conference on, pp. 537–544. IEEE, 1990.
  13. 13.Hinton, Geoffrey E, Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
  14. 14.Höhfeld, Markus and Fahlman, Scott E. Probabilistic rounding in neural network learning with limited precision. Neurocomputing, 4(6):291–299, 1992.
  15. 15.Holt, JL and Hwang, Jenq-Neng. Finite precision error analysis of neural network hardware implementations. Computers, IEEE Transactions on, 42(3):281–290, 1993.
  16. 16.Iwata, Akira, Yoshida, Yukio, Matsuda, Satoshi, Sato, Yukimasa, and Suzumura, Nobuo. An artificial neural network accelerator using general purpose 24 bit floating point digital signal processors. In Neural Networks, 1989. IJCNN., International Joint Conference on, pp. 171–175. IEEE, 1989.
  17. 17.Kim, Jonghong, Hwang, Kyuyeon, and Sung, Wonyong. X1000 real-time phoneme recognition vlsi using feed-forward deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, pp. 7510–7514. IEEE, 2014.
  18. 18.Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 1(4):7, 2009.
  19. 19.Kung, H.T. Why systolic architectures? Computer, 15(1):37–46, Jan 1982. doi: 10.1109/MC.1982.1653825.
  20. 20.Lecun, Yann and Cortes, Corinna. The MNIST database of handwritten digits. URL http://yann.lecun.com/exdb/mnist/.
  21. 21.LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  22. 22.Merolla, Paul A, Arthur, John V, Alvarez-Icaza, Rodrigo, Cassidy, Andrew S, Sawada, Jun, Akopyan, Filipp, Jackson, Bryan L, Imam, Nabil, Guo, Chen, Nakamura, Yutaka, et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668–673, 2014.
  23. 23.Murray, Alan F and Edwards, Peter J. Enhanced mlp performance and fault tolerance resulting from synaptic weight noise during training. Neural Networks, IEEE Transactions on, 5(5):792–802, 1994.
  24. 24.Recht, Benjamin, Re, Christopher, Wright, Stephen, and Niu, Feng. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, pp. 693–701, 2011.
  25. 25.Vanhoucke, Vincent, Senior, Andrew, and Mao, Mark Z. Improving the speed of neural networks on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011.
  26. 26.Wu, Ren, Yan, Shengen, Shan, Yi, Dang, Qingqing, and Sun, Gang. Deep image: Scaling up image recognition. arXiv preprint arXiv:1501.02876, 2015.

Citation

MLA
Gupta, S., et al. “Deep Learning with Limited Numerical Precision”. arXiv, 2015, http://arxiv.org/abs/1502.02551v1.
APA
Gupta, S., Agrawal, A., Gopalakrishnan, K., & Narayanan, P. (2015). Deep Learning with Limited Numerical Precision. arXiv. http://arxiv.org/abs/1502.02551v1
Chicago
Gupta, S., A. Agrawal, K. Gopalakrishnan, and P. Narayanan. 2015. “Deep Learning with Limited Numerical Precision”. arXiv. http://arxiv.org/abs/1502.02551v1.
Harvard
Gupta, S. et al. (2015) “Deep Learning with Limited Numerical Precision”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1502.02551v1.
Vancouver
1. Gupta S, Agrawal A, Gopalakrishnan K, Narayanan P (2015) Deep Learning with Limited Numerical Precision. arXiv

BibTeX

@article{gupta2015deep,
  title = {Deep Learning with Limited Numerical Precision},
  author = {Gupta, Suyog and Agrawal, Ankur and Gopalakrishnan, Kailash and Narayanan, Pritish},
  year = {2015},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1502.02551v1},
  eprint = {1502.02551}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors