Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

quantized neural networks

Quantized neural networks are artificial neural networks that employ low-precision numerical representations, such as reduced-bit integers or binary values, rather than standard full-precision floating-point numbers for parameters like weights and activations. By mapping continuous or high-precision values to a discrete set of lower-bit formats, these architectures significantly reduce memory footprint, storage requirements, and memory bandwidth demands. The reduced precision also allows computationally intensive floating-point arithmetic to be replaced by faster, more energy-efficient fixed-point or bitwise calculations during inference and training. Consequently, quantized neural networks enable the efficient execution of deep learning models on resource-constrained platforms, such as mobile and embedded edge devices, while maintaining task performance and accuracy comparable to their full-precision counterparts.

1 item

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio

OrganizationsColumbia UniversityTechnion – Israel Institute of TechnologyUniversité de Montréal

Why you should read this

Demonstrates how to train neural networks using low-precision weights and activations down to 1 bit, replacing standard arithmetic with fast bitwise operations to drastically reduce memory footprint and power consumption without sacrificing competitive accuracy.

We introduce a method to train Quantized Neural Networks (QNNs) --- neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At train-time the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves 51%51\% top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.

Added

2026-09-18