DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning

Robert HönigYiren ZhaoRobert Mullins

article2022ICML93 citations

Proposes DAdaQuant, a communication-efficient federated learning algorithm that dynamically adjusts parameter quantization levels across training rounds and individual clients to achieve up to 2.8× higher uplink compression without sacrificing accuracy.

Listen

Federated learning enables organizations to train artificial intelligence models across decentralized edge devices, such as smartphones and sensors, while keeping raw user data private. However, repeatedly transmitting large model updates over networks with restricted upload bandwidth creates severe communication bottlenecks, driving up energy consumption and prolonging training cycles. While parameter quantization reduces message size by rounding model numbers into discrete bins, conventional approaches apply a single, fixed precision level across all devices and throughout the entire training process, leaving substantial efficiency gains untapped.

The article introduces and evaluates DAdaQuant, a novel compression algorithm designed to drastically reduce client-to-server communication in federated learning without sacrificing model accuracy or convergence speed. The framework achieves this through doubly-adaptive quantization, dynamically adjusting precision over time and across individual participating clients.

To assess the method, the authors developed a baseline called Federated QSGD—adapting stochastic fixed-point quantization with difference coding and lossless compression for federated systems—and integrated DAdaQuant into standard federated optimization workflows. They evaluated performance across five diverse benchmark datasets and model architectures, including logistic regression, convolutional neural networks, and recurrent neural networks, across image and natural language tasks while simulating device compute heterogeneity.

The evaluation demonstrates that DAdaQuant significantly improves communication efficiency, outperforming the strongest non-adaptive quantization baselines by up to 2.8 times across various tasks while maintaining target accuracy. The client-adaptive component alone delivers substantial gains on imbalanced datasets by allocating higher precision to heavily weighted clients and coarser precision to smaller ones, matching analytical variance bounds. Concurrently, the time-adaptive component saves bandwidth by starting with coarse precision and progressively increasing resolution as training stabilizes, adding negligible computational overhead of approximately 1%.

These findings indicate that communication bottlenecks in federated learning can be significantly mitigated through dynamic precision scheduling rather than static compression schemes. By cutting total data transfer by orders of magnitude compared to uncompressed baselines, DAdaQuant enables organizations to reduce mobile data costs, lower energy footprints on edge devices, and accelerate distributed training cycles.

Engineering teams deploying federated learning systems should adopt doubly-adaptive quantization strategies for client-to-server updates, particularly in heterogeneous edge environments where client data sizes vary widely. Further work should explore applying DAdaQuant principles to emerging vector quantizers and evaluating end-to-end performance in real-world cellular deployments with unstable connectivity.

Confidence in these findings is supported by rigorous mathematical proofs of optimality for fixed-point quantization and consistent empirical results across diverse model architectures. Readers should note that bandwidth savings depend partly on local dataset size variations, and tasks requiring high initial precision may require conservative starting quantization settings to prevent early convergence slowdowns.

ETH Zurich University of Cambridge client server communication overheads federated learning DAdaQuant doubly-adaptive quantization communication-efficient federated learning

arXiv: 2111.00465

Paper: Federated Learning: Strategies for Improving Communication Efficiency, Jakub Konečný et al. (2016). This foundational paper introduces communication-reduction strategies for federated learning, including structured updates and probabilistic quantization, providing essential background for adaptive quantization schemes.
Paper: QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding, Dan Alistarh et al. (2016). It establishes the theoretical principles and convergence guarantees of randomized gradient quantization and encoding in distributed SGD that underpin federated model compression.
Paper: Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data, Felix Sattler et al. (2019). It analyzes the vulnerability of standard gradient quantization methods under non-IID client distributions, motivating the need for robust, client-adaptive compression protocols.
Paper: signSGD: compressed optimisation for non-convex problems, Jeremy Bernstein et al. (2018). It analyzes 1-bit sign-based gradient compression and theoretical convergence in distributed non-convex optimization, serving as a core non-adaptive compression baseline.
Paper: Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training, Yujun Lin et al. (2018). It details gradient compression and error-feedback techniques to reduce communication bandwidth during distributed neural network training.
Paper: Communication-Efficient Learning of Deep Networks from Decentralized Data, H. B. McMahan et al. (2016). It establishes the foundational FederatedAveraging framework and defines the communication-efficiency challenges in decentralized optimization across edge clients.

Paper: FedNL: Making Newton-Type Methods Applicable to Federated Learning, Mher Safaryan et al. (2022). It extends communication-efficient federated learning to second-order Newton-type methods by compressing curvature matrices and adapting to heterogeneous edge clients.

Cover for DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning

Abstract

Federated Learning (FL) is a powerful technique to train a model on a server with data from several clients in a privacy-preserving manner. FL incurs significant communication costs because it repeatedly transmits the model between the server and clients. Recently proposed algorithms quantize the model parameters to efficiently compress FL communication. We find that dynamic adaptations of the quantization level can boost compression without sacrificing model quality. We introduce DAdaQuant as a doubly-adaptive quantization algorithm that dynamically changes the quantization level across time and different clients. Our experiments show that DAdaQuant consistently improves client→server compression, outperforming the strongest non-adaptive baselines by up to 2.8×.

Topics

client server communication overheads federated learning privacy preserving quantization

Keywords

DAdaQuant doubly-adaptive quantization communication-efficient federated learning client-adaptive quantization time-adaptive quantization quantization-based compression quantization error lossless compression

1. Introduction
2. Related Work
3. The DAdaQuant method
3.1. Federated Learning
3.2. Federated Averaging (FedAvg)
3.3. Quantization with Federated QSGD
3.4. Time-adaptive quantization
3.5. Client-adaptive quantization
3.6. Doubly-adaptive quantization (DAdaQuant)
4. Experiments
4.1. Experimental details
4.2. Results
5. Conclusion
6. Reproducibility Statement
References
A. Additional simulation details and experiments
A.1. Additional simulation details
A.2. Additional communication-accuracy trade-off curves
A.3. Computational overhead of DAdaQuant
A.4. Additional UVeQFed experiments
A.5. Additional AdaQuantFL experiments
B. Proofs
B.1. Proof of Theorem 1

Knowls

Knowl 1 — Doubly-Adaptive Quantization (DAdaQuant) Algorithm
algorithm
DAdaQuant (Doubly-Adaptive Quantization) combines time-adaptive base quantization level adjustments with client-adaptive quantization level allocations across participating clients to minimize uplink communication in federated learning without sacrificing model accuracy.
Input: Set of $N$ clients $C = \{c_1, \dots, c_N\}$ , initial parameters $p_0$ , rounds $T$ , sampled clients per round $K$ , local epochs $E$ , learning rate $\eta$ , minimum quantization level $q_{\min}$ , maximum quantization level $q_{\max}$ , check window $\phi$ , smoothing factor $\psi \in (0, 1)$
Output: Final model parameters $p_T$
Initialize client weights $w_i \leftarrow \frac{|D_i|}{\sum_{j=1}^N |D_j|}$ for all $i \in \{1, \dots, N\}$
Initialize base quantization level $q_0 \leftarrow q_{\min}$ , loss moving average $\hat{\bar{G}}_0 \leftarrow 0$
for round $t = 0, 1, \dots, T-1$ do
    Uniformly sample a subset $S_t \subset C$ of $K$ clients
    if $t = 0$ then
         $q_t \leftarrow q_{\min}$
    else if $t > \phi$ and $\hat{\bar{G}}_{t-1} \ge \hat{\bar{G}}_{t-\phi}$ and $2 q_{t-1} < q_{\max}$ and $q_{t-1} = q_{t-\phi}$ then
         $q_t \leftarrow 2 q_{t-1}$
    else
         $q_t \leftarrow q_{t-1}$
    end if

     $a \leftarrow \sum_{j \in S_t} w_j^{2/3}$
     $b \leftarrow \sum_{j \in S_t} \frac{w_j^2}{q_t^2}$
    for each client $c_k \in S_t$ do
         $q_t^k \leftarrow \max\left(1, \text{round}\left( \sqrt{\frac{a}{b}} \cdot w_k^{2/3} \right)\right)$
        Send model parameters $p_t$ and quantization level $q_t^k$ to $c_k$
    end for

    for each client $c_k \in S_t$ in parallel do
        Evaluate local loss $\hat{G}_t^k \leftarrow \frac{1}{|D_k|} \sum_{d \in D_k} f_{p_t}(d)$
        Compute updated parameters $p_{t+1}^k$ via $E$ epochs of local optimization (SGD or FedProx) with learning rate $\eta$
        Compute quantized parameter difference $\Delta_t^k \leftarrow Q_{q_t^k}(p_{t+1}^k - p_t)$ via Federated QSGD
        Send $(\Delta_t^k, \hat{G}_t^k)$ to server
    end for

    Server updates global model: $p_{t+1} \leftarrow p_t + \sum_{k \in S_t} w_k \Delta_t^k$
    Compute current round loss estimate: $\hat{G}_t \leftarrow \sum_{k \in S_t} w_k \hat{G}_t^k$
    if $t = 0$ then
         $\hat{\bar{G}}_0 \leftarrow \hat{G}_0$
    else
         $\hat{\bar{G}}_t \leftarrow \psi \hat{\bar{G}}_{t-1} + (1 - \psi) \hat{G}_t$
    end if
end for
return $p_T$
The algorithm executes on a client-server architecture where client aggregation weights $w_i = |D_i| / \sum_j |D_j|$ reflect local dataset sizes. At each round, the server adjusts the global base quantization level $q_t$ based on moving average loss stagnation, determines client-specific quantization levels $q_t^k$ , and distributes them. Clients compute updates via local training, compress parameter differences using Federated QSGD, and transmit quantized updates along with evaluation losses back to the server.
Knowl 2 — Optimal Client-Adaptive Quantization Level Allocation
theoretical result
Let $K$ participating clients have model parameter vectors $p_1, \dots, p_K$ whose elements are independently and identically distributed according to a uniform distribution $\mathcal{U}[-t, t]$ for $t > 0$ , with corresponding aggregation weights $w_1, \dots, w_K > 0$ such that $\sum_{i=1}^K w_i = 1$ . Let $Q_q$ denote an unbiased stochastic fixed-point quantizer with quantization level $q$ , and define the accumulation quantization error as $e_p^{q_1\dots q_K} = |p - \sum_{i=1}^K w_i Q_{q_i}(p_i)|$ .

To minimize the total communication budget $Q = \sum_{i=1}^K q_i$ subject to maintaining the expected accumulation error variance equal to that of a static baseline quantization level $q$ :

$\min_{q_1, \dots, q_K} \sum_{i=1}^K q_i \quad \text{subject to} \quad \mathbb{E}_{p_1, \dots, p_K}\left[\operatorname{Var}(e_p^{q_1\dots q_K})\right] = \mathbb{E}_{p_1, \dots, p_K}\left[\operatorname{Var}(e_p^q)\right]$

the optimal real-valued quantization level $q_i$ for each client $i \in \{1, \dots, K\}$ is given analytically by:

$q_i = \sqrt{\frac{a}{b}} \cdot w_i^{2/3}$

where

$a = \sum_{j=1}^K w_j^{2/3} \quad \text{and} \quad b = \sum_{j=1}^K \frac{w_j^2}{q^2}.$

To ensure natural numbers for practical quantizers, the discrete client quantization levels are assigned as:

$q_i = \max\left(1, \operatorname{round}\left(\sqrt{\frac{a}{b}} \cdot w_i^{2/3}\right)\right).$
Knowl 3 — Expected Accumulation Variance of Stochastic Fixed-Point Quantization
theoretical result
Let client parameters $p_1, \dots, p_K$ have components sampled independently and uniformly from $\mathcal{U}[-t, t]$ for $t > 0$ . Let $Q_{q_i}$ be an unbiased stochastic fixed-point quantizer with integer quantization level $q_i \ge 1$ , which divides $[-t, t]$ into uniform intervals of width $s_i = t / q_i$ . The quantizer rounds a parameter $p_i$ to the nearest endpoints $c s_i$ and $(c + 1)s_i$ with probabilities $u_i / s_i$ and $b_i / s_i$ , where $b_i = \operatorname{rem}(p_i, s_i)$ and $u_i = s_i - b_i$ .

For weighted aggregation with client weights $w_1, \dots, w_K$ , the expected variance of the quantization error $e_p^{q_1\dots q_K} = |\sum_{i=1}^K w_i p_i - \sum_{i=1}^K w_i Q_{q_i}(p_i)|$ over the parameter distribution evaluates to:

$\mathbb{E}_{p_1, \dots, p_K}\left[\operatorname{Var}(e_p^{q_1\dots q_K})\right] = \frac{t^2}{6} \sum_{i=1}^K \frac{w_i^2}{q_i^2}.$

When all clients share a single static quantization level $q_i = q$ , this expected variance reduces to:

$\mathbb{E}_{p_1, \dots, p_K}\left[\operatorname{Var}(e_p^q)\right] = \frac{t^2}{6} \sum_{i=1}^K \frac{w_i^2}{q^2}.$
Knowl 4 — Federated QSGD Compression Scheme
model/method
Federated QSGD adapts the stochastic gradient quantization algorithm (QSGD) to compress uplink model parameter updates in federated learning via difference coding combined with lossless entropy encoding.

Given the current global model parameters $p_t$ and updated local parameters $p_{t+1}^k$ computed by client $c_k$ , the compression procedure operates as follows:
1. Difference Coding: Client $c_k$ computes the parameter update vector $\Delta_t^k = p_{t+1}^k - p_t$ .
2. Lossy Stochastic Quantization: The update vector is normalized by its Euclidean norm $\|\Delta_t^k\|_2$ . For a configured quantization level $q \in \mathbb{N}$ , the non-negative interval $[0, 1]$ is split into $q$ uniform bins. Each component $|\Delta_{t, j}^k| / \|\Delta_t^k\|_2$ falling in $[l/q, (l+1)/q]$ is stochastically rounded to $l/q$ with probability $1 - (|\Delta_{t, j}^k| / \|\Delta_t^k\|_2 - l/q) \cdot q$ or $(l+1)/q$ with the remaining probability. The signs and norm $\|\Delta_t^k\|_2$ are stored separately.
3. Zero Run-Length Encoding (Lossless): Consecutive sequences of zero-valued quantized bins are encoded into run lengths.
4. Elias $\omega$ Universal Coding (Lossless): The resulting run lengths and non-zero integer values are compressed into variable-length binary codes.
The server decodes the bitstreams to recover unbiased estimates $\Delta_t^k$ and updates the global parameters according to $p_{t+1} = p_t + \sum_{k \in S_t} w_k Q_q(p_{t+1}^k - p_t)$ .
Knowl 5 — Smoothed Loss-Driven Time-Adaptive Quantization
model/method
Time-adaptive quantization dynamically increases the quantization level across training rounds to exploit the property that early federated training rounds tolerate coarse quantization without degrading final model quality, whereas later convergence requires higher precision.

Because the exact global loss $G(p_t)$ cannot be evaluated without querying all federation clients, the server estimates the global loss in round $t$ from the sampled subset of clients $S_t$ ( $|S_t| = K$ ) using an exponentially smoothed running average:

$\hat{\bar{G}}_t = \begin{cases} \hat{G}_0, & t = 0 \\ \psi \hat{\bar{G}}_{t-1} + (1 - \psi) \hat{G}_t, & t > 0 \end{cases}$

where $\hat{G}_t = \sum_{k \in S_t} w_k F_k(p_t)$ is the weighted sample average of local client losses $F_k(p_t)$ , and $\psi \in (0, 1)$ is the smoothing factor (set to $\psi = 0.9$ ).

The base quantization level $q_t$ is updated monotonically according to:

$q_t = \begin{cases} q_{\min}, & t = 0 \\ 2 q_{t-1}, & t > \phi \text{ and } \hat{\bar{G}}_{t-1} \ge \hat{\bar{G}}_{t-\phi} \text{ and } 2 q_{t-1} < q_{\max} \text{ and } q_{t-1} = q_{t-\phi} \\ q_{t-1}, & \text{otherwise} \end{cases}$

where $\phi \in \mathbb{N}$ is a grace period (set to $1/10$ of the total training rounds) during which $q_t$ remains fixed to allow loss reductions to manifest in $\hat{\bar{G}}_t$ . For tasks requiring high initial precision to avoid early divergence, binary time adaptation is used with $q_{\min} = q_{\max} / 2$ .

Knowl 6 — Top-1 Accuracy and Communication Compression Performance Across Benchmarks

data/table

The table below compares Top-1 test accuracy difference (percentage relative to uncompressed training) and total client-to-server uplink communication across five federated benchmarks for baseline methods, Federated QSGD, DAdaQuant, and its standalone ablations DAdaQuant $_{\text{time}}$ and DAdaQuant $_{\text{clients}}$ .

Method	Synthetic	FEMNIST	Sent140	Shakespeare	CelebA
Uncompressed	$78.3 \pm 0.3\%$ (12.2 MB)	$77.7 \pm 0.4\%$ (132.1 GB)	$69.7 \pm 0.5\%$ (43.9 GB)	$49.9 \pm 0.3\%$ (267.0 MB)	$90.4 \pm 0.0\%$ (12.6 GB)
Federated QSGD	$-0.1 \pm 0.1\%$ ( $17\times$ )	$+0.7 \pm 0.5\%$ ( $2809\times$ )	$-0.0 \pm 0.5\%$ ( $90\times$ )	$-0.5 \pm 0.6\%$ ( $9.5\times$ )	$-0.1 \pm 0.1\%$ ( $648\times$ )
FP8	$+0.1 \pm 0.4\%$ ( $4.0\times / 0.23\times\times$ )	$-0.1 \pm 0.4\%$ ( $4.0\times / 0.00\times\times$ )	$-0.2 \pm 0.5\%$ ( $4.0\times / 0.04\times\times$ )	$-0.2 \pm 0.4\%$ ( $4.0\times / 0.42\times\times$ )	$+0.0 \pm 0.1\%$ ( $4.0\times / 0.01\times\times$ )
FedPAQ (FxPQ)	$-0.1 \pm 0.1\%$ ( $6.4\times / 0.37\times\times$ )	$+0.7 \pm 0.5\%$ ( $11\times / 0.00\times\times$ )	$-0.0 \pm 0.5\%$ ( $4.0\times / 0.04\times\times$ )	$-0.5 \pm 0.6\%$ ( $3.2\times / 0.34\times\times$ )	$-0.1 \pm 0.1\%$ ( $6.4\times / 0.01\times\times$ )
FxPQ + GZip	$-0.1 \pm 0.1\%$ ( $14\times / 0.82\times\times$ )	$+0.6 \pm 0.2\%$ ( $1557\times / 0.55\times\times$ )	$-0.0 \pm 0.6\%$ ( $71\times / 0.79\times\times$ )	$-0.5 \pm 0.6\%$ ( $9.3\times / 0.97\times\times$ )	$-0.1 \pm 0.2\%$ ( $494\times / 0.76\times\times$ )
UVeQFed	$-0.5 \pm 0.2\%$ ( $0.6\times / 0.03\times\times$ )	$-2.8 \pm 0.5\%$ ( $12\times / 0.00\times\times$ )	$+0.0 \pm 0.2\%$ ( $15\times / 0.16\times\times$ )	$-0.0 \pm 0.4\%$ ( $7.9\times / 0.83\times\times$ )	$-0.4 \pm 0.3\%$ ( $31\times / 0.05\times\times$ )
DAdaQuant	$-0.2 \pm 0.4\%$ ( $48\times / 2.81\times\times$ )	$+0.7 \pm 0.1\%$ ( $4772\times / 1.70\times\times$ )	$-0.1 \pm 0.4\%$ ( $108\times / 1.19\times\times$ )	$-0.6 \pm 0.5\%$ ( $21\times / 2.21\times\times$ )	$-0.1 \pm 0.1\%$ ( $775\times / 1.20\times\times$ )
DAdaQuant $_{\text{time}}$	$-0.1 \pm 0.5\%$ ( $37\times / 2.16\times\times$ )	$+0.8 \pm 0.2\%$ ( $4518\times / 1.61\times\times$ )	$-0.1 \pm 0.6\%$ ( $93\times / 1.03\times\times$ )	$-0.5 \pm 0.5\%$ ( $12\times / 1.29\times\times$ )	$-0.1 \pm 0.2\%$ ( $716\times / 1.10\times\times$ )
DAdaQuant $_{\text{clients}}$	$+0.0 \pm 0.3\%$ ( $26\times / 1.51\times\times$ )	$+0.7 \pm 0.4\%$ ( $3017\times / 1.07\times\times$ )	$+0.1 \pm 0.6\%$ ( $105\times / 1.16\times\times$ )	$-0.4 \pm 0.5\%$ ( $16\times / 1.67\times\times$ )	$-0.1 \pm 0.0\%$ ( $700\times / 1.08\times\times$ )

In the table, $p\times$ represents the compression factor relative to uncompressed communication, while $(q\times\times)$ represents the relative compression factor compared to static Federated QSGD. Results demonstrate that DAdaQuant achieves $1.19\times$ to $2.81\times$ higher compression than Federated QSGD (and up to $4772\times$ relative to uncompressed transmission) while maintaining final model accuracy within the experimental margin of error across all tasks.

Knowl 7 — Multiplicative Compression Synergy and Dataset Heterogeneity Correlation
empirical result
The overall compression factor achieved by DAdaQuant is approximately multiplicative with respect to the individual compression gains of its constituent modules, DAdaQuant $_{\text{time}}$ and DAdaQuant $_{\text{clients}}$ :
- On Synthetic: DAdaQuant $_{\text{time}}$ achieves $2.16\times\times$ and DAdaQuant $_{\text{clients}}$ achieves $1.51\times\times$ over Federated QSGD, yielding $2.81\times\times$ combined ( $2.16 \times 1.51 \approx 3.26$ ).
- On FEMNIST: DAdaQuant $_{\text{time}}$ achieves $1.61\times\times$ and DAdaQuant $_{\text{clients}}$ achieves $1.07\times\times$ , yielding $1.70\times\times$ combined ( $1.61 \times 1.07 \approx 1.72$ ).
- On Shakespeare: DAdaQuant $_{\text{time}}$ achieves $1.29\times\times$ and DAdaQuant $_{\text{clients}}$ achieves $1.67\times\times$ , yielding $2.21\times\times$ combined ( $1.29 \times 1.67 \approx 2.15$ ).
Furthermore, the compression benefit delivered by client-adaptive quantization (DAdaQuant $_{\text{clients}}$ ) is directly correlated with the coefficient of variation of the number of local client samples, defined as $c_v = \sigma / \mu$ where $\mu$ is the mean and $\sigma$ is the standard deviation of local dataset sizes:
- Datasets with high sample size heterogeneity exhibit large client-adaptive compression gains: Synthetic ( $c_v = 3.3$ , $1.51\times\times$ ) and Shakespeare ( $c_v = 1.7$ , $1.67\times\times$ ).
- Datasets with lower sample size variation yield moderate compression gains: FEMNIST ( $c_v = 0.4$ , $1.07\times\times$ ), Sent140 ( $c_v = 0.3$ , $1.16\times\times$ ), and CelebA ( $c_v = 0.3$ , $1.08\times\times$ ).
Knowl 8 — Scalability Advantage of DAdaQuant over AdaQuantFL
empirical result
Existing adaptive quantization algorithms such as AdaQuantFL require the exact global training loss to decide quantization adjustments, requiring model transmission to all $N$ clients in the federation in every round. As a consequence, AdaQuantFL's per-round uplink communication scales linearly with the total client population size $N$ .

In contrast, DAdaQuant evaluates its loss moving average estimate $\hat{\bar{G}}_t$ using only the sampled subset of $K$ clients per round ( $K \ll N$ ). Experiments on synthetic federated datasets with $N \in \{10, 100, 200, 400\}$ clients and $K = 10$ show that:
1. AdaQuantFL's per-round uplink communication grows linearly from $0.5$ KB (at $N=10$ ) to $>20$ KB (at $N=400$ ), whereas DAdaQuant's per-round communication remains constant at $\approx 0.5$ KB regardless of $N$ .
2. Although AdaQuantFL samples all $N$ clients per round, its convergence speed in terms of rounds is only slightly faster than DAdaQuant with $K=10$ , which fails to offset its linearly higher communication cost.
3. In a direct comparison on FEMNIST under partial client participation, AdaQuantFL required 50.5 MB of total uplink communication to achieve $78.7\%$ top-1 accuracy, whereas DAdaQuant $_{\text{time}}$ achieved $78.4\%$ accuracy with only 27.5 MB of total communication (a $1.84\times$ communication reduction).
Knowl 9 — Computational Overhead of DAdaQuant
empirical result
The computational runtime overhead introduced by DAdaQuant on participating edge clients and the coordinating server is approximately $1\%$ of total training time.

On the FEMNIST benchmark with a 2-layer CNN (6.6 million parameters), execution time profiling across one federated training round yields the following breakdown:
- Baseline Local Training (SGD optimization): 36.0 s per round ( $100\%$ )
- Time-Adaptive Quantization (Loss moving average computation and step logic): $< 1$ ms ( $0.00\%$ of training time)
- Client-Adaptive Quantization (Quantization level closed-form arithmetic per client): 0.17 s ( $0.47\%$ of training time)
- Federated QSGD Quantization and Encoding: 0.24 s ( $0.67\%$ of training time)
- Total Overhead: 0.41 s per round ( $1.14\%$ of local training time).
The minor overhead is dominated by an additional single evaluation pass over the local dataset per round to evaluate $F_k(p_t)$ for smoothed loss tracking.
Knowl 10 — Experimental Setup for Federated Quantization Evaluation
experimental setup
The experimental evaluation uses five federated learning tasks spanning synthetic data, computer vision, and natural language processing, simulated using PyTorch and the Flower federated learning framework:
1. Synthetic: Multinomial Logistic Regression (MLR, 610 parameters) on 10-class vectors in $\mathbb{R}^{60}$ with parameters $\alpha=1, \beta=1$ ; 30 clients, 9,600 total samples; trained for 500 rounds with learning rate $\eta=0.01$ and FedProx proximal weight $\mu=1.0$ .
2. FEMNIST (LEAF): 62-class handwritten character classification via a 2-layer CNN (6.6M parameters); 3,500 clients, 785,582 samples; 500 rounds, $\eta=0.003, \mu=1.0$ .
3. CelebA (LEAF): Binary facial smile detection via a 4-layer CNN (630,000 parameters with LayerNorm replacing BatchNorm); 9,343 clients, 200,288 samples; 500 rounds, $\eta=0.1, \mu=0.0$ .
4. Sent140 (LEAF): Binary sentiment analysis on Twitter text via a 2-layer LSTM (1.1M parameters); filtered to clients with $\ge 10$ samples (21,876 clients, 430,707 samples); 1000 rounds, $\eta=0.3, \mu=1.0$ .
5. Shakespeare (LEAF): Next-character prediction on dramatic text via a 2-layer LSTM (130,000 parameters); 1,129 clients, 4,226,158 samples; 50 rounds, $\eta=0.8, \mu=0.001$ .
General parameters: $K=10$ randomly sampled clients per round, local mini-batch size of 10, local train/test split of $80\% / 20\%$ . System compute heterogeneity is simulated by randomly selecting $90\%$ of sampled clients ( $|S_t'| = 0.9K$ ) in each round to run a reduced number of local epochs $E' \sim [1, \dots, E]$ .

Coverage note — Omitted the step-by-step mathematical proofs of intermediate lemmas (Lemmas 1, 2, 4) and Theorem 1, as well as secondary tuning experiments for baseline UVeQFed with coding rate R=1, in accordance with the extraction rules regarding proofs and non-primary baseline variations.

References

1.Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30, pp. 1709–1720. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/6768-qsgd-communication-efficient-sgd-via-gradient-quantization-and-encoding.pdf.
2.Amiri, M. M., Gunduz, D., Kulkarni, S. R., and Poor, H. V. Federated learning with quantized global model updates. arXiv:2006.10672, 2020.
3.Banner, R., Hubara, I., Hoffer, E., and Soudry, D. Scalable methods for 8-bit training of neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 5151–5159, 2018.
4.Beutel, D. J., Topal, T., Mathur, A., Qiu, X., Parcollet, T., and Lane, N. D. Flower: A friendly Federated Learning research framework. arXiv:2007.14390, 2020.
5.Bouacida, N., Hou, J., Zang, H., and Liu, X. Adaptive Federated Dropout: Improving communication efficiency and generalization for federated learning. In IEEE INFOCOM 2021-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 1–6. IEEE, 2021.
6.Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
7.Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. LEAF: A benchmark for federated settings. arXiv:1812.01097, 2018.
8.Cao, B., Zheng, L., Zhang, C., Yu, P. S., Piscitello, A., Zulueta, J., Ajilore, O., Ryan, K., and Leow, A. D. DeepMood: modeling mobile phone typing dynamics for mood detection. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 747–755, 2017.
9.Fu, Y., You, H., Zhao, Y., Wang, Y., Li, C., Gopalakrishnan, K., Wang, Z., and Lin, Y. FracTrain: Fractionally squeezing bit savings both temporally and spatially for efficient DNN training. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12127–12139. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/8dc5983b8c4ef1d8fcd5f325f9a65511-Paper.pdf.
10.Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P. Deep learning with limited numerical precision. In International conference on machine learning, pp. 1737–1746. PMLR, 2015.
11.Horváth, S., Kovalev, D., Mishchenko, K., Stich, S., and Richtárik, P. Stochastic distributed learning with gradient quantization and variance reduction. arXiv preprint arXiv:1904.05115, 2019.
12.Jhunjhunwala, D., Gadhikar, A., Joshi, G., and Eldar, Y. C. Adaptive quantization of model updates for communication-efficient federated learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3110–3114. IEEE, 2021.
13.Jiang, Y., Wang, S., Ko, B. J., Lee, W.-H., and Tassiulas, L. Model pruning enables efficient Federated Learning on edge devices. arXiv:1909.12326, 2019.
14.Li, H., De, S., Xu, Z., Studer, C., Samet, H., and Goldstein, T. Training quantized nets: A deeper understanding. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5813–5823, 2017.
15.Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. arXiv:1812.06127, 2018.
16.Malekijoo, A., Fadaeieslam, M. J., Malekijou, H., Homayounfar, M., Alizadeh-Shabdiz, F., and Rawassizadeh, R. FEDZIP: A compression framework for communication-efficient Federated Learning. arXiv:2102.01593, 2021.
17.McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017.
18.Nguyen, H. T., Sehwag, V., Hosseinalipour, S., Brinton, C. G., Chiang, M., and Poor, H. V. Fast-convergent federated learning. IEEE Journal on Selected Areas in Communications, 39(1):201–218, 2020.
19.Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
20.Qiu, X., Parcolle, T., Beutel, D. J., Topa, T., Mathur, A., and Lane, N. D. A first look into the carbon footprint of federated learning. arXiv preprint arXiv:2010.06537, 2020.
21.Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. FedPAQ: A communication-efficient Federated Learning method with periodic averaging and quantization. arXiv:1909.13014, 2019.
22.Rothchild, D., Panda, A., Ullah, E., Ivkin, N., Stoica, I., Braverman, V., Gonzalez, J., and Arora, R. FetchSGD: Communication-efficient federated learning with sketching. In International Conference on Machine Learning, pp. 8253–8265. PMLR, 2020.
23.Shen, J., Wang, Y., Xu, P., Fu, Y., Wang, Z., and Lin, Y. Fractional skipping: Towards finer-grained dynamic CNN inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5700–5708, 2020.
24.Shi, W. and Dustdar, S. The promise of edge computing. Computer, 49(5):78–81, 2016.
25.Shlezinger, N., Chen, M., Eldar, Y. C., Poor, H. V., and Cui, S. UVeQFed: Universal vector quantization for federated learning. IEEE Transactions on Signal Processing, 69:500–514, 2020.
26.Speedtest. Speedtest global index. https://www.speedtest.net/global-index. Accessed: 2021-05-12.
27.Sun, X., Choi, J., Chen, C.-Y., Wang, N., Venkataramani, S., Srinivasan, V., Cui, X., Zhang, W., and Gopalakrishnan, K. Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 4900–4909, 2019.
28.Voigt, P. and Von dem Bussche, A. The EU general data protection regulation (GDPR). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10: 3152676, 2017.
29.Wang, N., Choi, J., Brand, D., Chen, C.-Y., and Gopalakrishnan, K. Training deep neural networks with 8-bit floating point numbers. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 7675–7684. Curran Associates, Inc., 2018a. URL http://papers.nips.cc/paper/7994-training-deep-neural-networks-with-8-bit-floating-point-numbers.pdf.
30.Wang, P., Ye, F., and Chen, X. A smart home gateway platform for data collection and awareness. IEEE Communications magazine, 56(9):87–93, 2018b.

Citation

MLA

Hönig, R., et al. “DAdaQuant: Doubly-adaptive Quantization for Communication-efficient Federated Learning”. International Conference on Machine Learning, vol. 162, 2022, pp. 8852–66, https://proceedings.mlr.press/v162/honig22a.html.

APA

Hönig, R., Zhao, Y., & Mullins, R. (2022). DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning. International Conference on Machine Learning, 162, 8852–8866. https://proceedings.mlr.press/v162/honig22a.html

Chicago

Hönig, R., Y. Zhao, and R. Mullins. 2022. “DAdaQuant: Doubly-adaptive Quantization for Communication-efficient Federated Learning”. International Conference on Machine Learning 162: 8852–66. https://proceedings.mlr.press/v162/honig22a.html.

Harvard

Hönig, R., Zhao, Y. and Mullins, R. (2022) “DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning”, International Conference on Machine Learning. PMLR, pp. 8852–8866. Available at: https://proceedings.mlr.press/v162/honig22a.html.

Vancouver

1. Hönig R, Zhao Y, Mullins R (2022) DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning. In: International Conference on Machine Learning. PMLR, pp 8852–8866

BibTeX

@InProceedings{pmlr-v162-honig22a,
  title = 	 {{DA}da{Q}uant: Doubly-adaptive quantization for communication-efficient Federated Learning},
  author =       {H{\"o}nig, Robert and Zhao, Yiren and Mullins, Robert},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {8852--8866},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/honig22a/honig22a.pdf},
  url = 	 {https://proceedings.mlr.press/v162/honig22a.html},
  abstract = 	 {Federated Learning (FL) is a powerful technique to train a model on a server with data from several clients in a privacy-preserving manner. FL incurs significant communication costs because it repeatedly transmits the model between the server and clients. Recently proposed algorithms quantize the model parameters to efficiently compress FL communication. We find that dynamic adaptations of the quantization level can boost compression without sacrificing model quality. We introduce DAdaQuant as a doubly-adaptive quantization algorithm that dynamically changes the quantization level across time and different clients. Our experiments show that DAdaQuant consistently improves client$\rightarrow$server compression, outperforming the strongest non-adaptive baselines by up to $2.8\times$.}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Doubly-Adaptive Quantization (DAdaQuant) Algorithm

Knowl 2 — Optimal Client-Adaptive Quantization Level Allocation

Knowl 3 — Expected Accumulation Variance of Stochastic Fixed-Point Quantization

Knowl 4 — Federated QSGD Compression Scheme

Knowl 5 — Smoothed Loss-Driven Time-Adaptive Quantization

Knowl 6 — Top-1 Accuracy and Communication Compression Performance Across Benchmarks

Knowl 7 — Multiplicative Compression Synergy and Dataset Heterogeneity Correlation

Knowl 8 — Scalability Advantage of DAdaQuant over AdaQuantFL

Knowl 9 — Computational Overhead of DAdaQuant

Knowl 10 — Experimental Setup for Federated Quantization Evaluation

References

Citation

BibTeX

Access the Paper