Communication-Efficient Adaptive Federated Learning

Yujia WangLu LinJinghui Chen

article2022ICML113 citations

Proposes FedCAMS, a communication-compressed adaptive federated learning algorithm that combines error-feedback compression with adaptive optimization to significantly reduce bandwidth overhead while maintaining the theoretical convergence rate of uncompressed methods.

Listen

Federated learning enables multiple edge devices, such as mobile phones and local servers, to collaboratively train machine learning models without transferring raw private data. However, deploying these systems in real-world settings faces two major roadblocks: the excessive communication bandwidth consumed by repeated model exchanges between devices and the central server, and poor training stability when applying standard gradient descent techniques to complex modern architectures. While previous solutions tackled either communication compression or adaptive optimization in isolation, combining the two without causing optimization divergence has remained an unsolved challenge.

The article develops and analyzes FedCAMS, an adaptive federated learning framework that simultaneously achieves communication compression and adaptive optimization while maintaining rigorous mathematical guarantees of convergence. The researchers also introduce an uncompressed foundational variant, FedAMS, which improves upon existing adaptive methods by incorporating a numerical max-stabilization mechanism and supporting momentum during model updates.

To evaluate this approach, the authors established a comprehensive theoretical framework analyzing convergence under standard non-convex optimization conditions for both full and partial client participation. They paired this theory with empirical simulations using standard image classification benchmarks across 100 decentralized clients, testing both conventional neural networks and modern patch-based architectures. The evaluation benchmarked FedCAMS and FedAMS against standard federated baseline algorithms across varying compression techniques, participation rates, and local training epochs.

The findings demonstrate four primary results in order of significance. First, FedCAMS matches the theoretical convergence rate of uncompressed adaptive federated methods while transmitting several orders of magnitude fewer data bits. Second, empirical tests confirm that FedCAMS—particularly when paired with a scaled sign compressor—drastically cuts network data transmission with almost no loss in final prediction accuracy. Third, FedAMS consistently outperforms baseline federated optimizers in both final training loss and model accuracy across tested neural network architectures. Finally, both theoretical proofs and empirical tests confirm that increasing the number of participating clients per training round consistently accelerates convergence.

These results demonstrate that organizations can deploy advanced, large-scale deep learning models across bandwidth-constrained edge networks without incurring massive communication costs or sacrificing model accuracy. The findings resolve prior optimization trade-offs where practitioners had to choose between network efficiency and model stability. Because standard federated averaging struggled significantly on modern patch-based architectures, adopting adaptive federated methods is essential for organizations transitioning to newer model designs.

For practical implementation, organizations deploying federated learning systems over constrained networks should adopt FedCAMS and utilize scaled sign compression, which demonstrated the most reliable trade-off between compression ratio and model accuracy. System designers should also configure training rounds to include as many participating devices as bandwidth allows to accelerate convergence speeds. However, before deploying across bidirectional systems, practitioners should note that the current analysis is limited to one-way compression from client devices to the central server. Future validation is needed to extend synchronization and compression guarantees to two-way server-to-client communications, particularly under partial client participation.

Pennsylvania State University University of Virginia communication overheads federated learning nonconvex Communication-efficient adaptive federated learning FedAMS Gradient compression

arXiv: 2205.02719 jinghuichen/FedCAMS

Paper: Adaptive Federated Optimization, Sashank Reddi et al. (2020). Read this foundational treatment of server-side adaptive optimizers in federated learning first; FedCAMS builds on the adaptive-optimization framework and FedAdam-family methods it establishes.
Paper: Federated Learning: Strategies for Improving Communication Efficiency, Jakub Konečný et al. (2016). This early study lays out structured and sketched update compression for federated learning, the communication-reduction techniques that frame FedCAMS’s contribution.
Paper: Communication-Efficient Learning of Deep Networks from Decentralized Data, H. B. McMahan et al. (2016). Its introduction of FedAvg establishes the local-training and server-synchronization baseline that FedCAMS modifies to reduce communication while adding adaptivity.

Paper: DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning, Robert Hönig et al. (2022). DAdaQuant continues the search for communication-efficient federated training by adaptively varying quantization precision across clients and training rounds.
Paper: FedNL: Making Newton-Type Methods Applicable to Federated Learning, Mher Safaryan et al. (2022). FedNL extends communication-efficient federated optimization beyond adaptive first-order updates, using compressed curvature information to make Newton-type methods practical.

Cover for Communication-Efficient Adaptive Federated Learning

Abstract

Federated learning is a machine learning training paradigm that enables clients to jointly train models without sharing their own localized data. However, the implementation of federated learning in practice still faces numerous challenges, such as the large communication overhead due to the repetitive server-client synchronization and the lack of adaptivity by SGD-based model updates. Despite that various methods have been proposed for reducing the communication cost by gradient compression or quantization, and the federated versions of adaptive optimizers such as FedAdam are proposed to add more adaptivity, the current federated learning framework still cannot solve the aforementioned challenges all at once. In this paper, we propose a novel communication-efficient adaptive federated learning method (FedCAMS) with theoretical convergence guarantees. We show that in the nonconvex stochastic optimization setting, our proposed FedCAMS achieves the same convergence rate of O(1/(√(T K m))) as its non-compressed counterparts. Extensive experiments on various benchmarks verify our theoretical analysis.

Topics

communication overheads federated learning nonconvex optimization quantization stochastic stochastic optimization application servers privacy preserving

Keywords

Communication-efficient adaptive federated learning FedAMS Gradient compression Stochastic gradient descent Partial participation Non-convex optimization

Knowls

Knowl 1 — FedCAMS combines local SGD, error-feedback compression, and adaptive server updates
algorithm
FedCAMS minimizes the federated objective $f(x)=\frac{1}{m}\sum_{i=1}^{m}F_i(x)$ , where $x\in\mathbb{R}^d$ is the global model and $F_i$ is client $i$ ’s loss. It uses $K$ local SGD steps per round, a compressor $C$ , client-specific residuals $e_t^i$ , and an AMSGrad-style server update. The server broadcasts the model without compression; participating clients upload compressed model differences. A client not selected in a round keeps its residual unchanged.
Input: Initial model $x_1$ , local step size $\eta_l$ , server step size $\eta$ , local steps $K$ , rounds $T$ , momentum parameters $\beta_1,\beta_2$ , stabilization value $\epsilon>0$ , compressor $C$
Initialize $m_0=0$ , $v_0=0$ , and $e_1^i=0$ for every client $i$
for $t=1,\ldots,T$ do
    Randomly select the participating client set $S_t$
    Broadcast $x_t$ to clients in $S_t$
    for each client $i\in S_t$ in parallel do
        Set $x_{t,0}^i=x_t$
        for $k=0,\ldots,K-1$ do
            Draw a local stochastic gradient $g_{t,k}^i$ at $x_{t,k}^i$
            Set $x_{t,k+1}^i=x_{t,k}^i-\eta_l g_{t,k}^i$
        end for
        Set $\Delta_t^i=x_{t,K}^i-x_t$
        Set $\widehat{\Delta}_t^i=C(\Delta_t^i+e_t^i)$ and upload $\widehat{\Delta}_t^i$
        Set $e_{t+1}^i=\Delta_t^i+e_t^i-\widehat{\Delta}_t^i$
    end for
    for each client $j\notin S_t$ do
        Set $e_{t+1}^j=e_t^j$
    end for
    Set $\widehat{\Delta}_t=\frac{1}{|S_t|}\sum_{i\in S_t}\widehat{\Delta}_t^i$
    Set $m_t=\beta_1m_{t-1}+(1-\beta_1)\widehat{\Delta}_t$
    Set $v_t=\beta_2v_{t-1}+(1-\beta_2)\widehat{\Delta}_t^2$
    Set $\widehat{v}_t=\max(\widehat{v}_{t-1},v_t,\epsilon)$ coordinatewise
    Set $x_{t+1}=x_t+\eta m_t/\sqrt{\widehat{v}_t}$ coordinatewise
end for
Output: $x_{T+1}$
The residual records compression error so that information omitted in one upload can be included in a later one. For clients not selected, retaining the residual makes this mechanism compatible with partial participation. The paper’s convergence analysis for error feedback introduces an auxiliary sequence to control residual effects rather than applying the uncompressed analysis directly.
Knowl 2 — FedAMS stabilizes adaptive federated updates by taking a coordinatewise maximum
model/method
FedAMS first aggregates client model differences into a server-side pseudo-gradient $\Delta_t$ . It updates first and second moments coordinatewise as $m_t=\beta_1m_{t-1}+(1-\beta_1)\Delta_t$ and $v_t=\beta_2v_{t-1}+(1-\beta_2)\Delta_t^2$ , where $\beta_1,\beta_2\in[0,1)$ and squares are elementwise. Its max-stabilized update has two variants:

$\widehat v_t=\max(\widehat v_{t-1},v_t,\epsilon),\qquad x_{t+1}=x_t+\eta\,m_t/\sqrt{\widehat v_t},$

or

$\widehat v_t=\max(\widehat v_{t-1},v_t),\qquad x_{t+1}=x_t+\eta\,m_t/(\sqrt{\widehat v_t}+\epsilon).$

Here $x_t,m_t,v_t,\widehat v_t\in\mathbb{R}^d$ , $\eta>0$ is the server step size, and $\epsilon>0$ stabilizes the denominator. Maxima, division, and square roots are coordinatewise. The first variant puts $\epsilon$ inside the maximum and therefore floors only coordinates whose accumulated variance is small; the second is the AMSGrad-style update with $\epsilon$ added after taking the square root. The paper’s main convergence results are stated for the first variant.
Knowl 3 — FedAMS converges at the full-participation nonconvex rate with momentum
theoretical result
Consider $f(x)=\frac{1}{m}\sum_{i=1}^{m}F_i(x)$ with all $m$ clients participating in every round. Suppose each client loss is $L$ -smooth, stochastic gradients have norm at most $G$ , local stochastic-gradient variance is at most $\sigma_l^2$ , and client heterogeneity satisfies $\frac{1}{m}\sum_i\|\nabla F_i(x)-\nabla f(x)\|^2\leq\sigma_g^2$ . For FedAMS with $K$ local SGD steps per round, $\beta_1,\beta_2\in[0,1)$ , and stabilization value $\epsilon>0$ , the paper proves a convergence bound on $\min_{t\leq T}\mathbb{E}\|\nabla f(x_t)\|^2$ under a sufficiently small local step size. In particular, with $C_1=\beta_1/(1-\beta_1)$ , the stated sufficient condition is

$\eta_l\leq\min\!\left\{\frac{1}{8KL},\frac{\epsilon}{K\sqrt{\beta_2K^2G^2+\epsilon}\big((3+C_1^2)\eta L+2\sqrt{2(1-\beta_2)}G\big)}\right\}.$

Choosing server step size $\eta=\Theta(\sqrt{Km})$ and local step size $\eta_l=\Theta(1/\sqrt{TK})$ , for sufficiently large $T$ (the paper specifies $T\geq Km$ ), gives $\min_{t\leq T}\mathbb{E}\|\nabla f(x_t)\|^2=O(1/\sqrt{TKm})$ . This analysis covers positive momentum $\beta_1$ , unlike the FedAdam analysis discussed by the authors, which they describe as restricted to $\beta_1=0$ .
Knowl 4 — FedAMS partial participation has a heterogeneity-sensitive convergence rate
theoretical result
For the objective $f(x)=\frac{1}{m}\sum_{i=1}^{m}F_i(x)$ , suppose each round samples $n$ of the $m$ clients uniformly without replacement and averages their local model differences. Under client-loss $L$ -smoothness, stochastic-gradient norm bound $G$ , local gradient variance at most $\sigma_l^2$ , and client heterogeneity bound $\frac{1}{m}\sum_i\|\nabla F_i(x)-\nabla f(x)\|^2\leq\sigma_g^2$ , FedAMS has a convergence bound on $\min_{t\leq T}\mathbb{E}\|\nabla f(x_t)\|^2$ when the local step size is sufficiently small. With $\eta=\Theta(\sqrt{Kn})$ and $\eta_l=\Theta(1/\sqrt{TK})$ , the paper gives the rate $O(\sqrt{K/(Tn)})$ .

The bound includes extra sampling-variance terms compared with full participation. The authors identify the term involving $\sigma_g^2$ as particularly important: client heterogeneity can therefore have a stronger effect under partial participation, while increasing the number $n$ of participating clients improves the stated rate. The result concerns uniform sampling without replacement.
Knowl 5 — Biased compressors covered by FedCAMS include top-$k$ and scaled-sign
definition
FedCAMS assumes a compressor $C:\mathbb{R}^d\to\mathbb{R}^d$ for which, for every vector $u$ , $\mathbb{E}\|C(u)-u\|\leq q\|u\|$ for some $q\in[0,1]$ . The expectation allows randomized compressors; $q=0$ corresponds to no compression. The paper identifies top- $k$ sparsification and scaled-sign compression as examples.

For top- $k$ , retain the $k$ coordinates of $u$ with largest magnitudes and set the others to zero. With compression ratio $r=k/d$ , it satisfies $\|C(u)-u\|^2\leq(1-r)\|u\|^2$ , so one may take $q=\sqrt{1-r}$ . For scaled-sign compression, $C(u)=(\|u\|_1/d)\operatorname{sign}(u)$ , and the paper gives $q=\sqrt{1-\|u\|_1^2/(d\|u\|^2)}$ for nonzero $u$ . These biased operators can be used with FedCAMS’s error-feedback residuals; the paper contrasts this with direct compression approaches that generally require an unbiased compressor.
Knowl 6 — FedCAMS preserves the full-participation rate for fixed compression strength
theoretical result
For $f(x)=\frac{1}{m}\sum_{i=1}^{m}F_i(x)$ with all clients participating each round, assume client losses are $L$ -smooth, stochastic gradients have norm at most $G$ , local and global variances are bounded by $\sigma_l^2$ and $\sigma_g^2$ , respectively, and the compressor satisfies $\mathbb{E}\|C(u)-u\|\leq q\|u\|$ for $q<1$ . FedCAMS uses $K$ local SGD steps, error feedback, and max stabilization. Write $C_1=\frac{\beta_1}{1-\beta_1}+\frac{2q}{1-q^2}$ and $C_2=\frac{\beta_1^2}{(1-\beta_1)^2}+\frac{4q^2}{(1-q^2)^2}$ . A sufficient local-step-size condition in the theorem is

$\eta_l\leq\min\left\{\frac{1}{8KL},\frac{\epsilon}{K C_{\beta,q}\big((3+2C_2)\eta L+2\sqrt{2(1-\beta_2)}G\big)}\right\},\qquad C_{\beta,q}=\sqrt{\frac{4\beta_2(1+q^2)^3}{3(1-q^2)^2}K^2G^2+\epsilon}.$

Under these conditions, the theorem bounds $\min_{t\leq T}\mathbb{E}\|\nabla f(x_t)\|^2$ by a term proportional to $\sqrt{\frac{4\beta_2(1+q^2)^3}{3(1-q^2)^2}\eta_l^2K^2G^2+\epsilon}$ times the optimization and variance terms. With $\eta=\Theta(\sqrt{Km})$ and $\eta_l=\Theta(1/\sqrt{TK})$ , the resulting rate for fixed $q<1$ is $O(1/\sqrt{TKm})$ , matching the rate of uncompressed FedAMS. The paper notes that the compression-dependent constants worsen as $q$ approaches $1$ .
Knowl 7 — FedCAMS has a partial-participation bound with additional sampling and compression terms
theoretical result
Let $m$ clients optimize $f(x)=\frac{1}{m}\sum_{i=1}^{m}F_i(x)$ , with $n$ clients sampled uniformly without replacement each round. Assume each loss is $L$ -smooth, stochastic gradients have norm at most $G$ , local and global variances are bounded by $\sigma_l^2$ and $\sigma_g^2$ , and the biased compressor satisfies $\mathbb{E}\|C(u)-u\|\leq q\|u\|$ , with $q<1$ . For FedCAMS with $K$ local steps, define

$A=\frac{4\beta_2(1+q^2)^3}{3(1-q^2)^2}\eta_l^2K^2G^2+\epsilon,\qquad C_1=\frac{\beta_1}{1-\beta_1}+\frac{2q}{1-q^2},\qquad D=\eta L+\sqrt{2(1-\beta_2)}G.$

The theorem’s local-step-size condition is

$\eta_l\leq\min\left\{\frac{1}{8KL},\frac{n(m-1)\epsilon}{48m(n-1)K\sqrt{\frac{4\beta_2(1+q^2)^3}{3(1-q^2)^2}K^2G^2+\epsilon}\,D}\right\}.$

Under this condition, the paper establishes

$\min_{t\leq T}\mathbb{E}\|\nabla f(x_t)\|^2\leq 8\sqrt{A}\left[\frac{f(x_1)-f_*}{\eta\eta_lKT}+\frac{\Psi}{T}+\Phi\right],$

where $f_*$ is a lower bound on the objective, $\Psi=C_1G^2\sqrt{d/\epsilon}+2C_1^2\eta\eta_lKLG^2d/\epsilon$ , and

$\Phi=\frac{C_1\eta\eta_lKLG^2}{\epsilon}+\frac{5\eta_l^2KL^2}{\sqrt{2\epsilon}}(\sigma_l^2+6K\sigma_g^2)+\frac{D\eta_l\sigma_l^2}{n\epsilon}+\frac{D\eta_l(m-n)}{n(m-1)\epsilon}\left[15K^2L^2\eta_l^2(\sigma_l^2+6K\sigma_g^2)+3K\sigma_g^2\right].$

Here $d$ is the parameter dimension and $\beta_1,\beta_2\in[0,1)$ . Relative to full participation, the bound contains additional terms from client sampling; compression strength also enters through $q$ and $C_1$ .
Knowl 8 — FedAMS improves benchmark performance over several federated optimizers
empirical result
The experiments used 100 clients, with 10 selected per round (a 0.1 participation ratio), three local epochs per round, and batch size 20. The evaluated image datasets were CIFAR-10 and CIFAR-100; models were ResNet-18 and ConvMixer-256-8. Training loss and test accuracy were tracked over 500 global rounds.

Across the CIFAR-10 and CIFAR-100 comparisons, FedAMS generally achieved the best final training loss and test accuracy among the tested adaptive methods and FedAvg. On ResNet-18, FedAMS and FedYogi performed similarly and better overall than FedAdam and FedAMSGrad; FedAvg sometimes had lower training loss than those two baselines while reaching test accuracy closer to FedAMS and FedYogi. On ConvMixer-256-8, the adaptive methods consistently outperformed FedAvg in both metrics, with FedAMS reported as best. Separate CIFAR-10 curves showed faster convergence as the number of participating clients increased from 5 to 10 to 20. Increasing local epochs from 3 to 10, 30, or 100 sped up training-loss convergence, but did not yield a clear test-accuracy advantage.
Knowl 9 — FedCAMS reduces communication while retaining accuracy, with results depending on model and compressor
empirical result
On CIFAR-10, the authors compared FedCAMS using scaled-sign compression and top- $k$ compression at ratios $r\in\{1/64,1/128,1/256\}$ against uncompressed FedAMS, measuring training loss and test accuracy against both rounds and client-to-server communication bits. For ResNet-18, scaled-sign and top- $k$ with $r=1/64$ gave similar test-accuracy behavior and the best reported trade-off between accuracy and communication. More aggressive top- $k$ compression reduced communication but slowed convergence. For ConvMixer-256-8, scaled-sign FedCAMS reached roughly the training loss and test accuracy of FedAMS with substantially less communication, while top- $k$ variants performed worse; among those variants, $r=1/64$ performed best but used more bits than the more compressed settings.

For the ResNet-18, CIFAR-10, 500-round communication comparison, the paper reports these approximate bit counts as uncompressed / one-way compression / two-way compression: scaled-sign, $3.58\times10^{11}/1.84\times10^{11}/1.12\times10^{10}$ ; top- $k$ with $r=1/64$ , the same values; top- $k$ with $r=1/128$ , $3.58\times10^{11}/1.82\times10^{11}/5.59\times10^9$ ; and top- $k$ with $r=1/256$ , $3.58\times10^{11}/1.80\times10^{11}/2.79\times10^9$ . The one-way figures compress client uploads while retaining uncompressed server broadcasts; two-way figures are communication estimates, not the setting analyzed in the main convergence results. The paper also states that a $d$ -dimensional scaled-sign message costs $32+d$ bits and that its cost is roughly comparable to top- $k$ at $r=1/64$ .
Knowl 10 — The convergence analysis does not cover two-way compression under partial participation
limitation
FedCAMS’s convergence analysis compresses client-to-server communication while leaving server-to-client broadcasts uncompressed. The authors state that extending the analysis to two-way compression is nontrivial, particularly with partial participation: biased compression and error feedback can make it difficult to guarantee that clients’ copies of the global model remain synchronized. They note that compressing broadcasts is straightforward in the full-participation setting, but leave the partial-participation synchronization problem and its analysis for future work.

Coverage note — The paper’s detailed proof steps and supporting lemmas are omitted because they serve the stated convergence theorems rather than constituting standalone contributions; ancillary hyperparameter searches and the epsilon ablation are also omitted because they are not central to reconstructing the main method or findings.

References

1.Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. Qsgd: Communication-efficient sgd via gradient quantization and encoding. Advances in Neural Information Processing Systems, 30:1709–1720, 2017.
2.Basu, D., Data, D., Karakus, C., and Diggavi, S. Qsparse-local-sgd: Distributed sgd with quantization, sparsification, and local computations. arXiv preprint arXiv:1906.02367, 2019.
3.Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560–569. PMLR, 2018.
4.Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
5.Chen, C., Shen, L., Huang, H., and Liu, W. Quantized adam with error feedback. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1–26, 2021a.
6.Chen, J., Zhou, D., Tang, Y., Yang, Z., Cao, Y., and Gu, Q. Closing the generalization gap of adaptive gradient methods in training deep neural networks. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2020a.
7.Chen, M., Shlezinger, N., Poor, H. V., Eldar, Y. C., and Cui, S. Communication-efficient federated learning. Proceedings of the National Academy of Sciences, 118(17), 2021b.
8.Chen, X., Liu, S., Sun, R., and Hong, M. On the convergence of a class of adam-type algorithms for non-convex optimization. arXiv preprint arXiv:1808.02941, 2018.
9.Chen, X., Li, X., and Li, P. Toward communication efficient adaptive gradient method. In Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference, pp. 119–128, 2020b.
10.Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
11.Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
12.Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
13.Ghosh, A., Hong, J., Yin, D., and Ramchandran, K. Robust federated learning in a heterogeneous environment. arXiv preprint arXiv:1906.06629, 2019.
14.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
15.Haddadpour, F., Kamani, M. M., Mokhtari, A., and Mahdavi, M. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics, pp. 2350–2358. PMLR, 2021.
16.He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
17.Hsu, T.-M. H., Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.
18.Jhunjhunwala, D., Gadhikar, A., Joshi, G., and Eldar, Y. C. Adaptive quantization of model updates for communication-efficient federated learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3110–3114. IEEE, 2021.
19.Jin, R., Huang, Y., He, X., Dai, H., and Wu, T. Stochastic-sign sgd for federated learning with theoretical guarantees. arXiv preprint arXiv:2002.10940, 2020.
20.Karimireddy, S. P., Rebjock, Q., Stich, S., and Jaggi, M. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pp. 3252–3261. PMLR, 2019.
21.Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132–5143. PMLR, 2020.
22.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
23.Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
24.Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
25.Li, D. and Wang, J. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
26.Li, T., Sanjabi, M., Beirami, A., and Smith, V. Fair resource allocation in federated learning. arXiv preprint arXiv:1905.10497, 2019a.
27.Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60, 2020.
28.Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189, 2019b.
29.Lin, T., Stich, S. U., Patel, K. K., and Jaggi, M. Don’t use large mini-batches, use local sgd. arXiv preprint arXiv:1808.07217, 2018.
30.Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
31.Luo, L., Xiong, Y., Liu, Y., and Sun, X. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.09843, 2019.
32.McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273–1282. PMLR, 2017.
33.Nishio, T. and Yonetani, R. Client selection for federated learning with heterogeneous resources in mobile edge. In ICC 2019-2019 IEEE International Conference on Communications (ICC), pp. 1–7. IEEE, 2019.
34.Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
35.Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018.
36.Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaie, A., and Pedarsani, R. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics, pp. 2021–2031. PMLR, 2020.
37.Robbins, H. and Monro, S. A stochastic approximation method. The annals of mathematical statistics, pp. 400–407, 1951.
38.Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Fifteenth annual conference of the international speech communication association. Citeseer, 2014.
39.Stich, S. U. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018.
40.Stich, S. U. and Karimireddy, S. P. The error-feedback framework: Better rates for sgd with delayed gradients and compressed communication. arXiv preprint arXiv:1909.05350, 2019.
41.Stich, S. U., Cordonnier, J.-B., and Jaggi, M. Sparsified sgd with memory. arXiv preprint arXiv:1809.07599, 2018.
42.Tang, H., Gan, S., Awan, A. A., Rajbhandari, S., Li, C., Lian, X., Liu, J., Zhang, C., and He, Y. 1-bit adam: Communication efficient large-scale training with adam’s convergence speed. arXiv preprint arXiv:2102.02888, 2021.
43.Tieleman, T., Hinton, G., et al. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4 (2):26–31, 2012.
44.Tong, Q., Liang, G., and Bi, J. Effective federated adaptive gradient methods with non-iid decentralized data. arXiv preprint arXiv:2009.06557, 2020.
45.Trockman, A. and Kolter, J. Z. Patches are all you need? arXiv preprint arXiv:2201.09792, 2022.
46.Wang, J., Tantia, V., Ballas, N., and Rabbat, M. Slowmo: Improving communication-efficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643, 2019.
47.Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481, 2020.
48.Wang, Y., Lin, L., and Chen, J. Communication-compressed adaptive gradient method for distributed nonconvex optimization. In International Conference on Artificial Intelligence and Statistics, pp. 6292–6320. PMLR, 2022.
49.Yang, H., Fang, M., and Liu, J. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv preprint arXiv:2101.11203, 2021.
50.Yang, Z., Chen, M., Saad, W., Hong, C. S., and Shikh-Bahaei, M. Energy efficient federated learning over wireless communication networks. IEEE Transactions on Wireless Communications, 20(3):1935–1949, 2020.
51.Zeiler, M. D. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
52.Zhou, D., Chen, J., Cao, Y., Tang, Y., Yang, Z., and Gu, Q. On the convergence of adaptive gradient methods for non-convex optimization. arXiv preprint arXiv:1808.05671, 2018.

Citation

MLA

Wang, Y., et al. “Communication-Efficient Adaptive Federated Learning”. International Conference on Machine Learning, vol. 162, 2022, pp. 22802–38, https://proceedings.mlr.press/v162/wang22o.html.

APA

Wang, Y., Lin, L., & Chen, J. (2022). Communication-Efficient Adaptive Federated Learning. International Conference on Machine Learning, 162, 22802–22838. https://proceedings.mlr.press/v162/wang22o.html

Chicago

Wang, Y., L. Lin, and J. Chen. 2022. “Communication-Efficient Adaptive Federated Learning”. International Conference on Machine Learning 162: 22802–38. https://proceedings.mlr.press/v162/wang22o.html.

Harvard

Wang, Y., Lin, L. and Chen, J. (2022) “Communication-Efficient Adaptive Federated Learning”, International Conference on Machine Learning. PMLR, pp. 22802–22838. Available at: https://proceedings.mlr.press/v162/wang22o.html.

Vancouver

1. Wang Y, Lin L, Chen J (2022) Communication-Efficient Adaptive Federated Learning. In: International Conference on Machine Learning. PMLR, pp 22802–22838

BibTeX

@InProceedings{pmlr-v162-wang22o,
  title = 	 {Communication-Efficient Adaptive Federated Learning},
  author =       {Wang, Yujia and Lin, Lu and Chen, Jinghui},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {22802--22838},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/wang22o/wang22o.pdf},
  url = 	 {https://proceedings.mlr.press/v162/wang22o.html},
  abstract = 	 {Federated learning is a machine learning training paradigm that enables clients to jointly train models without sharing their own localized data. However, the implementation of federated learning in practice still faces numerous challenges, such as the large communication overhead due to the repetitive server-client synchronization and the lack of adaptivity by SGD-based model updates. Despite that various methods have been proposed for reducing the communication cost by gradient compression or quantization, and the federated versions of adaptive optimizers such as FedAdam are proposed to add more adaptivity, the current federated learning framework still cannot solve the aforementioned challenges all at once. In this paper, we propose a novel communication-efficient adaptive federated learning method (FedCAMS) with theoretical convergence guarantees. We show that in the nonconvex stochastic optimization setting, our proposed FedCAMS achieves the same convergence rate of $O(\frac{1}{\sqrt{TKm}})$ as its non-compressed counterparts. Extensive experiments on various benchmarks verify our theoretical analysis.}
}

Metadata:DOI registry

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Communication-Efficient Adaptive Federated Learning

Abstract

Topics

Keywords

Knowls

Knowl 1 — FedCAMS combines local SGD, error-feedback compression, and adaptive server updates

Knowl 2 — FedAMS stabilizes adaptive federated updates by taking a coordinatewise maximum

Knowl 3 — FedAMS converges at the full-participation nonconvex rate with momentum

Knowl 4 — FedAMS partial participation has a heterogeneity-sensitive convergence rate

Knowl 5 — Biased compressors covered by FedCAMS include top-$k$ and scaled-sign

Knowl 6 — FedCAMS preserves the full-participation rate for fixed compression strength

Knowl 7 — FedCAMS has a partial-participation bound with additional sampling and compression terms

Knowl 8 — FedAMS improves benchmark performance over several federated optimizers

Knowl 9 — FedCAMS reduces communication while retaining accuracy, with results depending on model and compressor

Knowl 10 — The convergence analysis does not cover two-way compression under partial participation

References

Citation

BibTeX

Source Code

Access the Paper