Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Felix SattlerKlaus-Robert MüllerWojciech Samek

article2019IEEE Transactions on Neural Networks and Learning Systems1,507 citations

Develops Clustered Federated Learning, a model-agnostic framework that groups clients by the geometry of their loss surfaces to train specialized deep neural networks on heterogeneous data without modifying standard federated communication protocols.

Listen

Federated learning has emerged as a leading framework for training machine learning models collaboratively across distributed devices without sharing private local data. However, standard federated learning assumes that a single global model can adequately serve all participating devices. In practical applications, this assumption often fails because local client data distributions can diverge significantly due to varying user behaviors, conflicting preferences, limited device capacity, or adversarial activity. When client data are fundamentally incongruent, forcing a single shared model results in compromised performance across the entire network.

The article introduces and evaluates clustered federated learning, a distributed multi-task optimization framework designed to address this problem. The primary objective is to demonstrate that a central server can automatically identify natural groupings of clients with mutually compatible data and train tailored, high-performing models for each group without compromising data privacy or modifying standard communication protocols.

To evaluate this framework, the authors combine mathematical proofs with experimental testing on benchmark image classification (MNIST and CIFAR-10) and text classification (Ag-News) datasets. The experiments simulate distributed networks of 20 clients grouped into four distinct data distributions, utilizing deep convolutional and recurrent neural networks. Rather than requiring the server to inspect client data, the proposed approach analyzes the geometric properties of the transmitted model weight updates at stationary convergence points, using cosine similarity metrics to detect conflicting training trajectories and recursively bipartition clients into optimal sub-clusters.

The findings confirm that clustered federated learning significantly outperforms traditional federated learning in heterogeneous environments. First, when client data distributions conflict, traditional federated learning plateaus at low accuracy—around 20% validation accuracy in image classification and a perplexity of 42 in language modeling. By identifying client clusters, clustered federated learning more than doubles image classification accuracy to nearly 60% and reduces language model perplexity to below 36. Second, the method achieves robust separation even with limited local data (as few as 20 samples per client on MNIST) and after only 10 communication rounds. Third, using accumulated weight updates instead of raw mathematical gradients yields superior separation quality without altering baseline communication protocols. Fourth, the framework successfully avoids splitting clients when their underlying data distributions are compatible, ensuring that performance is always equal to or better than standard federated learning.

These results demonstrate that organizations deploying distributed machine learning can achieve specialized model performance without incurring the excessive communication overhead or architectural modifications required by alternative multi-task methods. Because clustering operates as a top-down postprocessing mechanism, the framework can be deployed with encryption mechanisms that preserve client privacy and allows the server to construct hierarchical parameter trees to support dynamic client populations.

For deployment, the authors recommend adopting the dual-norm stopping criteria to determine exactly when a global model should be split into clusters, setting threshold parameters based on available computing budget and anticipated data heterogeneity. Organizations should also leverage parameter trees to seamlessly assign newly onboarded devices to the most appropriate cluster model. Future work should focus on integrating this framework with formal differential privacy protections and advanced model compression techniques.

While the theoretical guarantees and empirical results are strong, readers should note that the evaluation is conducted on simulated multi-task splits using standard benchmark datasets rather than live production networks. Confidence in the mathematical separation criteria is high, but real-world deployments involving extreme network latency or highly imbalanced client distributions should be validated through pilot implementations.

Fraunhofer Heinrich Hertz Institute Korea University Max Planck Institute for Informatics Technische Universität Berlin clustering quality communication protocols federated learning clustered federated learning distributed multitask optimization privacy constraints

arXiv: 1910.01991

Paper: Communication-Efficient Learning of Deep Networks from Decentralized Data, H. B. McMahan et al. (2016). Introduces the foundational FederatedAveraging algorithm and framework that the source adopts and extends with clustering techniques.
Paper: Federated Multi-Task Learning, Virginia Smith et al. (2017). Establishes the federated multi-task learning formulation that the source directly addresses and improves upon for non-convex neural networks.
Paper: Federated Learning with Non-IID Data, Yue Zhao et al. (2018). Analyzes weight divergence and severe accuracy drops in federated learning under non-IID client distributions, defining the core problem the source resolves.
Paper: Federated Optimization in Heterogeneous Networks, Tian Li et al. (2018). Provides fundamental convergence analysis and algorithmic modifications (FedProx) for optimization under non-IID federated data.
Paper: Federated Optimization: Distributed Machine Learning for On-Device Intelligence, Jakub Konečný et al. (2016). Formulates the core federated optimization problem and characteristics of distributed on-device intelligence.
Paper: LEAF: A Benchmark for Federated Settings, Sebastian Caldas et al. (2018). Introduces the standard federated datasets and benchmark suites used to empirically evaluate the source's clustered learning framework.

Paper: Personalized Federated Learning with Moreau Envelopes, Canh T. Dinh et al. (2020). Advances personalized federated learning under heterogeneous data through Moreau envelope bi-level optimization, extending beyond cluster-level specialization.
Paper: Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization, Jianyu Wang et al. (2020). Analyzes the objective inconsistency problem arising from client heterogeneity and introduces a normalized averaging framework (FedNova).
Paper: Model-Contrastive Federated Learning, Qinbin Li et al. (2021). Proposes model-contrastive learning to correct client drift on heterogeneous data by leveraging representation comparisons.
Paper: Adaptive Federated Optimization, Sashank Reddi et al. (2020). Develops adaptive optimization methods on the server side to accelerate convergence across heterogeneous federated client distributions.
Paper: Ensemble Distillation for Robust Model Fusion in Federated Learning, Tao Lin et al. (2020). Extends federated learning under heterogeneous client settings by using server-side ensemble knowledge distillation instead of parameter averaging.
Paper: Federated Learning on Non-IID Data Silos: An Experimental Study, Qinbin Li et al. (2021). Presents an extensive experimental benchmark analyzing diverse forms of non-IID data skews across advanced federated algorithms.
Paper: Inverting Gradients - How easy is it to break privacy in federated learning?, Jonas Geiping et al. (2020). Demonstrates gradient inversion attacks that evaluate the privacy bounds of sharing parameter updates in federated optimization.

Cover for Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Abstract

Federated Learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit it's popularity, it has been observed that Federated Learning yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present Clustered Federated Learning (CFL), a novel Federated Multi-Task Learning (FMTL) framework, which exploits geometric properties of the FL loss surface, to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general non-convex objectives (in particular deep neural networks) and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy preserving way. As clustering is only performed after Federated Learning has converged to a stationary point, CFL can be viewed as a post-processing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used Federated Learning datasets.

Topics

clustering quality communication protocols federated learning optimization recurrent neural networks multitask learning individual privacy

Keywords

clustered federated learning distributed multitask optimization privacy constraints differential privacy multitask learning nonconvex objectives information leakage local data distribution

I Introduction
II Cosine Similarity based Clustering
II-A Distinguishing Congruent and Incongruent Clients
III Clustered Federated Learning
IV Related Work
V Implementation Considerations
V-A Weight-Updates as generalized Gradients
V-B Preserving Privacy
V-C Varying Client Populations and Parameter Trees
VI Experiments
VI-A Practical Considerations
VI-B Distinguishing Congruent and Incongruent Clients
VI-C Clustered Federated Learning
VII Conclusion
References
VIII Supplement
VIII-A Proving the Separation Theorem

Knowls

Knowl 1 — Clustered Federated Learning Data Congruence Assumption
assumption
In federated learning with $M$ clients, let each client $i \in \{1, \dots, M\}$ have a local dataset $D_i$ drawn from a data generating distribution $\phi_i(x, y)$ , where $x \in \mathcal{X}$ and $y \in \mathcal{Y}$ . For a model $f_\theta: \mathcal{X} \to \mathcal{Y}$ parameterized by $\theta \in \Theta$ and a loss function $l: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}_{\ge 0}$ , the expected risk associated with distribution $\phi_i$ is:
$R_i(\theta) = \int l(f_\theta(x), y) \, d\phi_i(x, y)$
A set of clients is defined as congruent if there exists a parameter configuration $\theta^* \in \Theta$ that simultaneously locally minimizes the risk across all clients' distributions:
$R_i(\theta^*) \le \min_{\theta: \|\theta - \theta^*\| < \varepsilon} R_i(\theta), \quad \forall i = 1, \dots, M$
for some $\varepsilon > 0$ . If this condition is not met, the clients are termed incongruent, which occurs when the model capacity is insufficient to fit all distributions simultaneously or when clients possess conflicting conditional distributions $\phi_i(y|x) \ne \phi_j(y|x)$ .

The Clustered Federated Learning (CFL) Assumption generalizes this by postulating that there exists a partitioning of the client population $\mathcal{C} = \{c_1, \dots, c_K\}$ with $\dot{\bigcup}_{k=1}^K c_k = \{1, \dots, M\}$ such that every cluster subset $c \in \mathcal{C}$ is internally congruent.
Knowl 2 — Separation Theorem for Incongruent Client Distributions
theoretical result
Let $M$ clients hold local datasets $D_1, \dots, D_M$ sampled from $K$ distinct data generating distributions $\phi_1, \dots, \phi_K$ , with mapping $I: \{1, \dots, M\} \to \{1, \dots, K\}$ assigning client $i$ to distribution $\phi_{I(i)}$ . Let $r_i(\theta) = \frac{1}{|D_i|} \sum_{(x,y) \in D_i} l(f_\theta(x), y)$ denote client $i$ 's empirical risk, and let $\theta^*$ be a stationary solution of the global federated learning objective $F(\theta) = \sum_{i=1}^M \frac{|D_i|}{|D|} r_i(\theta)$ , where $D = \bigcup_{i=1}^M D_i$ .

Assume that on every client the empirical gradient approximates the true gradient such that:
$\|\nabla R_{I(i)}(\theta^*)\| > \|\nabla R_{I(i)}(\theta^*) - \nabla r_i(\theta^*)\|$
Define the relative approximation error $\gamma_i \in [0, 1)$ as:
$\gamma_i := \frac{\|\nabla R_{I(i)}(\theta^*) - \nabla r_i(\theta^*)\|}{\|\nabla R_{I(i)}(\theta^*)\|}$
and define:
$H_{i,j} := -\gamma_i \gamma_j + \sqrt{1 - \gamma_i^2}\sqrt{1 - \gamma_j^2} \in (-1, 1]$
Then, there exists a bipartitioning $c_1^* \;\dot{\cup}\; c_2^* = \{1, \dots, M\}$ such that the maximum cosine similarity $\alpha(u, v) = \frac{\langle u, v \rangle}{\|u\| \|v\|}$ between gradient updates of clients across different clusters is bounded from above by:
$\alpha_{\text{cross}}^{\max} := \max_{i \in c_1^*, j \in c_2^*} \alpha(\nabla r_i(\theta^*), \nabla r_j(\theta^*)) \le \begin{cases} \cos\left(\frac{\pi}{K-1}\right) H_{i,j} + \sin\left(\frac{\pi}{K-1}\right) \sqrt{1 - H_{i,j}^2} & \text{if } H_{i,j} \ge \cos\left(\frac{\pi}{K-1}\right) \\ 1 & \text{otherwise} \end{cases}$
Simultaneously, the minimum cosine similarity between updates from clients sharing the same data generating distribution is bounded from below by:
$\alpha_{\text{intra}}^{\min} := \min_{i, j: I(i) = I(j)} \alpha(\nabla r_i(\theta^*), \nabla r_j(\theta^*)) \ge \min_{i, j: I(i) = I(j)} H_{i,j}$
For $K=2$ and zero approximation noise ( $\gamma_i = 0$ ), $H_{i,j} = 1$ , yielding $\alpha_{\text{intra}}^{\min} = 1$ and $\alpha_{\text{cross}}^{\max} = -1$ .
Knowl 3 — Separation Gap and Guaranteed Correct Bipartitioning
theoretical result
Given a pairwise cosine-similarity matrix $\alpha \in [-1, 1]^{M \times M}$ where $\alpha_{i,j} = \frac{\langle \nabla r_i(\theta^*), \nabla r_j(\theta^*) \rangle}{\|\nabla r_i(\theta^*)\| \|\nabla r_j(\theta^*)\|}$ and distribution assignment mapping $I: \{1, \dots, M\} \to \{1, \dots, K\}$ , the separation gap $g(\alpha)$ is defined as:
$g(\alpha) := \alpha_{\text{intra}}^{\min} - \alpha_{\text{cross}}^{\max} = \min_{i, j: I(i)=I(j)} \alpha_{i,j} - \min_{c_1 \dot{\cup} c_2 = c} \left( \max_{i \in c_1, j \in c_2} \alpha_{i,j} \right)$
If the separation gap is strictly positive ( $g(\alpha) > 0$ ), then the bipartitioning obtained by solving the min-max similarity cut:
$c_1, c_2 \leftarrow \arg\min_{c_1 \dot{\cup} c_2 = c} \left( \max_{i \in c_1, j \in c_2} \alpha_{i,j} \right)$
is strictly correct, meaning that no two clients $i, j$ with the same data generating distribution ( $I(i) = I(j)$ ) are assigned to different clusters ( $i \in c_1, j \in c_2$ ).
Knowl 4 — Stationarity and Noise Thresholds for Federated Splitting Decisions
model/method
To prevent splitting congruent clients (which harms performance by restricting data sharing), Clustered Federated Learning checks whether a stationary solution $\theta^*$ of the federated learning objective is also stationary for individual clients.

A cluster $c \subseteq \{1, \dots, M\}$ is split if and only if both of the following conditions hold:
1. Global convergence of the cluster objective:
$\left\| \sum_{i \in c} \frac{|D_i|}{|D_c|} \nabla_\theta r_i(\theta^*) \right\| < \varepsilon_1$
1. Local non-stationarity of individual client risks:
$\max_{i \in c} \|\nabla_\theta r_i(\theta^*)\| > \varepsilon_2 > 0$
Furthermore, to ensure clustering correctness given a maximum expected approximation noise $\gamma_{\max} = \max_{i} \gamma_i \in [0, 1)$ , a computed bipartition is accepted only if:
$\gamma_{\max} < \sqrt{\frac{1 - \alpha_{\text{cross}}^{\max}}{2}}$
which mathematically guarantees $\alpha_{\text{intra}}^{\min} \ge 1 - 2\gamma_{\max}^2 > \alpha_{\text{cross}}^{\max}$ independently of the number of clusters $K$ .

Practical hyperparameter recommendations: $\varepsilon_1$ should be set to approximately $\frac{1}{10} \max_t \|\Delta \theta_c^t\|$ , where $\Delta \theta_c^t$ is the cluster server update at communication round $t$ , and $\varepsilon_2 \in [\varepsilon_1, 10\varepsilon_1]$ .
Knowl 5 — Agglomerative Optimal Bipartition Algorithm
algorithm
The optimal bipartitioning problem $\arg\min_{c_1 \dot{\cup} c_2 = c} (\max_{i \in c_1, j \in c_2} \alpha_{i, j})$ partition a set of $M$ clients into two clusters such that the maximum similarity between separated clients is minimized. It is solved in $O(M^3)$ time using an agglomerative merging procedure over sorted pairwise similarities.
Input: Similarity Matrix $\alpha \in [-1, 1]^{M \times M}$
Output: Bipartitioning $c_1, c_2$
$s \leftarrow \operatorname{argsort}(-\alpha[:]) \in \mathbb{N}^{M^2}$
$\mathcal{C} \leftarrow \{\{i\} \mid i = 1, \dots, M\}$
for $i = 1, \dots, M^2$ do
     $i_1 \leftarrow s_i \operatorname{div} M$
     $i_2 \leftarrow s_i \bmod M$
     $c_{\text{tmp}} \leftarrow \{\}$
    for $c \in \mathcal{C}$ do
        if $i_1 \in c$ or $i_2 \in c$ then
             $c_{\text{tmp}} \leftarrow c_{\text{tmp}} \cup c$
             $\mathcal{C} \leftarrow \mathcal{C} \setminus \{c\}$
        end if
    end for
     $\mathcal{C} \leftarrow \mathcal{C} \cup \{c_{\text{tmp}}\}$
    if $|\mathcal{C}| = 2$ then
        return $\mathcal{C}$
    end if
end for
Knowl 6 — Privacy-Preserving Clustered Federated Learning with Weight-Updates
algorithm
Clustered Federated Learning (CFL) recursively bipartitions client clusters in a top-down hierarchy. Clients communicate SGD weight-updates $\Delta \theta_i = \text{SGD}(\theta_i, D_i) - \theta_i$ instead of gradients. To preserve privacy against server model inversion, clients apply a shared random orthonormal permutation operator $P$ to their updates prior to transmission, which leaves cosine similarities and $L_2$ norms invariant.
Input: Initial parameters $\theta_0$ , splitting parameters $\varepsilon_1, \varepsilon_2 > 0$ , noise bound $\gamma_{\max} \in [0, 1)$ , local epochs $n$
Output: Improved local parameters $\theta_i$ for each client $i$
Initialize: $\mathcal{C} \leftarrow \{\{1, \dots, M\}\}$ , $\theta_i \leftarrow \theta_0$ for all $i=1,\dots,M$ , $\Delta \theta_c \leftarrow 0$ for all $c \in \mathcal{C}$
Optional: Clients agree on random seed for permutation operator $P$ (else $P = I$ )
while not converged do
    for $i = 1, \dots, M$ in parallel do
         $\theta_i \leftarrow \theta_i + P^{-1}(\Delta \theta_{c(i)})$
         $\Delta \theta_i \leftarrow P(\operatorname{SGD}(\theta_i, D_i) - \theta_i)$
    end for
     $\mathcal{C}_{\text{tmp}} \leftarrow \mathcal{C}$
    for $c \in \mathcal{C}$ do
         $\Delta \theta_c \leftarrow \frac{1}{|c|} \sum_{i \in c} \Delta \theta_i$
        if $\|\Delta \theta_c\| < \varepsilon_1$ and $\max_{i \in c} \|\Delta \theta_i\| > \varepsilon_2$ then
            Compute $\alpha_{i, j} \leftarrow \frac{\langle \Delta \theta_i, \Delta \theta_j \rangle}{\|\Delta \theta_i\| \|\Delta \theta_j\|}$ for all $i, j \in c$
             $c_1, c_2 \leftarrow \arg\min_{c_1 \dot{\cup} c_2 = c} (\max_{i \in c_1, j \in c_2} \alpha_{i, j})$
             $\alpha_{\text{cross}}^{\max} \leftarrow \max_{i \in c_1, j \in c_2} \alpha_{i, j}$
            if $\gamma_{\max} < \sqrt{\frac{1 - \alpha_{\text{cross}}^{\max}}{2}}$ then
                 $\mathcal{C}_{\text{tmp}} \leftarrow (\mathcal{C}_{\text{tmp}} \setminus \{c\}) \cup \{c_1\} \cup \{c_2\}$
            end if
        end if
    end for
     $\mathcal{C} \leftarrow \mathcal{C}_{\text{tmp}}$
end while
return $\theta$
Knowl 7 — Weight-Updates as Directional Approximations for Cosine Similarity
model/method
In federated learning, clients compute local parameter updates over $n_b$ disjoint minibatches $D = \bigcup_{\tau=0}^{n_b-1} D_\tau$ via stochastic gradient descent (SGD). For learning rate $\eta$ and smooth loss, Taylor expansion over one epoch approximates the full-batch empirical gradient:
$\Delta \theta = \text{SGD}(\theta_0, D) - \theta_0 = -\sum_{\tau=0}^{n_b-1} \eta \nabla_\theta r(\theta_\tau, D_\tau) \approx -\eta \nabla_\theta r(\theta_0, D)$
Cosine similarity between clients $i$ and $j$ within cluster $c$ is therefore computed directly from weight-updates:
$\alpha_{i,j} := \frac{\langle \Delta \theta_i, \Delta \theta_j \rangle}{\|\Delta \theta_i\| \|\Delta \theta_j\|}$
Using accumulated weight-updates over local epochs preserves the standard Federated Averaging communication protocol and empirically yields a wider separation gap $g(\alpha)$ at lower data regimes and greater distances from stationary points than using exact full-batch gradients.
Knowl 8 — Orthonormal Invariance for Privacy-Preserving Clustering
model/method
To prevent model inversion attacks that reconstruct client training inputs from transmitted weight-updates $\Delta \theta_i$ , clients apply an orthonormal transformation operator $P$ (such as an index permutation matrix determined by a shared random seed among clients). Because orthonormal operators satisfy $\langle P u, P v \rangle = \langle u, v \rangle$ and $\|P u\| = \|u\|$ , the cosine similarity and vector norms are strictly invariant:
$\frac{\langle P \Delta \theta_i, P \Delta \theta_j \rangle}{\|P \Delta \theta_i\| \|P \Delta \theta_j\|} = \frac{\langle \Delta \theta_i, \Delta \theta_j \rangle}{\|\Delta \theta_i\| \|\Delta \theta_j\|}$ $\|P \Delta \theta_i\| = \|\Delta \theta_i\|$
The server performs similarity evaluation and aggregation on transformed vectors $\frac{1}{|c|} \sum_{i \in c} P \Delta \theta_i$ , and clients invert the received broadcast update using $P^{-1}$ before applying local model updates:
$\frac{1}{|c|} \sum_{i \in c} \Delta \theta_i = P^{-1} \left( \frac{1}{|c|} \sum_{i \in c} P \Delta \theta_i \right)$
This enables clustering without server access to unmasked parameter updates.
Knowl 9 — Parameter Tree Structure and Online Client Assignment
algorithm
To support dynamic client participation and variable specificity models, CFL records clustering history as a directed parameter tree $T = (V, E)$ . Every node $v \in V$ stores an intermediate or leaf cluster $c_v$ and its converged stationary model $\theta_v^*$ . Each edge $e(v_{\text{parent}} \to v_{\text{child}})$ caches the pre-split client weight-updates:
$\Delta_e = \left\{ \text{SGD}(\theta_{v_{\text{parent}}}^*, D_i) - \theta_{v_{\text{parent}}}^* \;\middle|\; i \in c_{v_{\text{child}}} \right\}$
A newly joining client with dataset $D_{\text{new}}$ is routed to a specialized leaf model by traversing from $v_{\text{root}}$ downward according to maximum cosine similarity against cached edge updates:
Input: New client data $D_{\text{new}}$ , parameter tree $T = (V, E)$
Output: Assigned cluster $c_v$ , model $\theta_v^*$
$v \leftarrow v_{\text{root}}$
while $|\operatorname{Children}(v)| > 0$ do
     $v_0, v_1 \leftarrow \operatorname{Children}(v)$
     $\Delta \theta_{\text{new}} \leftarrow \operatorname{SGD}(\theta_v^*, D_{\text{new}}) - \theta_v^*$
     $\alpha_0 \leftarrow \max_{\Delta \theta \in \Delta_{(v \to v_0)}} \alpha(\Delta \theta_{\text{new}}, \Delta \theta)$
     $\alpha_1 \leftarrow \max_{\Delta \theta \in \Delta_{(v \to v_1)}} \alpha(\Delta \theta_{\text{new}}, \Delta \theta)$
    if $\alpha_0 > \alpha_1$ then
         $v \leftarrow v_0$
    else
        $v \leftarrow v_1
    end if
end while
return $c_v, \theta_v^*$
Knowl 10 — Empirical Performance on Incongruent Vision and Language Benchmarks
empirical result
Clustered Federated Learning was evaluated on two multitask benchmark settings with $M = 20$ clients partitioned into $K = 4$ clusters:
1. CIFAR-10 Permuted Labels: Training a 5-layer CNN with 3 local epochs per round (batch size 100). Clients in each cluster received data with cluster-specific label permutations. Standard FL converged after 50 rounds, stagnating at approximately 20% test accuracy. CFL executed three successive splits at communication rounds 50, 100, and 150 (whenever the separation gap $g(\alpha) > 0$ ), successfully isolating all 4 clusters and increasing test accuracy to nearly 60%.
2. AG-News Language Modeling: Training a 2-layer LSTM for next-word prediction on 20 subcorpora divided across 4 distinct news topics (World, Sports, Business, Sci/Tech). Standard FL converged around 30 rounds to a perplexity of 42. CFL executed splits at communication rounds 30, 60, and 90, identifying all four topic clusters and reducing client test perplexity to below 36 at round 100.
Knowl 11 — Client Similarity Information Leakage
limitation
Although applying orthonormal transformations prevents the central server from reconstructing raw client data or model weights via standard inversion attacks, CFL inherently leaks relational metadata. Specifically, computing pairwise cosine similarities of client updates allows the server to determine which clients possess similar data generating distributions and to reconstruct the clustering hierarchy of the client population.

Coverage note — None was omitted; all contributed definitions, theorems, separation criteria, algorithms (optimal bipartition, recursive CFL, new client assignment), implementation considerations (privacy, weight updates, parameter trees), experimental findings, and stated limitations are covered.

References

1.H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, "Communication-efficient learning of deep networks from decentralized data," 2016, arXiv:1602.05629. [Online]. Available: http://arxiv.org/abs/1602.05629
2.J. Konečný, H. B. McMahan, D. Ramage, and P. Richtárik, "Federated optimization: Distributed machine learning for on-device intelligence," 2016, arXiv:1610.02527. [Online]. Available: http://arxiv.org/abs/1610.02527
3.K. Bonawitz et al., "Practical secure aggregation for privacy preserving machine learning," in Proc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), 2017, pp. 1175–1191.
4.K. Bonawitz et al., "Towards federated learning at scale: System design," 2019, arXiv:1902.01046. [Online]. Available: http://arxiv.org/abs/1902.01046
5.T. Li, A. Kumar Sahu, A. Talwalkar, and V. Smith, "Federated learning: Challenges, methods, and future directions," 2019, arXiv:1908.07873. [Online]. Available: http://arxiv.org/abs/1908.07873
6.F. Sattler, K.-R. Müller, T. Wiegand, and W. Samek, "On the Byzantine robustness of clustered federated learning," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2020, pp. 8861–8865.
7.X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, "On the convergence of FedAvg on non-IID data," 2019, arXiv:1907.02189. [Online]. Available: http://arxiv.org/abs/1907.02189
8.T. Li, A. Kumar Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," 2018, arXiv:1812.06127. [Online]. Available: http://arxiv.org/abs/1812.06127
9.V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, "Federated multi-task learning," in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 4424–4434.
10.A. Ghosh, J. Hong, D. Yin, and K. Ramchandran, "Robust federated learning in a heterogeneous environment," 2019, arXiv:1906.06629. [Online]. Available: http://arxiv.org/abs/1906.06629
11.S. Caldas et al., "LEAF: A benchmark for federated settings," 2018, arXiv:1812.01097. [Online]. Available: http://arxiv.org/abs/1812.01097
12.Q. Yang, Y. Liu, T. Chen, and Y. Tong, "Federated machine learning: Concept and applications," ACM Trans. Intell. Syst. Technol., vol. 10, no. 2, pp. 12:1–12:19, 2019.
13.T. Lin, S. U. Stich, K. Kshitij Patel, and M. Jaggi, "Don't use large mini-batches, use local SGD," 2018, arXiv:1808.07217. [Online]. Available: http://arxiv.org/abs/1808.07217
14.F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, "Sparse binary compression: Towards distributed deep learning with minimal communication," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2019, pp. 1–8.
15.F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, "Robust and communication-efficient federated learning from non-i.I.d. data," IEEE Trans. Neural Netw. Learn. Syst., early access, Nov. 1, 2019, doi: 10.1109/TNNLS.2019.2944481.
16.Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, "Federated learning with non-IID data," 2018, arXiv:1806.00582. [Online]. Available: http://arxiv.org/abs/1806.00582
17.P. Jiang and G. Agrawal, "A linear speedup analysis of distributed deep learning with sparse and quantized communication," in Proc. Adv. Neural Inf. Process. Syst., 2018, pp. 2525–2536.
18.H. Yu, S. Yang, and S. Zhu, "Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning," 2018, arXiv:1807.06629. [Online]. Available: http://arxiv.org/abs/1807.06629
19.A. Koloskova, T. Lin, S. U. Stich, and M. Jaggi, "Decentralized deep learning with arbitrary communication compression," 2019, arXiv:1907.09356. [Online]. Available: http://arxiv.org/abs/1907.09356
20.A. Koloskova, S. U. Stich, and M. Jaggi, "Decentralized stochastic optimization and gossip algorithms with compressed communication," 2019, arXiv:1902.00340. [Online]. Available: http://arxiv.org/abs/1902.00340
21.S. U. Stich, J.-B. Cordonnier, and M. Jaggi, "Sparsified SGD with memory," in Proc. Adv. Neural Inf. Process. Syst., 2018, pp. 4447–4458.
22.V. Smith, S. Forte, C. Ma, M. Takac, M. I. Jordan, and M. Jaggi, "CoCoA: A general framework for communication-efficient distributed optimization," 2016, arXiv:1611.02189. [Online]. Available: http://arxiv.org/abs/1611.02189
23.R. Caruana, "Multitask learning," Mach. Learn., vol. 28, no. 1, pp. 41–75, 1997.
24.L. Jacob, J.-P. Vert, and F. R. Bach, "Clustered multi-task learning: A convex formulation," in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 745–752.
25.A. Kumar and H. Daume, III, "Learning task grouping and overlap in multi-task learning," 2012, arXiv:1206.6417. [Online]. Available: http://arxiv.org/abs/1206.6417
26.S. Ruder, "An overview of multi-task learning in deep neural networks," 2017, arXiv:1706.05098. [Online]. Available: http://arxiv.org/abs/1706.05098
27.L. Corinzia and J. M. Buhmann, "Variational federated multi-task learning," 2019, arXiv:1906.06268. [Online]. Available: http://arxiv.org/abs/1906.06268
28.Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, "Deep gradient compression: Reducing the communication bandwidth for distributed training," 2017, arXiv:1712.01887. [Online]. Available: http://arxiv.org/abs/1712.01887
29.A. Bhowmick, J. Duchi, J. Freudiger, G. Kapoor, and R. Rogers, "Protection against reconstruction and its applications in private federated learning," 2018, arXiv:1812.00984. [Online]. Available: http://arxiv.org/abs/1812.00984
30.B. Hitaj, G. Ateniese, and F. Perez-Cruz, "Deep models under the GAN: Information leakage from collaborative deep learning," in Proc. ACM SIGSAC Conf. Comput. Commun. Secur., Oct. 2017, pp. 603–618.
31.M. Fredrikson, S. Jha, and T. Ristenpart, "Model inversion attacks that exploit confidence information and basic countermeasures," in Proc. 22nd ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), 2015, pp. 1322–1333.
32.N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, "The secret sharer: Evaluating and testing unintended memorization in neural networks," 2018, arXiv:1802.08232. [Online]. Available: http://arxiv.org/abs/1802.08232
33.L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, "Exploiting unintended feature leakage in collaborative learning," 2018, arXiv:1805.04049. [Online]. Available: http://arxiv.org/abs/1805.04049
34.Y. LeCun. (1998). The MNIST Database of Handwritten Digits. [Online]. Available: http://yann. lecun. com/exdb/mnist/
35.A. Krizhevsky, V. Nair, and G. Hinton. (2014). The CIFAR-10 dataset. [Online]. Available: http://www.cs.toronto.edu/kriz/cifar.html
36.C. Dwork, "Differential privacy: A survey of results," in Proc. Int. Conf. Theory Appl. Models Comput. Cham, Switzerland: Springer, 2008, pp. 1–19.
37.M. Abadi et al., "Deep learning with differential privacy," in Proc. ACM SIGSAC Conf. Comput. Commun. Secur., Oct. 2016, pp. 308–318.
38.D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, "QSGD: Communication-efficient SGD via gradient quantization and encoding," in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 1709–1720.
39.S. Wiedemann et al., "DeepCABAC: A universal compression algorithm for deep neural networks," IEEE J. Sel. Topics Signal Process., vol. 14, no. 4, pp. 700–714, May 2020, doi: 10.1109/JSTSP.2020.2969554.
40.S. Wiedemann, K.-R. Müller, and W. Samek, "Compact and computationally efficient representation of deep neural networks," IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 3, pp. 772–785, Mar. 2020.
41.W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, and K.-R. Müller, Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (Lecture Notes in Computer Science), vol. 11700. Cham, Switzerland: Springer, 2019, doi: 10.1007/978-3-030-28954-6.
42.S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, "Unmasking clever hans predictors and assessing what machines really learn," Nature Commun., vol. 10, no. 1, p. 1096, Mar. 2019, doi: 10.1038/s41467-019-08987-4.

Citation

MLA

Sattler, F., et al. “Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization Under Privacy Constraints”. arXiv, 2019, http://arxiv.org/abs/1910.01991v1.

APA

Sattler, F., Müller, K.-R., & Samek, W. (2019). Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints. arXiv. http://arxiv.org/abs/1910.01991v1

Chicago

Sattler, F., K.-R. Müller, and W. Samek. 2019. “Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization Under Privacy Constraints”. arXiv. http://arxiv.org/abs/1910.01991v1.

Harvard

Sattler, F., Müller, K.-R. and Samek, W. (2019) “Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1910.01991v1.

Vancouver

1. Sattler F, Müller K-R, Samek W (2019) Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints. arXiv

BibTeX

@article{sattler2019clustered,
  title = {Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints},
  author = {Sattler, Felix and Müller, Klaus-Robert and Samek, Wojciech},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1910.01991v1},
  eprint = {1910.01991}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Clustered Federated Learning Data Congruence Assumption

Knowl 2 — Separation Theorem for Incongruent Client Distributions

Knowl 3 — Separation Gap and Guaranteed Correct Bipartitioning

Knowl 4 — Stationarity and Noise Thresholds for Federated Splitting Decisions

Knowl 5 — Agglomerative Optimal Bipartition Algorithm

Knowl 6 — Privacy-Preserving Clustered Federated Learning with Weight-Updates

Knowl 7 — Weight-Updates as Directional Approximations for Cosine Similarity

Knowl 8 — Orthonormal Invariance for Privacy-Preserving Clustering

Knowl 9 — Parameter Tree Structure and Online Client Assignment

Knowl 10 — Empirical Performance on Incongruent Vision and Language Benchmarks

Knowl 11 — Client Similarity Information Leakage

References

Citation

BibTeX

Access the Paper