Federated Multi-Task Learning

Virginia SmithChao-Kai ChiangMaziar SanjabiAmeet Talwalkar

article2017NeurIPS2,184 citations

Proposes MOCHA, a federated multi-task learning framework that simultaneously addresses statistical heterogeneity across distributed devices and practical systems challenges including high communication costs, stragglers, and fault tolerance.

Listen

Modern edge devices such as smartphones, wearables, and smart home sensors generate vast amounts of data that are increasingly valuable for machine learning. Training models directly on these distributed devices avoids transferring sensitive raw data to central servers, but it introduces major operational bottlenecks. The network faces statistical challenges because data across devices is non-identical and unbalanced, alongside severe systems challenges such as high communication latency, hardware and connectivity variability across devices (stragglers), and frequent device dropouts (fault tolerance).

The article aims to evaluate a multi-task learning framework designed specifically for federated edge settings and demonstrates a new systems-aware optimization method, MOCHA, that simultaneously addresses statistical heterogeneity, communication constraints, stragglers, and dropped devices.

To achieve this, the authors formulated a primal-dual multi-task optimization framework where individual tasks represent individual devices while a central server estimates relationships among them. They developed MOCHA, an optimization algorithm that allows each device to solve local subproblems to a variable degree of precision within a central clock cycle. The researchers proved theoretical convergence guarantees for both smooth and non-smooth loss functions under conditions where devices periodically drop out. They then evaluated the framework through extensive simulations on three real-world edge datasets (Google Glass activity recognition, smartphone human activity tracking, and vehicle sensor classification) under simulated cellular and wireless network constraints.

The findings show that multi-task learning consistently achieves higher accuracy than conventional global or isolated local models, reducing average prediction error substantially (for example, achieving a 0.46% error rate on human activity recognition compared to 2.23% for a global model and 1.34% for local models). Furthermore, MOCHA converged significantly faster than standard distributed alternatives such as mini-batch stochastic gradient descent and mini-batch stochastic dual coordinate ascent across 3G, LTE, and Wi-Fi communication regimes. By enabling flexible per-node approximation accuracy, MOCHA prevented straggler delays caused by differences in local dataset sizes or hardware performance. Finally, simulations verified that MOCHA remains robust and converges even when devices periodically fail to send updates, provided no device drops out permanently.

These results demonstrate that fitting personalized but structurally linked models via multi-task learning resolves the accuracy degradation typical of one-size-fits-all global models in heterogeneous networks. In practice, MOCHA reduces communication overhead and mitigates the risk of system stalls caused by slow or temporarily disconnected devices, making distributed training on edge devices computationally practical and reliable.

Organizations deploying machine learning across distributed edge fleets should adopt multi-task architectures rather than single global models when data distributions vary significantly across users. Teams should implement flexible local computation budgets based on fixed clock deadlines to absorb hardware and network disparities. Before deployment in complex deep learning pipelines, stakeholders should conduct pilot tests, as the current framework is tailored to convex formulations and requires maintaining a task relationship matrix at the central coordinator.

The conclusions are supported by rigorous theoretical proofs and realistic simulations matching real-world device bandwidth and clock speeds. However, the framework's primary limitations are its focus on convex models (excluding non-convex deep neural networks without kernelization) and the computational overhead of updating the relationship matrix when the number of devices becomes exceptionally large.

Carnegie Mellon University Stanford University University of Southern California communication overheads fault tolerance federated learning federated multi-task learning mini-batch stochastic dual coordinate ascent primal-dual optimization

arXiv: 1705.10467

Paper: A Survey on Multi-Task Learning, Yu Zhang et al. (2017). Reading this comprehensive survey of multi-task learning provides the foundational taxonomy and theoretical justification for applying multi-task principles to federated networks.
Paper: An Overview of Multi-Task Learning in Deep Neural Networks, Sebastian Ruder (2017). Understanding the mechanisms of hard and soft parameter sharing in deep multi-task networks is a prerequisite for grasping how MOCHA manages task relationships across distributed devices.
Paper: Multitask Learning, RICH CARUANA (1997). Caruana's seminal work on multitask learning establishes the core empirical finding that training related tasks jointly improves generalization when data is sparse.

Paper: Federated Learning: Challenges, Methods, and Future Directions, Tian Li et al. (2019). This broad literature survey extends the foundational multi-task optimization concepts of MOCHA into a comprehensive taxonomy of federated learning challenges and future research directions.
Paper: Towards Federated Learning at Scale: System Design, Keith Bonawitz et al. (2019). Building directly on the optimization strategies of the source paper, this work details the production-scale system design required to deploy federated algorithms on millions of real-world devices.
Paper: SCAFFOLD: Stochastic Controlled Averaging for Federated Learning, Sai Praneeth Karimireddy et al. (2019). This single-chapter ebook extends the study of heterogeneous federated objectives by introducing SCAFFOLD to correct client drift through server-side control variates.
Paper: Advances and Open Problems in Federated Learning, P. Kairouz et al. (2019). This follow-up survey expands upon MOCHA's optimization framework by cataloging a wide spectrum of open problems in federated systems, privacy, and non-IID data handling.
Paper: Federated Optimization in Heterogeneous Networks, Tian Li et al. (2018). This paper continues the exploration of heterogeneous federated networks by proposing FedProx as a generalization of FedAvg that adds proximal regularization to handle stragglers and systems variability.

Abstract

Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices. In this work, we show that multi-task learning is naturally suited to handle the statistical challenges of this setting, and propose a novel systems-aware optimization method, MOCHA, that is robust to practical systems issues. Our method and theory for the first time consider issues of high communication cost, stragglers, and fault tolerance for distributed multi-task learning. The resulting method achieves significant speedups compared to alternatives in the federated setting, as we demonstrate through simulations on real-world federated datasets.

Topics

communication overheads fault tolerance federated learning optimization optimization problems multitask learning online learning communication networks

Keywords

federated multi-task learning mini-batch stochastic dual coordinate ascent primal-dual optimization systems-aware optimization task relationships convergence guarantees duality gap convex formulation

1 Introduction
1.1 Contributions
2 Related Work
3 Federated Multi-Task Learning
3.1 General Multi-Task Learning Setup
3.2 Mocha: A Framework for Federated Multi-Task Learning
3.3 Federated Update of 𝐖{\bf W}
3.4 Practical Considerations
4 Convergence Analysis
5 Simulations
5.1 Federated Datasets
5.2 Multi-Task Learning for the Federated Setting
5.3 Straggler Avoidance
5.4 Tolerance to Dropped Nodes
6 Discussion
References
A Preliminaries
B Multi-Task Learning
B.1 Multi-Task Learning Formulations
B.2 Strong Convexity of MTL Regularizers
B.3 Optimizing 𝛀{\bf\Omega} in MTL Formulations
B.3.1 Reducing the Size of 𝛀{\bf\Omega} by Sharing Tasks
C Convergence Analysis
C.1 Convergence Analysis for Smooth Losses
C.1.1 Proof of Theorem
C.1.2 Asymptotic Convergence
C.2 Convergence Analysis for Lipschitz Losses: Proof for Theorem
D Choosing σ′\sigma^{\prime}
D.1 The Role of Aggregation Parameter γ\gamma
E Simulation Details
E.1 Datasets
E.2 Multi-Task Learning with Highly Skewed Data
E.3 Implementation Details

Knowls

Knowl 1 — Federated Multi-Task Learning Formulation
model/method
In federated learning over $m$ distributed nodes (devices), each node $t \in [m] = \{1, \dots, m\}$ holds a local dataset $X_t \in \mathbb{R}^{d \times n_t}$ and label vector $y_t \in \mathbb{R}^{n_t}$ generated from a distinct distribution $X_t \sim P_t$ . Rather than fitting a single global model across non-IID and unbalanced data, Federated Multi-Task Learning (FMTL) learns separate but related weight vectors $w_t \in \mathbb{R}^d$ for each node by solving:

$\min_{W, \Omega} \left\{ \sum_{t=1}^m \sum_{i=1}^{n_t} \ell_t(w_t^T x_t^i, y_t^i) + \mathcal{R}(W, \Omega) \right\}$

where $W := [w_1, \dots, w_m] \in \mathbb{R}^{d \times m}$ is the weight matrix whose $t$ -th column is the model parameter for task $t$ , $\ell_t: \mathbb{R} \times \mathbb{R} \to \mathbb{R}$ is a convex loss function (such as the hinge loss or squared error), and $\Omega \in \mathbb{R}^{m \times m}$ is a task relationship matrix that is either known a priori or learned simultaneously with $W$ . The regularizer $\mathcal{R}(W, \Omega)$ enforces structural coupling between tasks. The problem is solved via bi-convex alternating optimization: at each outer iteration, $\Omega$ is updated centrally using $W$ , and $W$ is updated distributedly across the $m$ nodes while keeping $\Omega$ fixed.
Knowl 2 — MOCHA Algorithm for Federated Multi-Task Learning
algorithm
MOCHA is a distributed, systems-aware optimization framework for federated multi-task learning. It allows worker nodes to compute variable-accuracy approximations to their local subproblems, which provides robustness against stragglers and dropped nodes while communicating only compressed gradient-offset vectors $v_t \in \mathbb{R}^d$ to the central coordinator.
Input: Local data matrices $X_t \in \mathbb{R}^{d \times n_t}$ for $t \in \{1, \dots, m\}$ distributed across $m$ nodes, and initial task relationship matrix $\Omega_0 \in \mathbb{R}^{m \times m}$
Initialize dual variables $\alpha^{(0)} := 0 \in \mathbb{R}^n$ (where $n = \sum_{t=1}^m n_t$ ) and shared vectors $v^{(0)} := 0 \in \mathbb{R}^{md}$
for outer iterations $i = 0, 1, \dots$ do
    Set subproblem difficulty parameter $\sigma'$ and number of federated rounds $H_i$
    for federated rounds $h = 0, 1, \dots, H_i$ do
        for tasks $t \in \{1, \dots, m\}$ in parallel on node $t$ do
            Compute a $\theta_t^h$ -approximate update $\Delta \alpha_t$ to the local dual subproblem
            Update local dual variables: $\alpha_t \leftarrow \alpha_t + \Delta \alpha_t$
            Compute local representation update: $\Delta v_t := X_t \Delta \alpha_t$
        end for
        Reduce and aggregate across nodes: $v_t \leftarrow v_t + \Delta v_t$
    end for
    Update task structure matrix $\Omega$ centrally based on current weights $w(\alpha) = \nabla \mathcal{R}^*(X\alpha)$
end for
Compute final weight vectors $w(\alpha) = [w_1, \dots, w_m]$
return $W := [w_1, \dots, w_m] \in \mathbb{R}^{d \times m}$
The local subproblem solver is typically dual coordinate ascent. By allowing per-node approximation qualities $\theta_t^h \in [0, 1]$ , workers solve within dynamic deadlines without stalling synchronous coordination rounds.
Knowl 3 — Dual Formulation and Data-Local Subproblems in MOCHA
model/method
For a fixed task relationship matrix $\Omega$ , the dual of the multi-task learning objective with total sample count $n = \sum_{t=1}^m n_t$ and block-diagonal data matrix $X := \mathrm{Diag}(X_1, \dots, X_m) \in \mathbb{R}^{md \times n}$ is:

$\min_{\alpha \in \mathbb{R}^n} \left\{ \mathcal{D}(\alpha) := \sum_{t=1}^m \sum_{i=1}^{n_t} \ell_t^*(-\alpha_t^i) + \mathcal{R}^*(X\alpha) \right\}$

where $\alpha = [\alpha_1^T, \dots, \alpha_m^T]^T \in \mathbb{R}^n$ , each $\ell_t^*$ is the Fenchel conjugate of $\ell_t$ , and $\mathcal{R}^*$ is the conjugate of $\mathcal{R}(\cdot, \Omega)$ . When $\mathcal{R}$ is strongly convex with respect to $M^{-1}$ for a symmetric positive definite matrix $M \in \mathbb{R}^{md \times md}$ , $\mathcal{R}^*$ is smooth with respect to $M$ .

To decouple optimization across devices without sharing raw data $X_t$ , each node $t$ solves a data-local quadratic approximation for an update $\Delta \alpha_t \in \mathbb{R}^{n_t}$ :

$\min_{\Delta \alpha_t \in \mathbb{R}^{n_t}} \mathcal{G}_{\sigma'}^t(\Delta \alpha_t; v_t, \alpha_t) := \sum_{i=1}^{n_t} \ell_t^*(-\alpha_t^i - \Delta \alpha_t^i) + \langle w_t(\alpha), X_t \Delta \alpha_t \rangle + \frac{\sigma'}{2} \|X_t \Delta \alpha_t\|_{M_t}^2 + c(\alpha)$

where $c(\alpha) := \frac{1}{m}\mathcal{R}^*(X\alpha)$ , $M_t \in \mathbb{R}^{d \times d}$ is the $t$ -th diagonal block of $M$ , $\|u\|_{M_t}^2 = u^T M_t u$ , and $\sigma' > 0$ is a data-partitioning parameter. The primal weight vector for task $t$ is recovered as the $t$ -th block of $w(\alpha) = \nabla \mathcal{R}^*(X\alpha)$ , which depends only on $v = X\alpha$ .
Knowl 4 — Per-Node Approximation Quality and Fault-Tolerance Model
assumption
At each round $h$ and on each device $t$ , the subproblem solution quality is measured by the approximation parameter $\theta_t^h \in [0, 1]$ :

$\theta_t^h := \frac{\mathcal{G}_{\sigma'}^t(\Delta \alpha_t^{(h)}; v^{(h)}, \alpha_t^{(h)}) - \mathcal{G}_{\sigma'}^t(\Delta \alpha_t^*; v^{(h)}, \alpha_t^{(h)})}{\mathcal{G}_{\sigma'}^t(0; v^{(h)}, \alpha_t^{(h)}) - \mathcal{G}_{\sigma'}^t(\Delta \alpha_t^*; v^{(h)}, \alpha_t^{(h)})}$

where $\Delta \alpha_t^*$ is the exact minimizer of $\mathcal{G}_{\sigma'}^t(\cdot; v^{(h)}, \alpha_t^{(h)})$ . A value $\theta_t^h = 0$ corresponds to an exact local solution, whereas $\theta_t^h = 1$ signifies that node $t$ made no progress (e.g., a dropped node due to battery exhaustion, network timeout, or straggler cut-off).

Let $\mathcal{H}_h := (\alpha^{(h)}, \alpha^{(h-1)}, \dots, \alpha^{(1)})$ denote the dual vector history. The system assumes that for all tasks $t \in [m]$ and all iterations $h$ :

$p_t^h := \mathbb{P}[\theta_t^h = 1] \le p_{\max} < 1 \quad \text{and} \quad \hat{\Theta}_t^h := \mathbb{E}[\theta_t^h \mid \mathcal{H}_h, \theta_t^h < 1] \le \Theta_{\max} < 1$

which ensures that the overall expected per-round approximation quality is strictly bounded away from one:

$\bar{\Theta} := p_{\max} + (1 - p_{\max}) \Theta_{\max} < 1$

This models node unreliability and dynamic systems constraints without requiring bounded delays or fixed uniform work across workers.
Knowl 5 — Convergence Rate of MOCHA for Smooth Losses
theoretical result
Assume that the loss functions $\ell_t$ are $(1/\mu)$ -smooth for all $t \in [m]$ , and that the regularizer $\mathcal{R}$ is strongly convex with respect to $M^{-1}$ . Under the fault-tolerant approximation condition where $\bar{\Theta} := p_{\max} + (1 - p_{\max}) \Theta_{\max} < 1$ , there exists a constant $s = \frac{\mu}{\mu + \sigma_{\max}\sigma'} \in (0, 1]$ , where $\sigma_{\max} := \max_{t \in [m]} \max_{\alpha_t \in \mathbb{R}^{n_t}} \frac{\|X_t \alpha_t\|_{M_t}^2}{\|\alpha_t\|^2}$ , such that for any target dual error $\epsilon_D > 0$ , running $H$ federated rounds satisfying:

$H \ge \frac{1}{(1 - \bar{\Theta}) s} \log \left( \frac{n}{\epsilon_D} \right)$

with total dataset size $n = \sum_{t=1}^m n_t$ , guarantees that the expected dual suboptimality satisfies:

$\mathbb{E}[\mathcal{D}(\alpha^{(H)}) - \mathcal{D}(\alpha^*)] \le \epsilon_D$

By duality, the same rate applies to the expected duality gap $\mathbb{E}[G(\alpha^{(H)})] \le \epsilon_D$ .
Knowl 6 — Convergence Rate of MOCHA for Lipschitz Non-Smooth Losses
theoretical result
When the task loss functions $\ell_t$ are non-smooth but $L$ -Lipschitz (such as the hinge loss for SVMs), and the regularizer $\mathcal{R}$ is strongly convex with respect to $M^{-1}$ , MOCHA achieves sublinear convergence.

Let $\sigma := \sum_{t=1}^m \sigma_t n_t$ , where $\sigma_t := \max_{\alpha_t \in \mathbb{R}^{n_t}} \frac{\|X_t \alpha_t\|_{M_t}^2}{\|\alpha_t\|^2}$ . For any target error $\epsilon_D > 0$ , choosing the number of federated rounds $H$ such that:

$H \ge H_0 + \frac{2}{1 - \bar{\Theta}} \max\left( 1, \frac{2 L^2 \sigma \sigma'}{n^2 \epsilon_D} \right)$

with

$H_0 \ge h_0 + \frac{16 L^2 \sigma \sigma'}{(1 - \bar{\Theta}) n^2 \epsilon_D} \quad \text{and} \quad h_0 = \left[ 1 + \frac{1}{1 - \bar{\Theta}} \log \left( \frac{2 n^2 (\mathcal{D}(\alpha^*) - \mathcal{D}(\alpha^{(0)}))}{4 L^2 \sigma \sigma'} \right) \right]_+$

guarantees that the averaged iterate $\bar{\alpha} := \frac{1}{H - H_0} \sum_{h = H_0 + 1}^H \alpha^{(h)}$ satisfies:

$\mathbb{E}[\mathcal{D}(\bar{\alpha}) - \mathcal{D}(\alpha^*)] \le \epsilon_D$
Knowl 7 — Selection of the Subproblem Difficulty Parameter
theoretical result
To ensure that data-local subproblems provide a valid upper bound on the global dual objective, the subproblem scaling parameter $\sigma'$ must satisfy:

$\sigma' \sum_{t=1}^m \|X_t \alpha_t\|_{M_t}^2 \ge \gamma \|X\alpha\|_M^2 \quad \forall \alpha \in \mathbb{R}^n$

where $\gamma \in (0, 1]$ is the global aggregation step size (with $\gamma = 1$ used in default MOCHA). When the metric matrix has Kronecker structure $M = \bar{M} \otimes I_{d \times d}$ for a positive definite matrix $\bar{M} \in \mathbb{R}^{m \times m}$ , a valid choice for $\sigma'$ is:

$\sigma' := \gamma \max_{t \in [m]} \sum_{t'=1}^m \frac{|\bar{M}_{tt'}|}{\bar{M}_{tt}}$

Alternatively, task-specific parameters $\sigma'_t := \gamma \sum_{t'=1}^m \frac{|\bar{M}_{tt'}|}{\bar{M}_{tt}}$ can be assigned to allow tasks that are more loosely correlated with other tasks to update more aggressively.
Knowl 8 — Supported Multi-Task Regularizers and Corresponding Metric Matrices
model/method
The federated multi-task learning framework supports several regularizers $\mathcal{R}(W, \Omega)$ , each determining the symmetric positive definite metric matrix $M$ with respect to which $\mathcal{R}^*$ is smooth (where $\bar{\Omega} := \Omega \otimes I_{d \times d}$ ):
1. Mean/Cluster-Regularized MTL: Tasks within a cluster share similar weights: $\mathcal{R}(W, \Omega) = \lambda_1 \mathrm{tr}(W \Omega W^T) + \lambda_2 \|W\|_F^2 \implies M^{-1} = \lambda_1 \bar{\Omega} + \lambda_2 I_{md \times md}$
2. Convex Clustered MTL: For learned clustering structure $\Omega \in \{Q \mid Q \succeq 0, \mathrm{tr}(Q) = k, Q \preceq I\}$ : $\mathcal{R}(W, \Omega) = \lambda \mathrm{tr}(W(\eta I + \Omega)^{-1} W^T) \implies M = \lambda^{-1} ((\eta I + \Omega) \otimes I_{d \times d})$
3. Probabilistic Matrix-Normal Prior: For task covariance $\Omega \in \{Q \mid Q \succeq 0, \mathrm{tr}(Q) = 1\}$ : $\mathcal{R}(W, \Omega) = \lambda \left( \frac{1}{\sigma_0^2} \|W\|_F^2 + \mathrm{tr}(W \Omega^{-1} W^T) \right) \implies M = \lambda^{-1} \left( \frac{1}{\sigma_0^2} I + \Omega^{-1} \otimes I_{d \times d} \right)^{-1}$
4. Graphical Model Precision Matrix Prior: Enforcing sparsity in the precision matrix $\Omega$ : $\mathcal{R}(W, \Omega) = \lambda \left( \frac{1}{\sigma_0^2} \|W\|_F^2 + \mathrm{tr}(W \Omega W^T) - d \log |\Omega| \right) + \lambda_1 \|W\|_1 + \lambda_2 \|\Omega\|_1 \implies M = \lambda^{-1} \left( \frac{1}{\sigma_0^2} I + \Omega \otimes I_{d \times d} \right)^{-1}$
Knowl 9 — Asymptotic Convergence Under Diminishing Node Availability
theoretical result
For $\mu$ -smooth task losses $\ell_t$ , MOCHA guarantees asymptotic dual convergence $\mathbb{E}[\mathcal{D}(\alpha^{(H)}) - \mathcal{D}(\alpha^*)] \to 0$ as $H \to \infty$ under relaxed participation conditions where node availability may vanish over time. Specifically, asymptotic convergence holds if either of the following is satisfied:
1. $\limsup_{h \to \infty} p_t^h < 1$ and $\limsup_{h \to \infty} \hat{\Theta}_t^h < 1$ for all tasks $t \in [m]$ .
2. For every task $t \in [m]$ , $(1 - p_t^h)(1 - \hat{\Theta}_t^h) = \omega\left(\frac{1}{h}\right)$ .
Under the second condition, the node dropout probability $p_t^h$ is permitted to approach 1 in the limit ( $\{h \to \infty\}$ ), provided the frequency of successful updates and their accuracy decay strictly slower than $1/h$ .

Knowl 10 — Empirical Prediction Error of Multi-Task versus Global and Local Modeling

data/table

Support Vector Machine models trained under the cluster-regularized multi-task learning framework were evaluated against fully global models (pooling all task data into a single model) and fully local models (fitting isolated models per node) across three real-world federated benchmarks using 10 random 75%/25% train/test splits (regularization parameter $\lambda$ tuned via 5-fold cross-validation):

Human Activity Recognition: 30 tasks, feature dimension $d = 561$ , samples per task $n_t \in [210, 306]$ .
Google Glass (GLEAM): 38 tasks, feature dimension $d = 180$ , samples per task $n_t \in [524, 581]$ .
Vehicle Sensor: 23 tasks, feature dimension $d = 100$ , samples per task $n_t \in [872, 1933]$ .

MTL achieves the lowest average classification error across all datasets. Global models perform the worst due to statistical heterogeneity across tasks. In skewed datasets where $n_t$ spans orders of magnitude (e.g., minimum $n_t = 3$ to $19$ ), MTL maintains a similarly dominant error reduction over global and local baselines.

Coverage note — Detailed simulation formulas for converting FLOPs and latency to execution time across network types (3G, LTE, WiFi), specific eigenvalue algorithms for the central Omega update, and intermediate proof lemmas from Appendix C have been omitted in favor of the core framework, algorithm, convergence theorems, parameter selections, and empirical benchmarks.

References

1.A. Ahmed, A. Das, and A. J. Smola. Scalable hierarchical multitask learning algorithms for conversion optimization in display advertising. In Conference on Web Search and Data Mining, 2014.
2.R. K. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853, 2005.
3.D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2013.
4.A. Argyriou, T. Evgeniou, and M. Pontil. Multi-task feature learning. In Neural Information Processing Systems, 2007.
5.A. Argyriou, T. Evgeniou, and M. Pontil. Convex multi-task feature learning. Machine Learning, 73(3):243–272, 2008.
6.Ö. Aslan, X. Zhang, and D. Schuurmans. Convex deep learning via normalized kernels. In Advances in Neural Information Processing Systems, 2014.
7.I. M. Baytas, M. Yan, A. K. Jain, and J. Zhou. Asynchronous multi-task learning. In International Conference on Data Mining, 2016.
8.F. Bonomi, R. Milito, J. Zhu, and S. Addepalli. Fog computing and its role in the internet of things. In SIGCOMM Workshop on Mobile Cloud Computing, 2012.
9.A. Carroll and G. Heiser. An analysis of power consumption in a smartphone. In USENIX Annual Technical Conference, 2010.
10.R. Caruana. Multitask learning. Machine Learning, 28:41–75, 1997.
11.J. Chen, J. Zhou, and J. Ye. Integrating low-rank and group-sparse structures for robust multi-task learning. In Conference on Knowledge Discovery and Data Mining, 2011.
12.A. Deshpande, C. Guestrin, S. R. Madden, J. M. Hellerstein, and W. Hong. Model-based approximate querying in sensor networks. VLDB Journal, 14(4):417–443, 2005.
13.M. F. Duarte and Y. H. Hu. Vehicle classification in distributed sensor networks. Journal of Parallel and Distributed Computing, 64(7):826–838, 2004.
14.T. Evgeniou and M. Pontil. Regularized multi-task learning. In Conference on Knowledge Discovery and Data Mining, 2004.
15.P. Garcia Lopez, A. Montresor, D. Epema, A. Datta, T. Higashino, A. Iamnitchi, M. Barcellos, P. Felber, and E. Riviere. Edge-centric computing: Vision and challenges. SIGCOMM Computer Communication Review, 45(5):37–42, 2015.
16.A. R. Gonçalves, F. J. Von Zuben, and A. Banerjee. Multi-task sparse structure learning with gaussian copula models. Journal of Machine Learning Research, 17(33):1–30, 2016.
17.J. Gorski, F. Pfeuffer, and K. Klamroth. Biconvex sets and optimization with biconvex functions: a survey and extensions. Mathematical Methods of Operations Research, 66(3):373–407, 2007.
18.K. Hong, D. Lillethun, U. Ramachandran, B. Ottenwälder, and B. Koldehofe. Mobile fog: A programming model for large-scale applications on the internet of things. In SIGCOMM Workshop on Mobile Cloud Computing, 2013.
19.C.-J. Hsieh, M. A. Sustik, I. S. Dhillon, and P. Ravikumar. Sparse Inverse Covariance Matrix Estimation Using Quadratic Approximation. In Neural Information Processing Systems 27, 2014.
20.J. Huang, F. Qian, Y. Guo, Y. Zhou, Q. Xu, Z. M. Mao, S. Sen, and O. Spatscheck. An in-depth study of lte: Effect of network protocol and application behavior on performance. In ACM SIGCOMM Conference, 2013.
21.L. Jacob, J.-p. Vert, and F. R. Bach. Clustered multi-task learning: A convex formulation. In Neural Information Processing Systems, 2009.
22.M. Jaggi, V. Smith, J. Terhorst, S. Krishnan, T. Hofmann, and M. I. Jordan. Communication-Efficient Distributed Dual Coordinate Ascent. In Neural Information Processing Systems, 2014.
23.X. Jin, P. Luo, F. Zhuang, J. He, and Q. He. Collaborating between local and global learning for distributed online multiple tasks. In Conference on Information and Knowledge Management, 2015.
24.S. Kim and E. P. Xing. Statistical estimation of correlated genome associations to a quantitative trait network. PLoS Genet, 5(8):e1000587, 2009.
25.J. Konecnˇ y, H. B. McMahan, and D. Ramage. Federated optimization: Distributed optimization beyond the datacenter. arXiv:1511.03575, 2015.
26.J. Konecnˇ y, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency. arXiv:1610.05492, 2016.
27.T. Kuflik, J. Kay, and B. Kummerfeld. Challenges and solutions of ubiquitous user modeling. In Ubiquitous display environments, pages 7–30. Springer, 2012.
28.A. Kumar and H. Daumé. Learning task grouping and overlap in multi-task learning. In International Conference on Machine Learning, 2012.
29.S. L. Lauritzen. Graphical Models, volume 17. Clarendon Press, 1996.
30.S. Liu, S. J. Pan, and Q. Ho. Distributed multi-task relationship learning. Conference on Knowledge Discovery and Data Mining, 2017.
31.C. Ma, V. Smith, M. Jaggi, M. I. Jordan, P. Richtárik, and M. Takác. Adding vs. averaging in distributed primal-dual optimization. In International Conference on Machine Learning, 2015.
32.S. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. TAG: A tiny aggregation service for ad-hoc sensor networks. In Symposium on Operating Systems Design and Implementation, 2002.
33.S. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. TinyDB: An acquisitional query processing system for sensor networks. ACM Transactions on Database Systems, 30(1):122–173, 2005.
34.J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid. Convolutional kernel networks. In Neural Information Processing Systems, 2014.
35.D. Mateos-Núñez and J. Cortés. Distributed optimization for multi-task learning via nuclear-norm approximation. In IFAC Workshop on Distributed Estimation and Control in Networked Systems, 2015.
36.H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Conference on Artificial Intelligence and Statistics, 2017.
37.H. B. McMahan and D. Ramage. http://www.googblogs.com/federated-learning-collaborative-machine-learning-without-centralized-training-data/. Google, 2017.
38.A. P. Miettinen and J. K. Nurminen. Energy efficiency of mobile clients in cloud computing. In USENIX Conference on Hot Topics in Cloud Computing, 2010.
39.A. Pantelopoulos and N. G. Bourbakis. A survey on wearable sensor-based systems for health monitoring and prognosis. IEEE Transactions on Systems, Man, and Cybernetics, 40(1):1–12, 2010.
40.H. Qi, E. R. Sparks, and A. Talwalkar. Paleo: A performance model for deep neural networks. In International Conference on Learning Representations, 2017.
41.S. A. Rahman, C. Merck, Y. Huang, and S. Kleinberg. Unintrusive eating recognition using google glass. In Conference on Pervasive Computing Technologies for Healthcare, 2015.
42.P. Rashidi and D. J. Cook. Keeping the resident in the loop: Adapting the smart home to the user. IEEE Transactions on systems, man, and cybernetics, 39(5):949–959, 2009.
43.M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. XNOR-Net: ImageNet classification using binary convolutional neural networks. In European Conference on Computer Vision, 2016.
44.S. Ravi. https://research.googleblog.com/2017/02/on-device-machine-intelligence.html. Google, 2017.
45.M. Razaviyayn, M. Hong, and Z.-Q. Luo. A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM Journal on Optimization, 23(2):1126–1153, 2013.
46.S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. International Conference on Machine Learning, June 2007.
47.S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14:567–599, 2013.
48.D. Singelée, S. Seys, L. Batina, and I. Verbauwhede. The communication and computation cost of wireless security. In ACM Conference on Wireless Network Security, 2011.
49.V. Smith, S. Forte, C. Ma, M. Takác, M. I. Jordan, and M. Jaggi. CoCoA: A general framework for communication-efficient distributed optimization. arXiv:1611.02189, 2016.
50.M. Takác, A. Bijral, P. Richtárik, and N. Srebro. Mini-Batch Primal and Dual Methods for SVMs. In International Conference on Machine Learning, 2013.
51.C.-Y. Tsai, A. M. Saxe, and D. Cox. Tensor switching networks. In Neural Information Processing Systems, 2016.
52.C. Van Berkel. Multi-core for mobile phones. In Proceedings of the Conference on Design, Automation and Test in Europe, pages 1260–1265. European Design and Automation Association, 2009.
53.H. Wang, A. Banerjee, C.-J. Hsieh, P. K. Ravikumar, and I. S. Dhillon. Large scale distributed sparse precision estimation. In Neural Information Processing Systems, 2013.
54.J. Wang, M. Kolar, and N. Srebro. Distributed multi-task learning. In Conference on Artificial Intelligence and Statistics, 2016.
55.J. Wang, M. Kolar, and N. Srebro. Distributed multi-task learning with shared representation. arXiv:1603.02185, 2016.
56.Y. Zhang, P. Liang, and M. J. Wainwright. Convexified convolutional neural networks. International Conference on Machine Learning, 2017.
57.Y. Zhang and D.-Y. Yeung. A convex formulation for learning task relationships in multi-task learning. In Conference on Uncertainty in Artificial Intelligence, 2010.
58.J. Zhou, J. Chen, and J. Ye. Clustered multi-task learning via alternating structure optimization. In Neural Information Processing Systems, 2011.

Citation

MLA

Smith, V., et al. “Federated Multi-Task Learning”. arXiv, 2017, http://arxiv.org/abs/1705.10467v2.

APA

Smith, V., Chiang, C.-K., Sanjabi, M., & Talwalkar, A. (2017). Federated Multi-Task Learning. arXiv. http://arxiv.org/abs/1705.10467v2

Chicago

Smith, V., C.-K. Chiang, M. Sanjabi, and A. Talwalkar. 2017. “Federated Multi-Task Learning”. arXiv. http://arxiv.org/abs/1705.10467v2.

Harvard

Smith, V. et al. (2017) “Federated Multi-Task Learning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1705.10467v2.

Vancouver

1. Smith V, Chiang C-K, Sanjabi M, Talwalkar A (2017) Federated Multi-Task Learning. arXiv

BibTeX

@article{smith2017federated,
  title = {Federated Multi-Task Learning},
  author = {Smith, Virginia and Chiang, Chao-Kai and Sanjabi, Maziar and Talwalkar, Ameet},
  year = {2017},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1705.10467v2},
  eprint = {1705.10467}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Model	Human Activity	Google Glass	Vehicle Sensor
Global	2.23 (0.30)	5.34 (0.26)	13.4 (0.26)
Local	1.34 (0.21)	4.92 (0.26)	7.81 (0.13)
MTL	0.46 (0.11)	2.02 (0.15)	6.59 (0.21)

Federated Multi-Task Learning

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Federated Multi-Task Learning Formulation

Knowl 2 — MOCHA Algorithm for Federated Multi-Task Learning

Knowl 3 — Dual Formulation and Data-Local Subproblems in MOCHA

Knowl 4 — Per-Node Approximation Quality and Fault-Tolerance Model

Knowl 5 — Convergence Rate of MOCHA for Smooth Losses

Knowl 6 — Convergence Rate of MOCHA for Lipschitz Non-Smooth Losses

Knowl 7 — Selection of the Subproblem Difficulty Parameter

Knowl 8 — Supported Multi-Task Regularizers and Corresponding Metric Matrices

Knowl 9 — Asymptotic Convergence Under Diminishing Node Availability

Knowl 10 — Empirical Prediction Error of Multi-Task versus Global and Local Modeling

References

Citation

BibTeX

Access the Paper