Federated Learning with Personalization Layers

Manoj Ghuhan ArivazhaganVinay AggarwalAaditya Kumar SinghSunav Choudhary

article2019arXiv1,350 citations

Introduces FedPer, a federated learning framework that splits deep neural networks into shared base layers and local personalization layers to overcome performance degradation caused by statistical data heterogeneity across edge devices.

Listen

Modern edge devices generate massive volumes of personalized user data, but strict privacy constraints prevent centralizing this raw information. Federated learning enables devices to train machine learning models collaboratively on-device without sharing raw data. However, individual users naturally possess diverse preferences and behaviors, creating severe statistical differences in local data distributions. Standard federated learning techniques attempt to train a single global model for every device, which causes severe performance degradation and unfairness when tasks require personalized outputs.

The article introduces and evaluates FedPer, a novel training approach for deep feedforward neural networks that splits models into shared base layers and private personalization layers. The main objective is to demonstrate that this split architecture can overcome the performance failures of standard federated learning on heterogeneous personalization tasks while preserving data privacy.

To evaluate the method, the authors conducted extensive simulated experiments using two standard deep neural network architectures: ResNet-34 and MobileNet-v1. The evaluations tested the framework across 100 global aggregation rounds on benchmark image classification datasets (CIFAR-10 and CIFAR-100 across 10 clients with varying degrees of non-identical data splits) and a real-world personalized image aesthetics dataset (FLICKR-AES across 30 clients). Under the proposed framework, shared base layers are aggregated across devices via a central server using federated averaging, while private personalization layers remain strictly local and are updated solely using local data.

The findings show that FedPer delivers superior performance, converging faster and achieving significantly higher steady-state accuracy than standard federated averaging under severe data heterogeneity. On the real-world personalized aesthetics task, standard federated learning completely failed—performing no better than random guessing—whereas FedPer successfully learned distinct user preferences. Additionally, FedPer substantially reduced the variation in model accuracy across individual devices, ensuring fairer outcomes. Control experiments confirmed that both components are essential: shared base layers successfully extract shared visual features that purely local training cannot learn due to limited data, while at least one personalization layer is vital to capture unique user preferences.

These results demonstrate that standard federated learning is fundamentally ill-suited for personalized applications where identical inputs receive different user labels. Adopting a split-layer approach allows organizations to deliver highly tailored, high-performing user experiences without transferring sensitive personal data off edge devices. Furthermore, keeping personalization layers private reduces communication bandwidth requirements between client devices and central servers, lowering operational costs and network overhead.

Organizations developing personalized edge applications should adopt split-layer architectures like FedPer rather than enforcing single global models. Implementation teams should select personalization depths based on task complexity; experimental results show that 1 to 2 personal layers provide optimal performance without excessive local computation. Before deploying at scale, practitioners should conduct pilot studies on target hardware to evaluate edge device memory limits and test optional local fine-tuning steps, which improved accuracy on standard classification benchmarks but showed negligible benefits on aesthetic tasks.

The evaluations were conducted under controlled assumptions, including static client datasets, synchronous communications, and all participating devices remaining active throughout training. Because real-world deployments frequently experience intermittent connectivity and dynamic data updates, organizations should validate the approach under realistic network constraints and varied device hardware profiles.

Adobe Indian Institute of Technology Kharagpur Indian Institute of Technology Roorkee data privacy deep learning federated learning personalization layers FedPer federated learning

arXiv: 1912.00818

Paper: Communication-Efficient Learning of Deep Networks from Decentralized Data, H. B. McMahan et al. (2016). Introduces the foundational FederatedAveraging (FedAvg) algorithm and decentralized edge training framework that FedPer extends by decoupling base layers from personalized layers.
Paper: Federated Learning with Non-IID Data, Yue Zhao et al. (2018). Analyzes weight divergence and severe performance degradation in standard federated learning caused by non-IID data distributions, establishing the core problem that FedPer directly addresses.
Paper: Federated Optimization in Heterogeneous Networks, Tian Li et al. (2018). Formulates the mathematical challenge of statistical heterogeneity in federated networks and serves as a key optimization baseline for mitigating non-IID client drift.
Paper: Federated Multi-Task Learning, Virginia Smith et al. (2017). Provides the foundational multi-task learning formulation for edge devices that underpins the paradigm of learning personalized client models rather than a single global model.
Paper: Federated Optimization: Distributed Machine Learning for On-Device Intelligence, Jakub Konečný et al. (2016). Formulates the foundational distributed optimization constraints and on-device intelligence settings that necessitate personalized federated architectures.

Paper: Towards Personalized Federated Learning, Alysa Ziying Tan et al. (2021). Surveys the broader landscape and taxonomy of personalized federated learning, categorizing parameter-decoupling approaches like FedPer alongside meta-learning and distillation strategies.
Paper: Personalized Federated Learning with Moreau Envelopes, Canh T. Dinh et al. (2020). Advances personalized federated learning by formulating bi-level optimization via Moreau envelopes as an alternative regularization approach to layer-split personalization.
Paper: Ditto: Fair and Robust Federated Learning Through Personalization, Tian Li et al. (2020). Extends client-level personalization to simultaneously achieve robustness against adversarial attacks and fairness across heterogeneous client devices.
Paper: Federated Learning on Non-IID Data: A Survey, Hangyu Zhu et al. (2021). Provides a comprehensive taxonomy and evaluation of non-IID mitigation strategies, contextualizing local-layer personalization methods within parametric federated learning.
Paper: Federated Learning on Non-IID Data Silos: An Experimental Study, Qinbin Li et al. (2021). Presents an experimental benchmark across distinct non-IID partition types to systematically test algorithm robustness against the label and feature skews targeted by personalization.
Paper: Model-Contrastive Federated Learning, Qinbin Li et al. (2021). Proposes model-contrastive learning to correct local representation drift in heterogeneous settings, building on the feature-representation perspective of federated deep networks.

Cover for Federated Learning with Personalization Layers

Abstract

The emerging paradigm of federated learning strives to enable collaborative training of machine learning models on the network edge without centrally aggregating raw data and hence, improving data privacy. This sharply deviates from traditional machine learning and necessitates the design of algorithms robust to various sources of heterogeneity. Specifically, statistical heterogeneity of data across user devices can severely degrade the performance of standard federated averaging for traditional machine learning applications like personalization with deep learning. This paper pro-posesFedPer, a base + personalization layer approach for federated training of deep feedforward neural networks, which can combat the ill-effects of statistical heterogeneity. We demonstrate effectiveness ofFedPerfor non-identical data partitions ofCIFARdatasetsand on a personalized image aesthetics dataset from Flickr.

Topics

data privacy deep learning federated learning feedforward neural networks correlation analysis privacy preserving personalized service online learning

Keywords

personalization layers FedPer federated learning fine-tuning personalization meta-learning multi-task learning transfer learning cross-client variation

1 Introduction
1.1 Contributions
2 Related Work
3 Modeling and Algorithmic Setup
3.1 Model
3.2 Algorithm
4 Experiments
4.1 Datasets, Model Architectures, and Implementation Details
4.2 Effect of Statistical Heterogeneity
4.3 Effect of Personalization Layers
4.4 Do Base Layers Learn Anything?
4.5 Performance on \textFLICKR-AES
5 Conclusion
References
A Appendix
A.1 Effect of Statistical Heterogeneity
A.2 Effect of Personalization Layers
A.3 Effect of FineTuning

Knowls

Knowl 1 — FedPer Deep Architecture: Base and Personalization Layers
model/method
The FedPer (Federated Personalization) architecture decomposes a deep feedforward neural network into two sub-components: shared base layers and client-specific personalization layers.

Let $N$ be the number of participating client devices. Each client $j \in \{1, \dots, N\}$ maintains a local neural network composed of $K_B$ base layers followed by $K_P$ personalization layers:
- Base layers: Represented collectively by weight tensor $W_B = (W_{B,K_B}, \dots, W_{B,1})$ and vector-valued activation functions $(a_{B,K_B}, \dots, a_{B,1})$ . These parameters are shared across all clients and aggregated globally by a central server to learn domain-wide representations from the collective dataset.
- Personalization layers: Represented for client $j$ by weight tensor $W_{P_j} = (W_{P_j,K_P}, \dots, W_{P_j,1})$ and activation functions $(a_{P_j,K_P}, \dots, a_{P_j,1})$ . These parameters reside strictly on the local device, are never transmitted to the parameter server, and adapt directly to client $j$ 's specific data distribution.
For an input $x$ at client $j$ , the end-to-end forward pass $\hat{y} = f(x; W_B, W_{P_j})$ is computed as: $\hat{y} = a_{P_j, K_P}\Big(W_{P_j, K_P} \cdots a_{P_j, 1}\big(W_{P_j, 1} a_{B, K_B}(W_{B, K_B} \cdots a_{B, 1}(W_{B,1}x)\cdots)\big)\cdots\Big)$
Knowl 2 — FedPer Population and Empirical Risk Optimization
equation
In the FedPer framework, the learning goal across $N$ client devices is to minimize the average personalized population risk function $L^{PR}$ with respect to the globally shared base weights $W_B$ and private client-specific personalization weights $(W_{P_1}, \dots, W_{P_N})$ :

$L^{PR}(W_B, W_{P_1}, \dots, W_{P_N}) = \frac{1}{N} \sum_{j=1}^N \mathbb{E}_{(x,y) \sim P_j}\big[l(y, f(x; W_B, W_{P_j}))\big]$

where $P_j$ denotes the local joint data-label distribution on client $j$ , $l(\cdot, \cdot)$ denotes a common sample loss function, and $f(x; W_B, W_{P_j})$ is the output prediction given base weights $W_B$ and local personalization weights $W_{P_j}$ .

Because the data-generating distributions $P_j$ are unknown, client $j$ optimizes the local empirical risk $L_j^{ER}(W_B, W_{P_j})$ over its $n_j$ local samples $\{(x_{j,i}, y_{j,i})\}_{i=1}^{n_j}$ :

$L_j^{ER}(W_B, W_{P_j}) = \frac{1}{n_j} \sum_{i=1}^{n_j} l\big(y_{j,i}, f(x_{j,i}; W_B, W_{P_j})\big)$
Knowl 3 — FedPer Training Algorithm
algorithm
FedPer trains deep neural networks in a federated environment by alternating between local stochastic gradient descent (SGD) updates across both base and personalization layers on edge clients, and parameter aggregation of only base layers on a central parameter server.
Algorithm: FedPer
Server Process:
Input: Initial base layer weights $W_B^{(0)}$ , number of rounds $K$ , client dataset sizes $\{n_1, \dots, n_N\}$
Output: Final global base layer weights $W_B^{(K)}$
1: Compute aggregation weights $\gamma_j = n_j / \sum_{m=1}^N n_m$ for each client $j \in \{1, \dots, N\}$
2: Send $W_B^{(0)}$ to all clients $j \in \{1, \dots, N\}$
3: for global round $k = 1, 2, \dots, K$ do
4: for each client $j \in \{1, \dots, N\}$ in parallel do
5: $W_{B,j}^{(k)}, W_{P_j}^{(k)} \leftarrow \text{ClientUpdate}(j, W_B^{(k-1)})$
6: end for
7: Receive local base weights $W_{B,j}^{(k)}$ from each client $j$
8: Aggregate base weights: $W_B^{(k)} \leftarrow \sum_{j=1}^N \gamma_j W_{B,j}^{(k)}$
9: Send $W_B^{(k)}$ to all clients
10: end for
ClientUpdate $(j, W_B^{(k-1)})$ :
Input: Client index $j$ , current global base weights $W_B^{(k-1)}$ , local dataset $\{(x_{j,i}, y_{j,i})\}_{i=1}^{n_j}$ , local epochs $e$ , batch size $b$ , learning rate $\eta_j^{(k)}$
Output: Updated base weights $W_{B,j}^{(k)}$ and updated personalization weights $W_{P_j}^{(k)}$
1: Set local model parameters $(W_B, W_{P_j}) \leftarrow (W_B^{(k-1)}, W_{P_j}^{(k-1)})$
2: for epoch $= 1, \dots, e$ do
3: for each batch $B$ of size $b$ drawn from local dataset do
4: Compute gradient of batch loss w.r.t. both $W_B$ and $W_{P_j}$
5: Update $(W_B, W_{P_j}) \leftarrow (W_B, W_{P_j}) - \eta_j^{(k)} \nabla L_B(W_B, W_{P_j})$
6: end for
7: end for
8: Send $W_{B,j}^{(k)} = W_B$ to server and retain $W_{P_j}^{(k)} = W_{P_j}$ locally
9: return $W_{B,j}^{(k)}, W_{P_j}^{(k)}$
In standard experimental settings, clients run $e = 4$ local epochs per round with learning rate $\eta_j^{(k)} = 0.01$ , sending only base parameters $W_{B,j}$ over the network while personal parameters $W_{P_j}$ remain on-device.
Knowl 4 — FedPer Performance and Client Fairness under Non-IID Class Partitions
empirical result
On image classification tasks under statistical heterogeneity, FedPer achieves superior convergence speed, higher steady-state test accuracy, and lower cross-client performance variance compared to standard Federated Averaging (FedAvg).

When evaluating ResNet-34 (16 residual blocks) and MobileNet-v1 (11 depthwise/pointwise blocks) on CIFAR-10 across $N = 10$ clients with non-identical partitions where each client possesses data from at most $k \in \{4, 8, 10\}$ classes:
- Under severe non-IID conditions ( $k = 4$ ), FedPer converges significantly faster and attains a higher client-averaged test accuracy at steady state than FedAvg.
- As the data distribution approaches IID ( $k \to 10$ ), the performance advantage of FedPer narrows, converging toward FedAvg.
- FedPer substantially reduces the variance of test accuracies across individual clients (inter-client fairness) under non-IID settings ( $k = 4$ ), whereas FedAvg exhibits broad fluctuations and disparity in client accuracy.
Knowl 5 — Personalization Failure of Global Federated Averaging on Conflicting User Labels
empirical result
On personalized tasks where identical inputs receive divergent subjective ratings from different users (such as personalized image aesthetics in the FLICKR-AES dataset), global Federated Averaging (FedAvg) fails fundamentally, whereas FedPer succeeds.

In experiments on FLICKR-AES ( $40{,}000$ images rated on a 1-to-5 scale across 210 users; evaluated on $N = 30$ client users, $b = 4$ , $e = 4$ ):
- FedAvg achieves only random-guessing-level accuracy (approximately $15\%\text{--}20\%$ ) on both MobileNet-v1 and ResNet-34 architectures. Because FedAvg enforces a single identical global model across all clients, it cannot produce different target predictions for the same input across distinct users.
- FedPer maintains local personalization layers that adapt to user-specific aesthetic preferences, achieving approximately $40\%$ test accuracy on both MobileNet-v1 and ResNet-34.
Knowl 6 — Effect of Personalization Layer Depth ($K_P$) on Task Accuracy
empirical result
Varying the number of personalized basic blocks $K_P$ (where $K_P = 1$ designates personalizing only the final classification fully connected layer, and $K_P > 1$ includes the classifier plus the last $K_P - 1$ basic convolutional/residual blocks) shows that maintaining at least one local personalization layer ( $K_P \ge 1$ ) provides the decisive gain over global aggregation ( $K_P = 0$ , FedAvg).

Empirical sweeps over $K_P \in \{1, 2, 3, 4\}$ on CIFAR datasets demonstrate:
- On CIFAR-10 (evaluated on ResNet-34 and MobileNet-v1), $K_P = 2$ (the classification layer plus one preceding basic block) achieves the highest client-averaged steady-state test accuracy.
- On CIFAR-100, $K_P = 1$ (personalizing solely the linear classification layer) achieves the highest test accuracy.
- Beyond $K_P \ge 1$ , increasing the depth of personalization layers further does not yield a monotonic increase in performance.
Knowl 7 — Representational Necessity of Deep Federated Base Layers
empirical result
The collaborative base layers in FedPer extract meaningful complex representations that cannot be replicated by shallow shared layers or purely isolated client training.

When replacing the deep base convolutional blocks of ResNet-34 and MobileNet-v1 with a single linear fully connected layer while keeping personalization layers identical:
- On CIFAR-100 and FLICKR-AES, the linear-base models suffer significant degradation in test accuracy compared to full deep base architectures.
- Purely local training on individual edge devices without federated base layer sharing results in poor accuracy because local sample sizes per client are insufficient to train deep models in isolation.
These results confirm that collaborative federated training of deep base layers is essential to leverage collective data while local personalization layers handle client-specific decision boundaries.
Knowl 8 — Effect of Local Pre-Fine-Tuning on Personalization Layers
empirical result
A variant of FedPer incorporates an explicit fine-tuning step at the beginning of each global round: upon receiving new global base weights $W_B^{(k-1)}$ , clients freeze $W_B^{(k-1)}$ and optimize only the local personalization weights $W_{P_j}$ for 1 epoch before resuming standard joint SGD over all layers for $e$ epochs.
- On CIFAR-100 (using MobileNet-v1 and ResNet-34), local pre-fine-tuning improves convergence stability and yields higher client test accuracy, confirming that allowing personalization layers to adapt to new base parameters before joint optimization helps accommodate inter-round shifts.
- On FLICKR-AES, pre-fine-tuning produces no noticeable change in final test accuracy compared to standard FedPer.

Coverage note — None omitted; all primary contributions, algorithmic formulations, architectural specifications, and empirical findings regarding FedPer have been covered.

References

1.Bottou, L., Curtis, F. E., and Nocedal, J. (2018). Optimization methods for large-scale machine learning. SIAM Review, 60(2):223–311.
2.Chen, F., Dong, Z., Li, Z., and He, X. (2018). Federated Meta-Learning for Recommendation. ArXiv, abs/1802.07876.
3.Chen, Y., Wang, J., Yu, C., Gao, W., and Qin, X. (2019). FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare. ArXiv, abs/1907.09173.
4.Garcia del Molino, A. and Gygli, M. (2018). PHD-GIFs: Personalized Highlight Detection for Automatic GIF Creation. In Proceedings of the 26th ACM International Conference on Multimedia, MM '18, pages 600–608. ACM.
5.Harper, F. M. and Konstan, J. A. (2015). The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst., 5(4):19:1–19:19.
6.Konečny, J., McMahan, H. B., and Ramage, D. (2015). Federated Optimization: Distributed Optimization Beyond the Datacenter. ArXiv, abs/1511.03575.
7.Konečny, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., and Bacon, D. (2018). Federated Learning: Strategies for Improving Communication Efficiency. ArXiv, abs/1610.05492.
8.Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical report, Citeseer.
9.Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. L. (2019). Federated Learning: Challenges, Methods, and Future Directions. ArXiv, abs/1908.07873.
10.McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data. In Singh, A. and Zhu, J., editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273–1282, Fort Lauderdale, FL, USA. PMLR.
11.Miao, X., Chu, C.-T., Tang, L., Zhou, Y., Young, J., and Bhasin, A. (2015). Distributed Personalization. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, pages 1989–1998. ACM.
12.Ren, J., Shen, X., Lin, Z., Mech, R., and Foran, D. J. (2017). Personalized Image Aesthetics. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 638–647.
13.Ruder, S. (2017). An Overview of Multi-Task Learning in Deep Neural Networks. ArXiv, abs/1706.05098.
14.Sahu, A. K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., and Smith, V. (2018). Federated Optimization in Heterogeneous Networks. ArXiv, abs/1812.06127.
15.Smith, V., Chiang, C.-K., Sanjabi, M., and Talwalkar, A. (2017). Federated Multi-task Learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4427–4437. Curran Associates Inc.
16.ud din, M. A., Ivannikova, E., Khan, S. A., Oyomno, W., Fu, Q., Tan, K. E., and Flanagan, A. (2019). Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System. ArXiv, abs/1901.09888.
17.Vepakomma, P., Gupta, O., Swedish, T., and Raskar, R. (2018). Split learning for health: Distributed deep learning without sharing raw patient data. ArXiv, abs/1812.00564.
18.Yang, Q., Liu, Y., Chen, T., and Tong, Y. (2019). Federated Machine Learning: Concept and Applications. ACM Trans. Intell. Syst. Technol., 10(2):12:1–12:19.
19.Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. (2018). Federated Learning with Non-IID Data. ArXiv, abs/1806.00582.

Citation

MLA

Arivazhagan, M. G., et al. “Federated Learning with Personalization Layers”. arXiv, 2019, http://arxiv.org/abs/1912.00818v1.

APA

Arivazhagan, M. G., Aggarwal, V., Singh, A. K., & Choudhary, S. (2019). Federated Learning with Personalization Layers. arXiv. http://arxiv.org/abs/1912.00818v1

Chicago

Arivazhagan, M. G., V. Aggarwal, A. K. Singh, and S. Choudhary. 2019. “Federated Learning with Personalization Layers”. arXiv. http://arxiv.org/abs/1912.00818v1.

Harvard

Arivazhagan, M.G. et al. (2019) “Federated Learning with Personalization Layers”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1912.00818v1.

Vancouver

1. Arivazhagan MG, Aggarwal V, Singh AK, Choudhary S (2019) Federated Learning with Personalization Layers. arXiv

BibTeX

@article{arivazhagan2019federated,
  title = {Federated Learning with Personalization Layers},
  author = {Arivazhagan, Manoj Ghuhan and Aggarwal, Vinay and Singh, Aaditya Kumar and Choudhary, Sunav},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1912.00818v1},
  eprint = {1912.00818}
}

Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Federated Learning with Personalization Layers

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — FedPer Deep Architecture: Base and Personalization Layers

Knowl 2 — FedPer Population and Empirical Risk Optimization

Knowl 3 — FedPer Training Algorithm

Knowl 4 — FedPer Performance and Client Fairness under Non-IID Class Partitions

Knowl 5 — Personalization Failure of Global Federated Averaging on Conflicting User Labels

Knowl 6 — Effect of Personalization Layer Depth ($K_P$) on Task Accuracy

Knowl 7 — Representational Necessity of Deep Federated Base Layers

Knowl 8 — Effect of Local Pre-Fine-Tuning on Personalization Layers

References

Citation

BibTeX

Access the Paper