Back

Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data

Timothy J. CastigliaAnirban DasShiqiang WangStacy Patterson

article2022ICML72 citations

Establishes theoretical convergence guarantees and practical techniques for vertical federated learning with compressed intermediate embeddings, cutting communication costs by over 90% without sacrificing model accuracy.

Listen

Organizations such as hospitals, banks, and insurers frequently need to train collaborative machine learning models on the same set of individuals without sharing sensitive, locally held features. This setup, known as vertical federated learning, enables collaborative modeling across partitioned feature sets while preserving data privacy and complying with strict regulations. However, exchanging the necessary intermediate vector representations, known as embeddings, across distributed networks creates massive communication bottlenecks that can require terabytes of bandwidth and significantly slow down training.

The article aims to evaluate whether introducing message compression and multiple local training updates to vertically partitioned federated learning can substantially reduce communication overhead without degrading model convergence or predictive accuracy.

To evaluate this framework, named Compressed Vertical Federated Learning, the authors established theoretical convergence proofs for complex, non-linear server models and non-convex objectives. They derived specific parameter bounds for common compression methods, including uniform scalar quantization, lattice vector quantization, and top-k sparsification. They then conducted extensive empirical experiments across diverse benchmark datasets, including healthcare records (MIMIC-III), 3D computer-aided design views (ModelNet10), and image classification sets (CIFAR-10 and ImageNet-100), simulating different party counts and network latencies.

The study established several key findings. First, theoretical analysis proved that the compressed vertical framework maintains standard convergence rates when compression error is properly bounded over training. Second, empirical tests demonstrated that compressing embeddings down to as few as 2 to 4 bits per component reduced overall communication costs by over 90% compared to uncompressed baselines while attaining nearly identical predictive accuracy and F1-scores. Third, across the evaluated compression schemes, 2-dimensional lattice vector quantization consistently delivered the strongest performance and reconstruction stability. Finally, executing multiple local iterations per communication round significantly cut the elapsed training time needed to hit target accuracy metrics, delivering the greatest speedups in high-latency network environments.

These findings indicate that organizations can train high-capacity vertical federated models across geographically dispersed entities at a fraction of standard bandwidth costs and time delays. By compressing intermediate representations rather than transmitting full-precision data, teams can overcome infrastructure limitations without compromising model quality or regulatory compliance.

Decision-makers should consider adopting compressed vertical federated learning when deploying distributed models across feature-partitioned organizations, prioritizing lattice vector quantization as the primary compression technique. Engineering teams should tune the number of local update iterations based on prevailing network latency, opting for higher local iteration counts when operating over high-latency connections. Additional pilot validation is advised before applying the approach in environments with unconstrained or unbounded embedding distributions, and future evaluations should explore adaptive compression mechanisms.

communicationVertical federated learninggradient compressionvector quantization
Castiglia et al (2022).pdf

No sufficiently relevant recommendations were found.

Cover for Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data

Abstract

We propose Compressed Vertical Federated Learning (C-VFL) for communication-efficient training on vertically partitioned data. In C-VFL, a server and multiple parties collaboratively train a model on their respective features utilizing several local iterations and sharing compressed intermediate results periodically. Our work provides the first theoretical analysis of the effect message compression has on distributed training over vertically partitioned data. We prove convergence of non-convex objectives at a rate of O( 1/√T ) when the compression error is bounded over the course of training. We provide specific requirements for convergence with common compression techniques, such as quantization and top-k sparsification. Finally, we experimentally show compression can reduce communication by over 90% without a significant decrease in accuracy over VFL without compression.

Topics

communication

Keywords

Vertical federated learninggradient compressionvector quantizationscalar quantizationnon-convex objectives

Table of Contents

  • 1. Introduction
  • 2. Problem Formulation
  • 3. Algorithm
  • 4. Analysis
  • 5. Experiments
  • 6. Conclusion
  • Acknowledgements
  • References
  • A. Proofs of Theorems 4.2 and 4.4
  • A.1. Additional Notation
  • A.2. Supporting Lemmas
  • A.3. Proof of Theorems 4.2 and 4.4
  • B. Common Compressors
  • C. Experimental Details
  • C.1. MIMIC-III
  • C.2. ModelNet10 and CIFAR10
  • C.3. ImageNet
  • D. Additional Plots and Experiments
  • D.1. Additional Plots
  • D.2. Additional Experiments With ImageNet
  • D.3. Comparison With Alternative C-VFL Algorithm For Q = 1

Knowls

  1. Knowl 1 — C-VFL trains on compressed snapshots between communication rounds

    model/method

    Compressed Vertical Federated Learning (C-VFL) trains a model whose features are divided among a server and MM parties. Let QQ be the number of local iterations per communication round, RR the number of rounds, and T=RQT=RQ the total number of local iterations. Each party mm has parameters θm\theta_m and an embedding function hmh_m; the server has parameters θ0\theta_0.

    At the start of each round t0t_0, sample a mini-batch of BB aligned examples. Each party sends the server its batch of embeddings compressed by a map CmC_m, and the server forms and broadcasts the collection of compressed party embeddings together with its compressed server-model parameters C0(θ0)C_0(\theta_0). For the next QQ local iterations, each block updates its own parameters in parallel by a stochastic gradient step of size ηt0\eta^{t_0}. When computing a block’s gradient, use its current, uncompressed local embedding or parameters, but use the most recently broadcast compressed snapshots for the other blocks. Reuse the same mini-batch and received snapshots throughout those QQ iterations; communicate again at the next round. Initialize all blocks’ parameters and stop after RR rounds. The compressors may be arbitrary maps Cm:RPm→RPmC_m:\mathbb{R}^{P_m}\to\mathbb{R}^{P_m}, where PmP_m is the embedding dimension of block mm.

    Without compression, the paper gives per-round communication cost O ⁣(M(B∑m=0MPm+∣θ0∣))O\!\left(M\left(B\sum_{m=0}^{M}P_m+|\theta_0|\right)\right); compression reduces the transmitted representation sizes. The method supports an arbitrary trainable server or fusion model and permits multiple local updates between communications.

  2. Knowl 2 — Non-convex objective for vertically partitioned learning

    definition

    Let NN aligned samples have labels yiy_i and feature vectors xi=[xi1,…,xiM]x_i=[x_{i1},\ldots,x_{iM}], where party mm holds only its disjoint feature block ximx_{im}. Party mm has parameters θm\theta_m and computes embedding hm(θm;xim)h_m(\theta_m;x_{im}); the server has parameters θ0\theta_0 and combines the embeddings using loss function ℓ\ell. The global parameter vector is Θ=[θ0T,…,θMT]T\Theta=[\theta_0^T,\ldots,\theta_M^T]^T. The training objective is the mean loss over the aligned samples:

    F(Θ;X;y)=1N∑i=1Nℓ ⁣(θ0,h1(θ1;xi1),…,hM(θM;xiM);yi).F(\Theta;X;y)=\frac{1}{N}\sum_{i=1}^{N}\ell\!\left(\theta_0,h_1(\theta_1;x_{i1}),\ldots,h_M(\theta_M;x_{iM});y_i\right).

    The paper assumes that the server and parties have access to the labels. The optimization problem is to minimize FF while each party keeps its raw feature block local.

  3. Knowl 3 — Fixed-step convergence bound for C-VFL

    theoretical result

    For the non-convex C-VFL objective FF, let MM denote the number of parties, with block m=0m=0 denoting the server; let QQ be local iterations per round, RR the number of global rounds, and T=RQT=RQ. Let BB be the mini-batch size, η\eta the fixed step size, and Θt0\Theta^{t_0} the model at the start of round t0t_0. Assume: the full gradient is LL-Lipschitz; each stochastic block gradient is LmL_m-Lipschitz in the model; mini-batch block gradients are unbiased at the start of a round and have variance at most σm2/B\sigma_m^2/B; the mini-batch loss Hessian with respect to block mm’s embedding has Frobenius norm at most HmH_m; and the embedding Jacobian with respect to θm\theta_m has Frobenius norm at most GmG_m. If

    0<η≤116Qmax⁡{L,max⁡mLm},0<\eta\leq\frac{1}{16Q\max\{L,\max_m L_m\}},

    then the average expected squared full-gradient norm at round starts satisfies

    1R∑t0=0R−1E ⁣[∥∇F(Θt0)∥2]≤4(F(Θ0)−E[F(ΘT)])ηT+6ηQL∑m=0Mσm2B+92Q2R∑m=0MHm2Gm2∑t0=0R−1∑j=0j≠mMEjt0.\frac{1}{R}\sum_{t_0=0}^{R-1}\mathbb{E}\!\left[\|\nabla F(\Theta^{t_0})\|^2\right] \leq \frac{4\left(F(\Theta^0)-\mathbb{E}[F(\Theta^T)]\right)}{\eta T} +6\eta QL\sum_{m=0}^{M}\frac{\sigma_m^2}{B} +\frac{92Q^2}{R}\sum_{m=0}^{M}H_m^2G_m^2\sum_{t_0=0}^{R-1}\sum_{\substack{j=0\\j\ne m}}^{M}E_j^{t_0}.

    Here Ejt0E_j^{t_0} is the expected squared Frobenius norm of the compressed-message error for block jj’s batch embedding in round t0t_0. Thus the bound accounts separately for objective decrease, stochastic-gradient variance, and compression error; the last contribution grows with the accumulated message error and with the number of local iterations.

    In particular, with QQ and BB fixed, choosing η=1/T\eta=1/\sqrt{T} and keeping the average compression error 1R∑t0=0R−1∑m=0MEmt0=O(1/T)\frac{1}{R}\sum_{t_0=0}^{R-1}\sum_{m=0}^{M}E_m^{t_0}=O(1/\sqrt{T}) gives an O(1/T)O(1/\sqrt{T}) rate for this average squared gradient measure.

  4. Knowl 4 — Compression error perturbs a party’s gradient through other parties’ embeddings

    theoretical result

    Suppose the mini-batch loss has embedding Hessian Frobenius norm bounded by HmH_m for block mm, and the Jacobian of its embedding with respect to its parameters has Frobenius norm bounded by GmG_m. Let Ejt0E_j^{t_0} be the expected squared Frobenius norm of the compression error in party jj’s batch embedding at communication round t0t_0. Compare block mm’s stochastic gradient computed with compressed embeddings from the other blocks against the gradient computed with the same models, batch, and uncompressed embeddings. The expected squared difference is bounded by

    E ⁣[∥∇mFB(Φ^mt)−∇mFB(Φmt)∥2]≤Hm2Gm2∑j=0j≠mMEjt0.\mathbb{E}\!\left[\left\|\nabla_m F_B(\widehat{\Phi}_m^t)-\nabla_m F_B(\Phi_m^t)\right\|^2\right] \leq H_m^2G_m^2\sum_{\substack{j=0\\j\ne m}}^{M}E_j^{t_0}.

    Here FBF_B is the mini-batch objective, Φ^mt\widehat{\Phi}_m^t is the input embedding collection with compression errors, and Φmt\Phi_m^t is the corresponding collection without those errors. Block mm’s own embedding is not compressed for its local gradient calculation, so only errors from blocks j≠mj\ne m enter this bound. This result quantifies why embedding compression in vertical federated learning affects gradients differently from directly compressing gradients.

  5. Knowl 5 — Diminishing steps and compression errors yield convergence to stationarity

    theoretical result

    Use the same smoothness, unbiased-gradient, bounded-variance, bounded-embedding-Hessian, and bounded-embedding-Jacobian assumptions as for C-VFL’s fixed-step analysis. Let ηt0\eta^{t_0} be the step size at round t0t_0, and let Emt0E_m^{t_0} be block mm’s expected squared Frobenius compression error for that round’s batch. If 0<ηt0<10<\eta^{t_0}<1 and

    ηt0≤116Qmax⁡{L,max⁡mLm},\eta^{t_0}\leq\frac{1}{16Q\max\{L,\max_m L_m\}},

    then the minimum expected squared gradient norm over round starts has the bound

    min⁡t0=0,…,R−1E ⁣[∥∇F(Θt0)∥2]=O ⁣(1∑t0=0R−1ηt0+∑t0=0R−1(ηt0)2∑t0=0R−1ηt0+∑t0=0R−1ηt0∑m=0MEmt0∑t0=0R−1ηt0).\min_{t_0=0,\ldots,R-1}\mathbb{E}\!\left[\|\nabla F(\Theta^{t_0})\|^2\right] =O\!\left( \frac{1}{\sum_{t_0=0}^{R-1}\eta^{t_0}} +\frac{\sum_{t_0=0}^{R-1}(\eta^{t_0})^2}{\sum_{t_0=0}^{R-1}\eta^{t_0}} +\frac{\sum_{t_0=0}^{R-1}\eta^{t_0}\sum_{m=0}^{M}E_m^{t_0}}{\sum_{t_0=0}^{R-1}\eta^{t_0}} \right).

    Consequently, the minimum expected squared gradient norm tends to zero as the number of rounds grows if ∑t0ηt0=∞\sum_{t_0}\eta^{t_0}=\infty, ∑t0(ηt0)2<∞\sum_{t_0}(\eta^{t_0})^2<\infty, and ∑t0ηt0∑mEmt0<∞\sum_{t_0}\eta^{t_0}\sum_m E_m^{t_0}<\infty. The condition concerns the step-size-weighted compression errors, not merely whether each individual message is compressed.

  6. Knowl 6 — Quantization and top-k settings that control C-VFL compression error

    model/method

    The paper derives parameter choices for three embedding compressors so that their expected batch compression error can scale as O(1/T)O(1/\sqrt{T}), as required to retain C-VFL’s stated fixed-step rate. Let PmP_m be embedding dimension, BB batch size, and TT total local iterations. For scalar quantization, qq is the number of bits per component and there are 2q2^q levels; hmax⁡h_{\max} and hmin⁡h_{\min} bound every embedding component over training. For 2-dimensional hexagonal lattice vector quantization, VV is the volume of a lattice cell. For top-kk sparsification, kk components are transmitted and (∥h∥2)max⁡(\|h\|^2)_{\max} bounds the squared embedding norm over training.

    Compressor Parameter choice Expected squared batch error bound
    Scalar quantization q=Ω ⁣(log⁡2 ⁣(BPm(hmax⁡−hmin⁡)2T))q=\Omega\!\left(\log_2\!\left(BP_m(h_{\max}-h_{\min})^2\sqrt{T}\right)\right) Emt0≤BPm(hmax⁡−hmin⁡)212 2−2qE_m^{t_0}\leq \dfrac{BP_m(h_{\max}-h_{\min})^2}{12}\,2^{-2q}
    2-D lattice vector quantization V=O ⁣(1BPmT)V=O\!\left(\dfrac{1}{BP_m\sqrt{T}}\right) Emt0≤VBPm24E_m^{t_0}\leq \dfrac{VBP_m}{24}
    Top-kk sparsification k=Ω ⁣(Pm−PmB(∥h∥2)max⁡T)k=\Omega\!\left(P_m-\dfrac{P_m}{B(\|h\|^2)_{\max}\sqrt{T}}\right) Emt0≤B(1−kPm)(∥h∥2)max⁡E_m^{t_0}\leq B\left(1-\dfrac{k}{P_m}\right)(\|h\|^2)_{\max}

    The scalar and lattice bounds require bounded embedding values; the paper notes that their error is unbounded if those values are unbounded. In the C-VFL top-kk implementation, the selected components are those with largest-magnitude embedding gradients, estimated using the previous iteration’s embedding gradient because the current gradient is unavailable until embeddings are exchanged. To obtain diminishing error for the diminishing-step convergence result, the parameter choices must become more accurate over time: increase quantization levels, shrink lattice cells, or transmit more components.

  7. Knowl 7 — MIMIC-III mortality results: two-bit embeddings greatly reduce communication

    empirical result

    For MIMIC-III in-hospital mortality prediction, C-VFL used four parties, each holding 19 of 76 features and training an LSTM; the server used two fully connected layers. Runs used batch size 1000, step size 0.01, 1000 epochs, and Q=10Q=10. Each embedding component was compressed to 2, 3, or 4 bits, or left at 32 bits for the uncompressed baseline. The table reports the mean and standard deviation over five runs for maximum test F1-score and communication cost to reach test F1-score 0.4; communication includes embedding uploads and downloads of embeddings and server parameters.

    Compressor Max F1-score reached Cost to reach F1-score 0.4 (MB)
    None, b=32b=32 0.448±0.0100.448\pm0.010 3830.0±558.23830.0\pm558.2
    Scalar, b=2b=2 0.441±0.0180.441\pm0.018 233.1±28.7233.1\pm28.7
    Vector, b=2b=2 0.451±0.0210.451\pm0.021 236.1±17.9236.1\pm17.9
    Top-kk, b=2b=2 0.431±0.0160.431\pm0.016 309.8±93.6309.8\pm93.6
    Scalar, b=3b=3 0.446±0.0110.446\pm0.011 343.1±18.8343.1\pm18.8
    Vector, b=3b=3 0.455±0.0200.455\pm0.020 330.5±10.6330.5\pm10.6
    Top-kk, b=3b=3 0.435±0.0300.435\pm0.030 470.7±116.8470.7\pm116.8
    Scalar, b=4b=4 0.451±0.0200.451\pm0.020 456.0±87.8456.0\pm87.8
    Vector, b=4b=4 0.446±0.0170.446\pm0.017 446.5±21.3446.5\pm21.3
    Top-kk, b=4b=4 0.453±0.0140.453\pm0.014 519.1±150.4519.1\pm150.4

    The maximum F1-scores are similar across compression levels and the uncompressed baseline, while every compressed configuration reaches the target with much less communication. At two bits, costs are about 233–310 MB instead of 3830.0 MB; scalar quantization cuts cost by about 94%. The epoch and communication-cost trajectories plotted on page 7 likewise show compressed training reaching similar test F1-scores with substantially fewer transmitted bytes.

  8. Knowl 8 — CIFAR-10 results: two-bit vector quantization reaches the target with about 91% less communication

    empirical result

    For CIFAR-10 image classification, four parties each held a different quadrant of every image and trained ResNet18 models; the server used a fully connected layer. Training used batch size 100, step size 0.0001, 200 epochs, and Q=10Q=10. The table gives mean and standard deviation over five runs for maximum test accuracy and communication cost to reach 70% test accuracy. Embedding components used b=2,3,4b=2,3,4 bits, with a 32-bit uncompressed baseline; a dash means the target was not reached during training.

    Compressor Max accuracy reached Cost to reach 70% accuracy (GB)
    None, b=32b=32 73.18%±0.44%73.18\%\pm0.44\% 7.69±0.357.69\pm0.35
    Scalar, b=2b=2 65.16%±1.85%65.16\%\pm1.85\% –
    Vector, b=2b=2 71.43%±0.47%71.43\%\pm0.47\% 0.68±0.060.68\pm0.06
    Top-kk, b=2b=2 66.02%±2.24%66.02\%\pm2.24\% –
    Scalar, b=3b=3 71.49%±1.05%71.49\%\pm1.05\% 1.22±0.171.22\pm0.17
    Vector, b=3b=3 72.50%±0.40%72.50\%\pm0.40\% 0.81±0.050.81\pm0.05
    Top-kk, b=3b=3 71.56%±0.81%71.56\%\pm0.81\% 1.24±0.221.24\pm0.22
    Scalar, b=4b=4 71.80%±1.18%71.80\%\pm1.18\% 1.72±0.261.72\pm0.26
    Vector, b=4b=4 73.17%±0.39%73.17\%\pm0.39\% 0.98±0.080.98\pm0.08
    Top-kk, b=4b=4 72.03%±1.77%72.03\%\pm1.77\% 1.43±0.261.43\pm0.26

    Two-bit vector quantization is the only two-bit method in this comparison to reach the target, doing so with 0.680.68 GB versus 7.697.69 GB uncompressed, a reduction of about 91%. The plotted test-accuracy trajectories on page 7 show vector quantization tracking the uncompressed model more closely than the other two-bit compressors, while the communication-cost plots show its advantage in bytes sent.

  9. Knowl 9 — ModelNet10 results: vector quantization preserves accuracy at sharply lower cost

    empirical result

    For ModelNet10 classification, the reported comparison used four parties, each receiving three of the 12 camera views of each CAD model. Each party used two convolutional layers and a fully connected layer, and the server used a fully connected layer. Training used batch size 64, step size 0.001, 100 epochs, and Q=10Q=10. The table reports mean and standard deviation over five runs for maximum test accuracy and communication cost to reach 75% test accuracy, using 2, 3, or 4 bits per embedding component and a 32-bit uncompressed baseline.

    Compressor Max accuracy reached Cost to reach 75% accuracy (MB)
    None, b=32b=32 85.68%±1.57%85.68\%\pm1.57\% 9604.80±2933.409604.80\pm2933.40
    Scalar, b=2b=2 76.94%±5.87%76.94\%\pm5.87\% 1932.00±674.301932.00\pm674.30
    Vector, b=2b=2 84.80%±2.58%84.80\%\pm2.58\% 593.40±170.98593.40\pm170.98
    Top-kk, b=2b=2 79.91%±2.86%79.91\%\pm2.86\% 1317.90±222.951317.90\pm222.95
    Scalar, b=3b=3 81.32%±1.61%81.32\%\pm1.61\% 1738.80±254.791738.80\pm254.79
    Vector, b=3b=3 85.66%±1.36%85.66\%\pm1.36\% 900.45±275.01900.45\pm275.01
    Top-kk, b=3b=3 81.63%±1.24%81.63\%\pm1.24\% 1593.90±225.341593.90\pm225.34
    Scalar, b=4b=4 81.19%±1.88%81.19\%\pm1.88\% 2194.20±266.882194.20\pm266.88
    Vector, b=4b=4 85.77%±1.69%85.77\%\pm1.69\% 1200.60±366.681200.60\pm366.68
    Top-kk, b=4b=4 83.50%±1.21%83.50\%\pm1.21\% 1821.60±241.401821.60\pm241.40

    Vector quantization gives the lowest target-reaching communication cost at each reported bit rate and maintains a maximum accuracy close to the uncompressed baseline. At two bits, it reaches 75% accuracy using 593.40593.40 MB rather than 9604.809604.80 MB. The accuracy-versus-epoch and accuracy-versus-communication-cost curves plotted on pages 7 and 9 show the same pattern: vector quantization approaches the uncompressed accuracy using substantially fewer communicated bytes.

  10. Knowl 10 — More local iterations reduce time to target when communication latency is high

    empirical result

    On MIMIC-III, C-VFL used vector quantization with b=3b=3 bits per component and targeted test F1-score 0.45. The experiment simulated 10 ms of computation per mini-batch at each party and round-trip embedding communication latency tct_c of 1, 10, 50, or 200 ms. The entries are elapsed seconds to reach the target, reported as mean ±\pm one standard deviation over five runs.

    Round-trip latency tct_c (ms) Q=1Q=1 Q=10Q=10 Q=25Q=25
    1 694.53±150.75694.53\pm150.75 470.86±235.35470.86\pm235.35 445.21±51.44445.21\pm51.44
    10 1262.78±274.101262.78\pm274.10 512.82±256.32512.82\pm256.32 461.17±53.29461.17\pm53.29
    50 3788.32±822.303788.32\pm822.30 699.30±349.53699.30\pm349.53 532.12±61.49532.12\pm61.49
    200 13259.14±2878.0413259.14\pm2878.04 1398.60±699.051398.60\pm699.05 798.19±92.23798.19\pm92.23

    Increasing QQ reduces communication frequency and improves time to target in this simulation, with the largest benefit when latency is high: at tc=200t_c=200 ms, increasing from Q=1Q=1 to Q=25Q=25 reduces mean time from 13259.14 s to 798.19 s. This is an empirical communication-versus-computation trade-off; the convergence bound also indicates that more local iterations can increase optimization error.

Coverage note — The paper’s separate $Q=1$ server-gradient-transfer variant and additional ImageNet-100 experiments were omitted because they are secondary extensions or supplementary evaluations rather than load-bearing components of the main C-VFL method, convergence analysis, and three primary dataset results.

References

  1. 1.Bennett, W. R. Spectra of quantized signals. Bell Syst. Tech. J., 27(3):446–472, 1948.
  2. 2.Bernstein, J., Wang, Y., Azizzadenesheli, K., and Anandkumar, A. SIGNSGD: compressed optimisation for non-convex problems. Proc. Int. Conf. on Machine Learn., 2018.
  3. 3.Bonawitz, K. A., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for federated learning on user-held data. arXiv:1611.04482, 2016.
  4. 4.Bonawitz, K. A., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, B., Overveldt, T. V., Petrou, D., Ramage, D., and Roselander, J. Towards federated learning at scale: System design. Proc. of Machine Learn. Sys., 2019.
  5. 5.Bottou, L., Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223–311, 2018.
  6. 6.Çatak, F. O. Secure multi-party computation based privacy preserving extreme learning machine algorithm over vertically distributed data. Proc. Adv. Neural Inf. Process. Syst., 9490:337–345, 2015.
  7. 7.Ceballos, I., Sharma, V., Mugica, E., Singh, A., Roman, A., Vepakomma, P., and Raskar, R. Splitnn-driven vertical partitioning. arXiv:2008.04137, 2020.
  8. 8.Cha, D., Sung, M., and Park, Y.-R. Implementing vertical federated learning using autoencoders: Practical application, generalizability, and utility study. JMIR Medical Informatics, 9(6):e26598, 2021.
  9. 9.Chen, T., Jin, X., Sun, Y., and Yin, W. VAFL: a method of vertical asynchronous federated learning. arXiv:2007.06081, 2020.
  10. 10.Cheng, K., Fan, T., Jin, Y., Liu, Y., Chen, T., Papadopoulos, D., and Yang, Q. Secureboost: A lossless federated learning framework. IEEE Intell. Syst., 36(6):87–98, 2021.
  11. 11.Das, A. and Patterson, S. Multi-tier federated learning for vertically partitioned data. Proc. IEEE Int. Conf. on Acoust., Speech, and Signal Process., pp. 3100–3104, 2021.
  12. 12.Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009.
  13. 13.Feng, S. and Yu, H. Multi-participant multi-class vertical federated learning. arXiv:2001.11154, 2020.
  14. 14.Geiping, J., Bauermeister, H., Droge, H., and Moeller, M. Inverting gradients - how easy is it to break privacy in federated learning? Adv. Neural Inf. Process. Syst., 2020.
  15. 15.Gu, B., Xu, A., Huo, Z., Deng, C., and Huang, H. Privacy-preserving asynchronous vertical federated learning algorithms for multiparty collaborative learning. IEEE Trans. on Neural Netw. Learn. Syst., pp. 1–13, 2021. doi: 10.1109/TNNLS.2021.3072238.
  16. 16.Gu, Y., Lyu, X., Sun, W., Li, W., Chen, S., Li, X., and Marsic, I. Mutual correlation attentive factors in dyadic fusion networks for speech emotion recognition. Proc. ACM Int. Conf. on Multimedia, 2019.
  17. 17.Gupta, O. and Raskar, R. Distributed learning of deep neural network over multiple agents. J. Netw. Comput. Appl., 116:1–8, 2018.
  18. 18.Han, D.-J., Bhatti, H. I., Lee, J., and Moon, J. Accelerating federated learning with split learning on locally generated losses. In ICML 2021 Workshop on Federated Learning for User Privacy and Data Confidentiality, 2021a.
  19. 19.Han, W., Chen, H., and Poria, S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. Proc. 2020 Conf. Empir. Methods in Nat. Lang. Process., pp. 9180–9192, 2021b.
  20. 20.Hardy, S., Henecka, W., Ivey-Law, H., Nock, R., Patrini, G., Smith, G., and Thorne, B. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv:1711.10677, 2017.
  21. 21.He, C., Annavaram, M., and Avestimehr, S. Group knowledge transfer: Federated learning of large cnns at the edge. Proc. Adv. Neural Inf. Process. Syst., 2020.
  22. 22.Hu, Y., Niu, D., Yang, J., and Zhou, S. FDML: A collaborative machine learning framework for distributed features. Proc. ACM Int. Conf. Knowl. Discov. Data Min., pp. 2232–2240, 2019.
  23. 23.Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G. MIMIC-III, a freely accessible critical care database. Nature, 2016.
  24. 24.Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K. A., Charles, Z., Cormode, G., Cummings, R., D’Oliveira, R. G. L., Eichner, H., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascon, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konečný, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Qi, H., Ramage, D., Raskar, R., Raykova, M., Song, D., Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramer, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F. X., Yu, H., and Zhao, S. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1–210, 2021. doi: 10.1561/2200000083.
  25. 25.Karimireddy, S. P., Rebjock, Q., Stich, S. U., and Jaggi, M. Error feedback fixes signSGD and other gradient compression schemes. Proc. Int. Conf. on Machine Learn., 2019.
  26. 26.Koehrsen, W. Book recommendation system. https://github.com/WillKoehrsen/wikipedia-data-science/blob/master/notebooks/Book2018.
  27. 27.Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
  28. 28.Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proc. of Machine Learn. Sys., 2020.
  29. 29.Lim, W. Y. B., Luong, N. C., Hoang, D. T., Jiao, Y., Liang, Y., Yang, Q., Niyato, D., and Miao, C. Federated learning in mobile edge networks: A comprehensive survey. IEEE Commun. Surveys Tuts., 2020.
  30. 30.Lin, T., Stich, S. U., Patel, K. K., and Jaggi, M. Don’t use large mini-batches, use local SGD. Proc. Int. Conf. on Learn. Representations, 2020.
  31. 31.Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, B. Deep gradient compression: Reducing the communication bandwidth for distributed training. Proc. Int. Conf. on Learn. Representations, 2018.
  32. 32.Liu, L., Zhang, J., Song, S., and Letaief, K. B. Client-edge-cloud hierarchical federated learning. Proc. IEEE Int. Conf. on Comm., 2020.
  33. 33.Liu, Y., Kang, Y., Zhang, X., Li, L., Cheng, Y., Chen, T., Hong, M., and Yang, Q. A communication efficient vertical federated learning framework. Adv. Neural Inf. Process. Syst., Workshop on Federated Learning for Data Privacy and Confidentiality, 2019.
  34. 34.Mahendran, A. and Vedaldi, A. Understanding deep image representations by inverting them. Proc. IEEE Int. Conf. Comput. Vis., pp. 5188–5196, 2015.
  35. 35.McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication-efficient learning of deep networks from decentralized data. Proc. 20th Int. Conf. on Artif. Intell., pp. 1273–1282, 2017.
  36. 36.Moritz, P., Nishihara, R., Stoica, I., and Jordan, M. I. Sparknet: Training deep networks in spark. Proc. Int. Conf. on Learn. Representations, 2016.
  37. 37.Nguyen, L. M., Nguyen, P. H., van Dijk, M., Richtarik, P., Scheinberg, K., and Takac, M. SGD and Hogwild! convergence without the bounded gradients assumption. Proc. Int. Conf. on Machine Learn., 80:3747–3755, 2018.
  38. 38.Nie, W., Liang, Q., Wang, Y., Wei, X., and Su, Y. MMFN: multimodal information fusion networks for 3d model classification and retrieval. ACM Trans. on Multimedia Computing, Communications, and Applications, 2021.
  39. 39.Phong, L. T., Aono, Y., Hayashi, T., Wang, L., and Moriai, S. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Trans. Inf. Forensics Security, 13(5):1333–1345, 2018.
  40. 40.Richtarik, P. and Takac, M. Parallel coordinate descent methods for big data optimization. Math. Program., 156 (1-2):433–484, 2016.
  41. 41.Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., Ourselin, S., Sheller, M., Summers, R. M., Trask, A., Xu, D., Baust, M., and Cardoso, M. J. Digital Medicine, 2020.
  42. 42.Romanini, D., Hall, A. J., Papadopoulos, P., Titcombe, T., Ismail, A., Cebere, T., Sandmann, R., Roehm, R., and Hoeh, M. A. PyVertical: A vertical federated learning framework for multi-headed SplitNN. Int. Conf. Learn. Representations, Workshop on Distributed and Private Machine Learn., 2021.
  43. 43.Shi, S., Zhao, K., Wang, Q., Tang, Z., and Chu, X. A convergence analysis of distributed SGD with communication-efficient gradient sparsification. Proc. Int. Joint Conf. on Artif. Intell., 2019.
  44. 44.Shlezinger, N., Chen, M., Eldar, Y. C., Poor, H. V., and Cui, S. Uveqfed: Universal vector quantization for federated learning. IEEE Trans. Signal Process., 69:500–514, 2021.
  45. 45.Stich, S. U., Cordonnier, J., and Jaggi, M. Sparsified SGD with memory. Adv. Neural Inf. Process. Syst., 2018.
  46. 46.Tsitsiklis, J., Bertsekas, D., and Athans, M. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Trans. Autom. Control, 31(9):803–812, 1986.
  47. 47.Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., and Chan, K. Adaptive federated learning in resource constrained edge computing systems. IEEE J. Sel. Areas Commun., 37(6):1205–1221, 2019.
  48. 48.Wannamaker, R. A. The Theory of Dithered Quantization. PhD thesis, 1997.
  49. 49.Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. Terngrad: Ternary gradients to reduce communication in distributed deep learning. Adv. Neural Inf. Process. Syst., 2017.
  50. 50.Woods, J. W. Multidimensional signal, image, and video processing and coding. Elsevier, 2006.
  51. 51.Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. 3D shapenets: A deep representation for volumetric shapes. Proc. IEEE Int. Conf. Comput. Vis., pp. 1912–1920, 2015.
  52. 52.Yang, Q., Liu, Y., Chen, T., and Tong, Y. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol., 10(2):12:1–12:19, 2019.
  53. 53.Zamir, R. and Feder, M. On lattice quantization noise. IEEE Trans. Inf. Theory, 42(4):1152–1159, 1996.
  54. 54.Zhang, X., Yin, W., Hong, M., and Chen, T. Hybrid federated learning: Algorithms and implementation. arXiv:2012.12420, 2020.
  55. 55.Zheng, F., Chen, C., Zheng, X., and Zhu, M. Towards secure and practical machine learning via secret sharing and random permutation. Knowl. Based Syst., 245:108609, 2022.

Citation

MLA
Castiglia, T. J., et al. “Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data”. International Conference on Machine Learning, vol. 162, 2022, pp. 2738–66, https://proceedings.mlr.press/v162/castiglia22a.html.
APA
Castiglia, T. J., Das, A., Wang, S., & Patterson, S. (2022). Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data. International Conference on Machine Learning, 162, 2738–2766. https://proceedings.mlr.press/v162/castiglia22a.html
Chicago
Castiglia, T. J., A. Das, S. Wang, and S. Patterson. 2022. “Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data”. International Conference on Machine Learning 162: 2738–66. https://proceedings.mlr.press/v162/castiglia22a.html.
Harvard
Castiglia, T.J. et al. (2022) “Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data”, International Conference on Machine Learning. PMLR, pp. 2738–2766. Available at: https://proceedings.mlr.press/v162/castiglia22a.html.
Vancouver
1. Castiglia TJ, Das A, Wang S, Patterson S (2022) Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data. In: International Conference on Machine Learning. PMLR, pp 2738–2766

BibTeX

@InProceedings{pmlr-v162-castiglia22a,
  title = 	 {Compressed-{VFL}: Communication-Efficient Learning with Vertically Partitioned Data},
  author =       {Castiglia, Timothy J and Das, Anirban and Wang, Shiqiang and Patterson, Stacy},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {2738--2766},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/castiglia22a/castiglia22a.pdf},
  url = 	 {https://proceedings.mlr.press/v162/castiglia22a.html},
  abstract = 	 {We propose Compressed Vertical Federated Learning (C-VFL) for communication-efficient training on vertically partitioned data. In C-VFL, a server and multiple parties collaboratively train a model on their respective features utilizing several local iterations and sharing compressed intermediate results periodically. Our work provides the first theoretical analysis of the effect message compression has on distributed training over vertically partitioned data. We prove convergence of non-convex objectives at a rate of $O(\frac{1}{\sqrt{T}})$ when the compression error is bounded over the course of training. We provide specific requirements for convergence with common compression techniques, such as quantization and top-$k$ sparsification. Finally, we experimentally show compression can reduce communication by over $90%$ without a significant decrease in accuracy over VFL without compression.}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/