Back

Secure Quantized Training for Deep Learning

Marcel KellerKe Sun

article2022ICML92 citations

Demonstrates practical deep neural network training within secure multi-party computation by introducing optimized protocols for exponentiation and inverse square roots that achieve near-plaintext accuracy on standard image benchmarks across various threat models.

Listen

Organizations in regulated domains such as healthcare and finance increasingly seek to train deep learning models collaboratively without exposing private underlying datasets. Secure multi-party computation enables multiple parties to jointly evaluate functions on private inputs while keeping the data confidential. However, prior attempts to train deep neural networks within this framework have suffered from severe performance bottlenecks, practical instability, or substantial drops in model accuracy due to poor approximations of complex mathematical functions and high communication overhead.

The article demonstrates an extensible framework for end-to-end neural network training entirely within secure multi-party computation. It evaluates how low-precision fixed-point representation combined with optimized cryptographic subroutines can achieve classification accuracy comparable to standard, unencrypted training while remaining computationally practical across diverse security setups.

To achieve this, the authors implemented neural network training using fixed-point arithmetic in native CPU code based on the open-source MP-SPDZ library. They developed novel mixed-circuit cryptographic protocols for exponentiation and inverse square root operations, which are essential for evaluating softmax functions and modern adaptive optimizers such as AMSGrad and Adam. The evaluation benchmarked standard image classification tasks—including MNIST and CIFAR-10—across multiple network architectures and security models, testing configurations with up to ten parties under both honest-majority and dishonest-majority assumptions.

The findings show that training purely within secure multi-party computation can match standard unencrypted performance within tight margins. On the MNIST dataset, a convolutional neural network achieved 99.2% accuracy in 3.5 hours (reaching 99.0% within one hour), trailing unencrypted training accuracy by less than 0.2 percentage points. The newly developed exponentiation protocol reduced communication overhead by roughly 30% compared to prior art, while the inverse square root protocol halved communication costs. Furthermore, theoretical analysis and empirical results confirmed that probabilistic rounding delivers unbiased matrix multiplication and injects useful noise, allowing a 16-bit precision parameter to perform as well as higher-precision configurations while minimizing data transfer. On CIFAR-10 using an AlexNet-style architecture with batch normalization, secure training converged within a few hours to 64.9% accuracy, tracking closely with unencrypted baselines.

These results demonstrate that privacy-preserving machine learning does not require sacrificing model accuracy or resorting to oversimplified, unstable mathematical replacements. By moving critical computations into native execution and minimizing communication rounds, secure multi-party computation becomes a viable alternative to trusted hardware or slower homomorphic encryption systems. The findings also highlight that network communication, rather than raw floating-point computing power, is the primary performance bottleneck in secure training, making optimized CPU architectures more effective than graphics processors in these environments.

Decision-makers considering collaborative, privacy-preserving machine learning should focus on implementations using native CPU execution and exact mathematical representations rather than unverified approximations. When designing systems, teams should adopt probabilistic rounding at 16-bit fixed-point precision to optimize both network bandwidth and classification performance. For architectures requiring adaptive gradient methods or normalization layers, incorporating optimized mixed-circuit protocols will prevent divergence and improve throughput.

The primary limitation of this work is that evaluations were conducted in local area network environments on small-to-moderate image datasets rather than massive industrial datasets containing millions of samples. While confidence in the mathematical correctness and stability of the system is high, performance will vary depending on network latency and bandwidth across wider geographic areas. Further pilot deployments on real-world distributed infrastructure are recommended to evaluate operational communication limits.

deep learningquantizerssecure quantized trainingsecure computationlow-precision training
Keller et al (2022).pdf

No sufficiently relevant recommendations were found.

Cover for Secure Quantized Training for Deep Learning

Abstract

We implement training of neural networks in secure multi-party computation (MPC) using quantization commonly used in said setting. We are the first to present an MNIST classifier purely trained in MPC that comes within 0.2 percent of the accuracy of the same convolutional neural network trained via plaintext computation. More concretely, we have trained a network with two convolutional and two dense layers to 99.2% accuracy in 3.5 hours (under one hour for 99% accuracy). We have also implemented AlexNet for CIFAR-10, which converges in a few hours. We develop novel protocols for exponentiation and inverse square root. Finally, we present experiments in a range of MPC security models for up to ten parties, both with honest and dishonest majority as well as semi-honest and malicious security.

Topics

deep learningquantizers

Keywords

secure quantized trainingsecure computationlow-precision trainingprobabilistic roundingbinary secret sharingoblivious selectionmixed-circuit computationstochastic gradient descent

Table of Contents

  • 1. Introduction
  • 2. Secure Computation Building Blocks
  • 2.1. Quantization
  • 2.2. Exponentiation
  • 3. Deep Learning Building Blocks
  • 4. An Analysis of Probabilistic Rounding
  • 5. Implementation and Benchmarks
  • 5.1. MNIST Classification
  • 5.1.1. SECURE COMPUTATION
  • 5.1.2. COMPARISON TO CLEARTEXT TRAINING
  • 5.2. CIFAR-10 Classification
  • 6. Conclusions
  • References
  • A. An Efficient Secure Three-Party Computation Protocol
  • B. An Efficient Multi-Party Computation Protocol Based on Homomorphic Encryption
  • B.1. Linear-cost Triple Generation with Semi-homomorphic Encryption
  • B.2. Replacing Homomorphic Encryption by a Dealer
  • C. High-Level Secure Computation Building Blocks
  • D. Deep Learning Building Blocks
  • E. Models
  • F. Fashion MNIST
  • G. More Experimental Results
  • H. Hyperparameter Settings
  • I. List of Symbols
  • J. Proofs
  • J.1. Proof of Proposition 4.1
  • J.2. A Lemma of Probabilistic Rounding
  • J.3. Proof of Proposition 4.2
  • J.4. Proof of Proposition 4.3

Knowls

  1. Knowl 1 — Secure LeNet training reaches 99.2% MNIST accuracy

    data/table

    The authors trained a LeNet-style convolutional network entirely in MPC on MNIST, using 16-bit fixed-point precision (f=16f=16), probabilistic rounding, and a three-party LAN protocol with one corruption. The network has two convolutional layers and two dense layers. In the five-epoch benchmark, the SGD run reached 98.5% accuracy at 343 seconds and 352 GB of communication per epoch; the AMSGrad run reached 99.0% at 513 seconds and 765 GB per epoch. In a longer secure run, AMSGrad reached 99.2% accuracy after 25 epochs in 3.6 hours, and reached 99% in about one hour. The paper reports this as within 0.2 percentage points of the same network trained in plaintext.

  2. Knowl 2 — Probabilistic fixed-point multiplication is unbiased and has concentration bounds

    theoretical result

    Let ff be the fixed-point precision and ϵ=2−f\epsilon=2^{-f}. Represent a real number xx by the integer Qf(x)=⌊x2f⌉Q_f(x)=\lfloor x2^f\rceil, where ⌊⋅⌉\lfloor\cdot\rceil denotes nearest-integer rounding. For a scalar product, set μ=Qf(x)Qf(y)2−f\mu=Q_f(x)Q_f(y)2^{-f} and output the integer Rf(xy)=⌊μ⌋+bR_f(xy)=\lfloor\mu\rfloor+b, where bb is an independent Bernoulli random variable with probability {μ}=μ−⌊μ⌋\{\mu\}=\mu-\lfloor\mu\rfloor of equaling 1. Thus the decoded product is 2−fRf(xy)2^{-f}R_f(xy). Apply this operation independently to every scalar product in a matrix product, summing the integer outputs to form Rf(AB)R_f(AB).

    For real matrices A∈Rm×nA\in\mathbb{R}^{m\times n} and B∈Rn×pB\in\mathbb{R}^{n\times p}, the matrix result is unbiased relative to the product of the quantized inputs: E[Rf(AB)]=2−fQf(A)Qf(B)\mathbb{E}[R_f(AB)]=2^{-f}Q_f(A)Q_f(B), where quantization is elementwise and expectation is over the rounding randomness. In contrast, deterministic nearest rounding is generally biased.

    If every entry of AA and BB has absolute value at most 2k2^k, for k≥0k\geq 0, then the worst-case Frobenius-norm error satisfies ∥Rf(AB)−2fAB∥F<mp n (2k+1+ϵ/4)\|R_f(AB)-2^fAB\|_F<\sqrt{mp}\,n\,(2^k+1+\epsilon/4). A probabilistic bound, which does not require the entry bound, is Pr⁡(∥Rf(AB)−2−fQf(A)Qf(B)∥F≤ιmnp)≥1−1/(4ι2)\Pr\bigl(\|R_f(AB)-2^{-f}Q_f(A)Q_f(B)\|_F\leq\iota\sqrt{mnp}\bigr)\geq 1-1/(4\iota^2) for ι>0\iota>0. The latter bound captures concentration around the exact expectation of the randomized product.

  3. Knowl 3 — Mixed-circuit exponentiation supports negative inputs with lower communication

    algorithm

    The paper’s base-two exponentiation protocol computes a fixed-point approximation to 2x2^x using arithmetic secret sharing over a large integer domain and binary secret sharing for bit operations. Its inputs are a secret-shared signed fixed-point value xx with precision ff and a kk-bit representation; its output is a secret-shared fixed-point approximation with the same precision. The protocol assumes the input is within the signed representable range and maps inputs below −(k−f−1)-(k-f-1) to zero, where the exponential is too small for the output representation. It uses arithmetic-to-binary decomposition (A2B\mathrm{A2B}), bit-to-arithmetic conversion (Bit2A\mathrm{Bit2A}), binary-to-arithmetic conversion (B2A\mathrm{B2A}), and a Taylor-series approximation Approx2(r)\mathrm{Approx2}(r) to 2r2^r for r∈[0,1]r\in[0,1].

    Input: Secret-shared signed fixed-point x, precision f, total bit length k
    Output: Secret-shared fixed-point approximation to 2^x
    1. Convert x to binary shares x_0,...,x_{k-1} with A2B; x_{k-1} is the sign bit.
    2. Compute z = [x < -(k-f-1)] by a binary comparison on the fixed-point value.
    3. Set ell = ceil(log2(k-f)). Convert bits x_f,...,x_{f+ell-1} to arithmetic shares.
    4. Compute d = product over j=0,...,ell-1 of (1 + x_{f+j}(2^(2^j)-1)). This encodes the integer-part exponential.
    5. Convert the low f bits to the arithmetic fractional part r using B2A.
    6. Compute u = Approx2(r), then g = u*d.
    7. Compute g_neg by fixed-point truncation of g by 2^ell bits, which divides its represented value by 2^(2^ell).
    8. Select h = g_neg when the sign bit x_{k-1} is 1, and h = g otherwise.
    9. Return 0 when z is 1; otherwise return h.

    The negative-input correction avoids computing a reciprocal of the exponential of the absolute value. In the paper’s communication comparison at f=16f=16 and k=31k=31, the earlier protocol versus this protocol used, respectively, 27 versus 16 kbit for three-party semi-honest security, 498 versus 323 kbit for three-party malicious security, 1,338 versus 813 kbit for two-party semi-honest security, and 214,476 versus 121,747 kbit for two-party malicious security. These results correspond to about a 30% communication reduction across the reported settings, despite the wider negative-input support.

  4. Knowl 4 — An optimized inverse-square-root protocol reduces conversion cost

    model/method

    The inverse-square-root method normalizes a positive secret-shared fixed-point input xx by locating its power-of-two interval: for an integer exponent ee with 2e−1≤x≤2e2^{e-1}\leq x\leq 2^e, the normalized value is u=x2−(e+1)∈[0.25,0.5)u=x2^{-(e+1)}\in[0.25,0.5). It approximates the inverse square root on this interval with the polynomial 3.14736+u(4.63887u−5.77789)3.14736+u(4.63887u-5.77789), then multiplies by a power-of-two compensation factor derived from the exponent encoding to approximate 1/x1/\sqrt{x}. The exponent is represented as a one-hot binary vector. The paper’s optimization computes paired exponent indicators with binary OR operations and obtains the parity needed for scaling with XOR, avoiding more costly binary-to-arithmetic conversions. The authors report that this optimization cuts the cost of the square-root compensation by roughly half relative to the prior method.

    For the complete inverse-square-root computation, total communication in kbit for the prior method versus the authors’ method was 19 versus 9 for three-party semi-honest security, 160 versus 114 for three-party malicious security, 481 versus 342 for two-party semi-honest security, and 25,456 versus 21,522 for two-party malicious security. These measurements use one corrupted party. In training, inverse square root is used repeatedly in optimizers such as Adam and AMSGrad.

  5. Knowl 5 — The MPC framework batches neural-network operations and controls numeric range

    model/method

    The implementation builds neural networks as successions of layers on MP-SPDZ. Protocol-specific operations provide secret input and output, addition and multiplication, and conversion between arithmetic sharing and binary sharing; higher-level operations include division, exponentiation, and inverse square root. A cleartext emulator executes the same computation with the same fixed-point precision, allowing the authors to assess training before running the more expensive secure protocol.

    Dense-layer and convolutional forward and backward computations are expressed as dot products. The implementation batches all dot products for a matrix multiplication into one communication batch, reducing communication rounds. It also postpones fixed-point truncation until after the dot-product sum, reducing both truncation cost and accumulated rounding error. ReLU and max pooling use secure comparisons and oblivious selection; max pooling uses a balanced comparison tree, and forward-pass comparison results can be retained as secret shares for back-propagation.

    For classification, the logits x1,…,xLx_1,\ldots,x_L are shifted by xmax⁡=max⁡jxjx_{\max}=\max_j x_j before exponentiation. The softmax probability for class ii is exp⁡(xi−xmax⁡)/∑jexp⁡(xj−xmax⁡)\exp(x_i-x_{\max})/\sum_j\exp(x_j-x_{\max}); because every shifted logit is nonpositive, its denominator lies between 1 and the number of classes LL, avoiding overflow in the fixed-point representation. The same construction implements sigmoid as a two-class softmax. For SGD, the mini-batch gradient is accumulated before division by the batch size; with the batch size of 128 used in the experiments, the final division can be implemented as probabilistic truncation rather than a general secure division.

  6. Knowl 6 — AlexNet-style CIFAR-10 training converges under MPC

    data/table

    The authors implemented CIFAR-10 training with a five-convolutional-layer, two-hidden-dense-layer AlexNet-derived network, including batch normalization. The input is 32×32 RGB; the convolutional filter counts are 96, 256, 384, 384, and 256, followed by two 256-unit dense layers and a 10-class output. Training used f=16f=16 fixed-point precision with probabilistic truncation in a three-party LAN setting with one corruption. After ten epochs, the authors’ SGD run achieved 64.9% accuracy at 1,603 seconds and 771 GB of communication per epoch; Adam achieved 64.7% at 2,431 seconds and 3,317 GB per epoch; AMSGrad achieved 63.6% at 2,473 seconds and 3,285 GB per epoch. The paper reports that secure training came within a few percentage points of cleartext training. The cited comparison systems did not provide directly comparable training-from-scratch accuracy figures.

  7. Knowl 7 — LeNet benchmarks quantify the cost across MPC security models

    data/table

    The paper benchmarks the LeNet network with AMSGrad across LAN security configurations, reporting seconds and GB of communication per epoch. Most values are estimates based on ten batch iterations. The entries show the substantial effect of party count, corruption tolerance, and security model on cost:

    • Two-party homomorphic-encryption protocol, 1 corruption, non-malicious: 196,745 seconds and 51,770 GB.
    • Three-party replicated-sharing protocol, 1 corruption, non-malicious: 513 seconds and 765 GB.
    • Three-party maliciously secure protocol, 1 corruption: 4,961 seconds and 9,101 GB.
    • Three-party homomorphic-encryption protocol, 2 corruptions, non-malicious: 357,214 seconds and 271,595 GB.
    • Four-party maliciously secure protocol, 1 corruption: 1,175 seconds and 2,945 GB.
    • Ten-party dealer-based protocol, 1/8 corruptions as reported: 29,078 seconds and 99,775 GB.
    • Ten-party protocol with 4 corruptions, non-malicious: 129,667 seconds and 434,138 GB.
    • Ten-party homomorphic-encryption protocol, 9 corruptions, non-malicious: 2,833,641 seconds and 13,875,834 GB.

    The comparison demonstrates that the paper’s three-party honest-majority protocol is much faster than the reported high-corruption alternatives, while also showing that the framework can be evaluated in settings ranging from three to ten parties and including malicious security.

  8. Knowl 8 — Distributed homomorphic encryption generates multiplication triples with linear party scaling

    algorithm

    For semi-honest MPC that tolerates up to n−1n-1 corrupted parties, the paper gives a multiplication-triple generation protocol based on distributed semi-homomorphic encryption. The encryption supports multiplying a ciphertext by a plaintext vector using the componentwise (Schur) product, and the parties have a distributed key setup. The output is additive shares of random vectors aa, bb, and cc satisfying c=a⊙bc=a\mathbin{\odot}b; these are Beaver multiplication triples. The protocol communicates through a designated party P1P_1 rather than requiring pairwise exchanges among every party.

    Input: Distributed encryption-key setup; parties P_1,...,P_n
    Output: Additive shares of random a, b, c with c = a ⊙ b
    1. Each party P_i samples a_i and sends Enc(a_i) to P_1.
    2. P_1 sums the ciphertexts to obtain C_a = sum_i Enc(a_i), then broadcasts C_a.
    3. Each party P_i samples b_i and sends C_i = C_a ⊙ b_i + Enc(0) to P_1, using a fresh encryption of zero.
    4. P_1 sums the ciphertexts C_i to obtain C_c and broadcasts C_c.
    5. The parties jointly decrypt C_c and obtain additive shares c_i.
    6. Interpret a_i, b_i, and c_i as shares of a = sum_i a_i, b = sum_i b_i, and c = sum_i c_i.

    The resulting shares satisfy c=(∑iai)⊙(∑ibi)c=(\sum_i a_i)\mathbin{\odot}(\sum_i b_i). Adding a fresh encryption of zero ensures that the designated party receives a fresh encryption in step 3. The paper describes this construction as linear-cost in the number of parties, in contrast to pairwise-communication approaches that scale quadratically.

  9. Knowl 9 — Sixteen-bit probabilistic rounding and AMSGrad were the strongest MNIST choices

    empirical result

    In the LeNet MNIST experiments, the authors compared nearest and probabilistic rounding at several fixed-point precisions using SGD with learning rate 0.01 and batch size 128. Precision f=16f=16 with probabilistic rounding was selected for the remaining benchmarks: 16-bit nearest rounding performed worse, increasing precision to 32 or 64 bits did not improve accuracy, and f=8f=8 caused divergence. Among the tested optimizers, AMSGrad gave the strongest convergence and final accuracy; adding Dropout produced at most a small improvement and did not consistently help. In the same-optimizer comparison with TensorFlow plaintext training, secure training performed only slightly worse. The main reported MNIST settings used learning rates 0.01 for SGD and 0.001 for AMSGrad.

  10. Knowl 10 — The evaluation is limited by fixed-point error and selected nonlinear protocols

    limitation

    The paper notes that the lower precision required for efficient MPC introduces a small increase in computational error. Its training results use one particular implementation of division and exponentiation, both important to the learning process through softmax; alternative secure approximations may produce different performance. The experiments focus on MNIST and CIFAR-10 because MPC costs make datasets with millions of examples less feasible than datasets with tens of thousands. These results therefore do not establish training cost or accuracy for substantially larger datasets or for other implementations of the nonlinear operations.

Coverage note — The detailed derivation of standard secure division and logarithm, secondary MNIST network curves, and Fashion-MNIST curves are omitted because they are supporting building blocks or supplementary experiments rather than distinct central contributions.

References

  1. 1.Agrawal, N., Shamsabadi, A. S., Kusner, M. J., and Gascón, A. QUOTIENT: Two-party secure neural network training and prediction. In Cavallaro, L., Kinder, J., Wang, X., and Katz, J. (eds.), ACM CCS 2019, pp. 1231–1247. ACM Press, November 2019. doi: 10.1145/3319535.3339819.
  2. 2.Aliasgari, M., Blanton, M., Zhang, Y., and Steele, A. Secure computation on floating point numbers. In NDSS 2013. The Internet Society, February 2013.
  3. 3.Aly, A. and Smart, N. P. Benchmarking privacy preserving scientific operations. In Deng, R. H., Gauthier-Umaña, V., Ochoa, M., and Yung, M. (eds.), ACNS 19, volume 11464 of LNCS, pp. 509–529. Springer, Heidelberg, June 2019. doi: 10.1007/978-3-030-21568-2_25.
  4. 4.Araki, T., Furukawa, J., Lindell, Y., Nof, A., and Ohara, K. High-throughput semi-honest secure three-party computation with an honest majority. In Weippl, E. R., Katzenbeisser, S., Kruegel, C., Myers, A. C., and Halevi, S. (eds.), ACM CCS 2016, pp. 805–817. ACM Press, October 2016. doi: 10.1145/2976749.2978331.
  5. 5.Araki, T., Barak, A., Furukawa, J., Keller, M., Lindell, Y., Ohara, K., and Tsuchida, H. Generalizing the SPDZ compiler for other protocols. In Lie, D., Mannan, M., Backes, M., and Wang, X. (eds.), ACM CCS 2018, pp. 880–895. ACM Press, October 2018. doi: 10.1145/3243734.3243854.
  6. 6.Barni, M., Orlandi, C., and Piva, A. A privacy-preserving protocol for neural-network-based computation. In Proceedings of the 8th workshop on Multimedia and security, pp. 146–151, 2006.
  7. 7.Beaver, D. Efficient multiparty protocols using circuit randomization. In Feigenbaum, J. (ed.), CRYPTO’91, volume 576 of LNCS, pp. 420–432. Springer, Heidelberg, August 1992. doi: 10.1007/3-540-46766-1_34.
  8. 8.Benaloh, J. C. and Leichter, J. Generalized secret sharing and monotone functions. In Goldwasser, S. (ed.), CRYPTO’88, volume 403 of LNCS, pp. 27–35. Springer, Heidelberg, August 1990. doi: 10.1007/0-387-34799-2_3.
  9. 9.Catrina, O. and Saxena, A. Secure computation with fixed-point numbers. In Sion, R. (ed.), FC 2010, volume 6052 of LNCS, pp. 35–50. Springer, Heidelberg, January 2010.
  10. 10.Cock, M. d., Dowsley, R., Nascimento, A. C., and Newman, S. C. Fast, privacy preserving linear regression over distributed datasets based on pre-distributed data. In Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security, AISec ’15, pp. 3–14, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450338264. doi: 10.1145/2808769.2808774.
  11. 11.Cramer, R., Damgård, I., and Nielsen, J. B. Multiparty computation from threshold homomorphic encryption. In Pfitzmann, B. (ed.), EUROCRYPT 2001, volume 2045 of LNCS, pp. 280–299. Springer, Heidelberg, May 2001. doi: 10.1007/3-540-44987-6_18.
  12. 12.Dahl, M., Mancuso, J., Dupis, Y., Decoste, B., Giraud, M., Livingstone, I., Patriquin, J., and Uhma, G. Private machine learning in TensorFlow using secure computation. CoRR, abs/1810.08130, 2018.
  13. 13.Dalskov, A., Escudero, D., and Keller, M. Fantastic four: Honest-majority four-party secure computation with malicious security. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
  14. 14.Dalskov, A. P. K., Escudero, D., and Keller, M. Secure evaluation of quantized neural networks. PoPETs, 2020(4):355–375, October 2020. doi: 10.2478/popets-2020-0077.
  15. 15.Damgård, I., Fitzi, M., Kiltz, E., Nielsen, J. B., and Toft, T. Unconditionally secure constant-rounds multi-party computation for equality, comparison, bits and exponentiation. In Halevi, S. and Rabin, T. (eds.), TCC 2006, volume 3876 of LNCS, pp. 285–304. Springer, Heidelberg, March 2006. doi: 10.1007/11681878_15.
  16. 16.Damgård, I., Pastro, V., Smart, N. P., and Zakarias, S. Multiparty computation from somewhat homomorphic encryption. In Safavi-Naini, R. and Canetti, R. (eds.), CRYPTO 2012, volume 7417 of LNCS, pp. 643–662. Springer, Heidelberg, August 2012. doi: 10.1007/978-3-642-32009-5_38.
  17. 17.Damgård, I., Escudero, D., Frederiksen, T. K., Keller, M., Scholl, P., and Volgushev, N. New primitives for actively-secure MPC over rings with applications to private machine learning. In 2019 IEEE Symposium on Security and Privacy, pp. 1102–1120. IEEE Computer Society Press, May 2019. doi: 10.1109/SP.2019.00078.
  18. 18.Demmler, D., Schneider, T., and Zohner, M. ABY - A framework for efficient mixed-protocol secure two-party computation. In NDSS 2015. The Internet Society, February 2015.
  19. 19.Eerikson, H., Keller, M., Orlandi, C., Pullonen, P., Puura, J., and Simkin, M. Use your brain! Arithmetic 3PC for any modulus with active security. In Kalai, Y. T., Smith, A. D., and Wichs, D. (eds.), ITC 2020, pp. 5:1–5:24. Schloss Dagstuhl, June 2020. doi: 10.4230/LIPIcs.ITC.2020.5.
  20. 20.Escudero, D., Ghosh, S., Keller, M., Rachuri, R., and Scholl, P. Improved primitives for MPC over mixed arithmetic-binary circuits. In Micciancio, D. and Ristenpart, T. (eds.), CRYPTO 2020, Part II, volume 12171 of LNCS, pp. 823–852. Springer, Heidelberg, August 2020. doi: 10.1007/978-3-030-56880-1_29.
  21. 21.Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics (AISTATS), pp. 249–256, 2010.
  22. 22.Goldschmidt, R. E. Applications of division by convergence. Master’s thesis, MIT, 1964.
  23. 23.Goyal, V., Li, H., Ostrovsky, R., Polychroniadou, A., and Song, Y. ATLAS: Efficient and scalable MPC in the honest majority setting. In Malkin, T. and Peikert, C. (eds.), CRYPTO 2021, Part II, volume 12826 of LNCS, pp. 244–274, Virtual Event, August 2021. Springer, Heidelberg. doi: 10.1007/978-3-030-84245-1_9.
  24. 24.Halevi, S. and Shoup, V. Algorithms in HElib. In Garay, J. A. and Gennaro, R. (eds.), CRYPTO 2014, Part I, volume 8616 of LNCS, pp. 554–571. Springer, Heidelberg, August 2014. doi: 10.1007/978-3-662-44371-2_31.
  25. 25.Hart, J. F. Computer approximations. Krieger Publishing Co., Inc., 1978.
  26. 26.Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Binarized neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 29. Curran Associates, Inc., 2016.
  27. 27.Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448–456, Lille, France, 2015. PMLR.
  28. 28.Juvekar, C., Vaikuntanathan, V., and Chandrakasan, A. GAZELLE: A low latency framework for secure neural network inference. In Enck, W. and Felt, A. P. (eds.), USENIX Security 2018, pp. 1651–1669. USENIX Association, August 2018.
  29. 29.Keller, M. MP-SPDZ: A versatile framework for multi-party computation. In Ligatti, J., Ou, X., Katz, J., and Vigna, G. (eds.), ACM CCS 2020, pp. 1575–1590. ACM Press, November 2020. doi: 10.1145/3372297.3417872.
  30. 30.Keller, M. and Sun, K. Effectiveness of MPC-friendly softmax replacement. In Privacy Preserving Machine Learning - PriML and PPML Joint Edition (NeurIPS 2020 workshop), 2020.
  31. 31.Keller, M., Pastro, V., and Rotaru, D. Overdrive: Making SPDZ great again. In Nielsen, J. B. and Rijmen, V. (eds.), EUROCRYPT 2018, Part III, volume 10822 of LNCS, pp. 158–189. Springer, Heidelberg, April / May 2018. doi: 10.1007/978-3-319-78372-7_6.
  32. 32.Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
  33. 33.Knott, B., Venkataraman, S., Hannun, A., Sengupta, S., Ibrahim, M., and van der Maaten, L. CrypTen: Secure multi-party computation meets machine learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
  34. 34.Kolesnikov, V., Sadeghi, A.-R., and Schneider, T. A systematic approach to practically efficient general two-party secure function evaluation protocols and their modular design. Journal of Computer Security, 21(2):283–315, 2013.
  35. 35.Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6):84–90, may 2017.
  36. 36.LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  37. 37.LeCun, Y., Cortes, C., and Burges, C. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010. Creative Commons Attribution-Share Alike 3.0 license, https://creativecommons.org/licenses/by-sa/3.0/.
  38. 38.Li, H., De, S., Xu, Z., Studer, C., Samet, H., and Goldstein, T. Training quantized nets: A deeper understanding. In Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017.
  39. 39.Lin, D., Talathi, S., and Annapureddy, S. Fixed point quantization of deep convolutional networks. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2849–2858, New York, New York, USA, 20–22 Jun 2016. PMLR.
  40. 40.Liu, J., Juuti, M., Lu, Y., and Asokan, N. Oblivious neural network predictions via MiniONN transformations. In Thuraisingham, B. M., Evans, D., Malkin, T., and Xu, D. (eds.), ACM CCS 2017, pp. 619–631. ACM Press, October / November 2017. doi: 10.1145/3133956.3134056.
  41. 41.Lou, Q., Feng, B., Fox, G. C., and Jiang, L. Glyph: Fast and accurately training deep neural networks on encrypted data. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
  42. 42.Lu, W.-j, Fang, Y., Huang, Z., Hong, C., Chen, C., Qu, H., Zhou, Y., and Ren, K. Faster secure multiparty computation of adaptive gradient descent. In Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, PPMLP’20, pp. 47–49, 2020.
  43. 43.Mishra, P., Lehmkuhl, R., Srinivasan, A., Zheng, W., and Popa, R. A. Delphi: A cryptographic inference service for neural networks. In Capkun, S. and Roesner, F. (eds.), USENIX Security 2020, pp. 2505–2522. USENIX Association, August 2020.
  44. 44.Mohassel, P. and Rindal, P. ABY3: A mixed protocol framework for machine learning. In Lie, D., Mannan, M., Backes, M., and Wang, X. (eds.), ACM CCS 2018, pp. 35–52. ACM Press, October 2018. doi: 10.1145/3243734.3243760.
  45. 45.Mohassel, P. and Zhang, Y. SecureML: A system for scalable privacy-preserving machine learning. In 2017 IEEE Symposium on Security and Privacy, pp. 19–38. IEEE Computer Society Press, May 2017. doi: 10.1109/SP.2017.12.
  46. 46.Nair, V. and Hinton, G. E. Rectified linear units improve Restricted Boltzmann machines. In International Conference on Machine Learning (ICML), pp. 807–814, 2010.
  47. 47.Quoc, D. L., Gregor, F., Arnautov, S., Kunkel, R., Bhatotia, P., and Fetzer, C. secureTF: A secure TensorFlow framework. CoRR, abs/2101.08204, 2021.
  48. 48.Rathee, D., Rathee, M., Kumar, N., Chandran, N., Gupta, D., Rastogi, A., and Sharma, R. CrypTFlow2: Practical 2-party secure inference. In Ligatti, J., Ou, X., Katz, J., and Vigna, G. (eds.), ACM CCS 2020, pp. 325–342. ACM Press, November 2020. doi: 10.1145/3372297.3417274.
  49. 49.Reddi, S. J., Kale, S., and Kumar, S. On the convergence of Adam and beyond. In International Conference on Learning Representations (ICLR), 2018.
  50. 50.Riazi, M. S., Weinert, C., Tkachenko, O., Songhori, E. M., Schneider, T., and Koushanfar, F. Chameleon: A hybrid secure computation framework for machine learning applications. In Kim, J., Ahn, G.-J., Kim, S., Kim, Y., López, J., and Kim, T. (eds.), ASIACCS 18, pp. 707–721. ACM Press, April 2018.
  51. 51.Rotaru, D. and Wood, T. MArBled circuits: Mixing arithmetic and Boolean circuits with active security. In Hao, F., Ruj, S., and Sen Gupta, S. (eds.), INDOCRYPT 2019, volume 11898 of LNCS, pp. 227–249. Springer, Heidelberg, December 2019. doi: 10.1007/978-3-030-35423-7_12.
  52. 52.Ryffel, T., Trask, A., Dahl, M., Wagner, B., Mancuso, J., Rueckert, D., and Passerat-Palmbach, J. A generic framework for privacy preserving deep learning. In Privacy Preserving Machine Learning (NeurIPS 2018 Workshop), 2018.
  53. 53.Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 15:1929–1958, 2014.
  54. 54.Tan, S., Knott, B., Tian, Y., and Wu, D. J. CryptGPU: Fast privacy-preserving machine learning on the GPU. In IEEE Symposium on Security and Privacy, 2021.
  55. 55.Wagh, S., Gupta, D., and Chandran, N. SecureNN: 3-party secure computation for neural network training. PoPETs, 2019(3):26–49, July 2019. doi: 10.2478/popets-2019-0035.
  56. 56.Wagh, S., Tople, S., Benhamouda, F., Kushilevitz, E., Mittal, P., and Rabin, T. Falcon: Honest-majority maliciously secure framework for private deep learning. PoPETs, 2021(1):188–208, January 2021. doi: 10.2478/popets-2021-0011.
  57. 57.Wang, N., Choi, J., Brand, D., Chen, C.-Y., and Gopalakrishnan, K. Training deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  58. 58.Xiao, H., Rasul, K., and Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.

Citation

MLA
Keller, M., and K. Sun. “Secure Quantized Training for Deep Learning”. International Conference on Machine Learning, vol. 162, 2022, pp. 10912–38, https://proceedings.mlr.press/v162/keller22a.html.
APA
Keller, M., & Sun, K. (2022). Secure Quantized Training for Deep Learning. International Conference on Machine Learning, 162, 10912–10938. https://proceedings.mlr.press/v162/keller22a.html
Chicago
Keller, M., and K. Sun. 2022. “Secure Quantized Training for Deep Learning”. International Conference on Machine Learning 162: 10912–38. https://proceedings.mlr.press/v162/keller22a.html.
Harvard
Keller, M. and Sun, K. (2022) “Secure Quantized Training for Deep Learning”, International Conference on Machine Learning. PMLR, pp. 10912–10938. Available at: https://proceedings.mlr.press/v162/keller22a.html.
Vancouver
1. Keller M, Sun K (2022) Secure Quantized Training for Deep Learning. In: International Conference on Machine Learning. PMLR, pp 10912–10938

BibTeX

@InProceedings{pmlr-v162-keller22a,
  title = 	 {Secure Quantized Training for Deep Learning},
  author =       {Keller, Marcel and Sun, Ke},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {10912--10938},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/keller22a/keller22a.pdf},
  url = 	 {https://proceedings.mlr.press/v162/keller22a.html},
  abstract = 	 {We implement training of neural networks in secure multi-party computation (MPC) using quantization commonly used in said setting. We are the first to present an MNIST classifier purely trained in MPC that comes within 0.2 percent of the accuracy of the same convolutional neural network trained via plaintext computation. More concretely, we have trained a network with two convolutional and two dense layers to 99.2% accuracy in 3.5 hours (under one hour for 99% accuracy). We have also implemented AlexNet for CIFAR-10, which converges in a few hours. We develop novel protocols for exponentiation and inverse square root. Finally, we present experiments in a range of MPC security models for up to ten parties, both with honest and dishonest majority as well as semi-honest and malicious security.}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/