HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption

Seewoo Lee, Garam Lee, Jung Woo Kim, Junbum Shin, Mun-Kyu Lee
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:19010-19035, 2023.

Abstract

Transfer learning is a de facto standard method for efficiently training machine learning models for data-scarce problems by adding and fine-tuning new classification layers to a model pre-trained on large datasets. Although numerous previous studies proposed to use homomorphic encryption to resolve the data privacy issue in transfer learning in the machine learning as a service setting, most of them only focused on encrypted inference. In this study, we present HETAL, an efficient Homomorphic Encryption based Transfer Learning algorithm, that protects the client’s privacy in training tasks by encrypting the client data using the CKKS homomorphic encryption scheme. HETAL is the first practical scheme that strictly provides encrypted training, adopting validation-based early stopping and achieving the accuracy of nonencrypted training. We propose an efficient encrypted matrix multiplication algorithm, which is 1.8 to 323 times faster than prior methods, and a highly precise softmax approximation algorithm with increased coverage. The experimental results for five well-known benchmark datasets show total training times of 567–3442 seconds, which is less than an hour.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-lee23m, title = {{HETAL}: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption}, author = {Lee, Seewoo and Lee, Garam and Kim, Jung Woo and Shin, Junbum and Lee, Mun-Kyu}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {19010--19035}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/lee23m/lee23m.pdf}, url = {https://proceedings.mlr.press/v202/lee23m.html}, abstract = {Transfer learning is a de facto standard method for efficiently training machine learning models for data-scarce problems by adding and fine-tuning new classification layers to a model pre-trained on large datasets. Although numerous previous studies proposed to use homomorphic encryption to resolve the data privacy issue in transfer learning in the machine learning as a service setting, most of them only focused on encrypted inference. In this study, we present HETAL, an efficient Homomorphic Encryption based Transfer Learning algorithm, that protects the client’s privacy in training tasks by encrypting the client data using the CKKS homomorphic encryption scheme. HETAL is the first practical scheme that strictly provides encrypted training, adopting validation-based early stopping and achieving the accuracy of nonencrypted training. We propose an efficient encrypted matrix multiplication algorithm, which is 1.8 to 323 times faster than prior methods, and a highly precise softmax approximation algorithm with increased coverage. The experimental results for five well-known benchmark datasets show total training times of 567–3442 seconds, which is less than an hour.} }
Endnote
%0 Conference Paper %T HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption %A Seewoo Lee %A Garam Lee %A Jung Woo Kim %A Junbum Shin %A Mun-Kyu Lee %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-lee23m %I PMLR %P 19010--19035 %U https://proceedings.mlr.press/v202/lee23m.html %V 202 %X Transfer learning is a de facto standard method for efficiently training machine learning models for data-scarce problems by adding and fine-tuning new classification layers to a model pre-trained on large datasets. Although numerous previous studies proposed to use homomorphic encryption to resolve the data privacy issue in transfer learning in the machine learning as a service setting, most of them only focused on encrypted inference. In this study, we present HETAL, an efficient Homomorphic Encryption based Transfer Learning algorithm, that protects the client’s privacy in training tasks by encrypting the client data using the CKKS homomorphic encryption scheme. HETAL is the first practical scheme that strictly provides encrypted training, adopting validation-based early stopping and achieving the accuracy of nonencrypted training. We propose an efficient encrypted matrix multiplication algorithm, which is 1.8 to 323 times faster than prior methods, and a highly precise softmax approximation algorithm with increased coverage. The experimental results for five well-known benchmark datasets show total training times of 567–3442 seconds, which is less than an hour.
APA
Lee, S., Lee, G., Kim, J.W., Shin, J. & Lee, M.. (2023). HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:19010-19035 Available from https://proceedings.mlr.press/v202/lee23m.html.

Related Material