Rethinking DP-SGD in Discrete Domain: Exploring Logistic Distribution in the Realm of signSGD

Jonggyu Jang, Seongjin Hwang, Hyun Jong Yang
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:21241-21265, 2024.

Abstract

Deep neural networks (DNNs) have a risk of remembering sensitive data from their training datasets, inadvertently leading to substantial information leakage through privacy attacks like membership inference attacks. DP-SGD is a simple but effective defense method, incorporating Gaussian noise into gradient updates to safeguard sensitive information. With the prevalence of large neural networks, DP-signSGD, a variant of DP-SGD, has emerged, aiming to curtail memory usage while maintaining security. However, it is noteworthy that most DP-signSGD algorithms default to Gaussian noise, suitable only for DP-SGD, without scant discussion of its appropriateness for signSGD. Our study delves into an intriguing question: "Can we find a more efficient substitute for Gaussian noise to secure privacy in DP-signSGD?" We propose an answer with a Logistic mechanism, which conforms to signSGD principles and is interestingly evolved from an exponential mechanism. In this paper, we provide both theoretical and experimental evidence showing that our method surpasses DP-signSGD.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-jang24a, title = {Rethinking {DP}-{SGD} in Discrete Domain: Exploring Logistic Distribution in the Realm of sign{SGD}}, author = {Jang, Jonggyu and Hwang, Seongjin and Yang, Hyun Jong}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {21241--21265}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/jang24a/jang24a.pdf}, url = {https://proceedings.mlr.press/v235/jang24a.html}, abstract = {Deep neural networks (DNNs) have a risk of remembering sensitive data from their training datasets, inadvertently leading to substantial information leakage through privacy attacks like membership inference attacks. DP-SGD is a simple but effective defense method, incorporating Gaussian noise into gradient updates to safeguard sensitive information. With the prevalence of large neural networks, DP-signSGD, a variant of DP-SGD, has emerged, aiming to curtail memory usage while maintaining security. However, it is noteworthy that most DP-signSGD algorithms default to Gaussian noise, suitable only for DP-SGD, without scant discussion of its appropriateness for signSGD. Our study delves into an intriguing question: "Can we find a more efficient substitute for Gaussian noise to secure privacy in DP-signSGD?" We propose an answer with a Logistic mechanism, which conforms to signSGD principles and is interestingly evolved from an exponential mechanism. In this paper, we provide both theoretical and experimental evidence showing that our method surpasses DP-signSGD.} }
Endnote
%0 Conference Paper %T Rethinking DP-SGD in Discrete Domain: Exploring Logistic Distribution in the Realm of signSGD %A Jonggyu Jang %A Seongjin Hwang %A Hyun Jong Yang %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-jang24a %I PMLR %P 21241--21265 %U https://proceedings.mlr.press/v235/jang24a.html %V 235 %X Deep neural networks (DNNs) have a risk of remembering sensitive data from their training datasets, inadvertently leading to substantial information leakage through privacy attacks like membership inference attacks. DP-SGD is a simple but effective defense method, incorporating Gaussian noise into gradient updates to safeguard sensitive information. With the prevalence of large neural networks, DP-signSGD, a variant of DP-SGD, has emerged, aiming to curtail memory usage while maintaining security. However, it is noteworthy that most DP-signSGD algorithms default to Gaussian noise, suitable only for DP-SGD, without scant discussion of its appropriateness for signSGD. Our study delves into an intriguing question: "Can we find a more efficient substitute for Gaussian noise to secure privacy in DP-signSGD?" We propose an answer with a Logistic mechanism, which conforms to signSGD principles and is interestingly evolved from an exponential mechanism. In this paper, we provide both theoretical and experimental evidence showing that our method surpasses DP-signSGD.
APA
Jang, J., Hwang, S. & Yang, H.J.. (2024). Rethinking DP-SGD in Discrete Domain: Exploring Logistic Distribution in the Realm of signSGD. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:21241-21265 Available from https://proceedings.mlr.press/v235/jang24a.html.

Related Material