Does Label Differential Privacy Prevent Label Inference Attacks?

Ruihan Wu, Jin Peng Zhou, Kilian Q. Weinberger, Chuan Guo
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:4336-4347, 2023.

Abstract

Label differential privacy (label-DP) is a popular framework for training private ML models on datasets with public features and sensitive private labels. Despite its rigorous privacy guarantee, it has been observed that in practice label-DP does not preclude label inference attacks (LIAs): Models trained with label-DP can be evaluated on the public training features to recover, with high accuracy, the very private labels that it was designed to protect. In this work, we argue that this phenomenon is not paradoxical and that label-DP is designed to limit the advantage of an LIA adversary compared to predicting training labels using the Bayes classifier. At label-DP $\epsilon=0$ this advantage is zero, hence the optimal attack is to predict according to the Bayes classifier and is independent of the training labels. Our bound shows the semantic protection conferred by label-DP and gives guidelines on how to choose $\epsilon$ to limit the threat of LIAs below a certain level. Finally, we empirically demonstrate that our result closely captures the behavior of simulated attacks on both synthetic and real world datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-wu23a, title = {Does Label Differential Privacy Prevent Label Inference Attacks?}, author = {Wu, Ruihan and Zhou, Jin Peng and Weinberger, Kilian Q. and Guo, Chuan}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {4336--4347}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/wu23a/wu23a.pdf}, url = {https://proceedings.mlr.press/v206/wu23a.html}, abstract = {Label differential privacy (label-DP) is a popular framework for training private ML models on datasets with public features and sensitive private labels. Despite its rigorous privacy guarantee, it has been observed that in practice label-DP does not preclude label inference attacks (LIAs): Models trained with label-DP can be evaluated on the public training features to recover, with high accuracy, the very private labels that it was designed to protect. In this work, we argue that this phenomenon is not paradoxical and that label-DP is designed to limit the advantage of an LIA adversary compared to predicting training labels using the Bayes classifier. At label-DP $\epsilon=0$ this advantage is zero, hence the optimal attack is to predict according to the Bayes classifier and is independent of the training labels. Our bound shows the semantic protection conferred by label-DP and gives guidelines on how to choose $\epsilon$ to limit the threat of LIAs below a certain level. Finally, we empirically demonstrate that our result closely captures the behavior of simulated attacks on both synthetic and real world datasets.} }
Endnote
%0 Conference Paper %T Does Label Differential Privacy Prevent Label Inference Attacks? %A Ruihan Wu %A Jin Peng Zhou %A Kilian Q. Weinberger %A Chuan Guo %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-wu23a %I PMLR %P 4336--4347 %U https://proceedings.mlr.press/v206/wu23a.html %V 206 %X Label differential privacy (label-DP) is a popular framework for training private ML models on datasets with public features and sensitive private labels. Despite its rigorous privacy guarantee, it has been observed that in practice label-DP does not preclude label inference attacks (LIAs): Models trained with label-DP can be evaluated on the public training features to recover, with high accuracy, the very private labels that it was designed to protect. In this work, we argue that this phenomenon is not paradoxical and that label-DP is designed to limit the advantage of an LIA adversary compared to predicting training labels using the Bayes classifier. At label-DP $\epsilon=0$ this advantage is zero, hence the optimal attack is to predict according to the Bayes classifier and is independent of the training labels. Our bound shows the semantic protection conferred by label-DP and gives guidelines on how to choose $\epsilon$ to limit the threat of LIAs below a certain level. Finally, we empirically demonstrate that our result closely captures the behavior of simulated attacks on both synthetic and real world datasets.
APA
Wu, R., Zhou, J.P., Weinberger, K.Q. & Guo, C.. (2023). Does Label Differential Privacy Prevent Label Inference Attacks?. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:4336-4347 Available from https://proceedings.mlr.press/v206/wu23a.html.

Related Material