Differentially Private Variational Inference for Non-conjugate Models

Joonas Jälkö, Onur Dikmen, Antti Honkela
Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, PMLR R15:291-300, 2017.

Abstract

Many machine learning applications are based on data collected from people, such as their tastes and behaviour as well as biological traits and genetic data. Regardless of how important the application might be, one has to make sure individuals’ identities or the privacy of the data are not compromised in the analysis. Differential privacy constitutes a powerful framework that prevents breaching of data subject privacy from the output of a computation. Differentially private versions of many important Bayesian inference methods have been proposed, but there is a lack of an efficient unified approach applicable to arbitrary models. In this contribution, we propose a differentially private variational inference method with a very wide applicability. It is built on top of doubly stochastic variational inference, a recent advance which provides a variational solution to a large class of models. We add differential privacy into doubly stochastic variational inference by clipping and perturbing the gradients. The algorithm is made more efficient through privacy amplification from subsampling. We demonstrate the method can reach an accuracy close to non-private level under reasonably strong privacy guarantees, clearly improving over previous sampling-based alternatives especially in the strong privacy regime. $*$AH is also with the Department of Mathematics and Statistics and Department of Public Health, University of Helsinki.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR15-jalko17a, title = {Differentially Private Variational Inference for Non-conjugate Models}, author = {J{\"a}lk{\"o}, Joonas and Dikmen, Onur and Honkela, Antti}, booktitle = {Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence}, pages = {291--300}, year = {2017}, editor = {Elidan, Gal and Kersting, Kristian}, volume = {R15}, series = {Proceedings of Machine Learning Research}, month = {11--15 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r15/main/assets/jalko17a/jalko17a.pdf}, url = {https://proceedings.mlr.press/r15/jalko17a.html}, abstract = {Many machine learning applications are based on data collected from people, such as their tastes and behaviour as well as biological traits and genetic data. Regardless of how important the application might be, one has to make sure individuals’ identities or the privacy of the data are not compromised in the analysis. Differential privacy constitutes a powerful framework that prevents breaching of data subject privacy from the output of a computation. Differentially private versions of many important Bayesian inference methods have been proposed, but there is a lack of an efficient unified approach applicable to arbitrary models. In this contribution, we propose a differentially private variational inference method with a very wide applicability. It is built on top of doubly stochastic variational inference, a recent advance which provides a variational solution to a large class of models. We add differential privacy into doubly stochastic variational inference by clipping and perturbing the gradients. The algorithm is made more efficient through privacy amplification from subsampling. We demonstrate the method can reach an accuracy close to non-private level under reasonably strong privacy guarantees, clearly improving over previous sampling-based alternatives especially in the strong privacy regime. $*$AH is also with the Department of Mathematics and Statistics and Department of Public Health, University of Helsinki.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Differentially Private Variational Inference for Non-conjugate Models %A Joonas Jälkö %A Onur Dikmen %A Antti Honkela %B Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2017 %E Gal Elidan %E Kristian Kersting %F pmlr-vR15-jalko17a %I PMLR %P 291--300 %U https://proceedings.mlr.press/r15/jalko17a.html %V R15 %X Many machine learning applications are based on data collected from people, such as their tastes and behaviour as well as biological traits and genetic data. Regardless of how important the application might be, one has to make sure individuals’ identities or the privacy of the data are not compromised in the analysis. Differential privacy constitutes a powerful framework that prevents breaching of data subject privacy from the output of a computation. Differentially private versions of many important Bayesian inference methods have been proposed, but there is a lack of an efficient unified approach applicable to arbitrary models. In this contribution, we propose a differentially private variational inference method with a very wide applicability. It is built on top of doubly stochastic variational inference, a recent advance which provides a variational solution to a large class of models. We add differential privacy into doubly stochastic variational inference by clipping and perturbing the gradients. The algorithm is made more efficient through privacy amplification from subsampling. We demonstrate the method can reach an accuracy close to non-private level under reasonably strong privacy guarantees, clearly improving over previous sampling-based alternatives especially in the strong privacy regime. $*$AH is also with the Department of Mathematics and Statistics and Department of Public Health, University of Helsinki. %Z Reissued by PMLR on 04 October 2026.
APA
Jälkö, J., Dikmen, O. & Honkela, A.. (2017). Differentially Private Variational Inference for Non-conjugate Models. Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R15:291-300 Available from https://proceedings.mlr.press/r15/jalko17a.html. Reissued by PMLR on 04 October 2026.

Related Material