MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy

Badr-Eddine Cherief-Abdellatif, Pierre Alquier
Proceedings of The 2nd Symposium on Advances in Approximate Bayesian Inference, PMLR 118:1-21, 2020.

Abstract

In some misspecified settings, the posterior distribution in Bayesian statistics may lead to inconsistent estimates. To fix this issue, it has been suggested to replace the likelihood by a pseudo-likelihood, that is the exponential of a loss function enjoying suitable robustness properties. In this paper, we build a pseudo-likelihood based on the Maximum Mean Discrepancy, dened via an embedding of probability distributions into a reproducing kernel Hilbert space. We show that this MMD-Bayes posterior is consistent and robust to model misspecication. As the posterior obtained in this way might be intractable, we also prove that reasonable variational approximations of this posterior enjoy the same properties. We provide details on a stochastic gradient algorithm to compute these variational approximations. Numerical simulations indeed suggest that our estimator is more robust to misspecication than the ones based on the likelihood.

Cite this Paper

BibTeX
@InProceedings{pmlr-v118-cherief-abdellatif20a, title = {MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy}, author = {Cherief-Abdellatif, Badr-Eddine and Alquier, Pierre}, booktitle = {Proceedings of The 2nd Symposium on Advances in Approximate Bayesian Inference}, pages = {1--21}, year = {2020}, editor = {Zhang, Cheng and Ruiz, Francisco and Bui, Thang and Dieng, Adji Bousso and Liang, Dawen}, volume = {118}, series = {Proceedings of Machine Learning Research}, month = {08 Dec}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v118/cherief-abdellatif20a/cherief-abdellatif20a.pdf}, url = {https://proceedings.mlr.press/v118/cherief-abdellatif20a.html}, abstract = {In some misspecified settings, the posterior distribution in Bayesian statistics may lead to inconsistent estimates. To fix this issue, it has been suggested to replace the likelihood by a pseudo-likelihood, that is the exponential of a loss function enjoying suitable robustness properties. In this paper, we build a pseudo-likelihood based on the Maximum Mean Discrepancy, dened via an embedding of probability distributions into a reproducing kernel Hilbert space. We show that this MMD-Bayes posterior is consistent and robust to model misspecication. As the posterior obtained in this way might be intractable, we also prove that reasonable variational approximations of this posterior enjoy the same properties. We provide details on a stochastic gradient algorithm to compute these variational approximations. Numerical simulations indeed suggest that our estimator is more robust to misspecication than the ones based on the likelihood.} }
Endnote
%0 Conference Paper %T MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy %A Badr-Eddine Cherief-Abdellatif %A Pierre Alquier %B Proceedings of The 2nd Symposium on Advances in Approximate Bayesian Inference %C Proceedings of Machine Learning Research %D 2020 %E Cheng Zhang %E Francisco Ruiz %E Thang Bui %E Adji Bousso Dieng %E Dawen Liang %F pmlr-v118-cherief-abdellatif20a %I PMLR %P 1--21 %U https://proceedings.mlr.press/v118/cherief-abdellatif20a.html %V 118 %X In some misspecified settings, the posterior distribution in Bayesian statistics may lead to inconsistent estimates. To fix this issue, it has been suggested to replace the likelihood by a pseudo-likelihood, that is the exponential of a loss function enjoying suitable robustness properties. In this paper, we build a pseudo-likelihood based on the Maximum Mean Discrepancy, dened via an embedding of probability distributions into a reproducing kernel Hilbert space. We show that this MMD-Bayes posterior is consistent and robust to model misspecication. As the posterior obtained in this way might be intractable, we also prove that reasonable variational approximations of this posterior enjoy the same properties. We provide details on a stochastic gradient algorithm to compute these variational approximations. Numerical simulations indeed suggest that our estimator is more robust to misspecication than the ones based on the likelihood.
APA
Cherief-Abdellatif, B. & Alquier, P.. (2020). MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy. Proceedings of The 2nd Symposium on Advances in Approximate Bayesian Inference, in Proceedings of Machine Learning Research 118:1-21 Available from https://proceedings.mlr.press/v118/cherief-abdellatif20a.html.

Related Material