Calibrated Approximate Bayesian Inference

Hanwen Xing, Geoff Nicholls, Jeong Lee
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6912-6920, 2019.

Abstract

We give a general purpose computational framework for estimating the bias in coverage resulting from making approximations in Bayesian inference. Coverage is the probability credible sets cover true parameter values. We show how to estimate the actual coverage an approximation scheme achieves when the ideal observation model and the prior can be simulated, but have been replaced, in the Monte Carlo, with approximations as they are intractable. Coverage estimation procedures given in Lee et al. (2018) work well on simple problems, but are biased, and do not scale well, as those authors note. For example, the methods of Lee et al. (2018) fail for calibration of an approximate completely collapsed MCMC algorithm for partition structure in a Dirichlet process for clustering group labels in a hierarchical model. By exploiting the symmetry of the coverage error under permutation of low level group labels and smoothing with Bayesian Additive Regression Trees, we are able to show that the original approximate inference had poor coverage and should not be trusted.

Cite this Paper

BibTeX
@InProceedings{pmlr-v97-xing19a, title = {Calibrated Approximate {B}ayesian Inference}, author = {Xing, Hanwen and Nicholls, Geoff and Lee, Jeong}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {6912--6920}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/xing19a/xing19a.pdf}, url = {https://proceedings.mlr.press/v97/xing19a.html}, abstract = {We give a general purpose computational framework for estimating the bias in coverage resulting from making approximations in Bayesian inference. Coverage is the probability credible sets cover true parameter values. We show how to estimate the actual coverage an approximation scheme achieves when the ideal observation model and the prior can be simulated, but have been replaced, in the Monte Carlo, with approximations as they are intractable. Coverage estimation procedures given in Lee et al. (2018) work well on simple problems, but are biased, and do not scale well, as those authors note. For example, the methods of Lee et al. (2018) fail for calibration of an approximate completely collapsed MCMC algorithm for partition structure in a Dirichlet process for clustering group labels in a hierarchical model. By exploiting the symmetry of the coverage error under permutation of low level group labels and smoothing with Bayesian Additive Regression Trees, we are able to show that the original approximate inference had poor coverage and should not be trusted.} }
Endnote
%0 Conference Paper %T Calibrated Approximate Bayesian Inference %A Hanwen Xing %A Geoff Nicholls %A Jeong Lee %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-xing19a %I PMLR %P 6912--6920 %U https://proceedings.mlr.press/v97/xing19a.html %V 97 %X We give a general purpose computational framework for estimating the bias in coverage resulting from making approximations in Bayesian inference. Coverage is the probability credible sets cover true parameter values. We show how to estimate the actual coverage an approximation scheme achieves when the ideal observation model and the prior can be simulated, but have been replaced, in the Monte Carlo, with approximations as they are intractable. Coverage estimation procedures given in Lee et al. (2018) work well on simple problems, but are biased, and do not scale well, as those authors note. For example, the methods of Lee et al. (2018) fail for calibration of an approximate completely collapsed MCMC algorithm for partition structure in a Dirichlet process for clustering group labels in a hierarchical model. By exploiting the symmetry of the coverage error under permutation of low level group labels and smoothing with Bayesian Additive Regression Trees, we are able to show that the original approximate inference had poor coverage and should not be trusted.
APA
Xing, H., Nicholls, G. & Lee, J.. (2019). Calibrated Approximate Bayesian Inference. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:6912-6920 Available from https://proceedings.mlr.press/v97/xing19a.html.

Related Material