Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference

Luca Rendsburg, Agustinus Kristiadi, Philipp Hennig, Ulrike Von Luxburg
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:1503-1526, 2022.

Abstract

Full Bayesian posteriors are rarely analytically tractable, which is why real-world Bayesian inference heavily relies on approximate techniques. Approximations generally differ from the true posterior and require diagnostic tools to assess whether the inference can still be trusted. We investigate a new approach to diagnosing approximate inference: the approximation mismatch is attributed to a change in the inductive bias by treating the approximations as exact and reverse-engineering the corresponding prior. We show that the problem is more complicated than it appears to be at first glance, because the solution generally depends on the observation. By reframing the problem in terms of incompatible conditional distributions we arrive at a natural solution: the Gibbs prior. The resulting diagnostic is based on pseudo-Gibbs sampling, which is widely applicable and easy to implement. We illustrate how the Gibbs prior can be used to discover the inductive bias in a controlled Gaussian setting and for a variety of Bayesian models and approximations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-rendsburg22a, title = { Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference }, author = {Rendsburg, Luca and Kristiadi, Agustinus and Hennig, Philipp and Von Luxburg, Ulrike}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {1503--1526}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/rendsburg22a/rendsburg22a.pdf}, url = {https://proceedings.mlr.press/v151/rendsburg22a.html}, abstract = { Full Bayesian posteriors are rarely analytically tractable, which is why real-world Bayesian inference heavily relies on approximate techniques. Approximations generally differ from the true posterior and require diagnostic tools to assess whether the inference can still be trusted. We investigate a new approach to diagnosing approximate inference: the approximation mismatch is attributed to a change in the inductive bias by treating the approximations as exact and reverse-engineering the corresponding prior. We show that the problem is more complicated than it appears to be at first glance, because the solution generally depends on the observation. By reframing the problem in terms of incompatible conditional distributions we arrive at a natural solution: the Gibbs prior. The resulting diagnostic is based on pseudo-Gibbs sampling, which is widely applicable and easy to implement. We illustrate how the Gibbs prior can be used to discover the inductive bias in a controlled Gaussian setting and for a variety of Bayesian models and approximations. } }
Endnote
%0 Conference Paper %T Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference %A Luca Rendsburg %A Agustinus Kristiadi %A Philipp Hennig %A Ulrike Von Luxburg %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-rendsburg22a %I PMLR %P 1503--1526 %U https://proceedings.mlr.press/v151/rendsburg22a.html %V 151 %X Full Bayesian posteriors are rarely analytically tractable, which is why real-world Bayesian inference heavily relies on approximate techniques. Approximations generally differ from the true posterior and require diagnostic tools to assess whether the inference can still be trusted. We investigate a new approach to diagnosing approximate inference: the approximation mismatch is attributed to a change in the inductive bias by treating the approximations as exact and reverse-engineering the corresponding prior. We show that the problem is more complicated than it appears to be at first glance, because the solution generally depends on the observation. By reframing the problem in terms of incompatible conditional distributions we arrive at a natural solution: the Gibbs prior. The resulting diagnostic is based on pseudo-Gibbs sampling, which is widely applicable and easy to implement. We illustrate how the Gibbs prior can be used to discover the inductive bias in a controlled Gaussian setting and for a variety of Bayesian models and approximations.
APA
Rendsburg, L., Kristiadi, A., Hennig, P. & Von Luxburg, U.. (2022). Discovering Inductive Bias with Gibbs Priors: A Diagnostic Tool for Approximate Bayesian Inference . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:1503-1526 Available from https://proceedings.mlr.press/v151/rendsburg22a.html.

Related Material