Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support

Tim Reichelt, Luke Ong, Tom Rainforth
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:829-837, 2024.

Abstract

The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-reichelt24a, title = {Beyond {B}ayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support}, author = {Reichelt, Tim and Ong, Luke and Rainforth, Tom}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {829--837}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/reichelt24a/reichelt24a.pdf}, url = {https://proceedings.mlr.press/v238/reichelt24a.html}, abstract = {The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.} }
Endnote
%0 Conference Paper %T Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support %A Tim Reichelt %A Luke Ong %A Tom Rainforth %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-reichelt24a %I PMLR %P 829--837 %U https://proceedings.mlr.press/v238/reichelt24a.html %V 238 %X The posterior in probabilistic programs with stochastic support decomposes as a weighted sum of the local posterior distributions associated with each possible program path. We show that making predictions with this full posterior implicitly performs a Bayesian model averaging (BMA) over paths. This is potentially problematic, as BMA weights can be unstable due to model misspecification or inference approximations, leading to sub-optimal predictions in turn. To remedy this issue, we propose alternative mechanisms for path weighting: one based on stacking and one based on ideas from PAC-Bayes. We show how both can be implemented as a cheap post-processing step on top of existing inference engines. In our experiments, we find them to be more robust and lead to better predictions compared to the default BMA weights.
APA
Reichelt, T., Ong, L. & Rainforth, T.. (2024). Beyond Bayesian Model Averaging over Paths in Probabilistic Programs with Stochastic Support. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:829-837 Available from https://proceedings.mlr.press/v238/reichelt24a.html.

Related Material