Automatic Reparameterisation of Probabilistic Programs

Maria Gorinova, Dave Moore, Matthew Hoffman
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:3648-3657, 2020.

Abstract

Probabilistic programming has emerged as a powerful paradigm in statistics, applied science, and machine learning: by decoupling modelling from inference, it promises to allow modellers to directly reason about the processes generating data. However, the performance of inference algorithms can be dramatically affected by the parameterisation used to express a model, requiring users to transform their programs in non-intuitive ways. We argue for automating these transformations, and demonstrate that mechanisms available in recent modelling frameworks can implement non-centring and related reparameterisations. This enables new inference algorithms, and we propose two: a simple approach using interleaved sampling and a novel variational formulation that searches over a continuous space of parameterisations. We show that these approaches enable robust inference across a range of models, and can yield more efficient samplers than the best fixed parameterisation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-gorinova20a, title = {Automatic Reparameterisation of Probabilistic Programs}, author = {Gorinova, Maria and Moore, Dave and Hoffman, Matthew}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {3648--3657}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/gorinova20a/gorinova20a.pdf}, url = {https://proceedings.mlr.press/v119/gorinova20a.html}, abstract = {Probabilistic programming has emerged as a powerful paradigm in statistics, applied science, and machine learning: by decoupling modelling from inference, it promises to allow modellers to directly reason about the processes generating data. However, the performance of inference algorithms can be dramatically affected by the parameterisation used to express a model, requiring users to transform their programs in non-intuitive ways. We argue for automating these transformations, and demonstrate that mechanisms available in recent modelling frameworks can implement non-centring and related reparameterisations. This enables new inference algorithms, and we propose two: a simple approach using interleaved sampling and a novel variational formulation that searches over a continuous space of parameterisations. We show that these approaches enable robust inference across a range of models, and can yield more efficient samplers than the best fixed parameterisation.} }
Endnote
%0 Conference Paper %T Automatic Reparameterisation of Probabilistic Programs %A Maria Gorinova %A Dave Moore %A Matthew Hoffman %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-gorinova20a %I PMLR %P 3648--3657 %U https://proceedings.mlr.press/v119/gorinova20a.html %V 119 %X Probabilistic programming has emerged as a powerful paradigm in statistics, applied science, and machine learning: by decoupling modelling from inference, it promises to allow modellers to directly reason about the processes generating data. However, the performance of inference algorithms can be dramatically affected by the parameterisation used to express a model, requiring users to transform their programs in non-intuitive ways. We argue for automating these transformations, and demonstrate that mechanisms available in recent modelling frameworks can implement non-centring and related reparameterisations. This enables new inference algorithms, and we propose two: a simple approach using interleaved sampling and a novel variational formulation that searches over a continuous space of parameterisations. We show that these approaches enable robust inference across a range of models, and can yield more efficient samplers than the best fixed parameterisation.
APA
Gorinova, M., Moore, D. & Hoffman, M.. (2020). Automatic Reparameterisation of Probabilistic Programs. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:3648-3657 Available from https://proceedings.mlr.press/v119/gorinova20a.html.

Related Material