How well does your sampler really work?

Ryan Turner, Brady Neal
Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, PMLR R16:72-81, 2018.

Abstract

We present a data-driven benchmark system to evaluate the performance of new MCMC samplers. Taking inspiration from the COCO benchmark in optimization, we view this benchmark as having critical importance to machine learning and statistics given the rate at which new samplers are proposed. The common hand-crafted examples to test new samplers are unsatisfactory; we take a meta- learning-like approach to generate realistic benchmark examples from a large corpus of data sets and models. Surrogates of posteriors found in real problems are created using highly flexible density models including modern neu- ral network models. We provide new insights into the real effective sample size of various samplers per unit time and the estimation effi- ciency of the samplers per sample. Addition- ally, we provide a meta-analysis to assess the predictive utility of various MCMC diagnos- tics and perform a nonparametric regression to combine them.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR16-turner18a, title = {How well does your sampler really work?}, author = {Turner, Ryan and Neal, Brady}, booktitle = {Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence}, pages = {72--81}, year = {2018}, editor = {Globerson, Amir and Silva, Ricardo}, volume = {R16}, series = {Proceedings of Machine Learning Research}, month = {06--10 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r16/main/assets/turner18a/turner18a.pdf}, url = {https://proceedings.mlr.press/r16/turner18a.html}, abstract = {We present a data-driven benchmark system to evaluate the performance of new MCMC samplers. Taking inspiration from the COCO benchmark in optimization, we view this benchmark as having critical importance to machine learning and statistics given the rate at which new samplers are proposed. The common hand-crafted examples to test new samplers are unsatisfactory; we take a meta- learning-like approach to generate realistic benchmark examples from a large corpus of data sets and models. Surrogates of posteriors found in real problems are created using highly flexible density models including modern neu- ral network models. We provide new insights into the real effective sample size of various samplers per unit time and the estimation effi- ciency of the samplers per sample. Addition- ally, we provide a meta-analysis to assess the predictive utility of various MCMC diagnos- tics and perform a nonparametric regression to combine them.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T How well does your sampler really work? %A Ryan Turner %A Brady Neal %B Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2018 %E Amir Globerson %E Ricardo Silva %F pmlr-vR16-turner18a %I PMLR %P 72--81 %U https://proceedings.mlr.press/r16/turner18a.html %V R16 %X We present a data-driven benchmark system to evaluate the performance of new MCMC samplers. Taking inspiration from the COCO benchmark in optimization, we view this benchmark as having critical importance to machine learning and statistics given the rate at which new samplers are proposed. The common hand-crafted examples to test new samplers are unsatisfactory; we take a meta- learning-like approach to generate realistic benchmark examples from a large corpus of data sets and models. Surrogates of posteriors found in real problems are created using highly flexible density models including modern neu- ral network models. We provide new insights into the real effective sample size of various samplers per unit time and the estimation effi- ciency of the samplers per sample. Addition- ally, we provide a meta-analysis to assess the predictive utility of various MCMC diagnos- tics and perform a nonparametric regression to combine them. %Z Reissued by PMLR on 04 October 2026.
APA
Turner, R. & Neal, B.. (2018). How well does your sampler really work?. Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R16:72-81 Available from https://proceedings.mlr.press/r16/turner18a.html. Reissued by PMLR on 04 October 2026.

Related Material