The Binomial Block Bootstrap Estimator for Evaluating Loss on Dependent Clusters

Matt Barnes, Artur Dubrawski
Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, PMLR R15:799-808, 2017.

Abstract

In this paper, we study the non-IID learn- ing setting where samples exhibit dependency within latent clusters. Our goal is to esti- mate a learner’s loss on new clusters, an ex- tension of the out-of-bag error. Previously developed cross-validation estimators are well suited to the case where the clustering of ob- served data is known a priori. However, as is often the case in real world problems, we are only given a noisy approximation of this clustering, likely the result of some clustering algorithm. This subtle yet potentially signifi- cant issue afflicts domains ranging from image classification to medical diagnostics, where naive cross-validation is an optimistically bi- ased estimator. We present a novel bootstrap technique and corresponding cross-validation method that, somewhat counterintuitively, in- jects additional dependency to asymptotically recover the loss in the independent setting.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR15-barnes17a, title = {The Binomial Block Bootstrap Estimator for Evaluating Loss on Dependent Clusters}, author = {Barnes, Matt and Dubrawski, Artur}, booktitle = {Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence}, pages = {799--808}, year = {2017}, editor = {Elidan, Gal and Kersting, Kristian}, volume = {R15}, series = {Proceedings of Machine Learning Research}, month = {11--15 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r15/main/assets/barnes17a/barnes17a.pdf}, url = {https://proceedings.mlr.press/r15/barnes17a.html}, abstract = {In this paper, we study the non-IID learn- ing setting where samples exhibit dependency within latent clusters. Our goal is to esti- mate a learner’s loss on new clusters, an ex- tension of the out-of-bag error. Previously developed cross-validation estimators are well suited to the case where the clustering of ob- served data is known a priori. However, as is often the case in real world problems, we are only given a noisy approximation of this clustering, likely the result of some clustering algorithm. This subtle yet potentially signifi- cant issue afflicts domains ranging from image classification to medical diagnostics, where naive cross-validation is an optimistically bi- ased estimator. We present a novel bootstrap technique and corresponding cross-validation method that, somewhat counterintuitively, in- jects additional dependency to asymptotically recover the loss in the independent setting.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T The Binomial Block Bootstrap Estimator for Evaluating Loss on Dependent Clusters %A Matt Barnes %A Artur Dubrawski %B Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2017 %E Gal Elidan %E Kristian Kersting %F pmlr-vR15-barnes17a %I PMLR %P 799--808 %U https://proceedings.mlr.press/r15/barnes17a.html %V R15 %X In this paper, we study the non-IID learn- ing setting where samples exhibit dependency within latent clusters. Our goal is to esti- mate a learner’s loss on new clusters, an ex- tension of the out-of-bag error. Previously developed cross-validation estimators are well suited to the case where the clustering of ob- served data is known a priori. However, as is often the case in real world problems, we are only given a noisy approximation of this clustering, likely the result of some clustering algorithm. This subtle yet potentially signifi- cant issue afflicts domains ranging from image classification to medical diagnostics, where naive cross-validation is an optimistically bi- ased estimator. We present a novel bootstrap technique and corresponding cross-validation method that, somewhat counterintuitively, in- jects additional dependency to asymptotically recover the loss in the independent setting. %Z Reissued by PMLR on 04 October 2026.
APA
Barnes, M. & Dubrawski, A.. (2017). The Binomial Block Bootstrap Estimator for Evaluating Loss on Dependent Clusters. Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R15:799-808 Available from https://proceedings.mlr.press/r15/barnes17a.html. Reissued by PMLR on 04 October 2026.

Related Material