Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data

Måns Magnusson, Aki Vehtari, Johan Jonasson, Michael Andersen
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:341-351, 2020.

Abstract

Recently, new methods for model assessment, based on subsampling and posterior approximations, have been proposed for scaling leave-one-out cross-validation (LOO-CV) to large datasets. Although these methods work well for estimating predictive performance for individual models, they are less powerful in model comparison. We propose an efficient method for estimating differences in predictive performance by combining fast approximate LOO surrogates with exact LOO sub-sampling using the difference estimator and supply proofs with regards to scaling characteristics. The resulting approach can be orders of magnitude more efficient than previous approaches, as well as being better suited to model comparison.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-magnusson20a, title = {Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data}, author = {Magnusson, M{\aa}ns and Vehtari, Aki and Jonasson, Johan and Andersen, Michael}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {341--351}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/magnusson20a/magnusson20a.pdf}, url = {https://proceedings.mlr.press/v108/magnusson20a.html}, abstract = {Recently, new methods for model assessment, based on subsampling and posterior approximations, have been proposed for scaling leave-one-out cross-validation (LOO-CV) to large datasets. Although these methods work well for estimating predictive performance for individual models, they are less powerful in model comparison. We propose an efficient method for estimating differences in predictive performance by combining fast approximate LOO surrogates with exact LOO sub-sampling using the difference estimator and supply proofs with regards to scaling characteristics. The resulting approach can be orders of magnitude more efficient than previous approaches, as well as being better suited to model comparison.} }
Endnote
%0 Conference Paper %T Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data %A Måns Magnusson %A Aki Vehtari %A Johan Jonasson %A Michael Andersen %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-magnusson20a %I PMLR %P 341--351 %U https://proceedings.mlr.press/v108/magnusson20a.html %V 108 %X Recently, new methods for model assessment, based on subsampling and posterior approximations, have been proposed for scaling leave-one-out cross-validation (LOO-CV) to large datasets. Although these methods work well for estimating predictive performance for individual models, they are less powerful in model comparison. We propose an efficient method for estimating differences in predictive performance by combining fast approximate LOO surrogates with exact LOO sub-sampling using the difference estimator and supply proofs with regards to scaling characteristics. The resulting approach can be orders of magnitude more efficient than previous approaches, as well as being better suited to model comparison.
APA
Magnusson, M., Vehtari, A., Jonasson, J. & Andersen, M.. (2020). Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:341-351 Available from https://proceedings.mlr.press/v108/magnusson20a.html.

Related Material