Fast and scalable score-based kernel calibration tests

Pierre Glaser, David Widmann, Fredrik Lindsten, Arthur Gretton
Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, PMLR 216:691-700, 2023.

Abstract

We introduce the Kernel Calibration Conditional Stein Discrepancy test (KCCSD test), a nonparametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores. In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error. We achieve these improvements by using a new family of kernels for score-based probabilities that can be estimated without probability density samples, and by using a Conditional Goodness of Fit criterion for the KCCSD test’s U-statistic. We demonstrate the properties of our test on various synthetic settings.

Cite this Paper

BibTeX
@InProceedings{pmlr-v216-glaser23a, title = {Fast and scalable score-based kernel calibration tests}, author = {Glaser, Pierre and Widmann, David and Lindsten, Fredrik and Gretton, Arthur}, booktitle = {Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence}, pages = {691--700}, year = {2023}, editor = {Evans, Robin J. and Shpitser, Ilya}, volume = {216}, series = {Proceedings of Machine Learning Research}, month = {31 Jul--04 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v216/glaser23a/glaser23a.pdf}, url = {https://proceedings.mlr.press/v216/glaser23a.html}, abstract = {We introduce the Kernel Calibration Conditional Stein Discrepancy test (KCCSD test), a nonparametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores. In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error. We achieve these improvements by using a new family of kernels for score-based probabilities that can be estimated without probability density samples, and by using a Conditional Goodness of Fit criterion for the KCCSD test’s U-statistic. We demonstrate the properties of our test on various synthetic settings.} }
Endnote
%0 Conference Paper %T Fast and scalable score-based kernel calibration tests %A Pierre Glaser %A David Widmann %A Fredrik Lindsten %A Arthur Gretton %B Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2023 %E Robin J. Evans %E Ilya Shpitser %F pmlr-v216-glaser23a %I PMLR %P 691--700 %U https://proceedings.mlr.press/v216/glaser23a.html %V 216 %X We introduce the Kernel Calibration Conditional Stein Discrepancy test (KCCSD test), a nonparametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores. In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error. We achieve these improvements by using a new family of kernels for score-based probabilities that can be estimated without probability density samples, and by using a Conditional Goodness of Fit criterion for the KCCSD test’s U-statistic. We demonstrate the properties of our test on various synthetic settings.
APA
Glaser, P., Widmann, D., Lindsten, F. & Gretton, A.. (2023). Fast and scalable score-based kernel calibration tests. Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 216:691-700 Available from https://proceedings.mlr.press/v216/glaser23a.html.

Related Material