A histogram based betting function for conformal martingales

Charalambos Eliades, Harris Papadopoulos
Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, PMLR 128:100-113, 2020.

Abstract

This paper investigates the use of Conformal Martingales (CM) for providing a numerical indication of how likely it is that the exchangeability assumption holds on a set of data. Reliable and fast testing of exchangeability is an important challenge because many machine learning algorithms rely on this assumption. Therefore a technique with only a few parameters to tune, that is able to reject the exchangeability assumption with respect to a significance level should be very beneficial for enhancing the reliability of such machine learning models. Our approach consists of a CM whose betting function is estimated on the previously seen p-values, we compare its computational efficiency and its performance with a kernel betting function and the Kolmogorov-Smirnoff test. We test our approach on two benchmark data-sets, USPS and Statlog Satellite data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v128-eliades20a, title = {A histogram based betting function for conformal martingales}, author = {Eliades, Charalambos and Papadopoulos, Harris}, booktitle = {Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications}, pages = {100--113}, year = {2020}, editor = {Gammerman, Alexander and Vovk, Vladimir and Luo, Zhiyuan and Smirnov, Evgueni and Cherubin, Giovanni}, volume = {128}, series = {Proceedings of Machine Learning Research}, month = {09--11 Sep}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v128/eliades20a/eliades20a.pdf}, url = {https://proceedings.mlr.press/v128/eliades20a.html}, abstract = {This paper investigates the use of Conformal Martingales (CM) for providing a numerical indication of how likely it is that the exchangeability assumption holds on a set of data. Reliable and fast testing of exchangeability is an important challenge because many machine learning algorithms rely on this assumption. Therefore a technique with only a few parameters to tune, that is able to reject the exchangeability assumption with respect to a significance level should be very beneficial for enhancing the reliability of such machine learning models. Our approach consists of a CM whose betting function is estimated on the previously seen p-values, we compare its computational efficiency and its performance with a kernel betting function and the Kolmogorov-Smirnoff test. We test our approach on two benchmark data-sets, USPS and Statlog Satellite data.} }
Endnote
%0 Conference Paper %T A histogram based betting function for conformal martingales %A Charalambos Eliades %A Harris Papadopoulos %B Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications %C Proceedings of Machine Learning Research %D 2020 %E Alexander Gammerman %E Vladimir Vovk %E Zhiyuan Luo %E Evgueni Smirnov %E Giovanni Cherubin %F pmlr-v128-eliades20a %I PMLR %P 100--113 %U https://proceedings.mlr.press/v128/eliades20a.html %V 128 %X This paper investigates the use of Conformal Martingales (CM) for providing a numerical indication of how likely it is that the exchangeability assumption holds on a set of data. Reliable and fast testing of exchangeability is an important challenge because many machine learning algorithms rely on this assumption. Therefore a technique with only a few parameters to tune, that is able to reject the exchangeability assumption with respect to a significance level should be very beneficial for enhancing the reliability of such machine learning models. Our approach consists of a CM whose betting function is estimated on the previously seen p-values, we compare its computational efficiency and its performance with a kernel betting function and the Kolmogorov-Smirnoff test. We test our approach on two benchmark data-sets, USPS and Statlog Satellite data.
APA
Eliades, C. & Papadopoulos, H.. (2020). A histogram based betting function for conformal martingales. Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, in Proceedings of Machine Learning Research 128:100-113 Available from https://proceedings.mlr.press/v128/eliades20a.html.

Related Material